# Localization of min-max minimal hypersurfaces

Douglas Stryker  Department of Mathematics, Princeton University, Princeton,
NJ 08540, USA<EMAIL_ADDRESS>
###### Abstract.

We show that min-max minimal hypersurfaces can be localized. As a consequence,
we obtain the sharp generalization to complete manifolds of the famous
Almgren-Pitts min-max theorem in closed manifolds. We use this result to prove
the existence of a complete embedded finite area minimal hypersurface of index
at most one in every balanced complete manifold.

##  1\. Introduction

The existence of an abundance of closed minimal hypersurfaces in any closed
Riemannian manifold has been established as the culmination of a Morse theory
framework for the area functional, called _min-max_ (we refer the reader to
[MN16, MN21, MN17, LMN18, IMN18, MNS19, Zho20, Son23]). This framework
additionally provides bounds on the area and the Morse index of the produced
minimal hypersurfaces. However, it is difficult to deduce any additional
information about these hypersurfaces.

One interesting feature of min-max minimal hypersurfaces that has received
substantial attention is their topology. For surfaces in 3-manifolds, a
modified Morse theory framework was developed in [Smi83] (see also [CDL03,
DLP10, Ket19]), which yields an upper bound on the genus of the resulting
minimal surface. We refer the reader to [Ket22, WZ23] for exciting recent
developments in this direction. Unfortunately, this modified framework does
not work in higher dimensions, due to limitations of the required regularity
theory.

In this paper, we focus on a different and somewhat overlooked feature of min-
max minimal hypersurfaces: their location in the ambient space. While
potentially useful in a variety of problems, we emphasize that localization of
min-max minimal hypersurfaces is _essential_ to prove the existence of minimal
hypersurfaces in _non-compact_ ambient spaces. Indeed, min-max sequences may
run off to infinity in general without some form of localization.

###  1.1. Previous work

There have been three particularly relevant results towards localization of
min-max minimal hypersurfaces in general ambient spaces111[Son23] also studies
the localization of minimal hypersurfaces in a stronger sense (i.e.
constraining the hypersurface the lie fully within a given set) in manifolds
with cylindrical ends. However, this result crucially uses the rigid structure
of these manifolds..

  * •

[Mon16]: If $(M,g)$ admits a bounded open subset $U$ with strictly mean
concave boundary (and $(M,g)$ satisfies some additional hypotheses about its
asymptotic geometry), then [Mon16] uses min-max to produce a closed embedded
minimal hypersurface intersecting $U$.

  * •

[CL20]: If $(M,g)$ admits a bounded open subset $U$ with the property that a
certain notion of min-max width (for 1-parameter sweepouts) is at least ten
times the area of the boundary of $U$, then [CL20] uses min-max to produce an
embedded complete minimal hypersurface of finite area intersecting any open
neighborhood of $\overline{U}$. Notably, this result suffices to prove the
existence of a complete minimal hypersurface of finite area in any complete
manifold of finite volume, generalizing to higher dimensions the analogous
result for geodesics due to [Tho78, Ban80].

  * •

[Son19]: If $(M,g)$ admits a compact domain $K$ that has no singular strictly
mean-convex foliation, then [Son19] uses min-max and the mean curvature flow
to produce an embedded complete minimal hypersurface of finite area and index
at most one intersecting $K$222See also [Gro14] for an earlier proof of a
similar result.. This result suffices to prove the existence of a complete
minimal hypersurface of finite area and _index at most one_ in any complete
manifold of finite volume, improving on the result of [CL20].

From the perspective of general existence problems, criteria related to the
nontriviality of some min-max invariant seems to be the most natural and
useful. One reason is that criteria related to the curvature of certain
submanifolds can be hard to verify. Moreover, the _triviality_ of certain min-
max invariants can be a useful hypothesis in proving existence by other means
(i.e. Lyusternik-Schnirelmann theory, see [LS47]), enabling proofs of
existence under weaker hypotheses by a dichotomy argument (for example, see
the dichotomy set up in our proof of Theorem 5.1).

In the “min-max invariant” framework, several questions remain regarding the
localization of min-max minimal hypersurfaces (including several questions
posed in [CL20]).

  * •

_Does the same result hold if the min-max width is only assumed to be greater
than the area of the boundary of $U$?_ ([CL20, §2.5 Question 1].)

  * •

_Does the same result hold for a minimal hypersurface intersecting
$\overline{U}$?_ ([CL20, §2.5 Question 3].)

  * •

_Does the same result hold for a minimal hypersurface with index at most 1?_
([CL20, §2.5 Question 2].)

  * •

_Does an analogous result hold for min-max using higher parameter sweepouts?_

We emphasize the none of the previous work in this area deals with higher
parameter min-max. While the modest applications we consider in this paper
only use 1-parameter min-max constructions, we expect that localization for
higher parameter sweepouts will be useful to prove the existence of more than
one minimal hypersurface in some non-compact settings.

###  1.2. Min-max localization

The following result is the sharp analogue in complete manifolds of the
Almgren-Pitts min-max theorem (see [Pit81]). Moreover, it answers all of the
previous questions affirmatively333We note that we use a different formulation
of min-max from [CL20]. For a comparison of the min-max frameworks considered
in [CL20] versus here, see §4.1. We emphasize that our proof will work in any
min-max framework for which the multiplicity one result of [Zho20] applies–we
only use this formulation for simplicity, as it is precisely the formulation
used in [Zho20].. We refer the reader to §2 for the precise min-max
definitions.

######  Theorem 1.1.

Let $(M^{n+1},g)$ be complete of dimension $3\leq n+1\leq 7$. Let
$X^{k}\subset I^{m}$ be a $k$-dimensional cubical complex, and let $Z\subset
X$ be a cubical subcomplex. Let $\Phi_{0}:X\to\mathcal{C}(M)$ be an
${\mathbf{F}}$-continuous map. If

${\mathbf{L}}(\Phi_{0},M)>\sup_{z\in Z}\mathcal{H}^{n}(\partial\Phi_{0}(z)),$

then there exists a smooth complete (possibly non-compact) embedded minimal
hypersurface $\Sigma\subset M$ satisfying

  * •

$\mathcal{H}^{n}(\Sigma)\leq{\mathbf{L}}(\Phi_{0},\mathrm{span}(\Phi_{0}))$,

  * •

$\operatorname{index}(\Sigma)\leq k$,

  * •

$\Sigma\cap\mathrm{span}(\Phi_{0})\neq\varnothing$, where
$\mathrm{span}(\Phi_{0}):=\overline{\bigcup_{x\in X}\partial\Phi_{0}(x)}$.

The localization aspect of this result (i.e. nonempty intersection with the
span) is new even for closed ambient spaces, and we expect that there may be
interesting applications in that setting alone. Moreover, we prove the
analogous statement in compact manifolds with strictly mean convex boundary in
the course of the proof of Theorem 1.1.

###  1.3. Balanced manifolds

As a modest application of this theory, we find that Theorem 1.1 applies to a
large class of complete manifolds defined only by their asymptotic behavior.
Roughly, we say that a complete manifold is _balanced_ if we can decompose
$M=E\sqcup K\sqcup F$ for $K$ compact so that the asymptotic minimal area of
cross sections of $E$ and $F$ agree; namely

$\displaystyle\lim_{r\to\infty}$
$\displaystyle\inf\\{\mathcal{H}^{n}(\Sigma)\mid\Sigma\text{\ is\ homologous\
to\ }\partial E\ \text{in\ }E,\ \Sigma\cap B_{r}(K)=\varnothing\\}$
$\displaystyle=\lim_{r\to\infty}\inf\\{\mathcal{H}^{n}(\Sigma)\mid\Sigma\text{\
is\ homologous\ to\ }\partial F\ \text{in\ }F,\ \Sigma\cap
B_{r}(K)=\varnothing\\}.$

We refer to §5 for the precise definition.

######  Theorem 1.2.

Every complete balanced $(M^{n+1},g)$ of dimension $3\leq n+1\leq 7$ admits a
smooth complete embedded minimal hypersurface of finite area and index at most
1.

In §5, we present an example to show the necessity of the balanced condition.

We emphasize that Theorem 1.2 applies to manifolds exhausted by compact
subsets with smooth boundaries whose area tends to zero, which includes finite
volume manifolds. Hence, we provide a simpler proof of [CL20, Corollary 1.3]
and [Son19, Corollary 2.2 (1)]:

######  Corollary 1.3.

Every complete finite volume $(M^{n+1},g)$ of dimension $3\leq n+1\leq 7$
admits a smooth complete embedded minimal hypersurface of finite area and
index at most 1.

In most cases (see Remark 5.3), Theorem 1.2 is completely new, as it requires
the sharpness of our localization result.

###  1.4. Sketch of the proof of Theorem 1.1

As compared to the careful cut-and-paste constructions of [CL20], our proof is
extremely geometrically simple. The ideas here somewhat resemble the “cutting
along hypersurfaces” arguments in [Son19], but our proof involves many fewer
complications and tools. For example, we do not require the use of the level
set flow.

We first reduce to the case of compact manifolds with smooth, compact,
strictly mean convex boundary by modifying the metric near the boundary of a
compact exhaustion of $M$ and using the compactness theory of [Sha17].

We further reduce to the case where the metric is bumpy, again using the
compactness theory of [Sha17].

By the resolution of the multiplicity one conjecture due to [Zho20] in bumpy
metrics and the quantitative minimality of strictly stable minimal
hypersurfaces due to [IM18] (see also [MN21]), min-max will produce a
connected _unstable_ 2-sided smooth closed embedded minimal hypersurface
$\Sigma$ with bounded area and bounded index. Suppose that $\Sigma$ does not
intersect $\mathrm{span}(\Phi_{0})$. Since each side of a small tubular
neighborhood of a connected 2-sided unstable minimal hypersurface has a
strictly mean convex foliation, there is a subdomain $W\subset M$ disjoint
from a component of $\Sigma$ and containing $\mathrm{span}(\Phi_{0})$ that has
strictly mean convex boundary, so we can now restart the argument in $W$.
Since there are only finitely many smooth closed embedded minimal
hypersurfaces satisfying the area and index bound (by bumpiness), we produce a
hypersurface that intersects $\mathrm{span}(\Phi_{0})$ after only finitely
many iterations of this argument. We refer the reader to Figure 1 for an
illustration of the argument.

$\mathrm{span}(\Phi_{0})$$M$$\Phi_{0}(0)$$\Phi_{0}(1)$$\Sigma$$\partial
W$$\partial M$$\partial M$ Figure 1. An illustration of the proof of Theorem
1.1.

######  Acknowledgements.

I am grateful to my advisor Fernando Codá Marques for his support and for
suggesting the topic of min-max in non-compact spaces. I am also grateful to
Lorenzo Sarnataro and Otis Chodosh for providing feedback on earlier versions
of this paper.

I was supported by an NDSEG Fellowship.

##  2\. Setup and Notation

Let $(M^{n+1},g)$ be a smooth Riemannian manifold, either closed, complete
non-compact, or compact with smooth compact boundary.

######  Definition 2.1 ($\mathcal{C}(M)$).

We define the appropriate class of finite perimeter sets in each case:

  * •

If $(M,g)$ is closed, we define $\mathcal{C}(M)$ to be the set of finite
perimeter sets $\Omega\subset M$.

  * •

If $(M,g)$ is complete non-compact, we define $\mathcal{C}(M)$ to be the set
of finite perimeter sets $\Omega\subset M$ so that $\partial\Omega$ is bounded
and nonempty. We emphasize that $\Omega\in\mathcal{C}(M)$ need not be bounded;
we only require that $\partial\Omega$ is bounded.

  * •

If $(M,g)$ is compact with smooth compact boundary, we define $\mathcal{C}(M)$
to be the set of finite perimeter sets $\Omega\subset M$ so that
$\partial\Omega\cap\mathrm{Int}(M)\neq\varnothing$ and
$\overline{\partial\Omega\cap\mathrm{Int}(M)}\cap\partial M=\varnothing$. In
other words, $\partial\Omega\cap\mathrm{Int}(M)$ is nonempty and contained in
a compact subset of $\mathrm{Int}(M)$.

We note that in any case, if $W\subset M$ is a bounded open domain with smooth
boundary, $\Omega\in\mathcal{C}(M)$, and $\partial\Omega\subset W$, then
$\Omega\cap W\in\mathcal{C}(W)$.

Henceforth, we follow the setup and notation of [Zho20].

######  Definition 2.2 (${\mathbf{F}}$-metric on $\mathcal{C}(M)$).

For $\Omega\in\mathcal{C}(M)$, we let $[\Omega]$ denote the integral
$(n+1)$-current associated to $\Omega$, and we let $|\partial\Omega|$ denote
the varifold associated to $\partial\Omega$. For $\Omega_{1},\
\Omega_{2}\in\mathcal{C}(M)$, we define

${\mathbf{F}}(\Omega_{1},\Omega_{2}):=\mathcal{F}([\Omega_{1}]-[\Omega_{2}])+{\mathbf{F}}(|\partial\Omega_{1}|,|\partial\Omega_{2}|),$

where $\mathcal{F}$ is the flat norm on currents and ${\mathbf{F}}$ is the
usual ${\mathbf{F}}$-metric on varifolds.

Let $X^{k}\subset I^{m}$ be a $k$-dimensional cubical complex, and let
$Z\subset X$ be a cubical subcomplex. Let $\Phi_{0}:X\to\mathcal{C}(M)$ be a
continuous map in the ${\mathbf{F}}$ topology.

######  Definition 2.3 (Span).

The _span_ of $\Phi_{0}$ is the set

$\mathrm{span}(\Phi_{0}):=\overline{\bigcup_{x\in X}\partial\Phi_{0}(x)}.$

By definition, $\mathrm{span}(\Phi_{0})\subset\mathrm{Int}(M)$ is compact.

######  Definition 2.4 (Homotopy class).

Let $\Pi(\Phi_{0},M)$ be the set of all sequences of ${\mathbf{F}}$-continuous
maps $\\{\Phi_{i}:X\to\mathcal{C}(M)\\}_{i\in\mathbb{N}}$ with the property
that there are flat-continuous homotopies $\\{\Psi_{i}:[0,1]\times
X\to\mathcal{C}(M)\\}_{i\in\mathbb{N}}$ satisfying
$\Psi_{i}(0,\cdot)=\Phi_{0}$, $\Psi_{i}(1,\cdot)=\Phi_{i}$, and

$\limsup_{i\to\infty}\sup\\{{\mathbf{F}}(\Psi_{i}(s,z),\Phi_{0}(z))\mid
s\in[0,1],\ z\in Z\\}=0.$

$\Pi(\Phi_{0},M)$ is called the _homotopy class of $\Phi_{0}$ in $M$_.

######  Definition 2.5 (Homotopy class on subdomain).

If $W\subset M$ is a bounded open domain with smooth boundary and
$\mathrm{span}(\Phi_{0})\subset W$, then we define

$\Pi(\Phi_{0},W):=\Pi(\Phi_{0}\cap W,W),$

where $(\Phi_{0}\cap W)(x):=\Phi_{0}(x)\cap W$. $\Pi(\Phi_{0},W)$ is called
the _homotopy class of $\Phi_{0}$ in $W$_.

######  Definition 2.6 (Width).

The _width_ of $\Pi(\Phi_{0},W)$ is

${\mathbf{L}}(\Phi_{0},W):=\inf_{\\{\Phi_{i}\\}\in\Pi(\Phi_{0},W)}\limsup_{i\to\infty}\sup_{x\in
X}\mathcal{H}^{n}(\partial\Phi_{i}(x)).$

######  Remark 2.7.

We emphasize the ambient space in the definition because we must consider
restrictions to subdomains. It follows from a straightforward extension
argument that if $\mathrm{span}(\Phi_{0})\subset W_{1}\subset W_{2}\subset M$
are bounded open domains with smooth boundary, then

${\mathbf{L}}(\Phi_{0},W_{1})\geq{\mathbf{L}}(\Phi_{0},W_{2}).$

######  Definition 2.8 (Width on subset).

If $\mathrm{span}(\Phi_{0})\subset C\subset M$ is an arbitrary subset, we
define

${\mathbf{L}}(\Phi_{0},C):=\sup\\{{\mathbf{L}}(\Phi_{0},W)\mid W\supset
C\text{\ bounded open domain with smooth boundary}\\},$

and the supremum is finite because ${\mathbf{L}}(\Phi_{0},W)\leq\sup_{x\in
X}\mathcal{H}^{n}(\partial\Phi_{0}(x))<\infty$ for any bounded open domain
with smooth boundary $W$ containing $\mathrm{span}(\Phi_{0})$.

In particular, this definition allows us to make sense of
${\mathbf{L}}(\Phi_{0},\mathrm{span}(\Phi_{0}))$.

##  3\. Localization in Compact Manifolds

Suppose $(M^{n+1},g)$ is a smooth compact Riemannian manifold of dimension
$3\leq n+1\leq 7$ with smooth compact (possibly empty) boundary.

It will be convenient to approximate arbitrary metrics by _bumpy_ metrics.

######  Definition 3.1.

The metric $g$ is _weakly bumpy_ if every smooth closed embedded minimal
hypersurface in $M$ is non-degenerate.

######  Lemma 3.2.

The set of weakly bumpy metrics on $M$ is $C^{\infty}$ dense.

###### Proof.

Let $g$ be any metric on $M$. By a standard extension argument, there is a
smooth closed Riemannian manifold $(N,\overline{g})$ that has $(M,g)$ as a
subset. By White’s bumpy metric theorem [Whi91, Theorem 2.2], there is a
sequence of bumpy metrics on $N$ converging smoothly to $\overline{g}$. The
restrictions of these metrics to $M$ are weakly bumpy by definition, and they
converge smoothly to $g$. ∎

We collect facts about min-max from many foundational works in the following
lemma.

######  Lemma 3.3.

Suppose the metric $g$ is weakly bumpy and $\partial M$ is strictly mean
convex. If

${\mathbf{L}}(\Phi_{0},M)>\sup_{z\in Z}\mathcal{H}^{n}(\partial\Phi_{0}(z)),$

then there is a 2-sided smooth closed embedded minimal hypersurface
$\Sigma\subset M$ satisfying

  * •

$\mathcal{H}^{n}(\Sigma)={\mathbf{L}}(\Phi_{0},M)$,

  * •

$1\leq\operatorname{index}(\Sigma)\leq k$.

###### Proof.

We first review the proof in the closed case (i.e. $\partial M=\varnothing$).
By [Zho20, Theorem 4.1], there is a 2-sided smooth closed embedded minimal
hypersurface $\Sigma\subset M$ satisfying
$\mathcal{H}^{n}(\Sigma)={\mathbf{L}}(\Phi_{0},M)$ and
$\operatorname{index}(\Sigma)\leq k$. Since $\Sigma$ is nondegenerate, [IM18,
Theorem 1.1] (see also [MN21, Theorem 6.1]) implies that $\Sigma$ cannot be
stable, so $\operatorname{index}(\Sigma)\geq 1$.

In the case of strictly mean convex boundary, we observe as in [Wan22] (noting
that by definition $\partial\Omega\cap\mathrm{Int}(M)$ for
$\Omega\in\mathcal{C}(M)$ is a cycle in $\mathrm{Int}(M)$) that we can perform
the pull-tight procedure in the min-max construction using only isotopies
generated by vector fields pointing into $M$ to produce constrained stationary
varifolds, which are strictly contained in the interior of $M$ by the strict
mean convexity of $\partial M$ and the strong maximum principle (see [Whi10,
Theorem 7]). The regularity theory, index lower bound, index upper bound, and
approximation by PMC hypersurfaces are all local arguments and still apply in
this setting444The observation that these local arguments extend to manifolds
with strictly mean convex boundary can be seen in the proof of [Son19, Theorem
6.6].. We note that we must choose the approximating prescribing function to
be small relative the the positive lower bound on the mean curvature of
$\partial M$ so that we can constrain the PMC solutions to the interior of
$M$. There is no problem because we ultimately take the prescribing function
to zero uniformly in the proof of [Zho20, Theorem 4.1]. ∎

The essential point of Lemma 3.3 is that the min-max hypersurface is
_unstable_. We can now remove the bumpiness assumption and argue by
approximation.

######  Theorem 3.4.

Suppose $\partial M$ is strictly mean convex. If

${\mathbf{L}}(\Phi_{0},M)>\sup_{z\in Z}\mathcal{H}^{n}(\partial\Phi_{0}(z)),$

then there exists a smooth closed embedded minimal hypersurface $\Sigma\subset
M$ satisfying

  * •

$\mathcal{H}^{n}(\Sigma)\leq{\mathbf{L}}(\Phi_{0},\mathrm{span}(\Phi_{0}))$,

  * •

$\operatorname{index}(\Sigma)\leq k$,

  * •

$\Sigma\cap\mathrm{span}(\Phi_{0})\neq\varnothing$.

###### Proof.

We begin by assuming that the metric $g$ is weakly bumpy.

Let $\mathcal{M}$ denote the set of connected 2-sided smooth closed embedded
minimal hypersurfaces $\Sigma\subset M$ satisfying

  * •

$\mathcal{H}^{n}(\Sigma)\leq{\mathbf{L}}(\Phi_{0},\mathrm{span}(\Phi_{0}))$,

  * •

$1\leq\operatorname{index}(\Sigma)\leq k$.

By Sharp’s compactness theorem [Sha17, Theorem 2.3] $\mathcal{M}$ is a finite
set.

By Lemma 3.3 applied to ${\mathbf{L}}(\Phi_{0},M)$,
$\mathcal{M}\neq\varnothing$.

Suppose for contradiction that $\Sigma\cap\mathrm{span}(\Phi_{0})=\varnothing$
for all $\Sigma\in\mathcal{M}$.

Let $\mathcal{D}\subset\mathcal{M}$ be a maximal disjoint subset; namely, (1)
if $\Sigma,\Sigma^{\prime}\in\mathcal{D}$ and
$\Sigma\cap\Sigma^{\prime}\neq\varnothing$, then $\Sigma=\Sigma^{\prime}$, and
(2) every $\Sigma\in\mathcal{M}$ satisfies
$\Sigma\cap\Sigma^{\prime}\neq\varnothing$ for some
$\Sigma^{\prime}\in\mathcal{D}$. Such a set $\mathcal{D}$ exists by induction.

Since each $\Sigma\in\mathcal{D}$ is 2-sided and has index at least 1, there
is a mean convex foliation (with curvature pointing away from $\Sigma$) in a
neighborhood of $\Sigma$ obtained by flowing $\Sigma$ in the normal direction
according to the positive first eigenfunction of the Jacobi operator (see
[HK19, Proof of Lemma 3.2]). Hence, by deleting a small neighborhood of each
$\Sigma\in\mathcal{D}$ (and then deleting any resulting components that do not
contain $\mathrm{span}(\Phi_{0})$), there is an open set $W\subset M$
satisfying

  * •

$\mathrm{span}(\Phi_{0})\subset W$,

  * •

$W$ is disjoint from $\Sigma$ for all $\Sigma\in\mathcal{D}$,

  * •

$W$ has smooth, strictly mean convex boundary.

By Lemma 3.3 for ${\mathbf{L}}(\Phi_{0},W)$, there is a smooth closed embedded
minimal hypersurface $\Sigma^{*}\subset W$ so that $\Sigma^{*}\in\mathcal{M}$,
contradicting the defining property of $\mathcal{D}$.

By Sharp’s compactness theorem [Sha17, Theorem A.6] for changing ambient
metrics and Lemma 3.2, the conclusion now follows for arbitrary metrics. ∎

##  4\. Localization in Complete Non-Compact Manifolds

Suppose $(M^{n+1},g)$ is a smooth complete non-compact Riemannian manifold of
dimension $3\leq n+1\leq 7$.

######  Theorem 4.1.

If

${\mathbf{L}}(\Phi_{0},M)>\sup_{z\in Z}\mathcal{H}^{n}(\partial\Phi_{0}(z)),$

then there exists a smooth complete (possibly non-compact) embedded minimal
hypersurface $\Sigma\subset M$ satisfying

  * •

$\mathcal{H}^{n}(\Sigma)\leq{\mathbf{L}}(\Phi_{0},\mathrm{span}(\Phi_{0}))$,

  * •

$\operatorname{index}(\Sigma)\leq k$,

  * •

$\Sigma\cap\mathrm{span}(\Phi_{0})\neq\varnothing$.

###### Proof.

Let $\\{M_{i}\\}_{i\in\mathbb{N}}$ be an exhaustion of $M$ by nested
precompact open subsets with smooth boundary satisfying
$\mathrm{span}(\Phi_{0})\subset M_{1}$.

We construct a modified metric $\tilde{g}_{i}$ on $M_{i}$ so that

  * •

$\tilde{g}_{i}$ agrees with $g$ outside $B_{1}(\partial M_{i})\setminus
B_{1}(\mathrm{span}(\Phi_{0}))$,

  * •

$\tilde{g}_{i}\geq g$ in the sense of bilinear forms,

  * •

$\partial M_{i}$ has strictly mean convex boundary in $\tilde{g}_{i}$.

Namely, using Fermi coordinates in a small tubular neighborhood of $\partial
M_{i}$, we can smoothly transition from the metric $g$ to a metric on the
cylinder $(0,1)\times\partial M_{i}$ that is pointwise larger than $g$ and so
that $\\{0\\}\times M_{i}$ has strictly mean convex boundary.

Since $\tilde{g}_{i}\geq g$, we have
${\mathbf{L}}(\Phi_{0},M_{i},\tilde{g}_{i})\geq{\mathbf{L}}(\Phi_{0},M_{i},g)\geq{\mathbf{L}}(\Phi_{0},M,g)$.
Moreover, since $\tilde{g}_{i}=g$ on $B_{1}(\mathrm{span}(\Phi_{0}))$, we have
${\mathbf{L}}(\Phi_{0},M_{i},\tilde{g}_{i})\leq{\mathbf{L}}(\Phi_{0},\mathrm{span}(\Phi_{0}),\tilde{g}_{i})={\mathbf{L}}(\Phi_{0},\mathrm{span}(\Phi_{0}),g)$.

By Theorem 3.4, there is a smooth closed embedded minimal hypersurface
$\Sigma_{i}\subset M_{i}$ (with respect to $\tilde{g}_{i}$) satisfying

  * •

$\mathcal{H}^{n}(\Sigma_{i})\leq{\mathbf{L}}(\Phi_{0},\mathrm{span}(\Phi_{0}),g)$,

  * •

$\operatorname{index}_{\tilde{g}_{i}}(\Sigma)\leq k$,

  * •

$\Sigma_{i}\cap\mathrm{span}(\Phi_{0})=\varnothing$.

By Sharp’s compactness theorem [Sha17, Theorem 2.3]555While the statement of
[Sha17, Theorem 2.3] concerns closed hypersurfaces in closed manifolds, the
arguments are local and apply on compact subsets of complete manifolds., the
sequence $\\{\Sigma_{i}\\}$ converges locally smoothly away from at most $k$
points to a smooth complete (not necessarily compact) embedded minimal
hypersurface $\Sigma\subset M$ satisfying the conclusions of the theorem. ∎

###  4.1. Comparison to [CL20]

We begin by observing that we use the Almgren-Pitts min-max framework as in
[Zho20] (for example), whereas [CL20] uses the constructions of [DLT13]. Since
the equivalence of these frameworks is besides the point, we make comparisons
by analogy only.

Let $(M^{n+1},g)$ be a smooth complete Riemannian manifold, and let $U\subset
M$ be a bounded open set with smooth boundary.

In [CL20], the _width_ of $U$, denoted $W(U)$, is defined to be the min-max
width over families of hypersurfaces
$\\{\Sigma_{t}=\partial\Omega_{t}\\}_{t\in[0,1]}$ as in [DLT13] so that
$U\cap\Omega_{0}=\varnothing$ and $U\subset\Omega_{1}$. The _relative width_
of $U$, denoted $W_{\partial}(U)$, is defined to be the min-max width over
families as above that are furthermore nested, and where we only compute the
area of each hypersurface inside $U$. Ultimately, [CL20, Theorem 8.2] shows
that for any $\delta>0$, there is a smooth complete (not necessarily compact)
embedded minimal hypersurface of area at most
$W_{\partial}(U)+\mathcal{H}^{n}(\partial U)$ intersecting the
$\delta$-neighborhood of $U$ so long as

$W_{\partial}(U)\geq 10\mathcal{H}^{n}(\partial U).$

In our framework, let $\\{\Sigma_{t}=\partial\Omega_{t}\\}$ be a nested family
of hypersurfaces satisfying $\Sigma_{0}\sqcup\Sigma_{1}=\partial U$. We can
view this family as a map $\Phi_{0}:[0,1]\to\mathcal{C}(M)$, which satisfies
$\mathrm{span}(\Phi_{0})=\overline{U}$. We show that there is a smooth
complete (not necessarily compact) embedded minimal hypersurface of area at
most ${\mathbf{L}}(\Phi_{0},\overline{U})$ and index at most 1 intersecting
$\overline{U}$ so long as

${\mathbf{L}}(\Phi_{0},M)>\max\\{\mathcal{H}^{n}(\Sigma_{0}),\mathcal{H}^{n}(\Sigma_{1})\\}.$

We draw the analogy between the two frameworks.

  * •

${\mathbf{L}}(\Phi_{0},M)$ is analogous to $W(U)$, and $W(U)\geq
W_{\partial}(U)$.

  * •

${\mathbf{L}}(\Phi_{0},\overline{U})$ is analogous to a quantity bounded from
above by $W_{\partial}(U)+\mathcal{H}^{n}(\partial U)$.

  * •

$\max\\{\mathcal{H}^{n}(\Sigma_{0}),\mathcal{H}^{n}(\Sigma_{1})\\}\leq\mathcal{H}^{n}(\partial
U)$.

Hence, our result has weaker hypotheses and stronger conclusions than [CL20].

######  Remark 4.2.

The result of [CL20] applies when the dimension is $n+1\geq 3$, with the usual
optimal non-smooth regularity when $n+1>7$. We emphasize that our proof only
works when $3\leq n+1\leq 7$, which are the dimensions where [CL20] produces a
_smooth_ minimal hypersurface.

##  5\. Existence in Balanced Manifolds

Let $(M^{n+1},g)$ be a smooth complete non-compact Riemannian manifold of
dimension $3\leq n+1\leq 7$.

Let $K\subset M$ be a compact set with smooth compact boundary. Let $E\subset
M\setminus K$ be the disjoint union of some of the components of $M\setminus
K$ (where we allow $E=\varnothing$ and $E=M\setminus K$). We let
$F:=(M\setminus K)\setminus E$.

Let $\alpha\in H_{n}(M,\mathbb{Z}/2\mathbb{Z})$ be the homology class of
$\partial E$ in $M$ (which is the same as the homology class of $\partial F$
in $M$). Let $\alpha_{E}\in H_{n}(E,\mathbb{Z}/2\mathbb{Z})$ be the homology
class of $\partial E$ in $E$. Let $\alpha_{F}\in
H_{n}(F,\mathbb{Z}/2\mathbb{Z})$ be the homology class of $\partial F$ in $F$.

Fix some $x_{0}\in K$.

We define

$\mathcal{A}_{E}:=\lim_{r\to\infty}\inf\\{\mathbb{M}(T)\mid
T\in\alpha_{E}\mathrm{\ and\ }\operatorname{spt}(T)\subset E\setminus
B_{r}(x_{0})\\},$ $\mathcal{A}_{F}:=\lim_{r\to\infty}\inf\\{\mathbb{M}(T)\mid
T\in\alpha_{F}\mathrm{\ and\ }\operatorname{spt}(T)\subset F\setminus
B_{r}(x_{0})\\}.$

For convenience, we let $\mathcal{A}_{E}$ (resp. $\mathcal{A}_{F}$) equal 0 if
$E$ (resp. $F$) is bounded.

We say $(M,g)$ is _balanced_ if there is a choice of $K$, $x_{0}$, $E$, and
$F$ so that $\mathcal{A}_{E}=\mathcal{A}_{F}$.

######  Theorem 5.1.

Every complete balanced $(M^{n+1},g)$ of dimension $3\leq n+1\leq 7$ admits a
smooth complete (not necessarily compact) embedded minimal hypersurface of
finite area and index at most 1.

######  Remark 5.2.

If $(M,g)$ is furthermore _thick at infinity_ , meaning that any connected
finite area complete minimal hypersurface in $(M,g)$ is closed (see [Son19]),
then the minimal embedding is closed.

######  Remark 5.3.

We emphasize that Theorem 5.1 is completely new when
$\mathcal{A}_{E}=\mathcal{A}_{F}>0$, as the localization results of [CL20] and
[Son19] are insufficient to handle these cases.

###  5.1. Examples

We present two simple classes of manifolds that satisfy the hypotheses of
Theorem 5.1.

######  Example 5.4 (Finite Volume Manifolds).

Suppose $(M^{n+1},g)$ has finite volume. We can define
$K=\overline{B}_{2\varepsilon}(x)\setminus B_{\varepsilon}(x)$ for some $x\in
M$ and $\varepsilon<\mathrm{inj}(M,p)$, and let $E=B_{\varepsilon}(x)$ and
$F=M\setminus\overline{B}_{2\varepsilon}(x)$. Since $E$ is bounded, we have
$\mathcal{A}_{E}=0$. Since $M$ has finite volume, $\mathcal{A}_{F}=0$.

In fact, the same argument applies to any manifold admitting an exhaustion by
compact sets with smooth boundaries whose areas tend to zero666In fact, the
proof in this case follows immediately from the relative isoperimetric
inequality in a compact subset with smooth boundary and Theorem 1.1., as in
[Son19].

######  Example 5.5 (Balanced Asymptotically Cylindrical Manifolds).

Let $(M^{n+1},g)$ be _asymptotically cylindrical_. Namely, there is a compact
subset $K\subset M$, a smooth closed (not necessarily connected) Riemannian
manifold $(N^{n},h)$, and a diffeomorphism $\phi:\mathbb{R}_{+}\times N\to
M\setminus K$ so that

$\limsup_{r\to\infty}\|(dt^{2}\oplus
h)-\phi^{*}g\|_{C^{\infty}((r,\infty)\times N)}=0.$

These manifolds generalize the manifolds with cylindrical ends considered by
[Son23], although they need not be rigid on any finite scale.

If there is a decomposition of $N=R\sqcup S$ so that each of $R$ and $S$ is
the disjoint union of components of $N$ and
$\mathcal{H}^{n}(R)=\mathcal{H}^{n}(S)=\frac{1}{2}\mathcal{H}^{n}(N)$, then
$(M,g)$ is balanced and Theorem 5.1 applies.

######  Example 5.6 (Unbalanced).

The balanced condition in Theorem 5.1 is necessary, as demonstrated by the
following example. Let $(\mathbb{R}\times S^{n},g)$ be a warped product metric
given by

$g=dt^{2}\oplus f(t)g_{S^{n}},$

where $g_{S^{n}}$ is the round metric on $S^{n}$ and $f:\mathbb{R}\to(1/2,2)$
is a strictly decreasing smooth function. Then $(M,g)$ has a strictly mean
convex foliation and bounded geometry, so $(M,g)$ does not admit any finite
area embedded minimal hypersurfaces. Importantly, since
$\lim_{t\to-\infty}f(t)>\lim_{t\to\infty}f(t)$, $(M,g)$ is not balanced.

###  5.2. Proof of Theorem 5.1

We define two geometric invariants.

######  Definition 5.7.

The _area_ of the homology class $\alpha\in H_{n}(M,\mathbb{Z}/2\mathbb{Z})$
is

$\operatorname{area}(\alpha):=\inf_{T\in\alpha}\mathbb{M}(T).$

######  Definition 5.8.

Let

$\mathcal{A}_{E}^{r}:=\inf\\{\mathbb{M}(T)\mid T\in\alpha_{E}\text{\ and\
}\operatorname{spt}(T)\subset E\setminus B_{r}(x_{0})\\},$
$\mathcal{A}_{F}^{r}:=\inf\\{\mathbb{M}(T)\mid T\in\alpha_{F}\text{\ and\
}\operatorname{spt}(T)\subset F\setminus B_{r}(x_{0})\\}.$

Let $\mathcal{S}_{r}$ denote the set of ${\mathbf{F}}$-continuous maps
$\Phi:[0,1]\to\mathcal{C}(M)$ so that777In the case that (without loss of
generality) $E$ is bounded, we instead require that $\partial\Phi(0)\subset E$
and satisfies $\mathcal{H}^{n}(\partial\Phi(0))\leq 1/r$.

  * •

$\partial\Phi(t)\in\alpha$ for all $t\in[0,1]$,

  * •

$\partial\Phi(0)\subset E\setminus B_{r}(x_{0})$ and satisfies
$\mathcal{H}^{n}(\partial\Phi(0))\leq\mathcal{A}_{E}^{r}+\frac{1}{r}$,

  * •

$\partial\Phi(1)\subset F\setminus B_{r}(x_{0})$ and satisfies
$\mathcal{H}^{n}(\partial\Phi(1))\leq\mathcal{A}_{F}^{r}+\frac{1}{r}$.

The _width_ of the homology class $\alpha\in H_{n}(M,\mathbb{Z}/2\mathbb{Z})$
is

$\operatorname{width}(\alpha):=\limsup_{r\to\infty}\sup_{\Phi\in\mathcal{S}_{r}}{\mathbf{L}}(\Phi,M).$

Note that we always have
$\operatorname{area}(\alpha)\leq\operatorname{width}(\alpha)$.

######  Lemma 5.9.

If $\operatorname{area}(\alpha)<\min\\{\mathcal{A}_{E},\mathcal{A}_{F}\\}$,
then there is a smooth complete (not necessarily compact) embedded minimal
hypersurface $\Sigma\subset M$ satisfying

  * •

$\mathcal{H}^{n}(\Sigma)\leq\operatorname{area}(\alpha)$,

  * •

$\operatorname{index}(\Sigma)=0$.

###### Proof.

Let $\\{\Omega_{i}\\}_{i\in\mathbb{N}}$ be an exhaustion of $M$ by precompact
connected nested open sets with smooth boundary. We suppose without loss of
generality that $K\subset\Omega_{i}$ for all $i$.

We define

$\operatorname{area}_{i}(\alpha):=\inf_{T\in\alpha\mid\operatorname{spt}(T)\subset\overline{\Omega}_{i}}\mathbb{M}(T).$

Since the sets $\Omega_{i}$ are nested, we have
$\operatorname{area}_{i}(\alpha)\geq\operatorname{area}_{j}(\alpha)$ if $i<j$.
Since $\Omega_{i}$ is an exhaustion of $M$ and each $T\in\alpha$ has compact
support, we have

$\lim_{i\to\infty}\operatorname{area}_{i}(\alpha)=\operatorname{area}(\alpha).$

By the standard compactness theory for currents (see [Sim14, Theorem 6.8.2]),
there is a current $T_{i}^{*}\in\alpha$ satisfying
$\operatorname{spt}(T_{i}^{*})\subset\overline{\Omega}_{i}$ and
$\mathbb{M}(T_{i}^{*})=\operatorname{area}_{i}(\alpha)$. Since $T_{i}^{*}$ is
area minimizing in $\Omega_{i}$, $T_{i}^{*}\llcorner\Omega_{i}$ is the current
of a smooth stable embedded minimal hypersurface with integer multiplicity
(see [Sim14, Theorem 7.5.8]).

Since $\min\\{\mathcal{A}_{E},\mathcal{A}_{F}\\}>\operatorname{area}(\alpha)$,
there is a compact set $C\subset M$ so that $\operatorname{spt}(T_{i}^{*})\cap
C\neq\varnothing$ for all $i$ sufficiently large.

By the curvature estimates for stable minimal immersions [SS81, Corollary 1],
there is a subsequence of $\\{T_{i}^{*}\\}_{i\in\mathbb{N}}$ that converges
locally smoothly to a nontrivial smooth complete (not necessarily closed)
stable embedded minimal hypersurface $\Sigma\subset M$ satisfying
$\mathcal{H}^{n}(\Sigma)\leq\operatorname{area}(\alpha)$. ∎

######  Lemma 5.10.

If $\operatorname{width}(\alpha)>\max\\{\mathcal{A}_{E},\mathcal{A}_{F}\\}$,
then there is a smooth complete (not necessarily compact) embedded minimal
hypersurface $\Sigma\subset M$ satisfying

  * •

$\mathcal{H}^{n}(\Sigma)<\infty$,

  * •

$\operatorname{index}(\Sigma)\leq 1$.

###### Proof.

By assumption, there is an ${\mathbf{F}}$-continuous map
$\Phi:[0,1]\to\mathcal{C}(M)$ satisfying

$\operatorname{width}(\alpha)\geq{\mathbf{L}}(\Phi,M)>\max\\{\mathcal{H}^{n}(\partial\Phi(0)),\mathcal{H}^{n}(\partial\Phi(1))\\}.$

Hence, Theorem 4.1 applies, and the conclusion follows. ∎

######  Lemma 5.11.

If $\operatorname{area}(\alpha)=\operatorname{width}(\alpha)$, then for every
$x\in M$ there is a smooth complete (not necessarily compact) embedded minimal
hypersurface $\Sigma_{x}\subset M$ satisfying

  * •

$\mathcal{H}^{n}(\Sigma_{x})\leq\operatorname{area}(\alpha)$,

  * •

$\operatorname{index}(\Sigma_{x})=0$,

  * •

$x\in\Sigma_{x}$.

In particular, there are infinitely many such minimal embeddings.

###### Proof.

Fix $x\in M$. Fix $D\subset M$ a compact set with smooth boundary whose
interior contains $x$. By the relative isoperimetric inequality, there is a
constant $c>0$ so that for any $\Omega\in\mathcal{C}(M)$ satisfying
$\mathcal{H}^{n+1}(\Omega\cap D)=\frac{1}{2}\mathcal{H}^{n+1}(D)$, we have
$\mathcal{H}^{n}(\partial\Omega\cap D)\geq c$.

Let $r_{j}\to\infty$, and let $\Phi^{j}\in\mathcal{S}_{r_{j}}$. Let
$\\{\Phi_{i}^{j}\\}_{i\in\mathbb{N}}\in\Pi(\Phi^{j},M)$ so that

$\limsup_{i\to\infty}\sup_{x\in
X}\mathcal{H}^{n}(\partial\Phi_{i}^{j}(x))={\mathbf{L}}(\Phi^{j},M),$

which exists by taking a diagonal sequence. Since $D\subset B_{r_{j}}(x_{0})$
for all $j$ (without loss of generality), there is some $t_{i}^{j}\in[0,1]$ so
that $\mathcal{H}^{n+1}(\Phi_{i}^{j}(t_{i}^{j})\cap
D)=\frac{1}{2}\mathcal{H}^{n+1}(D)$. Moreover, by the assumption that
$\operatorname{width}(\alpha)=\operatorname{area}(\alpha)$, there is a
diagonal sequence so that
$\Sigma_{j}:=\partial\Phi_{i_{j}}^{j}(t_{i_{j}}^{j})$ is a minimizing sequence
for area in $\alpha$ and satisfies $\mathcal{H}^{n}(\Sigma_{j}\cap D)\geq c$
for all $j$. Hence, there is a smooth complete (not necessarily compact)
embedded stable minimal hypersurface $\Sigma\subset M$ satisfying
$\mathcal{H}^{n}(\Sigma)\leq\operatorname{area}(\alpha)$ and $\Sigma\cap
D\neq\varnothing$.

By taking the set $D$ smaller, the compactness theory for stable minimal
hypersurfaces with uniform area bounds (see [SS81, Corollary 1]) produces the
desired minimal hypersurface. ∎

Since $\mathcal{A}_{E}=\mathcal{A}_{F}$ for balanced manifolds, either Lemma
5.11 or one of Lemma 5.9 or 5.10 applies, so Theorem 5.1 follows.

## References

  * [Ban80] Victor Bangert, _Closed geodesics on complete surfaces_ , Math. Ann. 251 (1980), no. 1, 83–96.
  * [CDL03] Tobias H. Colding and Camillo De Lellis, _The min-max construction of minimal surfaces_ , Surveys in differential geometry 8 (2003), 75–107.
  * [CL20] Gregory R. Chambers and Yevgeny Liokumovich, _Existence of minimal hypersurfaces in complete manifolds of finite volume_ , Invent. Math. 219 (2020), no. 1, 179–217.
  * [DLP10] Camillo De Lellis and Filippo Pellandini, _Genus bounds for minimal surfaces arising from min-max constructions_ , J. Reine Angew. Math. 644 (2010), 47–99.
  * [DLT13] Camillo De Lellis and Dominik Tasnady, _The existence of embedded minimal hypersurfaces_ , J. Differential Geom. 95 (2013), no. 3, 355–388.
  * [Gro14] Misha Gromov, _Plateau-Stein manifolds_ , Cent. Eur. J. Math. 12 (2014), no. 7, 923–951.
  * [HK19] Robert Haslhofer and Daniel Ketover, _Minimal 2-spheres in 3-spheres_ , Duke Math. J. 168 (2019), no. 10, 1929–1975.
  * [IM18] Dominik Inauen and Andrea Marchese, _Quantitative minimality of strictly stable extremal submanifolds in a flat neighbourhood_ , J. Funct. Anal. 275 (2018), no. 6, 1532–1550.
  * [IMN18] Kei Irie, Fernando C. Marques, and André Neves, _Density of minimal hypersurfaces for generic metrics_ , Ann. of Math. (2) 187 (2018), no. 3, 963–972.
  * [Ket19] Daniel Ketover, _Genus bounds for min-max minimal surfaces_ , J. Differential Geom. 112 (2019), no. 3, 555–590.
  * [Ket22] by same author, _Flipping Heegaard splittings and minimal surfaces_ , arXiv e-prints (2022), arXiv:2211.03745.
  * [LMN18] Yevgeny Liokumovich, Fernando C. Marques, and André Neves, _Weyl law for the volume spectrum_ , Ann. of Math. (2) 187 (2018), no. 3, 933–961.
  * [LS47] Lazar Lyusternik and Lev Schnirelmann, _Topological methods in variational problems and their application to the differential geometry of surfaces_ , Uspehi Matem. Nauk (N.S.) 2 (1947), no. 1(17), 166–217.
  * [MN16] Fernando C. Marques and André Neves, _Morse index and multiplicity of min-max minimal hypersurfaces_ , Camb. J. Math. 4 (2016), no. 4, 463–511.
  * [MN17] by same author, _Existence of infinitely many minimal hypersurfaces in positive Ricci curvature_ , Invent. Math. 209 (2017), no. 2, 577–616.
  * [MN21] by same author, _Morse index of multiplicity one min-max minimal hypersurfaces_ , Adv. Math. 378 (2021), Paper No. 107527, 58.
  * [MNS19] Fernando C. Marques, André Neves, and Antoine Song, _Equidistribution of minimal hypersurfaces for generic metrics_ , Invent. Math. 216 (2019), no. 2, 421–443.
  * [Mon16] Rafael Montezuma, _Min-max minimal hypersurfaces in non-compact manifolds_ , J. Differential Geom. 103 (2016), no. 3, 475–519.
  * [Pit81] Jon T. Pitts, _Existence and regularity of minimal surfaces on Riemannian manifolds_ , Mathematical Notes, vol. 27, Princeton University Press, Princeton, NJ; University of Tokyo Press, Tokyo, 1981.
  * [Sha17] Ben Sharp, _Compactness of minimal hypersurfaces with bounded index_ , J. Differential Geom. 106 (2017), no. 2, 317–339.
  * [Sim14] Leon Simon, _Lectures on geometric measure theory_ , Lecture Notes, 2014.
  * [Smi83] Francis R. Smith, _On the existence of embedded minimal 2-spheres in the 3-sphere, endowed with an arbitrary metric_ , PhD thesis, Supervisor: Leon Simon, University of Melbourne (1983).
  * [Son19] Antoine Song, _A dichotomy for minimal hypersurfaces in manifolds thick at infinity_ , arXiv e-prints (2019), arXiv:1902.06767.
  * [Son23] by same author, _Existence of infinitely many minimal hypersurfaces in closed manifolds_ , Ann. of Math. (2) 197 (2023), no. 3, 859 – 895.
  * [SS81] Richard Schoen and Leon Simon, _Regularity of stable minimal hypersurfaces_ , Comm. Pure Appl. Math. 34 (1981), no. 6, 741–797.
  * [Tho78] Gudlaugur Thorbergsson, _Closed geodesics on non-compact Riemannian manifolds_ , Math. Z. 159 (1978), no. 3, 249–258.
  * [Wan22] Zhihan Wang, _Min-max minimal hypersurfaces with obstacle_ , Calc. Var. Partial Differential Equations 61 (2022), no. 5, Paper No. 175, 26.
  * [Whi91] Brian White, _The space of minimal submanifolds for varying Riemannian metrics_ , Indiana Univ. Math. J. 40 (1991), no. 1, 161–200.
  * [Whi10] by same author, _The maximum principle for minimal varieties of arbitrary codimension_ , Comm. Anal. Geom. 18 (2010), no. 3, 421–432.
  * [WZ23] Zhichao Wang and Xin Zhou, _Existence of four minimal spheres in $S^{3}$ with a bumpy metric_, arXiv e-prints (2023), arXiv:2305.08755.
  * [Zho20] Xin Zhou, _On the multiplicity one conjecture in min-max theory_ , Ann. of Math. (2) 192 (2020), no. 3, 767–820.


# What’s Behind the Mask: Estimating
Uncertainty in Image-to-Image Problems

Gilad Kutiel, Regev Cohen, Michael Elad & Daniel Freedman
Verily Research
Haifa, Israel
<EMAIL_ADDRESS>
###### Abstract

Estimating uncertainty in image-to-image networks is an important task,
particularly as such networks are being increasingly deployed in the
biological and medical imaging realms. In this paper, we introduce a new
approach to this problem based on masking. Given an existing image-to-image
network, our approach computes a mask such that the distance between the
masked reconstructed image and the masked true image is guaranteed to be less
than a specified threshold, with high probability. The mask thus identifies
the more certain regions of the reconstructed image. Our approach is agnostic
to the underlying image-to-image network, and only requires triples of the
input (degraded), reconstructed and true images for training. Furthermore, our
method is agnostic to the distance metric used. As a result, one can use
$L_{p}$-style distances or perceptual distances like LPIPS, which contrasts
with interval-based approaches to uncertainty. Our theoretical guarantees
derive from a conformal calibration procedure. We evaluate our mask-based
approach to uncertainty on image colorization, image completion, and super-
resolution tasks, demonstrating high quality performance on each.

##  1 Introduction

Deep Learning has been very successful in many applications, spanning computer
vision, speech recognition, natural language processing, and beyond. For many
years, researchers were mainly content to develop new techniques to achieve
unprecedented accuracy, without concern for understanding the uncertainty
implicit in such models. More recently, however, there has been a concerted
effort within the research community to understand and quantify the
uncertainty of deep models.

This paper addresses the problem of estimating uncertainty in the realm of
image-to-image (sometimes referred to as image reconstruction) tasks. Such
tasks include super-resolution, deblurring, colorization, and image
completion, amongst others. Computing the uncertainty is important generally,
but is particularly so in application domains such as biological and medical
imaging, in which fidelity to the ground truth is paramount. If there is an
area of the reconstructed image where such fidelity is unlikely or unreliable
due to high uncertainty, this is crucial to convey.

Our approach to uncertainty estimation is based on masking. Specifically, we
are interested in the possibility of computing a mask such that the uncertain
regions in the image are masked out. More formally, we would like the distance
between the masked reconstructed image and the masked true image to be small
in expectation. Ideally, the method should be agnostic to the choice of
distance function, which should be dictated by the application. A high level
overview of our approach is illustrated in Figure 1.

$x$Colorization Model$\hat{y}$Masking Model$m$$y$$\odot$$m$$=$$y\odot
m$$\hat{y}$$\odot$$m$$=$$\hat{y}\odot m$

Figure 1: A high level view of our approach: $x$ is the degraded input
(colorization in this example) to the regression model. $x$ and $\hat{y}$ (the
colorized image) serve as the input to the masking model which predicts the
uncertainty mask $m$. The ground-truth image is denoted by $y$. With high
probability, the distortion between the masked images, $y\odot m$ and
$\hat{y}\odot m$ is low.

We show a direct connection between the mask and a theoretically well-founded
definition of uncertainty that matches image-to-image tasks. We then derive an
algorithm for computing such a mask which can apply to any existing (i.e. pre-
trained) image-to-image network, and any distance function between image
pairs. All that is required for training the mask network is triplets of the
input (degraded), reconstructed and true images. Using a procedure based on
conformal prediction (Angelopoulos & Bates, 2021a), we can guarantee that the
masks so produced satisfy the following criterion: the distance between the
masked reconstructed image and the masked true image is guaranteed to be less
than a specified threshold, with high probability. We demonstrate the power of
the method on image colorization, image completion, and super-resolution
tasks.

Our contributions are as follows:

  * •

We present an approach to uncertainty computation based on masking. We show
that moving from binary to continuous masks preserves the connection with a
theoretically well-founded definition of uncertainty that is relevant for
image-to-image tasks.

  * •

We derive an algorithm for computing the masks which works for arbitrary
image-to-image networks, and any distance function between image pairs. The
masks are guaranteed to yield a small distance between masked ground truth and
reconstructed images, with high probability.

  * •

We demonstrate the effectiveness of the method on image colorization, image
completion, and super-resolution tasks attaining high quality performance on
each.

### Notations

Throughout the paper we represent an image as a column-stacked vector
$x\in\mathbb{R}^{n}$. We use $x_{k}$ to denote the $k$th image of a
collection, while $x_{(i)}$ marks the $i$th entry of $x$. We define
$\vv{0}\in\mathbb{R}^{n}$ and $\vv{1}\in\mathbb{R}^{n}$ to be vectors of all
zeros and ones respectively. The operation $x\odot y$ stands for the Hadamard
(point-wise) product between $x$ and $y$. For $p\geq 1$, we denote the
$\ell_{p}$-norm of $x$ by
$\norm{x}_{p}^{p}\triangleq\sum_{i=1}^{n}|x_{(i)}|^{p}$. The symbol
$d(\cdot,\cdot)$ represents a general distortion measure between two images.
We define a continuous mask as a vector $m\in[0,1]^{n}$, where the _average
size_ of a mask equals $\norm{\vv{1}-m}_{1}/n$ such that the mask $\vv{1}$
(no-masking) has size 0 while the mask $\vv{0}$ has size of 1. Given a mask
$m$, we define $d_{m}(x,y)\triangleq d(m\odot x,m\odot y)$ as the distortion
measure between the masked versions of images $x$ and $y$. For a natural
number $n\geq 1$ we define the set $[n]\triangleq\\{1,...,n\\}$.

##  2 Related Work

Bayesian Uncertainty Quantification The Bayesian paradigm defines uncertainty
by assuming a distribution over the model parameters and/or activation
functions. The most prevalent approach is Bayesian neural networks (MacKay,
1992; Valentin Jospin et al., 2020; Izmailov et al., 2020), which are
stochastic models trained using Bayesian inference. Yet, as the number of
model parameters has grown rapidly, computing the exact posteriors has became
computationally intractable. This shortcoming has led to the development of
approximation methods such as Monte Carlo dropout (Gal & Ghahramani, 2016; Gal
et al., 2017a), stochastic gradient Markov chain Monte Carlo (Salimans et al.,
2015; Chen et al., 2014), Laplacian approximations (Ritter et al., 2018) and
variational inference (Blundell et al., 2015; Louizos & Welling, 2017; Posch
et al., 2019). Alternative Bayesian techniques include deep Gaussian processes
(Damianou & Lawrence, 2013), deep ensembles (Ashukha et al., 2020; Hu et al.,
2019), and deep Bayesian active learning (Gal et al., 2017b), to name just a
few. For a recent comprehensive review on Bayesian uncertainty quantification
we refer the readers to Abdar et al. (2021).

Distribution-Free Methods and Conformal Prediction Unlike Bayesian methods,
the frequentist approach assumes the true model parameters are fixed with no
underlying distribution. Examples of such distribution-free techniques are
model ensembles (Lakshminarayanan et al., 2017; Pearce et al., 2018),
bootstrap (Kim et al., 2020; Alaa & Van Der Schaar, 2020), interval regression
(Pearce et al., 2018; Kivaranovic et al., 2020; Wu et al., 2021) and quantile
regression (Gasthaus et al., 2019; Romano et al., 2019). An important
distribution-free technique which is most relevant to our work is conformal
prediction (Angelopoulos & Bates, 2021b; Shafer & Vovk, 2008). The conformal
approach relies on a labeled calibration dataset to convert point estimations
into prediction regions. Conformal methods can be used with any estimator,
require no retraining, are computationally efficient and provide coverage
guarantees in finite samples (Lei et al., 2018). Recent development includes
conformalized quantile regression (Romano et al., 2019; Sesia & Candès, 2020;
Angelopoulos et al., 2022b), conformal risk control (Angelopoulos et al.,
2022a; Bates et al., 2021; Angelopoulos et al., 2021) and semantic uncertainty
intervals for generative adversarial networks (Sankaranarayanan et al., 2022).
Sun (2022) provides an extensive survey on distribution-free conformal
prediction methods.

Uncertainty for Image-to-image Regression Tasks A work that stands out in this
realm is (Angelopoulos et al., 2022b), which experiments with the conformal
prediction technique. They evaluate and compare several models that address
both the prediction of a value per each pixel, and an accompanying confidence
interval that should contain the true value. After calibration, a guarantee is
provided such that with high probability the true values of the pixels are
indeed within these predicted confidence intervals. They find that quantile
regression gives the best results among the four methods considered. As our
work addresses the same breed of image-to-image problems, we provide extensive
comparison to this work.

##  3 Mask-Based Uncertainty Estimation

###  3.1 Problem Formulation

We consider the problem of recovering an image $y\in\mathbb{[}0,1]^{n}$ from
its observations $x\in\mathbb{[}0,1]^{n_{x}}$ and denote by
$\hat{y}(x)\in\mathbb{[}0,1]^{n}$ a given estimator of $y$. We define the
uncertainty of $\hat{y}$ given $x$ as follow:

$U(\hat{y}|x)\triangleq\mathbb{E}_{y|x}[d(y,\hat{y})],$ (1)

where $d(\cdot,\cdot)$ denotes an arbitrary distortion measure between the
ground truth image $y$ and its estimation $\hat{y}$, and $\mathbb{E}$ is the
expectation over the random variable $y|x$. Ideally, we would like to design
$\hat{y}$ such that

$P\Big{(}U(\hat{y}|x)\leq\alpha\Big{)}\geq\beta$ (2)

where $0\leq\alpha$ and $0\leq\beta\leq 1$ are user-defined, and the
probability $P(\cdot)$ is computed over the joint probability of the problem
pairs $(y,x)$. However, in practice, condition (2) might not hold. Hence, our
goal is to identify areas within $\hat{y}$ which have the most adverse effect
on the uncertainty and lead to the violation of (2). Formally, we aim to
construct an uncertainty map given by a continuous mask $m\in[0,1]^{n}$ such
that

$P\Big{(}\mathbb{E}_{y|x}[d_{m}(y,\hat{y})]\leq\alpha\Big{)}\geq\beta$ (3)

where $d_{m}(y,\hat{y})\triangleq d(m\odot y,m\odot\hat{y})$. Notice we must
avoid trivial and undesired solutions, e.g. $m=0$, which satisfy (3) but
provide no true information regarding the uncertainty. To that end, we
consider solving the following problem

$\displaystyle\begin{split}\max_{m}\;\norm{m}_{1}\quad\text{subject
to}\quad&\mathbb{E}_{y|x}[d_{m}(y,\hat{y})]\leq\alpha,\\\ &\vv{0}\leq
m\leq\vv{1}.\end{split}$ (P1)

In words, we wish to find the minimal masking required to satisfy the
distortion constraint. Notice that Problem (P1) does not depend on $\beta$ as
we consider an instance regression problem for a given $x$ and an originating
$y$. We bring back the probabilistic condition using $\beta$ in the next
section, when addressing the whole image manifold.

Under certain conditions, the optimal solution is readily given in a closed-
form expression as shown by the next theorem.

######  Theorem 1.

Consider Problem (P1) with the distortion measure
$d(y,\hat{y})=\norm{y-\hat{y}}_{p}^{p}$ where $p>1$. Then, for sufficiently
small $\alpha$, the optimal mask $m^{*}$ admits a closed-form solution given
by

$m_{(i)}^{*}=\alpha^{\frac{1}{p}}\frac{q_{(i)}}{\big{(}\sum_{j=1}^{n}q_{(j)}\big{)}^{\frac{1}{p}}}.$

where for all $j\in[n]$

$q_{(j)}\triangleq\frac{1}{\left(\mathbb{E}_{y|x}[|\hat{y}_{(j)}-y_{(j)}|^{p}]\right)^{1/(p-1)}}.$

###### Proof.

See Appendix B. ∎

Note that the above implies the following:

######  Corollary 1.

The optimal mask, as defined by the solution of (P1), is directly related to
the uncertainty measure in Equation (1) via

$U(\hat{y}|x)\propto\sum_{i=1}^{n}(m_{(i)}^{*})^{-(p-1)}.$

###### Proof.

$\sum_{i=1}^{n}(m_{(i)}^{*})^{-(p-1)}\propto\sum_{i=1}^{n}(q_{(i)}^{*})^{-(p-1)}=\sum_{i=1}^{n}\mathbb{E}_{y|x}[|\hat{y}_{(j)}-y_{(j)}|^{p}]=U(\hat{y}|x).$

∎

Thus, the above establishes a correspondence between the optimal mask and
uncertainty as defined in (1). Furthermore, it can be shown that the result of
Theorem 1 can be extended to $p=1$, leading to a binary mask (see Appendix A).

The result of Theorem 1 serves only as a general motivation in our work, since
obtaining the optimal mask directly via the above equations is impractical for
a number of reasons. First, $\alpha$ may vary depending on the user and the
application. Second, for arbitrary distortion measure Problem (P1) does not
admit a closed-form solution. Last and most importantly, at inference time the
ground truth $y$ is unknown. Hence, our next step is to provide an estimation
for the mask which requires no knowledge of $y$, can be computed for any
differentiable distortion measure, and is easily adapted for various values of
$\alpha$ and $\beta$.

###  3.2 Method

Here, we aim to design an estimator for the the uncertainty mask defined by
Problem (P1). We start by modeling the mask as a neural network
$m_{\theta}(x,\hat{y})$with parameters $\theta$, which we refer to as the
masking model (see details in Section 4.2). Similar to Problem (P1), we define
the following optimization problem

$\displaystyle\begin{split}\min_{\theta}\;||m_{\theta}(x,\hat{y})-\vv{1}||_{2}^{2}\quad\text{subject
to}\quad&\mathbb{E}_{y|x}[d_{m_{\theta}}(y,\hat{y})]\leq\alpha,\\\ &\vv{0}\leq
m_{\theta}(x,\hat{y})\leq\vv{1}.\end{split}$ (P2)

Ignoring (for now) the second constraint, the Lagrangian of Problem (P2) is
given by

$L(\theta,\mu)=||m_{\theta}(x,\hat{y})-\vv{1}||_{2}^{2}+\mu\Big{(}\mathbb{E}_{y|x}[d_{m_{\theta}}(y,\hat{y})]-\alpha\Big{)}$
(4)

where $\mu>0$ is the dual variable which is considered as an hyperparameter.
Given $\mu$, the optimal mask can be found by minimizing $L(\theta,\mu)$ with
respect to $\theta$, which is equivalent to minimizing

$||m_{\theta}(x,\hat{y})-\vv{1}||_{2}^{2}+\mu\mathbb{E}_{y|x}[d_{m_{\theta}}(y,\hat{y})]$
(5)

since $\alpha$ does not depend on $\theta$. Thus, motivated by the above, we
train our masking model using the following loss function:

$\ell(\mathcal{D},\theta)=\sum_{(x,\hat{y},y)\in\mathcal{D}}||m_{\theta}(x,\hat{y})-\vv{1}||_{2}^{2}+\mu
d_{m_{\theta}}(y,\hat{y})$ (6)

where $\mathcal{D}\triangleq\\{(x_{k},\hat{y}_{k},y_{k})\\}_{k}$ is a dataset
of triplets of the degraded input, recovered output and the true image
respectively. To enforce that the mask is within $[0,1]$ we simply clip the
output of masking model. Note that the proposed approach facilitates the use
of any differentiable distortion measure. Furthermore, at inference time, our
masking model provides an approximation of the optimal uncertainty mask – the
solution of (P1) – without requiring $y$.

Following the above, notice that the loss function is independent of $\alpha$
and $\beta$, hence, we cannot guarantee that $m_{\theta}$ satisfies condition
(3). To overcome this, we follow previous work on conformal prediction and
consider the output of our masking model as an initial estimation of the
uncertainty which can be calibrated to obtain strong statistical guarantees.
In general, the calibrated mask $m_{\lambda}$ follows the form

$m_{\lambda(i)}\triangleq F(m_{\theta(i)};\lambda),\quad\forall i=1,...,n,$
(7)

where $F(\cdot;\lambda)$ is a monotonic non-decreasing function, parameterized
by a global scalar which is tuned throughout the calibration process. As our
goal is to reduce the distortion below a predefined fixed level, the
calibrated mask need to be inversely proportional to the distortion. Since our
predicted mask is proportional to the distortion we use the following
formula111We add a small value, $\epsilon$, to the denominator for numerical
stability and clip the value of $m$ to be in $[0,1]$.

$m_{\lambda(i)}=\frac{\lambda}{{\epsilon}+1-m_{\theta(i)}}\quad\forall
i=1,...,n,$ (8)

which was found empirically to perform well in our experiments. To set the
value of $\lambda$ we assume a given calibration dataset
$\mathcal{C}=\\{(x_{k},\hat{y}_{k},y_{k})\\}_{k}$ and perform the following
calibration procedure:

  1. 1.

For each pair $(x_{k},\hat{y}_{k})$ we predict a mask
$m_{k}=m_{\theta}(x_{k},\hat{y}_{k})$.

  2. 2.

We define a calibrated mask $m_{\lambda_{k}}$ according to (8) where
$\lambda_{k}$ is the maximal value such that

$d\big{(}m_{\lambda_{k}}\odot y,m_{\lambda_{k}}\odot\hat{y}\big{)}\leq\alpha.$

  3. 3.

Given all $\\{\lambda_{k}\\}$, we set $\lambda$ to be the $1-\beta$ quantile
of $\\{\lambda_{k}\\}$, i.e. the maximal value for which at least $\beta$
fraction of the calibration set satisfies condition (P1).

The final mask, calibrated according to $\alpha$ and $\beta$, is guaranteed to
satisfy condition (3) above.

The benefits of our masking approach are as follows. First, it provides a
measure of uncertainty which can be easily interpreted. In addition, our
training process can accept any distortion measure and learns the
relationships and correlations between different pixels in the image. Note
thatour model is trained only once, irrespective of the values of $\alpha$ and
$\beta$. Thus, our model can be easily adapted for different values of
$\alpha$ and $\beta$ via our simple calibration process. Finally, at
inference, we can produce our uncertainty map without the knowledge of the
ground truth while still providing strong statistical guarantees on our output
mask.

##  4 Experiments

###  4.1 Datasets and Tasks

Datasets Two data-sets are used in our experiments:

[1em]0em

Places365 (Zhou et al., 2017): A large collection of 256x256 images from 365
scene categories. We use 1,803,460 images for training and 36,500 images for
validation/test.

Rat Astrocyte Cells (Ljosa et al., 2012): A dataset of 1,200 uncompressed
images of scanned rat cells of resolution 990$\times$708\. We crop the images
into 256$\times$256 tiles, and randomly split them into train and
validation/test sets of sizes 373,744 and 11,621 respectively. The tiles are
partially overlapped as we use stride of 32 pixels when cropping the images.

Tasks We consider the following image-to-image tasks:

[1em]0em

Completion: Using grayscale version of Places365, we remove a middle vertical
and horizontal stripe of 32 pixel width, and aim to reconstruct the missing
part.

Super Resolution: We experiment with this task on the two data-sets. The
images are scaled down to 64 $\times$ 64 images and the goal is to reconstruct
the original image.

Colorization: We convert the Places365 images to grayscale and aim to recover
their colors.

Figure 4 shows a sample input and output for each of the above tasks.

###  4.2 Experimental Details and Settings

Image-to-Image Models We start by training models for the above three tasks.
Note that these models are not intended to be state of the art, but rather
used to demonstrate the uncertainty estimation technique proposed in this
work. We use the same model architecture for all tasks: an 8 layer U-Net. For
each task we train two versions of the network: (i) A simple regressor; and
(ii) A conditional GAN, where the generator plays the role of the
reconstruction network. For the GAN, the discriminator is implemented as a 4
layer CNN. We use the L1 loss as the objective for the regressor, and augment
it with an adversarial loss in the conditional GAN ($\lambda=20$), as in Isola
et al. (2017). All models are trained for ten epochs using the Adam optimizer
with a learning rate equal to $1\text{e-}5$ and a batch size equal to $50$.

Mask Model For our masking model we also use an 8 layer U-Net architecture.
This choice was made for simplicity and compatibility with previous work. The
input for the mask model is the concatenation of the degraded image and the
predicated image on the channel axis. It’s output is a mask having the same
shape as the predicted image, with values in the range $[0,1]$. The masking
model is trained using the loss function described in Section 3.2 with
$\mu=2$, learning rate of $1\text{e-}5$ and a batch size of 25.

Experiments We consider the L1, L2, SSIM and LPIPS distances. We set aside
$1,000$ samples from each validation set for calibration and use the remaining
samples for evaluation. We demonstrate the flexibility of our approach by
conducting experiments on a variety of $12$ settings: (i) Image Completion:
{Regressor, GAN} $\times$ {L1, LPIPS}; (ii) Super Resolution: {Regressor, GAN}
$\times$ {L1, SSIM}; and (iii) Colorization: {Regressor, GAN} $\times$ {L1,
L2}.

Thresholds Recall that given a predicted image, our goal is to find a mask
that, when applied to both the prediction and the (unknown) reference image,
reduces the distortion between them to a predefined threshold with high
probability $\beta$. Here we fix $\beta=0.9$ and choose the threshold to be
the $0.1$-quantile of each measure computed on a random sample from the
validation set, i.e. roughly $10\%$ of the predictions are already considered
sufficiently good and do not require masking at all.

###  4.3 Competitor Techniques

Quantile – Interval-Based Technique We compare our method to the quantile
regression option presented in (Angelopoulos et al., 2022b). As described in
Section 2, their uncertainty is formulated by pixel-wise confidence intervals,
calibrated so as to guarantee that with high probability the true value of the
image is indeed within the specified range. While their predicted confidence
intervals are markedly different from the expected distortion we consider, we
can use these intervals as a heuristic mask. For completeness, we also report
the performance of the quantile regression even when it is less suitable, i.e.
when the underlying model is a GAN and when the distortion loss is different
from L1. We note again that for the sake of a fair comparison, our
implementation of the masking model uses exactly the same architecture as the
quantile regressor.

Opt – Oracle We also compare our method with an oracle, denoted Opt, which
computes an optimal mask using the following optimization procedure that has
access to the reference image $y$:

$\ell(m)=\min_{m}||m-\vv{1}||_{2}^{2}+\mu d_{m}(y,\hat{y}).$ (9)

This minimization is obtained via gradient descent, using the Adam optimizer
with a step-size $0.01$, iterating until the destination distortion value is
attained. Clearly, this approach does not require calibration, as the above
procedure ensures that all the masked samples are below the target distortion
threshold.

###  4.4 Results and Discussion

We now show a series of results that demonstrate our proposed uncertainty
masking approach, and its comparison with Opt and Quantile.222Due to space
limitations, we show more extensive experimental results in Appendix B, while
presenting a selected portion of them here. We begin with a representative
visual illustration of our proposed mask for several test cases, see Figure 2.
As can be seen, the mask aligns well with the true prediction error, while
still identifying sub-regions of high certainty.

Figure 2: Example of uncertainty masks for image completion (top),
colorization (middle) and super resolution (bottom): the images from left to
right are the input to the model, the reference image, the output of the
model, the calibrated uncertainty mask trained on the L1 loss and the L1
distance heat-map. In the image completion task the bottom left corner is
richer in details and thus there is high uncertainty regarding this part in
the reconstructed image. For the colorization task, the clothes and the
colored area of the bus are the most uncertained regions respectively as these
regions can be colorized with a large variaty of colors. In the super
resolution task the uncertained regions are around the edges and text while
smooth surfaces are more certain parts. Note that the actual error (right most
image) is not equivalent to our definition of uncertainty and we only present
it as a reference.

In Figure 3 we explore the performance of the three compared methods with
respect to (i) the probabilistic guarantee, (ii) the size of the masks
obtained, and (iii) the correlation between mask sizes – Quantile and Ours
versus Opt. As can be seen from the top row, all three methods meet the
distortion threshold with fewer than $10\%$ exceptions, as required.
Naturally, Opt does not have outliers since each mask is optimally calibrated
by its computation. The spread of loss values tends to be higher with
Quantile, indicating weaker performance. The colorization results, here and
below, seem to be more challenging, with a smaller performance increase for
our method.

The above probabilistic results are not surprising as they are the outcome of
the calibration stage. The main question remaining relates to the size of the
mask created so as to meet this threshold distortion. The middle row in Figure
3 presents histograms of these sizes, showing that our approach tends to
produce masks that are close in size to those of Opt; while Quantile produces
larger, and thus inferior, masked areas. Again, our colorization results show
smaller if any improvement.

The bottom row in Figure 3 shows the correlation between mask sizes, and as
can be observed, a high such correlation exists between our method and Opt,
while Quantile produces less correlated and much heavier masks.

Quantitative results are presented in Table 1 for the completion, super
resolution, and colorization tasks. First and foremost, the results obtained
by our method surpass those of Quantile across the board. Our method exhibits
smaller mask size $\mathbf{\|M\|}$, aligned well with the masks obtained by
Opt. Quantile, as expected, produces larger masks. In terms of the correlation
between the obtained mask-size and the unmasked distortion value
$\mathbf{C(M,D)}$, our method shows high such agreement, while Quantile lags
behind. This correlation indicates a much desired adaptivity of the estimated
mask to the complexity of image content and thus to the corresponding
uncertainty. The correlation $\mathbf{C(M,M_{opt})}$ between Opt’s mask size
and the size of the mask obtained by our method or Quantile show similar
behavior, aligned well with behavior we see in the bottom of Figure 3.

|  |
---|---|---
|  |
|  |
Figure 3: Summary of results for image completion (left), super-resolution (middle) and colorization (right). The top row shows the distribution of distortion values after masking versus the user-chosen threshold. The middle row presents histograms of mask sizes for the three compared methods. The bottom row describes the inter-relation between Opt’s mask size and the sizes obtained by Our method and Quantile’s.  |  | $\mathbf{\|M\|\quad(\downarrow)}$ | $\mathbf{C(M,D)\quad(\uparrow)}$ | $\mathbf{C(M,M_{opt})\quad(\uparrow)}$
---|---|---|---|---
Network | Distance | Opt | Ours | Quantile | Ours | Quantile | Ours | Quantile
Regression | L1 | $0.09$ | $\mathbf{0.10}$ | $0.15$ | $\mathbf{0.89}$ | $0.78$ | $\mathbf{0.89}$ | $0.76$
Regression | LPIPS | $0.01$ | $\mathbf{0.01}$ | $0.20$ | $\mathbf{0.54}$ | $0.51$ | $\mathbf{0.89}$ | $0.77$
GAN | L1 | $0.09$ | $\mathbf{0.09}$ | $0.14$ | $\mathbf{0.95}$ | $0.85$ | $\mathbf{0.94}$ | $0.80$
GAN | LPIPS | $0.01$ | $\mathbf{0.01}$ | $0.08$ | $\mathbf{0.31}$ | $0.24$ | $\mathbf{0.50}$ | $0.23$
Rat Astrocyte Cells
Regression | L1 | $0.24$ | $\mathbf{0.26}$ | $0.28$ | $\mathbf{0.99}$ | $0.54$ | $\mathbf{0.95}$ | $0.88$
Regression | SSIM | $0.03$ | $\mathbf{0.03}$ | $0.13$ | $\mathbf{0.66}$ | $0.64$ | $\mathbf{0.82}$ | $0.57$
GAN | L1 | $0.26$ | $\mathbf{0.30}$ | $0.40$ | $\mathbf{0.94}$ | $0.63$ | $\mathbf{0.80}$ | $0.72$
GAN | SSIM | $0.03$ | $\mathbf{0.03}$ | $0.13$ | $\mathbf{0.79}$ | $0.63$ | $\mathbf{0.83}$ | $0.63$
Places365
Regression | L1 | $0.30$ | $\mathbf{0.36}$ | $0.39$ | $\mathbf{0.99}$ | $0.97$ | $\mathbf{0.95}$ | $0.94$
Regression | SSIM | $0.10$ | $\mathbf{0.23}$ | $0.48$ | $\mathbf{0.89}$ | $0.85$ | $\mathbf{0.94}$ | $0.84$
GAN | L1 | $0.37$ | $\mathbf{0.38}$ | $0.47$ | $\mathbf{0.97}$ | $0.81$ | $\mathbf{0.95}$ | $0.67$
GAN | SSIM | $0.10$ | $\mathbf{0.12}$ | $0.51$ | $\mathbf{0.86}$ | $0.81$ | $\mathbf{0.92}$ | $0.86$
Regression | L1 | $0.27$ | $\mathbf{0.37}$ | $0.40$ | $\mathbf{0.68}$ | $0.43$ | $\mathbf{0.57}$ | $0.46$
Regression | L2 | $0.18$ | $\mathbf{0.37}$ | $0.38$ | $\mathbf{0.57}$ | $0.30$ | $\mathbf{0.60}$ | $0.48$
GAN | L1 | $0.27$ | $\mathbf{0.38}$ | $0.40$ | $\mathbf{0.58}$ | $0.40$ | $\mathbf{0.60}$ | $0.52$
GAN | L2 | $0.18$ | $\mathbf{0.36}$ | $0.38$ | $\mathbf{0.42}$ | $0.28$ | $\mathbf{0.59}$ | $0.49$
Table 1: Image completion (top), super-resolution (middle) and colorization
(bottom), results. $\mathbf{\|M\|}$ indicates average mask size;
$\mathbf{C(M,D)}$ the correlation between predicted mask size and the amount
of distortion; and $\mathbf{C(M,M_{opt})}$ the correlation between the size of
the calibrated mask and the size of the optimal mask. Arrows indicate which
direction is better. Best results shown in blue. 95% confidence intervals were
calculated for all the experiments and are omitted due to lack of space. All
presented results were verified for their statistical significance.

##  5 Conclusions

Uncertainty assessment in image-to-image regression problems is a challenging
task, due to the implied complexity, the high dimensions involved, and the
need to offer an effective and meaningful visualization of the estimated
results. This work proposes a novel approach towards these challenges by
constructing a continuous mask that highlights the trust-worthy regions in the
estimated image. This mask provides a measure of uncertainty accompanied by an
accuracy guarantee, stating that with high probability, the distance between
the original and the estimated images over the non-masked regions is below a
user-specified value. The presented paradigm is flexible, being agnostic to
the choice of distance function used and the regression method employed, while
yielding masks of small area.

## References

  * Abdar et al. (2021) Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U Rajendra Acharya, et al.  A review of uncertainty quantification in deep learning: Techniques, applications and challenges.  _Information Fusion_ , 76:243–297, 2021.
  * Alaa & Van Der Schaar (2020) Ahmed Alaa and Mihaela Van Der Schaar.  Frequentist uncertainty in recurrent neural networks via blockwise influence functions.  In _International Conference on Machine Learning_ , pp. 175–190. PMLR, 2020.
  * Angelopoulos & Bates (2021a) Anastasios N. Angelopoulos and Stephen Bates.  A gentle introduction to conformal prediction and distribution-free uncertainty quantification.  _CoRR_ , abs/2107.07511, 2021a.  URL https://arxiv.org/abs/2107.07511.
  * Angelopoulos & Bates (2021b) Anastasios N Angelopoulos and Stephen Bates.  A gentle introduction to conformal prediction and distribution-free uncertainty quantification.  _arXiv preprint arXiv:2107.07511_ , 2021b.
  * Angelopoulos et al. (2021) Anastasios N Angelopoulos, Stephen Bates, Emmanuel J Candès, Michael I Jordan, and Lihua Lei.  Learn then test: Calibrating predictive algorithms to achieve risk control.  _arXiv preprint arXiv:2110.01052_ , 2021.
  * Angelopoulos et al. (2022a) Anastasios N Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster.  Conformal risk control.  _arXiv preprint arXiv:2208.02814_ , 2022a.
  * Angelopoulos et al. (2022b) Anastasios N Angelopoulos, Amit P Kohli, Stephen Bates, Michael I Jordan, Jitendra Malik, Thayer Alshaabi, Srigokul Upadhyayula, and Yaniv Romano.  Image-to-image regression with distribution-free uncertainty quantification and applications in imaging.  _arXiv preprint arXiv:2202.05265_ , 2022b.
  * Ashukha et al. (2020) Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov.  Pitfalls of in-domain uncertainty estimation and ensembling in deep learning.  _arXiv preprint arXiv:2002.06470_ , 2020.
  * Bates et al. (2021) Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael Jordan.  Distribution-free, risk-controlling prediction sets.  _Journal of the ACM (JACM)_ , 68(6):1–34, 2021\.
  * Blundell et al. (2015) Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra.  Weight uncertainty in neural network.  In _International conference on machine learning_ , pp. 1613–1622. PMLR, 2015.
  * Chen et al. (2014) Tianqi Chen, Emily Fox, and Carlos Guestrin.  Stochastic gradient hamiltonian monte carlo.  In _International conference on machine learning_ , pp. 1683–1691. PMLR, 2014.
  * Damianou & Lawrence (2013) Andreas Damianou and Neil D Lawrence.  Deep gaussian processes.  In _Artificial intelligence and statistics_ , pp. 207–215. PMLR, 2013.
  * Gal & Ghahramani (2016) Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _international conference on machine learning_ , pp. 1050–1059. PMLR, 2016.
  * Gal et al. (2017a) Yarin Gal, Jiri Hron, and Alex Kendall.  Concrete dropout.  _Advances in neural information processing systems_ , 30, 2017a.
  * Gal et al. (2017b) Yarin Gal, Riashat Islam, and Zoubin Ghahramani.  Deep bayesian active learning with image data.  In _International Conference on Machine Learning_ , pp. 1183–1192. PMLR, 2017b.
  * Gasthaus et al. (2019) Jan Gasthaus, Konstantinos Benidis, Yuyang Wang, Syama Sundar Rangapuram, David Salinas, Valentin Flunkert, and Tim Januschowski.  Probabilistic forecasting with spline quantile function rnns.  In _The 22nd international conference on artificial intelligence and statistics_ , pp. 1901–1910. PMLR, 2019.
  * Hu et al. (2019) Ruihan Hu, Qijun Huang, Sheng Chang, Hao Wang, and Jin He.  The MBPEP: a deep ensemble pruning algorithm providing high quality uncertainty prediction.  _Applied Intelligence_ , 49(8):2942–2955, 2019\.
  * Isola et al. (2017) Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 1125–1134, 2017.
  * Izmailov et al. (2020) Pavel Izmailov, Wesley J Maddox, Polina Kirichenko, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson.  Subspace inference for Bayesian deep learning.  In _Uncertainty in Artificial Intelligence_ , pp. 1169–1179. PMLR, 2020.
  * Kim et al. (2020) Byol Kim, Chen Xu, and Rina Barber.  Predictive inference is free with the jackknife+-after-bootstrap.  _Advances in Neural Information Processing Systems_ , 33:4138–4149, 2020.
  * Kivaranovic et al. (2020) Danijel Kivaranovic, Kory D Johnson, and Hannes Leeb.  Adaptive, distribution-free prediction intervals for deep networks.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 4346–4356. PMLR, 2020.
  * Lakshminarayanan et al. (2017) Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  _Advances in neural information processing systems_ , 30, 2017.
  * Lei et al. (2018) Jing Lei, Max G’Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman.  Distribution-free predictive inference for regression.  _Journal of the American Statistical Association_ , 113(523):1094–1111, 2018.
  * Ljosa et al. (2012) Vebjorn Ljosa, Katherine L Sokolnicki, and Anne E Carpenter.  Annotated high-throughput microscopy image sets for validation.  _Nature methods_ , 9(7):637–637, 2012.
  * Louizos & Welling (2017) Christos Louizos and Max Welling.  Multiplicative normalizing flows for variational bayesian neural networks.  In _International Conference on Machine Learning_ , pp. 2218–2227. PMLR, 2017.
  * MacKay (1992) David JC MacKay.  Bayesian interpolation.  _Neural computation_ , 4(3):415–447, 1992.
  * Pearce et al. (2018) Tim Pearce, Alexandra Brintrup, Mohamed Zaki, and Andy Neely.  High-quality prediction intervals for deep learning: A distribution-free, ensembled approach.  In _International conference on machine learning_ , pp. 4075–4084. PMLR, 2018.
  * Posch et al. (2019) Konstantin Posch, Jan Steinbrener, and Jürgen Pilz.  Variational inference to measure model uncertainty in deep neural networks.  _arXiv preprint arXiv:1902.10189_ , 2019.
  * Ritter et al. (2018) Hippolyt Ritter, Aleksandar Botev, and David Barber.  A scalable Laplace approximation for neural networks.  In _6th International Conference on Learning Representations, ICLR 2018-Conference Track Proceedings_ , volume 6. International Conference on Representation Learning, 2018.
  * Romano et al. (2019) Yaniv Romano, Evan Patterson, and Emmanuel Candes.  Conformalized quantile regression.  _Advances in neural information processing systems_ , 32, 2019.
  * Salimans et al. (2015) Tim Salimans, Diederik Kingma, and Max Welling.  Markov chain monte carlo and variational inference: Bridging the gap.  In _International conference on machine learning_ , pp. 1218–1226. PMLR, 2015.
  * Sankaranarayanan et al. (2022) Swami Sankaranarayanan, Anastasios N Angelopoulos, Stephen Bates, Yaniv Romano, and Phillip Isola.  Semantic uncertainty intervals for disentangled latent spaces.  _arXiv preprint arXiv:2207.10074_ , 2022.
  * Sesia & Candès (2020) Matteo Sesia and Emmanuel J Candès.  A comparison of some conformal quantile regression methods.  _Stat_ , 9(1):e261, 2020.
  * Shafer & Vovk (2008) Glenn Shafer and Vladimir Vovk.  A tutorial on conformal prediction.  _Journal of Machine Learning Research_ , 9(3), 2008.
  * Sun (2022) Sophia Sun.  Conformal methods for quantifying uncertainty in spatiotemporal data: A survey.  _arXiv preprint arXiv:2209.03580_ , 2022.
  * Valentin Jospin et al. (2020) Laurent Valentin Jospin, Wray Buntine, Farid Boussaid, Hamid Laga, and Mohammed Bennamoun.  Hands-on Bayesian neural networks–a tutorial for deep learning users.  _arXiv e-prints_ , pp. arXiv–2007, 2020.
  * Wu et al. (2021) Dongxia Wu, Liyao Gao, Xinyue Xiong, Matteo Chinazzi, Alessandro Vespignani, Yi-An Ma, and Rose Yu.  Quantifying uncertainty in deep spatiotemporal forecasting.  _arXiv preprint arXiv:2105.11982_ , 2021.
  * Zhou et al. (2017) Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Places: A 10 million image database for scene recognition.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2017.

##  Appendix A Task Illustrations

|  |
---|---|---
|  |
Figure 4:  The three tasks we experimented with: 1) Image Colorization on the
left column 2) Gray-scale Image Completion on the middle column, and 3) Super
Resolution on the right column.

##  Appendix B Proof of Theorem 1

Recall that we have defined the following optimization problem as defining the
optimal mask:

$\displaystyle\begin{split}\max_{m}\;\norm{m}_{1}\quad\text{subject
to}\quad&\mathbb{E}_{y|x}[d_{m}(y,\hat{y})]\leq\alpha,\\\ &\vv{0}\leq
m\leq\vv{1}.\end{split}$ (P1)

Theorem 1: Consider Problem (P1) with the distortion measure
$d(y,\hat{y})=\norm{y-\hat{y}}_{p}^{p}$ where $p>1$. Then, for sufficiently
small $\alpha$, the optimal mask $m^{*}$ admits a closed-form solution given
by

$m_{(i)}^{*}=\alpha^{\frac{1}{p}}\frac{q_{(i)}}{\big{(}\sum_{j=1}^{n}q_{(j)}\big{)}^{\frac{1}{p}}}.$

where for all $j\in[n]$

$q_{(j)}\triangleq\frac{1}{\left(\mathbb{E}_{y|x}[|\hat{y}_{(j)}-y_{(j)}|^{p}]\right)^{1/(p-1)}}.$

###### Proof.

We start with reformulating Problem (P1) as follows:

$\displaystyle\begin{split}\min_{m}\;-\norm{m}_{1}\quad\text{subject
to}\quad&\mathbb{E}_{y|x}[d_{m}(y,\hat{y})]\leq\alpha,\\\
&m_{(i)}\in[0,1].\end{split}$

Next, we define the Lagrangian of the above problem,

$L(m,\mu)\triangleq-\norm{m}_{1}+\mu\Big{(}\mathbb{E}_{y|x}[\norm{m\odot(y-\hat{y})}_{p}^{p}]-\alpha\Big{)},$

where $\mu>0$ is the dual variable, and we invoke the chosen metric
$d(y,\hat{y})=\norm{y-\hat{y}}_{p}^{p}$ for $p>1$. According to the
Karush–Kuhn–Tucker (KKT) conditions, the optimal solution $(m^{*},\mu^{*})$
must satisfy

$\displaystyle\frac{\partial L}{\partial m_{(i)}}=0,\quad\forall i=1,...,n,$
$\displaystyle\frac{\partial L}{\partial\mu}=0.$

Furthermore, notice that the Lagrangian is separable with respect to the
coordinates of $m$, thus, we can write

$\displaystyle\frac{\partial L}{\partial m_{(i)}}$
$\displaystyle=\frac{\partial}{\partial
m_{(i)}}\Big{(}-|m_{(i)}|+\mu\big{(}\mathbb{E}_{y|x}[|m_{(i)}\cdot(y_{(i)}-\hat{y}_{(i)})|^{p}]-\alpha\big{)}\Big{)}$
$\displaystyle=\frac{\partial}{\partial
m_{(i)}}\Big{(}-|m_{(i)}|+\mu|m_{(i)}|^{p}\cdot\mathbb{E}_{y|x}[|y_{(i)}-\hat{y}_{(i)}|^{p}]\Big{)}$
$\displaystyle=\frac{\partial}{\partial m_{(i)}}\Big{(}-m_{(i)}+\mu
m_{(i)}^{p}\cdot\mathbb{E}_{y|x}[|y_{(i)}-\hat{y}_{(i)}|^{p}]\Big{)}$
$\displaystyle=-1+\mu p\mathbb{E}_{y|x}[|y_{(i)}-\hat{y}_{(i)}|^{p}]\cdot
m_{(i)}^{p-1}=0.$

In the above we assume that $m_{(i)}\geq 0$, a fact that should be verified
once a solution is formed. Hence, we obtain

$m_{(i)}^{p-1}=\Big{(}\mu
p\mathbb{E}_{y|x}[|y_{(i)}-\hat{y}_{(i)}|^{p}]\Big{)}^{-1}.$

To simplify our derivation, we define

$\displaystyle d_{(i)}\triangleq\mathbb{E}_{y|x}[|y_{(i)}-\hat{y}_{(i)}|^{p}]$
$\displaystyle q_{(i)}\triangleq d_{(i)}^{-1/(p-1)}.$

Thus, we can rewrite the above result equivalently as

$\displaystyle m_{(i)}^{p-1}=(\mu pd_{(i)})^{-1}\text{ or }m_{(i)}=(\mu
p)^{-1/(p-1)}\cdot q_{(i)}.$

Now, consider the second KKT condition:

$\displaystyle\frac{\partial L}{\partial\mu}$
$\displaystyle=\mathbb{E}_{y|x}[\norm{m\odot(y-\hat{y})}_{p}^{p}]-\alpha$
$\displaystyle=\sum_{j=1}^{n}\mathbb{E}_{y|x}[|m_{(j)}\cdot(y_{(j)}-\hat{y}_{(j)})|^{p}]-\alpha$
$\displaystyle=\sum_{j=1}^{n}m_{(j)}^{p}d_{(j)}-\alpha$
$\displaystyle=\sum_{j=1}^{n}m_{(j)}(\mu pd_{(j)})^{-1}d_{(j)}-\alpha$
$\displaystyle=(\mu p)^{-1}\sum_{j=1}^{n}m_{(j)}-\alpha$ $\displaystyle=(\mu
p)^{-1}(\mu p)^{-1/(p-1)}\sum_{j=1}^{n}q_{(j)}-\alpha=0.$
$\displaystyle\Rightarrow\mu
p=\bigg{(}\frac{\alpha}{\sum_{j=1}^{n}q_{(j)}}\bigg{)}^{-(p-1)/p}.$

Therefore, the optimal solution $m^{*}$ is given by

$m_{(i)}^{*}=\alpha^{\frac{1}{p}}\frac{q_{(i)}}{\big{(}\sum_{j=1}^{n}q_{(j)}\big{)}^{\frac{1}{p}}}.$

Note that as assumed, the obtained solution $m_{(i)}^{*}$ is non-negative.
Finally, assuming $\alpha$ is sufficiently small such that

$\alpha^{\frac{1}{p}}\leq\min_{i}\;\Bigg{(}\frac{q_{(i)}}{\big{(}\sum_{j=1}^{n}q_{(j)}\big{)}^{\frac{1}{p}}}\Bigg{)}^{-1},$

the solution satisfies the second constraint, $m_{(i)}^{*}\leq 1$, which
completes the proof. ∎

We now turn to present a closely related result, referring to the case
$d(y,\hat{y})=\norm{y-\hat{y}}_{1}$.

Theorem 2: Consider Problem (P1) with the distortion measure
$d(y,\hat{y})=\norm{y-\hat{y}}_{1}$. Then the optimal mask $m^{*}$ admits a
closed-form solution given by

$m_{(i)}=\begin{cases}1,&d_{(i)}<d^{*}\\\ 0,&\text{otherwise.}\end{cases}$

where $d_{(i)}\triangleq\mathbb{E}_{y|x}[|y_{(i)}-\hat{y}_{(i)}|]$ and $d^{*}$
is defined by

$\sum_{i}d_{(i)}\cdot\\{d_{(i)}\leq d^{*}\\}=\alpha.$

###### Proof.

Consider the Lagrangian of the above problem when
$d(y,\hat{y})=\norm{y-\hat{y}}_{1}$:

$L(m,\mu)\triangleq-\norm{m}_{1}+\mu\Big{(}\mathbb{E}_{y|x}[\norm{m\odot(y-\hat{y})}_{1}]-\alpha\Big{)},$

where $\mu>0$ is the dual variable. As before, the problem is separable with
respect to the coordinates of $m$, leading to the following one-dimensional
problem

$\min_{0\leq m(i)\leq 1}\;-m_{(i)}+\mu m_{(i)}d_{(i)}=\min_{0\leq m(i)\leq
1}\;m_{(i)}\Big{(}\mu d_{(i)}-1\Big{)},$

where $d_{(i)}\triangleq\mathbb{E}_{y|x}[|y_{(i)}-\hat{y}_{(i)}|]$. Thus,

$m_{(i)}=\begin{cases}1,&d_{(i)}<\frac{1}{\mu}\\\
0,&\text{otherwise.}\end{cases}$

Observe that this result is intuitive in retrospect, as smaller distances
$d_{(i)}$ correspond to un-masked areas. In addition, regardless of $\mu$, the
choice of the mask value is dictated by the distance, a fact that we leverage
hereafter. Recall that the solution should satisfy

$\displaystyle\sum_{i}^{n}m_{(i)}d_{(i)}=\alpha,$

where $m_{(i)}$ are 1-es for the smaller values of $d_{(i)}$. Thus,

$\displaystyle\sum_{i}m_{(i)}d_{(i)}=\sum_{i:d_{(i)}<d^{*}}d_{(i)}=\alpha.$

Hence, the above provides a definition for the threshold value $d^{*}$, and
$\mu=1/d^{*}$, which completes the proof. ∎

##  Appendix C More Results

|
---|---
|
Figure 5: Image Completion - The distribution of the masked distortion values versus the chosen threshold (shown as a horizontal dashed line) for the three tested methods - Opt, Ours and Quantile. |
---|---
|
Figure 6: Super Resolution - The distribution of the masked distortion values versus the chosen threshold (shown as a horizontal dashed line) for the three tested methods - Opt, Ours and Quantile. |
---|---
|
Figure 7: Colorization - The distribution of the masked distortion values versus the chosen threshold (shown as a horizontal dashed line) for the three tested methods - Opt, Ours and Quantile. |
---|---
|
Figure 8: Image Completion - Histograms of the calibrated mask sizes for the three tested methods - Opt, Ours and Quantile. |
---|---
|
Figure 9: Super Resolution - Histograms of the calibrated mask sizes for the three tested methods - Opt, Ours and Quantile. |
---|---
|
Figure 10: Colorization - Histograms of the calibrated mask sizes for the three tested methods - Opt, Ours and Quantile. |
---|---
|
Figure 11: Image Completion - The correlation between the Ours and Quantile mask sizes versus Opt’s mask size. |
---|---
|
Figure 12: Super Resolution - The correlation between the Ours and Quantile mask sizes versus Opt’s mask size. |
---|---
|
Figure 13: Colorization - The correlation between the Ours and Quantile mask
sizes versus Opt’s mask size.

In this appendix we bring the results corresponding to all 12 settings
explored in our work, corresponding to three inverse problems, two regression
techniques and two metrics per each, along the following breakdown:

  * •

Image Completion: {Regressor, GAN} $\times$ {L1, LPIPS};

  * •

Super Resolution: {Regressor, GAN} $\times$ {L1, SSIM}; and

  * •

Colorization: {Regressor, GAN} $\times$ {L1, L2}.

We start in Figures 5-7 with the obtained distributions of masked distortion
values for Opt, Ours and Quantile. The goal here is to show that all three
methods meet the required distortion threshold with exceptions that do not
surpass the destination probability $\beta=0.1$.

Figures 8-10 present histograms of the obtained mask sizes for the three
methods. The goal is to get minimal areas in these masks, so as to keep most
of the image content unmasked.

Figures 11-13 conclude these results by bringing graphs showing the inter-
relation between the Opt mask size and the ones given by Ours and Quantile.

Analysis and conclusions drawn from these graphs are brought in the
experimental part of the paper.