# Regularization by random translation of potentials for the continuous PAM
and related models in arbitrary dimension

Florian Bechtold  Florian Bechtold: Fakultät für Mathematik, Universität
Bielefeld, 33501 Bielefeld, Germany<EMAIL_ADDRESS>
###### Abstract.

We study a regularization by noise phenomenon for the continuous parabolic
Anderson model with a potential shifted along paths of fractional Brownian
motion. We demonstrate that provided the Hurst parameter is chosen
sufficiently small, this shift allows to establish well-posedness and
stability to the corresponding problem - without the need of renormalization -
in any dimension. We moreover provide a robustified Feynman-Kac type formula
for the unique solution to the regularized problem building upon regularity
estimates for the local time of fractional Brownian motion as well as non-
linear Young integration.

###  1\. Introduction

Consider the problem

$\partial_{t}u=\frac{1}{2}\Delta u-Vu,\qquad u(0)=f.$

As it is well known, provided $f$ and $V$ are sufficiently smooth, we obtain
the unique solution to the above problem via the Feynman-Kac formula

$u(t,x)=\mathbb{E}^{x}\left[f(W_{t})\exp{\left(-\int_{0}^{t}V(s,W_{t-s})ds\right)}\right],$

where $(W_{t})_{t}$ is a standard Brownian motion in $\mathbb{R}^{d}$ on a
stochastic basis $(\Omega,\mathcal{F},\mathbb{P},(\mathcal{F}_{t})_{t})$ and
$\mathbb{E}^{x}[(\cdot)]$ denotes the expectation conditional on the Brownian
motion starting in $x\in\mathbb{R}^{d}$. In the case of $V$ enjoying only
distributional regularity, this reasoning is no longer applicable. A famous
example in this context is the continuous parabolic Anderson model (PAM)

$\partial_{t}u=\frac{1}{2}\Delta u-\xi u,\qquad u(0)=f,$ (1.1)

where $\xi$ is spatial white noise on $\mathbb{T}^{d}$ or $\mathbb{R}^{d}$.
While in the case $d=1$ (1.1) is still well posed due to the regularizing
effect of the Laplacian, already $d=2,3$ require renormalization and the
implementation of advanced tools from the theory of singular SPDEs such as
regularity structures or paracontrolled distributions [Lab19], [HL15a],
[HL15b], [GIP15], [AC15], [Kö16, Example 1.21]. These approaches break down
for $d\;\geqslant\;4$.

In the following, we intend to study evolution problems of the above form with
a potential shifted along paths of fractional Brownian motion $w^{H}$ of Hurst
parameter $H$, i.e.

$\partial_{t}u=\frac{1}{2}\Delta u-\widetilde{V}u,\qquad u(0)=f,$ (1.2)

where we denote $\widetilde{V}(t,x):=V(x-w^{H}_{t})$ in case $V$ enjoys
sufficient regularity to admit pointwise evaluations or respectivelty
$\widetilde{V}(t,\varphi)=\langle V,\varphi(\cdot+w^{H}_{t})\rangle$ for any
smooth test function $\varphi$ in case $V$ only enjoys some distributional
regularity. We show that by shifting the reference frame of the potential in
such a way, we can restore well-posedness to (1.2) and obtain stability
results even in the setting of distributional valued $V$. As an application,
we establish well-posedness of a shifted parabolic Anderson model

$\partial_{t}u=\frac{1}{2}\Delta u-\widetilde{\xi}u,\qquad u(0)=f$ (1.3)

in arbitrary dimension, provided the Hurst parameter $H$ is chosen
sufficiently small.

Towards this end, we exploit a pathwise regularization phenomenon on the level
of the Feynman-Kac formula building mainly on [HP21]. More concretely, we
establish in Lemma 2.2 below that for smooth potentials $V^{\epsilon}$, we may
rewrite the Feynman-Kac formula for (1.2) as

$u^{\epsilon}(t,x)=\mathbb{E}^{x}\left[f(W_{t})\exp{\left(-(\mathcal{I}A^{t,\epsilon})_{t}\right)}\right],$
(1.4)

where $\mathcal{I}$ denotes Gubinelli’s Sewing operator111Refer to [Gub04] and
Lemma 3.3 in the Appendix., $A^{t,\epsilon}$ the germ

$A^{t,\epsilon}_{s,r}=(V^{\epsilon}*L_{s,r})(W_{t-s}),$

and $L$ the local time associated with the fractional Brownian motion $w^{H}$.
Due to Young’s inequality in Besov spaces, (1.4) might be well defined even
for distributional potentials $V$, provided the local time $L$ enjoys
sufficient spatial regularity222This is precisely what [HP21, Theorem 17],
also cited explicitly below, provides us with.. We therefore refer to (1.4) as
a robustified Feynman-Kac formula. This main observation allows us to
subsequently pass by a mollification argument: Given a distributional $V$, we
consider first a mollification $V^{\epsilon}=\rho^{\epsilon}*V$. For such
$V^{\epsilon}$, we may express the unique solution $u^{\epsilon}$ to the
associated PDE as in (1.4) by Lemma 2.2. Next, in Lemma 2.4 we establish that
due to the regularizing effect of the local time $L$,
$(u^{\epsilon})_{\epsilon}$ will converge in appropriate topologies to

$u(t,x)=\mathbb{E}^{x}\left[f(W_{t})\exp{\left(-(\mathcal{I}A^{t})_{t}\right)}\right],$
(1.5)

that then becomes a candidate for a solution to (1.2). Notice that the
expression $\widetilde{V}u$ appearing in the equation is however a priori ill-
defined. Yet, exploiting our explicit robustified Feynman-Kac representation
(1.5), we can easily establish higher spatial regularity of $u$. In
particular, we demonstrate that provided the initial condition $f$ is chosen
sufficiently smooth and $H$ sufficiently small, $u$ enjoys sufficient
regularity for the product

$V(\cdot)u(t,\cdot+w^{H}_{t})$

to be well defined in the sense of Lemma 1.4 for any $t\in[0,T]$. Under these
more restrictive conditions, we are thus able to identify (1.5) as a weak
solution to (1.2).

####  1.1. Formulation of the main result

######  Theorem 1.1.

For $\eta\;\geqslant\;0$ and $d\in\mathbb{N}$, let $V\in
H^{-\eta}(\mathbb{R}^{d})$ and $f\in C^{1}(\mathbb{R}^{d})$. Let $w^{H}$ be a
$d$-dimensional fractional Brownian motion on
$(\Omega^{H},\mathcal{F}^{H},\mathbb{P}^{H})$ whose Hurst parameter satisfies
$H<\frac{1}{2}(1+\eta+d/2)^{-1}$. Then for any mollification
$V^{\epsilon}=V*\rho^{\epsilon}$ and
$\widetilde{V^{\epsilon}}(t,x):=V^{\epsilon}(x-w^{H}_{t})$, the sequence
$(u^{\epsilon})_{\epsilon}$ of unique solutions to the the problem

$\partial_{t}u^{\epsilon}=\frac{1}{2}\Delta
u^{\epsilon}-\widetilde{V^{\epsilon}}u^{\epsilon},\qquad u(0)=f$

is Cauchy in $C([0,T]\times\mathbb{R}^{d})$ equipped with the topology of
uniform convergence, $\mathbb{P}^{H}$-almost surely. Moreover, provided
further $\eta\not\in\mathbb{N}$ and
$H<\frac{1}{2}(1+\eta+\lceil\eta\rceil+d/2)^{-1}$ and $f\in
C^{\lceil\eta\rceil}$, we have for the limit $u$ that $u(t,\cdot)\in
C^{\lceil\eta\rceil}$ uniformly in $[0,T]$, $\mathbb{P}^{H}$-almost surely. In
particular the product $V(\cdot)u(t,\cdot+w^{H}_{t})$ is well defined for any
$t\in[0,T]$ in the sense of Lemma 1.4 and $u$ is a weak solution to

$\partial_{t}u=\frac{1}{2}\Delta u-\widetilde{V}u,\qquad u(0)=f$ (1.6)

i.e. for any $\varphi\in C_{c}^{\infty}(\mathbb{R}^{d})$ and $t\in[0,T]$ we
have

$\langle u_{t}-f,\varphi\rangle=\int_{0}^{t}\langle
u_{s},\frac{1}{2}\Delta\varphi\rangle ds+\int_{0}^{t}\langle
V(\cdot)u(\cdot+w^{H}_{s}),\varphi(\cdot+w^{H}_{s})\rangle ds.$

######  Corollary 1.2 (Regularized PAM).

Consider spatial white noise on the $d$-dimensional torus $\mathbb{T}^{d}$.
Its realizations are known to lie in $H^{-(d/2+\epsilon)}$ for any
$\epsilon>0$ almost surely [Ver11]. Hence, imposing $H<\frac{1}{2}(1+d)^{-1}$,
we can apply the first part of our main result Theorem 1.1. Demanding further
$H<\frac{1}{2}(1+d+\lceil d/2+1/4\rceil)^{-1}$, we may employ the second part,
yielding a weak solution.

######  Remark 1.3.

Note that due to the robustness of our approach, several canonical extensions
to the above statements follow readily mutatis mutandis: Instead of
considering the Laplacian, more general non-degenerate diffusion operators can
be considered in (1.6). Moreover, instead of shifting the potential $V$ along
paths of fractional Brownian motion, shifts along any path that admits a
sufficiently regular local time are conceivable.

####  1.2. Short overview of existing literature

The idea of employing a robustified Feynman-Kac formula in the study of heat
equations with some form of multiplicative noise can be traced back to at
least [MV05], [HNS11], [HLN12] for various types of space-time fractional
Brownian motions. [HL16] combines these considerations with non-linear Young
theory similar in spirit to the setting presented in this article. Remark
however that our qualitatively different approach of considering random
translations of the potential allows us to treat considerably more singular
potentials. Furthermore, robustifications of the Feynman-Kac formula have been
employed in the study of rough stochastic PDEs for example in [FH14, Chapter
12] or [DFS17]. Regularization by additive noise for the multiplicative
stochastic heat equation was recently established by [CH21] building upon
ideas on pathwise regularization by noise in the spirit of [CG16], [GG21],
[HP21]. Let us mention that this approach to regularization by noise has
recently seen numerous interesting applications for example to interacting
particle systems [HM20], distribution dependent SDEs [GHM21], [GHM22] and
multiplicative SDEs [GH20], [BH22].

####  1.3. Notation

We employ the standard notation necessary to formulate and apply the Sewing
Lemma for which we refer to Appendix B. Let $\mathcal{S}^{\prime}$ denote the
space of tempered distributions. For $\eta\in\mathbb{R}$, we denote by
$H^{\eta}$ the inhomogeneous Bessel potential space of order $\eta$, i.e.

$H^{\eta}:=\left\\{f\in\mathcal{S}^{\prime}|\ \left\lVert
f\right\rVert_{H^{\eta}}=\left\lVert(1+|(\cdot)|)^{\eta}\hat{f}\right\rVert_{L^{2}}<\infty\right\\}.$

Moreover, for $\alpha>0$ and $\alpha\not\in\mathbb{N}$ we denote by
$\mathcal{C}^{\alpha}$ the Hölder space

$\mathcal{C}^{\alpha}:=\left\\{f\in\mathcal{S}^{\prime}|\ \left\lVert
f\right\rVert_{\mathcal{C}^{\alpha}}=\left\lVert
f\right\rVert_{C^{\lfloor\alpha\rfloor}}+\sup_{x\neq
y}\frac{|(D^{k}f)(x)-(D^{k}f)(y)|}{|x-y|^{\alpha-\lfloor\alpha\rfloor}}<\infty\right\\},$

where for $n\in\mathbb{N}_{0}$

$\left\lVert f\right\rVert_{C^{n}}=\sum_{k=0}^{n}\left\lVert
D^{k}f\right\rVert_{\infty}.$ (1.7)

We denote by $C^{n}$ the space of $n$-times continuously differentiable
functions such that (1.7) is finite. Let us remark that the first two spaces
above are related to more general Besov spaces in the sense that
$H^{\alpha}=B^{\alpha}_{2,2}$ for any $\alpha\in\mathbb{R}$ and
$\mathcal{C}^{\alpha}=B^{\alpha}_{\infty,\infty}$ for $\alpha>0$ and
$\alpha\not\in\mathbb{N}$. In particular, note that by Young’s inequality in
Besov spaces [KS21], we have

$\left\lVert f*g\right\rVert_{\mathcal{C}^{\alpha-\eta}}\lesssim\left\lVert
f\right\rVert_{H^{\alpha}}\left\lVert g\right\rVert_{H^{-\eta}}$ (1.8)

for any $\alpha-\eta$ such that $\alpha-\eta>0$ and
$\alpha-\beta\not\in\mathbb{N}$. Let us also recall the multiplication theorem
for Besov spaces (see e.g. [Mar18, Corollary 2.1.35], [vZ20, Theorem 19.7])
adapted to our setting:

######  Lemma 1.4.

Let $\alpha>0$ such that $\alpha-\eta>0$. Then for any $\epsilon>0$ and
$u,v\in\mathcal{S}^{\prime}$ we have

$\left\lVert u\cdot V\right\rVert_{H^{-\eta-\epsilon}}\lesssim\left\lVert
u\right\rVert_{\mathcal{C}^{\alpha}}\left\lVert V\right\rVert_{H^{-\eta}},$

i.e. the multiplication operator extends to a continuous bilinear map
$\cdot:\mathcal{C}^{\alpha}\times H^{-\eta}\to H^{-\eta-\epsilon}$

###  2\. Proof of Theorem 1.1

For the readers convenience we begin by stating the following result on the
regularity of local times associated with fractional Brownian motion that we
will use throughout. We refer to the Appendix A the basic definitions of
occupation measures, local times and the occupation times formula.

######  Lemma 2.1 ([HP21, Theorem 17] ).

Let $w^{H}$ be a $d$-dimensional fractional Brownian motion of Hurst parameter
$H<1/d$ on $(\Omega^{H},\mathcal{F}^{H},\mathbb{P}^{H})$. Then there exists a
null set $\mathcal{N}$ such that for all $\omega\in\mathcal{N}^{c}$, the path
$w^{H}(\omega)$ has a local time $L(\omega)$ and for
$\lambda<\frac{1}{2H}-\frac{d}{2}$ and $\gamma\in[0,1-(\lambda+\frac{d}{2})H)$
we have

$\left\lVert
L_{s,t}(\omega)\right\rVert_{H^{\lambda}}\;\leqslant\;C_{T}(\omega)|t-s|^{\gamma}.$
(2.1)

for any $s,t\in[0,T]$, where $L_{s,t}=L_{t}-L_{s}$.

Throughout the remainder of the paper and for $H<1/d$ satisfying the
conditions demanded in the statements below, we shall fix a realization of
fractional Brownian motion $w^{H}$ on
$(\Omega^{H},\mathcal{F}^{H},\mathbb{P}^{H})$ that admits a local time $L$ and
for which Lemma 2.1 can be applied in the corresponding regularity regime.

####  2.1. Solutions to the mollified equation converge

Let $V^{\epsilon}$ be a mollification of some given
$V\in\mathcal{S}^{\prime}$. Then for

$\widetilde{V^{\epsilon}}(t,x):=V^{\epsilon}(x-w^{H}_{t})$

we know that the unique solution to the problem

$\partial_{t}u^{\epsilon}=\frac{1}{2}\Delta
u^{\epsilon}-(\widetilde{V^{\epsilon}})u^{\epsilon},\qquad u^{\epsilon}(0)=f,$
(2.2)

is given by

$\displaystyle u^{\epsilon}(t,x)$
$\displaystyle=\mathbb{E}^{x}\left[f(W_{t})\exp{\left(-\int_{0}^{t}V^{\epsilon}(W_{t-s}-w^{H}_{s})ds\right)}\right],$

where $(W_{t})_{t}$ is a standard Brownian motion in $\mathbb{R}^{d}$ on a
stochastic basis $(\Omega,\mathcal{F},\mathbb{P},(\mathcal{F}_{t})_{t})$ and
$\mathbb{E}^{x}[(\cdot)]$ denotes the expectation conditional on the Brownian
motion starting in $x\in\mathbb{R}^{d}$. We first establish that we may
replace the above Lebesgue integral in time by an appropriate sewing that is
capable of leveraging the regularizing effect due to the highly oscillating
fractional Brownian motion. Towards this end, we exploit the Sewing Lemma 3.3.

######  Lemma 2.2 (Identification of Riemann integral as Sewing).

For $\delta>0$, let $V\in H^{(1\vee d/2)+\delta}(\mathbb{R}^{d})$. Let $W$ be
a standard Brownian motion on $\mathbb{R}^{d}$ on a stochastic basis
$(\Omega,\mathcal{F},\mathbb{P},(\mathcal{F}_{t})_{t})$. Then for almost every
$\omega\in\Omega$ the germ

$A^{t}_{s,r}:=(V*L_{s,r})(W_{t-s}(\omega))=\int_{s}^{r}V(W_{t-s}(\omega)-w^{H}_{v})dv$

admits a Sewing $(\mathcal{I}A^{t})$ on $[0,t]$ and moreover for any
$t\in[0,T]$ we have

$\int_{0}^{t}V(W_{t-s}-w^{H}_{s})ds=(\mathcal{I}A^{t})_{t}$

###### Proof.

Remark first that by Lemma 2.1 we have for some $\epsilon>0$ and any
$(s,t)\in\Delta_{2}([0,T])$:

$\left\lVert
L_{s,t}\right\rVert_{L^{2}}\;\leqslant\;C_{T}|t-s|^{1/2+\epsilon}.$

Moreover, by Young’s inequality in Besov spaces (1.8), we have that

$\left\lVert V*L_{s,t}\right\rVert_{\mathcal{C}^{1+\delta}}\lesssim\left\lVert
V\right\rVert_{H^{1+\delta}}\left\lVert L_{s,t}\right\rVert_{L^{2}}$

Moreover, we have for almost every $\omega\in\Omega$ that $W\in
C^{1/2-\epsilon/2}$. We therefore obtain for $(s,u,r,t)\in\Delta_{4}([0,T])$
that

$\displaystyle|(\delta A^{t})_{s,u,r}|$
$\displaystyle=|V*L_{u,r}(W_{t-s})-V*L_{u,r}(W_{t-u})|$
$\displaystyle\lesssim\left\lVert V\right\rVert_{H^{1+\delta}}\left\lVert
L_{s,t}\right\rVert_{L^{2}}|r-u|^{1/2+\epsilon}|u-s|^{1/2-\epsilon/2}.$

We conclude that $A^{t}$ does indeed admit a Sewing on $[0,t]$ and for
$(s,u,t)\in\Delta_{3}([0,T])$ we have

$|A^{t}_{s,u}-(\mathcal{I}A^{t})_{s,u}|=O(|u-s|^{1+\epsilon/2}).$

Next, observe that the germ

$\widetilde{A}^{t}_{s,u}:=\int_{s}^{u}V(W_{t-v}-w^{H}_{v})dv$

trivially admits a Sewing, as $\delta\widetilde{A}^{t}=0$ wherefore we have
$(\mathcal{I}\widetilde{A}^{t})=\widetilde{A}^{t}$. Moreover note that because
of $V\in H^{d/2+\delta}\hookrightarrow\mathcal{C}^{\delta}$ we have for
$(s,u,t)\in\Delta_{3}$

$\displaystyle\left|\widetilde{A}^{t}_{s,u}-A^{t}_{s,u}\right|\lesssim\int_{s}^{u}|W_{t-s}-W_{t-v}|^{\delta}dv\lesssim|u-s|^{1+\delta(1-\epsilon/2)}$

allowing to conclude

$\displaystyle\left|\int_{s}^{u}V(W_{t-v}-w^{H}_{v})dv-(\mathcal{I}A^{t})_{s,u}\right|\;\leqslant\;|\widetilde{A}^{t}_{s,u}-A^{t}_{s,u}|+|A^{t}_{s,u}-(\mathcal{I}A^{t})_{s,u}|\lesssim|u-s|^{1+\delta(1-\epsilon/2)}$

Hence the function

$s\in[0,t]\to\int_{0}^{s}V(W_{t-r}-w^{H}_{r})dr-(\mathcal{I}A^{t})_{s}$

is constant. Since it moreover starts in zero, this establishes the claim.

∎

######  Remark 2.3.

Remark that in the above statement, we did not exploit any regularization from
the local time, but instead demanded regularity of the potential $V$. As a
consequence, the only constraint on the Hurst parameter at this stage is
$H<1/d$, which simply assures the existence of a local time. In the following,
we will impose further restrictions on the Hurst parameter, allowing to pass
to less regular potentials $V$.

By the previous Lemma 2.2, we have that indeed

$\displaystyle u^{\epsilon}(t,x)$
$\displaystyle=\mathbb{E}^{x}\left[f(W_{t})\exp{\left(-\int_{0}^{t}V^{\epsilon}(W_{t-s}-w^{H}_{s})ds\right)}\right]$
$\displaystyle=\mathbb{E}^{x}\left[f(W_{t})\exp{\left(-(\mathcal{I}A^{t,\epsilon})_{t}\right)}\right],$

where

$A^{t,\epsilon}_{s,u}:=(V^{\epsilon}*L_{s,u})(W_{t-s}).$

In the next Lemma we address the question: Under which condition on $V$ and
$H$ is it possible to pass to a limit $\epsilon\to 0$?

######  Lemma 2.4 (Convergence of mollifications).

For $\eta\;\geqslant\;0$, $d\in\mathbb{N}$, let $f\in C^{1}(\mathbb{R}^{d})$,
$V\in H^{-\eta}(\mathbb{R}^{d})$ and $H<\frac{1}{2}(1+\eta+d/2)^{-1}$. Let
$u^{\epsilon}$ be the unique solution to the mollified problem (2.2) and set

$u(t,x):=\mathbb{E}^{x}\left[f(W_{t})\exp{\left(-(\mathcal{I}A^{t})_{t}\right)}\right]$
(2.3)

where

$A^{t}_{s,u}:=(V*L_{s,u})(W_{t-s}).$

Then $u^{\epsilon}$ converges uniformly to $u$ on $[0,T]\times\mathbb{R}^{d}$.

###### Proof.

Let us start by establishing that $u$ is well defined under the conditions
stated in the Lemma. By Lemma 2.1, we have for some $\delta>0$

$\left\lVert
L_{s,t}\right\rVert_{H^{1+\eta+\delta}}\lesssim|t-s|^{1/2+\delta}.$

Note that again, by Young’s inequality in Besov spaces, we have

$\displaystyle\left\lVert
V*L_{s,t}\right\rVert_{\mathcal{C}^{1+\delta}}\;\leqslant\;\left\lVert
V\right\rVert_{H^{-\eta}}\left\lVert L_{s,t}\right\rVert_{H^{1+\eta+\delta}}$

meaning again in particular that $V*L_{s,t}$ lies in $C^{1}$. Similar to the
previous Lemma 2.2, we can conclude that indeed, $A^{t}$ admits a sewing,
since

$\displaystyle|(\delta A^{t})_{s,u,r}|$
$\displaystyle=|V*L_{u,r}(W_{t-s})-V*L_{u,r}(W_{t-u})|$
$\displaystyle\lesssim\underbrace{\left(\sup_{x\neq
y\in[0,T]}\frac{|W_{x}(\omega)-W_{y}(\omega)|}{|x-y|^{1/2-\delta/2}}\right)}_{=:c_{\delta}(\omega)}|r-u|^{1/2+\delta}|u-s|^{1/2-\delta/2}.$

The above ensures that for almost every $\omega\in\Omega$ the expression
$(\mathcal{I}A^{t})_{0,t}$ is well defined. We further demonstrate that
$(\mathcal{I}A^{t})_{0,t}$ admits exponential moments with respect to the
measure $\mathbb{P}$, allowing to establish well posedness of (2.3). By the
Sewing Lemma 3.3, we have for $t\in[0,T]$ the a priori bound

$\begin{split}|(\mathcal{I}A^{t})_{0,t}|&\;\leqslant\;|A^{t}_{0,t}|+|(\mathcal{I}A^{t})_{0,t}-A^{t}_{0,t}|\\\
&\;\leqslant\;\left\lVert V*L_{0,t}\right\rVert_{\infty}+\left\lVert\delta
A^{t}\right\rVert_{1+\delta/2}T^{1+\delta/2}\\\ &\lesssim
1+c_{\delta}(\omega).\end{split}$ (2.4)

Hence, we conclude that for some $a>0$, we have

$\displaystyle|\mathbb{E}^{x}\left[f(W_{t})\exp{(-(\mathcal{I}A)_{0,t})}\right]|\lesssim\left\lVert
f\right\rVert_{\infty}\mathbb{E}^{x}[\exp{(ac_{\delta}(\omega))}]<\infty$

by Lemma 3.5. This shows that the function $u$ in (2.3) is well defined as a
function in $C^{0}([0,T]\times\mathbb{R}^{d})$. Towards establishing
convergence, let us first remark that similar to (2.4), we have

$|(\mathcal{I}A^{t,\epsilon})_{0,t}|\lesssim 1+c_{\delta}(\omega)$

uniformly in $\epsilon>0$. This permits the following bound

$\displaystyle|u^{\epsilon}_{t}(x)-u_{t}(x)|$
$\displaystyle\;\leqslant\;\left|\mathbb{E}^{x}\left[f(W_{t})(e^{-(\mathcal{I}A^{t})_{t}}-e^{-(\mathcal{I}A^{t,\epsilon})_{t}})\right]\right|$
$\displaystyle\;\leqslant\;\mathbb{E}^{x}\left[|f(W_{t})|e^{ac_{\delta}}|(\mathcal{I}A^{t})_{t}-(\mathcal{I}A^{t,\epsilon})_{t}|)\right].$

Next, note that due to the linearity of the Sewing operator $\mathcal{I}$

$\displaystyle|(\mathcal{I}A^{t})_{s,t}-(\mathcal{I}A^{t,\epsilon})_{t}|\;\leqslant\;\left\lVert
A^{t}-A^{t,\epsilon}\right\rVert_{1/2}T^{1/2}+\left\lVert\delta(A^{t}-A^{t,\epsilon})\right\rVert_{1+\delta/2}T^{1+\delta/2}.$

We have moreover that

$\displaystyle(A^{t}-A^{t,\epsilon})_{s,r}$
$\displaystyle\;\leqslant\;\left\lVert(V-V^{\epsilon})*L_{s,r}\right\rVert_{\infty}\lesssim\left\lVert
V-V^{\epsilon}\right\rVert_{H^{-\eta}}\left\lVert
L_{s,r}\right\rVert_{H^{\eta}+\delta}\lesssim\left\lVert
V-V^{\epsilon}\right\rVert_{H^{-\eta}}|r-s|^{1/2},$

as well as similar to the above calculations

$\displaystyle(\delta(A^{t}-A^{t,\epsilon}))_{s,u,r}$
$\displaystyle=|(V-V^{\epsilon})*L_{u,r}(W_{t-s})-(V-V^{\epsilon})*L_{u,r}(W_{t-u})|$
$\displaystyle\lesssim c_{\delta}(\omega)\left\lVert
V-V^{\epsilon}\right\rVert_{H^{-\eta}}|r-u|^{1/2+\delta}|u-s|^{1/2-\delta/2}.$

Overall, this permits to conclude that

$\displaystyle|u^{\epsilon}_{t}(x)-u_{t}(x)|\lesssim\left\lVert
V-V^{\epsilon}\right\rVert_{H^{-\eta}}\left\lVert
f\right\rVert_{\infty}\mathbb{E}^{x}[c_{\delta}\exp{(a(1+c_{\delta}))}]\lesssim\left\lVert
V-V^{\epsilon}\right\rVert_{H^{-\eta}}\left\lVert f\right\rVert_{\infty}$

exploiting Lemma 3.5. Hence we have established the claim. ∎

####  2.2. Weak solutions

In the following section, we undertake to establish in what sense and under
what conditions the function $u$ obtained in the previous section solves our
original problem. We recall again that the main obstacle to address lies in
the appearance of the product $\widetilde{V}u$. This obstacle will be overcome
by establishing higher spatial regularity of $u$ building upon our robustified
Feynman-Kac representation for $u$ in Lemma 2.4 provided we demand more
regularity in the initial condition $f$ as well as the local time $L$. In this
way, Lemma 1.4 allows us to give a meaning to the product $\widetilde{V}u$,
thereby allowing to conclude that $u$ satisfies the original problem in the
weak sense of Theorem 1.1.

######  Lemma 2.5 (Spatial regularity of $u$).

For $n\in\mathbb{N}$ suppose $H<\frac{1}{2}(1+\eta+n+d/2)^{-1}$, $f\in C^{n}$
and $V\in H^{-\eta}$. Then for every $t\in[0,T]$, the function

$u(t,x):=\mathbb{E}^{x}\left[f(W_{t})\exp{\left(-(\mathcal{I}A^{t})_{0,t}\right)}\right],$
(2.5)

where

$A^{t}_{s,r}=(V*L_{s,r})(W_{t-s})$

lies in $C^{n}$. Moreover, $u^{\epsilon}\to u$ in $C^{n}$, uniformly in
$t\in[0,T]$.

###### Proof.

Fix $t\in[0,T]$. We show that the function

$x\to(\mathcal{I}A^{t}(x))_{t}$

where

$A_{s,r}^{t}(x)=(V*L_{s,r})(W_{t-s}+x)$

is $n$ times differentiable and that moreover, all derivatives up to order $n$
are uniformly bounded in space, integrable with respect to $\mathbb{P}$.
Towards this end, let us note that

$D^{k}_{x}(\mathcal{I}A^{t}(x))_{t}=(\mathcal{I}(D^{k}_{x}A^{t})(x))_{t}.$
(2.6)

This can be established by using Lemma 3.4. For the sake of conciseness, we
restrict ourselves to the case $d=1$ and one derivative. Let us define

$A^{t,n}_{s,r}=n\left((V*L_{s,r})(W_{t-s}+x+1/n)-(V*L_{s,r})(W_{t-s}+x)\right).$

Then it can be seen easily that

$\left\lVert A^{t,n}-D_{x}A^{t}\right\rVert_{1/2}\to 0$

uniformly in $t\in[0,T]$. Moreover,

$\displaystyle\delta A^{t,n}_{s,u,r}$
$\displaystyle=n\left((V*L_{s,u})(W_{t-r}+x+1/n)-(V*L_{s,u})(W_{t-r}+x)\right)$
$\displaystyle-n\left((V*L_{s,u})(W_{t-u}+x+1/n)+(V*L_{s,u})(W_{t-u}+x)\right)$
$\displaystyle=(V*D_{x}L_{s,u})(W_{t-r}+x)-(V*D_{x}L_{s,u})(W_{t-u}+x)+O(1/n)|u-s|^{1+2\delta}$
$\displaystyle\lesssim
c_{\delta}(\omega)|u-s|^{1/2+\delta}|r-u|^{1/2-\delta/2}+O(1/n)|u-s|^{1+2\delta}$

meaning that indeed $\sup_{n}\left\lVert\delta
A^{t,n}\right\rVert_{1+\delta/2}<\infty$, allowing to conclude (2.6) by Lemma
3.4. By the Faà di Bruno formula, we have

$\displaystyle\frac{d^{n}}{dx^{n}}\left(f(W_{t}+x)\exp{\left(-(\mathcal{I}A^{t}(x))_{t}\right)}\right)$
$\displaystyle=$
$\displaystyle\exp{\left(-(\mathcal{I}A^{t}(x))_{t}\right)}\sum_{k=0}^{n}\binom{n}{k}\left(D^{n-k}_{x}f(W_{t}+x)\right)B_{k}(-(\mathcal{I}D_{x}A^{t}(x))_{t},\dots-(\mathcal{I}D^{k}_{x}A^{t}(x))_{t}),$

where $B_{k}$ denotes the $k$-th complete Bell polynomial with the convention
$B_{0}=1$. Note in particular that $(\mathcal{I}D^{k}_{x}A^{t}(x))_{t}$ is
uniformly bounded in space due to the regularity of the local time in this
setting. Moreover, we have for any $k\;\leqslant\;n$ the a priori bound

$|(\mathcal{I}D^{k}_{x}A^{t}(x))_{t}|\lesssim(1+c_{\delta}(\omega)).$

We therefore have for any $a>0$

$|B_{k}(-D(\mathcal{I}A^{t})_{0,t},\dots
D^{k}(\mathcal{I}A^{t})_{0,t})|\lesssim\exp{(ac_{\gamma}(\omega))}$

wherefore

$\frac{d^{n}}{dx^{n}}\left(f(B_{t}+x)\exp{\left(-(\mathcal{I}A^{t})_{0,t}\right)}\right)$

is well defined, uniformly bounded in $x\in\mathbb{R}^{d}$ and integrable with
respect to $\mathbb{P}$. Overall, this allows to conclude that indeed
$u(t,\cdot)\in C^{n}$ for any $t\in[0,T]$. Finally, going through similar
considerations for $u^{\epsilon}$ and remarking that

$(\mathcal{I}D^{k}_{x}(A^{t}-A^{t,\epsilon}(x)))_{t}\to 0$

uniformly in $t,x\in[0,T]\times\mathbb{R}^{d}$, we infer that
$u^{\epsilon}(t,\cdot)\to u(t,\cdot)$ in $C^{n}$ uniformly in $t\in[0,T]$. ∎

As a corollary, invoking Lemma 1.4, we can now conclude the proof of Theorem
1.1 by observing the following:

######  Corollary 2.6.

Suppose $H<\frac{1}{2}(1+\eta+\lceil\eta\rceil+d/2)^{-1}$, $f\in
C^{\lceil\eta\rceil}$ and $V\in H^{-\eta}$. Then for any $t\in[0,T]$, the
product

$\langle(\widetilde{V}u)_{t},\varphi\rangle:=\langle
V(\cdot)u(t,\cdot+w^{H}_{t}),\varphi(\cdot+w^{H}_{t})\rangle$

is well defined in the sense of Lemma 1.4. In particular, we infer that $u$ is
a weak solution to the original problem.

###  3\. Appendix

##  Appendix A Local time and occupation times formula

We recall for the reader the basic concepts of occupation measures, local
times and the occupation times formula. A comprehensive review paper on these
topics is [GH80].

######  Definition 3.1.

Let $w:[0,T]\to\mathbb{R}^{d}$ be a measurable path. Then the occupation
measure at time $t\in[0,T]$, written $\mu^{w}_{t}$ is the Borel measure on
$\mathbb{R}^{d}$ defined by

$\mu^{w}_{t}(A):=\lambda(\\{s\in[0,t]:\ w_{s}\in A\\}),\quad
A\in\mathcal{B}(\mathbb{R}^{d}),$

where $\lambda$ denotes the standard Lebesgue measure.

The occupation measure thus measures how much time the process $w$ spends in
certain Borel sets. Provided for any $t\in[0,T]$, the measure is absolutely
continuous with respect to the Lebesgue measure on $\mathbb{R}^{d}$, we call
the corresponding Radon-Nikodym derivative local time of the process $w$:

######  Definition 3.2.

Let $w:[0,T]\to\mathbb{R}^{d}$ be a measurable path. Assume that there exists
a measurable function $L^{w}:[0,T]\times\mathbb{R}^{d}\to\mathbb{R}_{+}$ such
that

$\mu^{w}_{t}(A)=\int_{A}L^{w}_{t}(z)dz,$

for any $A\in\mathcal{B}(\mathbb{R}^{d})$ and $t\in[0,T]$. Then we call
$L^{w}$ local time of $w$.

Note that by the definition of the occupation measure, we have for any bounded
measurable function $f:\mathbb{R}^{d}\to\mathbb{R}$ that

$\int_{0}^{t}f(w_{s})ds=\int_{\mathbb{R}^{d}}f(z)\mu^{w}_{t}(dz).$ (3.1)

The above equation (3.1) is called occupation times formula. Remark that in
particular, provided $w$ admits a local time, we also have for any
$x\in\mathbb{R}^{d}$

$\int_{0}^{t}f(x-w_{s})ds=\int_{\mathbb{R}^{d}}f(x-z)\mu^{w}_{t}(dz)=\int_{\mathbb{R}^{d}}f(x-z)L^{w}_{t}(z)dz=(f*L^{w}_{t})(x).$
(3.2)

##  Appendix B The Sewing Lemma

We recall the Sewing Lemma due to [Gub04] (see also [FH14, Lemma 4.2]). Let
$E$ be a Banach space, $[0,T]$ a given interval. Let $\Delta_{n}$ denote the
$n$-th simplex of $[0,T]$, i.e.
$\Delta_{n}:=\\{(t_{1},\dots,t_{n})|0\;\leqslant\;t_{1}\dots\;\leqslant\;t_{n}\;\leqslant\;T\\}$.
For a function $A:\Delta_{2}\to E$ define the mapping $\delta A:\Delta_{3}\to
E$ via

$(\delta A)_{s,u,t}:=A_{s,t}-A_{s,u}-A_{u,t}.$

Provided $A_{t,t}=0$ we say that for $\alpha,\beta>0$ we have $A\in
C^{\alpha,\beta}_{2}(E)$ if $\left\lVert A\right\rVert_{\alpha,\beta}<\infty$,
where

$\left\lVert
A\right\rVert_{\alpha}:=\sup_{(s,t)\in\Delta_{2}}\frac{\left\lVert
A_{s,t}\right\rVert_{E}}{|t-s|^{\alpha}},\qquad\left\lVert\delta
A\right\rVert_{\beta}:=\sup_{(s,u,t)\in\Delta_{3}}\frac{\left\lVert(\delta
A)_{s,u,t}\right\rVert_{E}}{|t-s|^{\beta}}$

and $\left\lVert A\right\rVert_{\alpha,\beta}:=\left\lVert
A\right\rVert_{\alpha}+\left\lVert\delta A\right\rVert_{\beta}$. For a
function $f:[0,T]\to E$, we denote $f_{s,t}:=f_{t}-f_{s}$. Moreover, if for
any sequence $(\mathcal{P}^{n}([s,t]))_{n}$ of partitions of $[s,t]$ whose
mesh size goes to zero, the quantity

$\lim_{n\to\infty}\sum_{[u,v]\in\mathcal{P}^{n}([s,t])}A_{u,v}$

converges to the same limit, we note

$(\mathcal{I}A)_{s,t}:=\lim_{n\to\infty}\sum_{[u,v]\in\mathcal{P}^{n}([s,t])}A_{u,v}.$

######  Lemma 3.3 (Sewing, [Gub04]).

Let $0<\alpha\;\leqslant\;1<\beta$. Then for any $A\in
C^{\alpha,\beta}_{2}(E)$, $(\mathcal{I}A)$ is well defined (we say that $A$
admits the sewing $(\mathcal{I}A)$). Moreover, denoting
$(\mathcal{I}A)_{t}:=(\mathcal{I}A)_{0,t}$, we have $(\mathcal{I}A)\in
C^{\alpha}([0,T],E)$ and $(\mathcal{I}A)_{0}=0$ and for some constant $c>0$
depending only on $\beta$ we have

$\left\lVert(\mathcal{I}A)_{t}-(\mathcal{I}A)_{s}-A_{s,t}\right\rVert_{E}\;\leqslant\;c\left\lVert\delta
A\right\rVert_{\beta}|t-s|^{\beta}.$

######  Lemma 3.4 (Lemma A.2 [Gal21]).

For $0<\alpha\;\leqslant\;1<\beta$ and $E$ a Banach space, let $A\in
C^{\alpha,\beta}_{2}(E)$ and $(A^{n})_{n}\subset C^{\alpha,\beta}_{2}(E)$ such
that for some $R>0$ $\sup_{n\in\mathbb{N}}\left\lVert\delta
A^{n}\right\rVert_{\beta}\;\leqslant\;R$ and such that $\left\lVert
A^{n}-A\right\rVert_{\alpha}\to 0$. Then

$\left\lVert\mathcal{I}(A-A^{n})\right\rVert_{\alpha}\to 0.$

##  Appendix C Exponential moments for the Hölder modulus of continuity of
Brownian motion

For the sake of completeness, we provide a sketch of the proof that the
$\gamma$-Hölder modulus of continuity of Brownian motion is exponentially
integrable for $\gamma\in(0,1/2)$. Refer also to [HV08] for more refined
integrability statements.

######  Lemma 3.5.

Let $B$ be a standard Brownian motion. Then for any $\gamma<1/2$ and $a>0$, we
have

$\mathbb{E}\left[\exp{\left(a\sup_{s\neq
t\in[0,T]}\frac{|B_{t}-B_{s}|}{|t-s|^{\gamma}}\right)}\right]<\infty.$

###### Proof.

Without loss of generality, set $T=1$. Remark first that we have for
$k\in\mathbb{N}$

$\mathbb{E}[|B_{t}-B_{s}|^{k}]\;\leqslant\;|t-s|^{k/2}(k-1)!!$

We follow the classical proof of Kolmogorov’s continuity theorem (refer for
example to [SP12, Theorem 10.1]). For $D_{m}:=2^{-m}\mathbb{N}_{0}\cap[0,1)$
and $D=\cup_{m}D_{m}$ set

$\Delta_{m}=\\{(s,t)\in D_{m}\times D_{m}:\ |t-s|\;\leqslant\;2^{-m}\\}.$

We then have for $\sigma_{j}:=\sup_{(s,t)\in\Delta_{j}}|B(t)-B(s)|$ the bound

$\mathbb{E}[\sigma_{j}^{k}]\;\leqslant\;\sum_{(s,t)\in\Delta_{j}}\mathbb{E}[|B_{t}-B_{s}|^{k}]\;\leqslant\;2\cdot
2^{j(1-k/2)})(k-1)!!$

Following further the proof of Kolmogorov’s continuity theorem as in [SP12,
Theorem 10.1], we obtain

$\displaystyle\mathbb{E}\left[\left(\sup_{s\neq t\in
D}\frac{|B_{t}-B_{s}|}{|t-s|^{\gamma}}\right)^{k}\right]^{1/k}$
$\displaystyle\;\leqslant\;2^{1+\gamma}\sum_{j=0}^{\infty}2^{j\gamma}\mathbb{E}[\sigma_{j}^{k}]^{1/k}$
$\displaystyle\;\leqslant\;2^{1/k}\cdot
2^{1+\gamma}((k-1)!!)^{1/k}\sum_{j=0}^{\infty}2^{j\gamma}(2^{-j(1/2-1/k)}).$

Now let $k_{0}$ be the smallest natural such that $\gamma<1/2-1/k_{0}$. We
then obtain for any $k\;\geqslant\;k_{0}$

$\displaystyle\mathbb{E}\left[\left(\sup_{s\neq t\in
D}\frac{|B_{t}-B_{s}|}{|t-s|^{\gamma}}\right)^{k}\right]^{1/k}$
$\displaystyle\;\leqslant\;2^{1/k}\cdot
2^{1+\gamma}((k-1)!!)^{1/k}\sum_{j=0}^{\infty}2^{j(\gamma+1/k-1/2)}$
$\displaystyle\;\leqslant\;2^{1/k}\cdot
2^{1+\gamma}((k-1)!!)^{1/k}\sum_{j=0}^{\infty}2^{j(\gamma+1/k_{0}-1/2)}$
$\displaystyle\;\leqslant\;C2^{1/k}\cdot 2^{1+\gamma}((k-1)!!)^{1/k}.$

Therefore, we have

$\displaystyle\sum_{k=k_{0}}^{\infty}\frac{a^{k}}{k!}\mathbb{E}\left[\left(\sup_{s\neq
t\in[0,1]}\frac{|B_{t}-B_{s}|}{|t-s|^{\gamma}}\right)^{k}\right]$
$\displaystyle\;\leqslant\;2\sum_{k=k_{0}}^{\infty}\frac{(2^{1+\gamma}Ca)^{k}}{k!!}<\infty,$

which proves the claim. ∎

### Acknowledgement

The author wishes to thank Cyril Labbé for early discussions and Martina
Hofmanová, Jörn Wichmann and Emanuela Gussetti for comments and suggestions on
a preliminary version of this work. This project has received funding from the
European Research Council under the European Union’s Horizon 2020 research and
innovation programme (grant agreement No.754362 and No.949981).

## References

  * [AC15] R. Allez and K. Chouk.  The continuous anderson hamiltonian in dimension two, 2015.  arXiv:1511.02718.
  * [BH22] F. Bechtold and M. Hofmanová.  Weak solutions for singular multiplicative sdes via regularization by noise, 2022.  arXiv:2203.13745.
  * [CG16] R. Catellier and M. Gubinelli.  Averaging along irregular curves and regularisation of ODEs.  Stochastic Processes and their Applications, 126(8):2323–2366, 2016\.
  * [CH21] R. Catellier and F. Harang.  Pathwise regularization of the stochastic heat equation with multiplicative noise through irregular perturbation, 2021.  arXiv:2101.00915.
  * [DFS17] J. Diehl, P. Friz, and W. Stannat.  Stochastic partial differential equations: a rough paths view on weak solutions via feynman–kac.  Annales de la Faculté des sciences de Toulouse : Mathématiques, Ser. 6, 26(4):911–947, 2017.
  * [FH14] P. Friz and M. Hairer.  A course on rough paths.  Universitext. Springer, Cham, 2014.
  * [Gal21] L. Galeati.  Nonlinear young differential equations: A review.  Journal of Dynamics and Differential Equations, February 2021.
  * [GG21] L. Galeati and M. Gubinelli.  Noiseless regularisation by noise.  Revista Matemática Iberoamericana, 38(2):433–502, 2021.
  * [GH80] D. Geman and J. Horowitz.  Occupation Densities.  The Annals of Probability, 8(1):1 – 67, 1980.
  * [GH20] L. Galeati and F. Harang.  Regularization of multiplicative sdes through additive noise, 2020.  arXiv:2008.02335.
  * [GHM21] L. Galeati, F. Harang, and A. Mayorcas.  Distribution dependent sdes driven by additive fractional brownian motion, 2021.  arXiv:2105.14063.
  * [GHM22] L. Galeati, F. Harang, and A. Mayorcas.  Distribution dependent SDEs driven by additive continuous noise.  Electronic Journal of Probability, 27:1 – 38, 2022.
  * [GIP15] M. Gubinelli, P. Imkeller, and N. Perkowski.  Paracontrolled distributions and singular pdes.  Forum of Mathematics, Pi, 3e6:1–75, 2015.
  * [Gub04] M Gubinelli.  Controlling rough paths.  J. Func. Anal., 216(1):86 – 140, 2004.
  * [HL15a] M. Hairer and C. Labbé.  A simple construction of the continuum parabolic Anderson model on $\mathbf{R}^{2}$.  Electronic Communications in Probability, 20(none):1 – 11, 2015\.
  * [HL15b] M. Hairer and C. Labbé.  Multiplicative stochastic heat equations on the whole space.  Journal of the European Mathematical Society, 20, 04 2015.
  * [HL16] Y. Hu and K. Lê.  Nonlinear young integrals and differential systems in hölder media.  Transactions of the American Mathematical Society, 369(3):1935–2002, May 2016.
  * [HLN12] Y. Hu, F. Lu, and D. Nualart.  Feynman–Kac formula for the heat equation driven by fractional noise with Hurst parameter $H<1/2$.  The Annals of Probability, 40(3):1041 – 1068, 2012.
  * [HM20] F. Harang and A. Mayorcas.  Pathwise regularisation of singular interacting particle systems and their mean field limits, 2020.  arXiv:2010.15517.
  * [HNS11] Y. Hu, D. Nualart, and J. Song.  Feynman–Kac formula for heat equation driven by fractional white noise.  The Annals of Probability, 39(1):291 – 326, 2011.
  * [HP21] F. Harang and N. Perkowski.  C$\infty$\- regularization of ODEs perturbed by noise.  Stochastics and Dynamics, page 2140010, 2021.
  * [HV08] T. P. Hytönen and M. C. Veraar.  On besov regularity of brownian motions in infinite dimensions.  Probability and Mathematical Statistics, Vol. 28, Fasc. 1:143–162, 2008.
  * [KS21] F. Kühn and R. Schilling.  Convolution inequalities for besov and triebel–lizorkin spaces, and applications to convolution semigroups, 2021.
  * [Kö16] W. König.  The Parabolic Anderson Model.  Springer International Publishing, 2016.
  * [Lab19] C. Labbé.  The continuous anderson hamiltonian in $d\;\leqslant\;3$.  Journal of Functional Analysis, 277(9):3187–3235, 2019.
  * [Mar18] J. Martin.  Refinements of the Solution Theory for Singular SPDEs.  PhD thesis, Humboldt-Universität zu Berlin, 2018.
  * [MV05] O. Mocioalca and F. Viens.  Skorohod integration and stochastic calculus beyond the fractional brownian scale.  Journal of Functional Analysis, 222(2):385–434, 2005.
  * [SP12] R. Schilling and L. Partzsch.  Brownian Motion: An Introduction to Stochastic Processes.  De Gruyter, 2012.
  * [Ver11] M. Veraar.  Regularity of gaussian white noise on the d-dimensional torus.  Banach Center Publications, 95:385–398, 2011.
  * [vZ20] W. van Zuijlen.  Theory of function spaces.  Lecture notes, 2020.  http://www.wias-berlin.de/people/vanzuijlen/LN_theory_of_function_spaces.pdf.


# Hologram: Realtime Holographic Overlays via LiDAR Augmented Reconstruction

Ekansh Agrawal  UC Berkeley EECSUSA<EMAIL_ADDRESS>
(2024)

###### Abstract.

Guided by the hologram technology of the infamous Star Wars franchise, I
present an application that creates real-time holographic overlays using LiDAR
augmented 3D reconstruction. Prior attempts involve SLAM or NeRFs which either
require highly calibrated scenes, incur steep computation costs, or fail to
render dynamic scenes. I propose 3 high-fidelity reconstruction tools that can
run on a portable device, such as a iPhone 14 Pro, which can allow for metric
accurate facial reconstructions. My systems enable interactive and immersive
holographic experiences that can be used for a wide range of applications,
including augmented reality, telepresence, and entertainment.

Computer vision, LiDAR, SLAM, 3D reconstruction, Deep learning

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††conference:
UC Berkeley EECS; May ’24; Berkeley, CA††isbn: 978-1-4503-XXXX-X/24/04††ccs:
Computing methodologies Structured outputs††ccs: Computing methodologies
Neural networks††ccs: Computing methodologies Volumetric models

##  1\. Motivation

As a kid, an engineer, and a researcher, the world of Stars Wars has never
stopped to pique my curiousity. Whether it’s the fast-than-light intergalatic
space travel or the iconic laser sword ”lightsabers”, the Star Wars universe
has always been a source of inspiration for me. One of the most iconic
technologies in the Star Wars universe is the hologram as shown in. Soldiers,
politicians, and civilians alike all use portable devices to project 3D images
of people or objects in real-time to communicate in real-time. This technology
has been a staple in the Star Wars franchise and in this project, I aim to
bring this technology to life. My goal is to create a system that can project
real-time holographic overlays using LiDAR augmented 3D reconstruction. The
idea of a hologram is not just a cool visual effect, but a powerful tool that
can be used for a wide range of applications, including augmented reality,
telepresence, and entertainment.

While there have been many notable hardware advancements in 3D reconstructions
and holographic displays, there are still many challenges that need to be
addressed before they can be proposed as an everyday consumer electronic
(Lucente, 1994) (Lucente, 1995) (Peng et al., 2020) (Shi et al., 2021). For
the scope of this project, I focus solely on software as a vehicle of real-
time reconstructions. I also further narrow the scope by mostly focusing on
trying to reconstruct faces. Existing methods rely on highly calibrated
environments, incur steep computation costs, or fail to render dynamic scenes
(Zander et al., 2012). In this project, I propose three high-fidelity
reconstruction tools that can run on a portable device, such as an iPhone 14
Pro, which can allow for metric accurate facial reconstructions. These systems
enable interactive and immersive holographic experiences that can be used for
a wide range of applications, including augmented reality, telepresence, and
entertainment.

Figure 1. Hologram technology from the Star Wars universe

##  2\. Prior Works

###  2.1. Simultaneous Localization and Mapping

With the introduction of the Kinect Sensor, low-resolution depth and visual
sensing introduced novel approaches for real-time real-time dense 3D
reconstruction (Han et al., 2013). Initial research employs a volumetric
truncated signed distance function (TSDF) to fuse depth data into a global
model. KinectFusion achieves impressive results but is limited to relatively
small-scale environments due to memory constraints (Izadi et al., 2011).
ElasticFusion extends KinectFusion by introducing a more flexible and scalable
map representation using surfels (surface elements) (Whelan et al., 2015). It
can handle larger environments and loop closures, enabling robust and
consistent reconstructions. Real-time 3D reconstruction system that combines
depth and RGB data. It leverages the complementary strengths of geometric and
photometric information, resulting in high-quality reconstructions with
accurate geometry and detailed texture (Dai et al., 2017). Research has
allowed large-scale, real-time 3D reconstructions via multiple Kinect sensors
by efficiently allocating and updating only the occupied regions in the volume
(Nießner et al., 2013) (Scona et al., 2018) (Prisacariu et al., 2017).

While these methods have made significant progress in real-time 3D
reconstruction, they are limited to static environments and struggle to handle
dynamic scenes in high resolution. This makes live reconstruction of a high
detail object like a subject’s face temperamental. They rely on geometric
priors and are sensitive to noise and outliers in the input data. Furthermore,
most of these methods rely on the Kinect Sensor which is not as portable as
the project goal.

###  2.2. Depth Estimation

Depth estimation of a scene allows us to perform classical image
transformations to scale coordinates from the image plane to the world plane.
This is a crucial step in 3D reconstruction as it allows us to map the 2D
image to a 3D point cloud quite effortlessly given the frame depth and
calibration parameters. Traditional methods for depth estimation include
stereo matching, structure from motion, and depth from de-focus. Stereo
matching involves finding correspondences between two images taken from
different viewpoints and computing a disparity map of some kind (Scharstein
and Szeliski, 2003) (Birchfield and Tomasi, 1999). This techniques usually are
contingent on a highly calibrated environment in order to use epipolar
geometry to extrapolate between the 3D points and their projections onto the
2D images (Kytö et al., 2011) (Marr and Poggio, 1976). More recent advances in
stereo vision include the use of CNNs for disparity estimation (Zbontar and
LeCun, 2016) (Kendall et al., 2017). Significant progress has also been made
in the realm of monocular depth estimation via deep learning approaches that
learn to extract relevant features and depth cues from large-scale datasets of
RGB images and corresponding depth maps (Eigen et al., 2014) (Liu et al.,
2015). Advancements like encoder-decoder architectures (Laina et al., 2016)
(Hu et al., 2019), attention mechanisms and multi-task learning (Chen et al.,
2019) (Wang et al., 2020), transformers-based architectures (Ranftl et al.,
2021) (Chen et al., 2022), and even diffusion-based architecture (Saxena et
al., 2023) (Ke et al., 2023) have all attempted to reduce the disparity of a
neural based approach for depth estimation.

Structure from motion techniques aim to reconstruct 3D structures from a
sequence of 2D images captured from different viewpoints. Basic techniques
involve extrapolating an essential matrix between the images and then using
the epipolar geometry to triangulate the 3D points (Longuet-Higgins, 1981)
(Tomasi and Kanade, 1992) (Pollefeys et al., 2008). More recent advances in
structure from motion include the use of deep learning techniques to estimate
the camera poses and 3D points (Ummenhofer et al., 2017) (Zhou et al., 2017).
My major qualm with adopting these techniques is the need for a highly
calibrated environment. These techniques also rely heavily on using more than
one camera to capture the scene (Zhao et al., 2018). I introduce engineering
constraints later in this project which make it difficult to adopt this
approach.

3D reconstructions via LiDAR involve processing and interpreting these point
cloud data to create accurate 3D models or representations of objects, scenes,
or environments. Through time-of-flight calculations, LiDAR technology can
create efficient point cloud registration (Besl and McKay, 1992) (Yang et al.,
2016) which can be used for surface reconstruction (Kazhdan et al., 2006)
(Oztireli et al., 2009). Deep learning approaches (Huang et al., 2018) (Jiang
et al., 2022) are instrumental for large-scale scene reconstruction (Pan et
al., 2013) (Oehler et al., 2011). The main problem with consumer grade LiDAR
sensors are their lack of precision (Ma et al., 2022), and industrial grade
sensors can cost tens of thousands of dollars making their usage cost
prohibitive (Yang et al., 2018).

###  2.3. Neural Radiance Fields

The advent of NeRFs introduced an approach to synthesizing novel views
(Mildenhall et al., 2020), highlighting their abilities to capture and render
complex scenes with high fidelity (Müller et al., 2022) (Tancik et al., 2022),
including intricate geometric details, view-dependent effects (e.g.,
specularity, transparency), and challenging lighting conditions (Trevithick
and Yang, 2021) (Yu et al., 2021). This is achieved by the high-dimensional
and continuous nature of the learned representation, which can encode rich
scene properties without relying on explicit 3D geometry or texture mapping
(Mildenhall et al., 2020). NeRFs have been extended to handle dynamic scenes
by incorporating temporal information (Li et al., 2021) (Pumarola et al.,
2021), enabling the reconstruction of moving objects and scenes. Further
optimizations introduced high quality renders with sparse input views (Garbin
et al., 2021) (Liu et al., 2020) and close to real-time rendering speeds for
individual scenes (Hedman et al., 2021) (Rebain et al., 2022) (Sitzmann et
al., 2020) (Srinivasan et al., 2021). However, in order to achieve high-
fidelity facial reconstructions, NeRFs require a large number of input views
and high-resolution images, which can be computationally expensive and time-
consuming. This makes it difficult to achieve real-time performance on
portable devices, like the iPhone 14 Pro, since I would need to render a new
scene for every frame.

##  3\. Methodology

Figure 2. Attempt 1 utilizes monocular depth estimation alongside classical
image projections in order to transfer pixels from their image representation
into there voxel representation. Figure 3. Attempt 2 utilizes the LiDAR and
TrueDepth data streamed from the iPhone 14 Pro’s front facing sensors. This
data is then upscaled with a SRCNN model trained on facial depth maps. I fuse
this data and then use classical projections using the phone’s IMU sensor for
camera poses. Figure 4. Attempt 3

I propose three approaches to generating real-time 3D facial reconstructions
in this paper. While these individual techniques may not be novel in nature,
the hybrid approach along with the constraint of the real-time reconstruction
requires quite a bit of engineering to get them to work with high fidelity. I
self-imposed the following engineering constraints on my methodology to ensure
that the engineered solutions were novel and challenging to implement:

  1. (1)

Monocular estimation: Reconstruction cannot be performed with more than a
single set of sensors. (IE. only one camera, one LiDAR sensor, etc..)

  2. (2)

Portability: If the clone troopers carried around tiny portable devices, I
want to be able to do same. No backup RTX 4090s.

  3. (3)

High fidelity: The final rendered voxel grid cannot be insanely sparse. It
should be extremely clear what’s being rendered, and we should be able to tell
exactly who’s face is being reconstructed.

  4. (4)

Instantaneous: A viable solution must achieve at least 30 FPS during
reconstruction and must occur online.

###  3.1. Attempt 1: Monocular Depth Estimation

My initial approach, as shown in Figure 2, is quite simple in nature. Given an
image frame, I want to use our classical projection formulas in order to
transform image from the uv coordinate system to the world coordinate system.
Calibrated environments can utilize stereo imaging to leverage disparity maps
to compute depth over a calibrated scene. I can replace this depth approach
with a monocular-depth estimation model. I use Intel MiDaS as the core
backbone of my depth estimation pipeline, consisting of a ViT backbone pre-
trained off of ImageNet (Ranftl et al., 2020) (Birkl et al., 2023). A single
frame is passed into this model, and the output is a $H$ by $W$ matrix
consisting of the relative depth of each pixel from the input image frame as
$z_{c}$. I then extract the camera intrinsics from the webcam and apply
Equation 1 to extract the camera points. I use identify transformations and
apply Equation 2 to project the points into the world coordinate system.

(1) $\begin{bmatrix}x_{c}\\\ y_{c}\\\
z_{c}\end{bmatrix}=\begin{bmatrix}f_{x}&0&c_{x}\\\ 0&f_{y}&c_{y}\\\
0&0&1\end{bmatrix}\begin{bmatrix}u\\\ v\\\ 1\end{bmatrix}+\begin{bmatrix}0\\\
0\\\ z_{c}-1\end{bmatrix}$

(2) $\begin{bmatrix}x_{w}\\\ y_{w}\\\
z_{w}\end{bmatrix}=R\begin{bmatrix}x_{c}\\\ y_{c}\\\ z_{c}\end{bmatrix}+T$

As a means of comparison, I also used the Marigold diffusion model for depth
estimation which utilizes a fine-tuned Stable Diffusion model (Ke et al.,
2023). I used 10 denoising steps and preserved the image resolution during
model inferences.

###  3.2. Attempt 2: LiDAR + TrueDepth

Figure 5. Projected light coordinates for the TrueDepth and LiDAR sensors
shipped with the Apple iPhones.

The release of Apple’s iPhone 12 Pro introduced a consumer-grade LiDAR sensor
and TrueDepth infrared depth sensor. The combination of these sensors project
points onto the subject and record the time it takes for points to reflect
light back into the sensors (Luetzenburg et al., 2021). The LiDAR sensor works
well with coarse objects while the TrueDepth sensors works well with
differentiating distance between fine objects as shown in Figure 5.

For the second attempt, as shown in Figure 3, I replace the monocular depth
estimation stack with the LiDAR sensor stack, using an iPhone 14 Pro as my
test device. This required a rewrite in Swift in order to leverage the native
Apple SDKs for GPU and LiDAR programming. Due the low resolution of the LiDAR
sensor, a custom trained SRCNN model was employed to upscale the LiDAR depth
frame. This model was initially pre-trained on the Set5 super-resolution
dataset (Dong et al., 2015) and then fine-tuned on a custom dataset of 10,000
face depth images. This dataset was constructed by using the Generated Photos
Academic Dataset and feeding it through Marigold with 15 denoising steps for a
depth frame. These images were then down scaled by a factor of 4 and anti-
aliased for their lower resolution representations. The trained model was
deployed via CoreML in order to leverage the Apple Neural Engine on the iPhone
(Ahremark and Bazso, 2022). The results of SRCNN deployed to the LiDAR
collection stack is shown in Figure 6. Figure 6(b) shows the depth results
after a 4x upscale via basic bilinear interpolation. The occluded edges are
more dramatic and the overall texture around the edges is more noisy. Figure
6(c) shows the depth results after a forward-inference through the SRCNN
model, which shows a more consistent and higher fidelity depth rendering with
a 4x upscaler. These screenshots were collected via my Swift application.

After passing the LiDAR data through the model and fusing it with the
TrueDepth sensor data via weighted averaging, I used the classical
transformation equations from above in order to project the voxels into their
world coordinate representations. However, this time, instead of ignoring the
extrinsic transformations, I was able to use the IMU sensor on the iPhone to
extract the extract camera rotation and translation for the final
transformations. I used Metal shaders to render the reconstructed voxel map in
real-time via Apple’s GPU in my app.

(a) Raw image frame

(b) LiDAR + True Depth fused frame upscaled with bi-linear interpolation

(c) LiDAR + True Depth fused frame upscaled with SRCNN model

Figure 6. Lidar + TrueDepth data from iPhone 14 Pro generated from
reconstruction app programmed in Swift + Metal Figure 7. Reconstruction via
monocular depth estimation with different modle backbones. In this figure we
show MiDaS with low fidelity depth estimations and Marigold as high quality
depth estimation.

###  3.3. Attempt 3: Template Modeling

The final attempt, as shown in Figure 4, was to leverage a deep-learning
approach to render a facial reconstruction and apply texture + mesh
transformations directly through an end-to-end approach. The intuition behind
this approach was that a deep learning method would enable us to learn the
general shape of a face and apply internal transformations to render real-time
facial reconstructions given a webcam stream. This would potentially allow for
more dynamic and responsive interactions in virtual environments, as the
system could adjust and adapt to the user’s facial expressions and movements
in real time. The initial goal was to use INSTA for volumetric avatar
rendering given a single RGB frame, however the inference time was too slow
for real-time rendering (Zielonka et al., 2022a). I instead used MICA, which
tracks facial landmarks on a face, and then renders a 3D mesh that skews the
landmarks from a global mean (Zielonka et al., 2022b). I modified this model
slightly to use the fused and upscaled LiDAR + TrueDepth sensor data from the
prior attempt to aid the model with more accurate metric reconstructions. I
retrained MICA on the original dataset, which were a combination of images
from LYHM (Pears and Duncan, 2016), and FaceWarehouse (Cao et al., 2013), but
fused a depth dimension generated via Marigold with 10 de-noising steps.

For real-time rendering, I pass the RGB image frame along with the pixel-wise
depth frame through my modified version of MICA. This inference produces a
mesh that I rasterize to receive its voxelized representation. Using the depth
frame as a heuristic, I project pixels in the background into the rendered
scene. Once again using the iPhone’s IMU sensor to extrapolate camera rotation
and translation to apply final transformation to the rendered voxel grid. As
noted in the Results sections, this approach wasn’t fully deployed on the
iPhone due to some technical issues. Therefore, I was unable to fully render
the scene which this approach and was limited to reconstructing the face.

##  4\. Results

###  4.1. Attempt 1: Monocular Depth Estimation

With the monocular depth estimation approach, reconstruction averaged around
~2 FPS with MiDAS and ~0.25 FPS when using Marigold on 10 de-noising steps.
The other backbones available on Torch Hub resulted in the same fidelity
reconstruction but larger model size, so I defaulted to using the default ViT
backbone for my tests. These models were deployed to an iPhone using the
PyTorch to CoreML pipeline for quantization and scheduling. An open-source
CoreML diffusion model loader application was monkey patched to run inference
on these deployed models on an iPhone 14 Pro. For Marigold, decreasing the
number of steps introduced quite a few artifacts, and increasing the steps
after 15 denoising steps didn’t seem to affect the reconstruction quite a lot.
Figure 7 shows these reconstructions, with the 2nd and 4th row showcasing
results of MiDaS reconstructions, and the 3rd and 5th row corresponding to the
results of Marigold reconstruction. Figure 8 shows the result of
reconstruction of a pair of legs and a coffee table with the MiDAS model.
Unfortunately, offloading data from the open-source app was incredibly buggy
therefore the quality of the image is a little low.

Figure 8. Live reconstruction of a leg and coffee table with MiDAS

From a visual perspective, the main shortcoming I saw with this approach was
that the inferred distance from the image plane to the facial features was
largely inconsistent. For example, the subject in the 2nd and 4th column of
Figure 7 are roughly equidistant from the camera. Yet the corresponding
reconstructions of these faces are incredibly varied. The 2nd subject has a
significantly flatter reconstruction around the cheeks that’s not existent in
the reconstruction of the 4th subject. Simultaneously we can see that both
reconstruction models seemed to have estimated the depth incorrectly for
subject in columns 9 and 10. Specifically, the depth model predicted the front
cowlicks to extrude outwards which results in a mohawk. From a technical
perspective, this makes sense because both MiDaS and Marigold were not trained
on a metric accurate datasets. Due to the diversity in the source of the
ground truth, all depth values were normalized between 0 to 1, effectively
prompting the models to predict relative depth instead of metric accurate
depth.

The overall fidelity of reconstructions is not the greatest, and sometimes
it’s difficult to ascertain whose face is being reconstructed. Increasing the
quality of the reconstruction models removes noised by a bit, but sacrifices
FPS. We need a way to collect metric accurate sensor data before using our
transformation pipeline to project the data. Due to the scope of the project,
I wasn’t able to allocate enough time + resources to further finetune and
quantize the depth estimation models which would have likely helped with
increasing the FPS.

###  4.2. Attempt 2: LiDAR + TrueDepth

Figure 9. Lidar + TrueDepth reconstruction from iPhone 14 Pro

Reading the fused LiDAR + TrueDepth data, passing it through the custom
upscaled model, and performing the transformations averaged about ~50 frames
per second. The main reason behind the throughput being tripled with this
approach, was that the majority of the data collection was being done with
live sensors instead of any of kind of large-scale neural network inference. I
also leveraged the Metal shaders which offloaded a bulk of the parallel
computation to the GPU. The unified memory architecture allowed to me perform
zero-copy computations which dramatically reduced non-compute latency. The
upscaler ML model was run via CoreML so most of the computation was offloaded
to the Apple Neural Engine; a forward inference was almost instantaneous.

Figure 9 shows screenshots of the live reconstruction through the custom Swift
app. In general, the fidelity of the reconstructions is insanely high. It is
quite clear that my face is being reconstructed in Figure 9. From the naked
eye, the proportions for the reconstructions also appear to be metric
accurate. Even though the upscaler model was finetuned with mostly facial
images, it still performs well in reconstructing other subjects like a couch
or a cat.

The main shortcoming of this approach, however, was the inability to render
any occluded pixels. Naturally, the easiest technical solution to this would
be to increase the number of LiDAR and camera sensors to fuse data from
multiple viewpoints. However, due to the self-imposed engineering constraint
of using one device, simply increasing the number of imaging sensors feels a
little bit like cheating.

###  4.3. Attempt 3: Template Modeling

Figure 10. Fused depth + facial landmarks predictions generated by MICA to
render 3D mesh of reconstructed face

Figure 11. INSTA end-to-end reconstruction with mesh and texture generated
given an image of a face (I can already hear Aleyosha’s voice in my head
saying ”texture is everything” but regardless the reconstructions look
incredibly realistic)

As an FYI, due to some issues while deploying the CoreML model, this attempt
wasn’t incredibly reliable as even loading the model would completely crash
the iPhone. I had to run these tests on my Nvidia Jetson (8 CPU cores, 32 GB
memory, Ampere Nvidia GPU) (Karumbunathan, 2022) in order to create
visualizations for this report, but was still unable to generate GIFs.

I used saved frames of depth and image data to create the facial
reconstructions. Figure 10 visualizes reconstruction via MICA which shows how
the facial landmark detection and LiDAR depth estimation come together to
render the mesh structure. The figure shows occluded edges through red dots
which is important to estimating the face shape for the occluded sections of
the images. The corresponding occluded structures on the images are clearly
shown in the reconstructions in Figure 11.

The fidelity of these reconstructions is pretty high as shown in Figure 11.
The facial features are rendered proportionately; although, I noticed that the
MICA more or less ignored rendering the hair. This could be easily solved by
fine-tuning MICA with high-fidelity LiDAR scans to measure the depth of
individual hair strands. The back of the head is no longer hollow since the
model use a texture model as a baseline to extrapolate off of.

I believe that the technical failure of this attempt was more indicative of
the limitations of the hardware and software stack rather than the underlying
approach itself. Maybe with a bit more of software maturity and/or retooling
of the model architecture, I would be able to get more promising results with
this hybrid approach over the pure LiDAR-based reconstruction.

##  5\. Further Study

Upon achieving decent reconstructions from Attempt 2 with the LiDAR +
TrueDepth sensor fusion, it would be interesting to explore basic hardware
implementations of projecting holograms into the real-world. Existing research
with novel approaches exist (Ritter et al., 1999) (Seo et al., 2011)
(Shimobaba et al., 2000) (Shimobaba et al., 2002) (Author7 and Author9, 2019),
however the ultimate goal should be use a novel approach for multi-dimensional
light projection.

###### Acknowledgements.

Special thanks to Aleyosha, Lisa, and Suzie for helping provide the groundwork
for this project.

## References

  * (1)
  * Ahremark and Bazso (2022) Jens Ahremark and Simon Bazso. 2022.  Benchmarking a machine learning model in the transformation from PyTorch to CoreML.
  * Author7 and Author9 (2019) Author8 Author7 and Author9. 2019.  Real-time dynamic holographic display using phase-only spatial light modulator. In _Proceedings of the SPIE Photonics West_ , Vol. 10944. 109440D.
  * Besl and McKay (1992) Paul J Besl and Neil D McKay. 1992.  A method for registration of 3-D shapes. In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , Vol. 14. IEEE, 239–256.
  * Birchfield and Tomasi (1999) Stan Birchfield and Carlo Tomasi. 1999.  Depth discontinuities by pixel-to-pixel stereo.  _International Journal of Computer Vision_ 35 (1999), 269–293.
  * Birkl et al. (2023) Reiner Birkl, Diana Wofk, and Matthias Müller. 2023.  Midas v3. 1–a model zoo for robust monocular relative depth estimation.  _arXiv preprint arXiv:2307.14460_ (2023).
  * Cao et al. (2013) Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou. 2013.  Facewarehouse: A 3d facial expression database for visual computing.  _IEEE Transactions on Visualization and Computer Graphics_ 20, 3 (2013), 413–425.
  * Chen et al. (2019) Xiaotian Chen, Xuejin Chen, and Zheng-Jun Zha. 2019.  Structure-Aware Residual Pyramid Depth Estimation. In _Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence_.
  * Chen et al. (2022) Xiaotian Chen, Junchen Zhou, Xuejin Chen, and Zheng-Jun Zha. 2022.  MonoDepthFormers: Transformers for Computationally Efficient Monocular Depth Estimation.  _arXiv preprint arXiv:2209.07688_ (2022).
  * Dai et al. (2017) Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Christian Theobalt. 2017.  Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration.  _ACM Transactions on Graphics (ToG)_ 36, 4 (2017), 1.
  * Dong et al. (2015) Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2015.  Image super-resolution using deep convolutional networks.  _IEEE transactions on pattern analysis and machine intelligence_ 38, 2 (2015), 295–307.
  * Eigen et al. (2014) David Eigen, Christian Puhrsch, and Rob Fergus. 2014.  Depth map prediction from a single image using a multi-scale deep network. In _Advances in Neural Information Processing Systems_. 2366–2374.
  * Garbin et al. (2021) Stephan J Garbin, Davide Schoenmakers, Matteo Leonetti, Rem Hofstede, Thomas Weyrich, and Hans-Peter Seidel. 2021.  Efficient neural radiance fields. In _arXiv preprint arXiv:2111.07931_.
  * Han et al. (2013) Jungong Han, Ling Shao, Dong Xu, and Jamie Shotton. 2013.  Enhanced computer vision with microsoft kinect sensor: A review.  _IEEE transactions on cybernetics_ 43, 5 (2013), 1318–1334.
  * Hedman et al. (2021) Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. 2021.  SneRG: Synthesizing normals to render geometry with learned differentiable intersection. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 8506–8516.
  * Hu et al. (2019) Junjie Hu, Muhammed Ozkan, Yu Seiken, Andrew Zisserman, and Marc Pollefeys. 2019.  Revisiting single image depth estimation: Toward higher resolution maps with semi-supervised learning. In _2019 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 1949–1958.
  * Huang et al. (2018) Qiangui Huang, Weiyue Wang, and Ulrich Neumann. 2018.  Recurrent slice networks for 3d segmentation of point clouds. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 2626–2635.
  * Izadi et al. (2011) Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, et al. 2011\.  Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In _Proceedings of the 24th annual ACM symposium on User interface software and technology_. 559–568.
  * Jiang et al. (2022) Ren Jiang, Guannan Luo, Wenyuan Liu, Ngan Kwok, and Qizhi Zhou. 2022.  LiDAR Point Cloud Reconstruction and Completion via RGB-Guided Voxel Transformers. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 8766–8775.
  * Karumbunathan (2022) Leela S Karumbunathan. 2022.  NVIDIA Jetson AGX Orin Series.
  * Kazhdan et al. (2006) Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. 2006.  Poisson surface reconstruction. In _Proceedings of the Fourth Eurographics Symposium on Geometry Processing_ , Vol. 7.
  * Ke et al. (2023) Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. 2023.  Repurposing diffusion-based image generators for monocular depth estimation.  _arXiv preprint arXiv:2312.02145_ (2023).
  * Kendall et al. (2017) Alex Kendall, David B Grimes, and Roberto Cipolla. 2017.  GC-Net: Geometric correspondence network for disparity estimation.  _The British Machine Vision Conference (BMVC)_ (2017).
  * Kytö et al. (2011) Mikko Kytö, Mikko Nuutinen, and Pirkko Oittinen. 2011.  Method for measuring stereo camera depth accuracy based on stereoscopic vision. In _Three-dimensional imaging, interaction, and measurement_ , Vol. 7864. SPIE, 168–176.
  * Laina et al. (2016) Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. 2016.  Deeper depth prediction with fully convolutional residual networks. In _2016 Fourth International Conference on 3D Vision (3DV)_. IEEE, 239–248.
  * Li et al. (2021) Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. 2021.  Neural 3D video synthesis from a single semantic image. In _arXiv preprint arXiv:2103.07953_.
  * Liu et al. (2015) Fayao Liu, Chunhua Shen, Guosheng Lin, and Ian Reid. 2015.  Learning depth from single monocular images using deep convolutional neural fields. In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , Vol. 38. IEEE, 2024–2039.
  * Liu et al. (2020) Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. 2020.  Neural sparse voxel fields. In _Advances in Neural Information Processing Systems_ , Vol. 33. 15651–15663.
  * Longuet-Higgins (1981) Hugh Christopher Longuet-Higgins. 1981.  A computer algorithm for reconstructing a scene from two projections.  _Nature_ 293, 5828 (1981), 133–135.
  * Lucente (1994) Michael Lucente. 1994.  Holographic optical elements for optical data processing.  _Optical Engineering_ 33, 9 (1994), 2768–2779.
  * Lucente (1995) Michael Lucente. 1995.  Optimization of hologram computation for real-time photorealistic augmented reality. In _Practical Holography XI_ , Vol. 2652. International Society for Optics and Photonics, 309–314.
  * Luetzenburg et al. (2021) Gregor Luetzenburg, Aart Kroon, and Anders A Bjørk. 2021.  Evaluation of the Apple iPhone 12 Pro LiDAR for an application in geosciences.  _Scientific reports_ 11, 1 (2021), 1–9.
  * Ma et al. (2022) Siheng Ma, Yonghua Bao, Xuanchen Gao, and Dong Cao. 2022.  LiDAR-Guided Autonomous Vehicle Trajectory Prediction. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 1375–1384.
  * Marr and Poggio (1976) David Marr and Tomaso Poggio. 1976.  Cooperative computation of stereo disparity.  _Science_ 194, 4262 (1976), 283–287.
  * Mildenhall et al. (2020) Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2020.  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In _European Conference on Computer Vision (ECCV)_. Springer.
  * Müller et al. (2022) Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022.  Instant neural graphics primitives with a multiresolution hash encoding. In _ACM Transactions on Graphics (TOG)_ , Vol. 41. ACM, 1–15.
  * Nießner et al. (2013) Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Marc Stamminger. 2013.  Real-time 3D reconstruction at scale using voxel hashing.  _ACM Transactions on Graphics (ToG)_ 32, 6 (2013), 1–11.
  * Oehler et al. (2011) Stefan Oehler, Martin Jaenicke, Ingo Wald, Wolfgang Worner, and Alfredo Delgado-Guzarahah. 2011.  Efficient Parallel Hierarchical Radiosity on a Cluster. In _Proceedings of the ACM SIGGRAPH Symposium on High Performance Graphics_. 143–152.
  * Oztireli et al. (2009) Cengiz Oztireli, Gael Guennebaud, and Markus Gross. 2009.  A computational model of fresco for 3D reconstruction. In _ACM Transactions on Graphics (TOG)_ , Vol. 28. ACM, 1–9.
  * Pan et al. (2013) Quannan Pan, Gerhard Reitmayr, and Tom Drummond. 2013.  Probuilding: A robust and memory-efficient voxel reconstruction engine for 3d-scanning systems. In _Computer Graphics Forum_ , Vol. 32. Wiley Online Library, 137–147.
  * Pears and Duncan (2016) Nick Pears and Christian Duncan. 2016.  Automatic 3D modelling of craniofacial form.  arXiv:1601.05593 [cs.CV]
  * Peng et al. (2020) Yifan Peng, Seung-Hwan Choi, Andrzej Czyzewski, and Aydogan Ozcan. 2020.  Deep learning for computer-generated holography.  _Advanced Photonics_ 2, 4 (2020), 046005.
  * Pollefeys et al. (2008) Marc Pollefeys, David Nistér, Jan-Michael Frahm, Afshin Akbarzadeh, Philippos Mordohai, Brian Clipp, Christopher Engels, David Gallup, Suk Jin Kim, Paul Merrell, et al. 2008\.  Detailed real-time urban 3D reconstruction from video.  _International Journal of Computer Vision_ 78, 2 (2008), 142–167.
  * Prisacariu et al. (2017) Victor Adrian Prisacariu, Olaf Kähler, Stuart Golodetz, Michael Sapienza, Tommaso Cavallari, Philip HS Torr, and David W Murray. 2017.  Infinitam v3: A framework for large-scale 3d reconstruction with loop closure.  _arXiv preprint arXiv:1708.00783_ (2017).
  * Pumarola et al. (2021) Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. 2021.  D-nerf: Neural radiance fields for dynamic scenes. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10318–10327.
  * Ranftl et al. (2021) René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. 2021.  Vision transformers for dense prediction. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 12179–12193.
  * Ranftl et al. (2020) René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. 2020.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  _IEEE transactions on pattern analysis and machine intelligence_ 44, 3 (2020), 1623–1637.
  * Rebain et al. (2022) Denis Rebain, Wei Jiang, Soroosh Yazdanpanah, Oussama Ammar, Mathieu Johannink, Baohua Sun, and Bernt Schiele. 2022.  DERF: Deformable Neural Radiance Fields.  _arXiv preprint arXiv:2208.00885_ (2022).
  * Ritter et al. (1999) Alf Ritter, Joachim Böttger, Oliver Deussen, Matthias König, and Thomas Strothotte. 1999.  Hardware-based rendering of full-parallax synthetic holograms.  _Applied Optics_ 38, 8 (1999), 1364–1369.
  * Saxena et al. (2023) Saurabh Saxena, Abhishek Kar, Mohammad Norouzi, and David J Fleet. 2023.  Monocular depth estimation using diffusion models.  _arXiv preprint arXiv:2302.14816_ (2023).
  * Scharstein and Szeliski (2003) Daniel Scharstein and Richard Szeliski. 2003.  High-accuracy stereo depth maps using structured light. In _2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings._ , Vol. 1. IEEE, I–I.
  * Scona et al. (2018) Raluca Scona, Mariano Jaimez, Yvan R Petillot, Maurice Fallon, and Daniel Cremers. 2018.  Staticfusion: Background reconstruction for dense rgb-d slam in dynamic environments. In _2018 IEEE international conference on robotics and automation (ICRA)_. IEEE, 3849–3856.
  * Seo et al. (2011) Young-Ho Seo, Hyun-Jun Choi, Ji-Sang Yoo, and Dong-Wook Kim. 2011.  Cell-based hardware architecture for full-parallel generation algorithm of digital holograms.  _Optics Express_ 19, 9 (2011), 8750–8761.
  * Shi et al. (2021) Liming Shi, Songjiang Ye, Chao Zuo, Ravi S Jonnal, and Laura Waller. 2021.  Towards real-time photorealistic 3D holography with deep generative networks.  _Nature_ 591, 7849 (2021), 258–262.
  * Shimobaba et al. (2002) Tomoyoshi Shimobaba, Sinsuke Hishinuma, and Tomoyoshi Ito. 2002.  Special-purpose computer for holography HORN-4 with recurrence algorithm.  _Computer physics communications_ 148, 2 (2002), 160–170.
  * Shimobaba et al. (2000) Tomoyoshi Shimobaba, Nobuyuki Masuda, Takashige Sugie, Satoru Hosono, Shinobu Tsukui, and Tomoyoshi Ito. 2000.  Special-purpose computer for holography HORN-3 with PLD technology.  _Computer physics communications_ 130, 1-2 (2000), 75–82.
  * Sitzmann et al. (2020) Vincent Sitzmann, Julien NP Martel, Alexander W Bergman, David B Lindell, and Gordon Wetzstein. 2020.  Implicit neural representations with periodic activation functions. In _Advances in Neural Information Processing Systems_ , Vol. 33. 7145–7156.
  * Srinivasan et al. (2021) Pratul P Srinivasan, Ben Mildenhall, Matthew Tancik, Jonathan T Barron, Richard Tucker, and Noah Snavely. 2021.  Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 7495–7504.
  * Tancik et al. (2022) Matthew Tancik, Armen Sadat, Ben Hershey, Charlie Donow, Ben Mildenhall, Ben Fridovich-Keil, and Jonathan T Barron. 2022.  Barf: Bundle-adjusting neural radiance fields.  _arXiv preprint arXiv:2205.11975_ (2022).
  * Tomasi and Kanade (1992) Carlo Tomasi and Takeo Kanade. 1992.  Shape and motion from image streams under orthography: a factorization method.  _International Journal of Computer Vision_ 9, 2 (1992), 137–154.
  * Trevithick and Yang (2021) Alex Trevithick and Bo Yang. 2021.  GRF: Learning a general radiance field for 3D scene representation and rendering. In _arXiv preprint arXiv:2010.04595_.
  * Ummenhofer et al. (2017) Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Niloy Mujica-Parodi, Tammy Berman, Andrew Duryea, Laurent Kneip, Laura Leal-Taixé, Alina Kirillov, Qiuren Tang, et al. 2017\.  DeMoN: Depth and motion network for learning monocular stereo. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 5038–5047.
  * Wang et al. (2020) Zhongqin Wang, Yatong Zhou, Haiyan Zhai, Zhang Lin, Leyuan Wu, Haihan Zhang, Jian Cheng, You Zhou, and Yujun Wang. 2020.  DFNet: Exploiting depth-wise separable convolutions for monocular depth estimation. In _IEEE Transactions on Image Processing_ , Vol. 30. IEEE, 4886–4898.
  * Whelan et al. (2015) Thomas Whelan, Stefan Leutenegger, Renato F Salas-Moreno, Ben Glocker, and Andrew J Davison. 2015.  ElasticFusion: Dense SLAM without a pose graph.. In _Robotics: science and systems_ , Vol. 11. Rome, Italy, 3.
  * Yang et al. (2018) Fan Yang, Guoxuan Li, Xin Zhou, Yuli Wong, and Yifan Zhou. 2018.  Towards complete 3d scene reconstruction from egocentric RGB-D semantic mapping. In _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 3556–3563.
  * Yang et al. (2016) Jiaolong Yang, Hongdong Li, Dylan Campbell, and Yunde Jia. 2016.  Go-ICP: A globally optimal solution to 3D ICP point-set registration. In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , Vol. 38. IEEE, 2241–2254.
  * Yu et al. (2021) Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. 2021.  Pixelnerf: Neural radiance fields from one or few images. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 4578–4587.
  * Zander et al. (2012) Stefan Zander, Chris Chiu, and Gerhard Sageder. 2012.  A computational model for the integration of linked data in mobile augmented reality applications. In _Proceedings of the 8th International Conference on Semantic Systems_. 133–140.
  * Zbontar and LeCun (2016) Jure Zbontar and Yann LeCun. 2016.  Stereo matching by training a convolutional neural network to compare image patches.  _The journal of machine learning research_ 17, 1 (2016), 2287–2318.
  * Zhao et al. (2018) Pengcheng Zhao, Qingwu Hu, Shaohua Wang, Mingyao Ai, and Qingzhou Mao. 2018.  Panoramic image and three-axis laser scanner integrated approach for indoor 3D mapping.  _Remote Sensing_ 10, 8 (2018), 1269.
  * Zhou et al. (2017) Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. 2017.  Unsupervised learning of depth and ego-motion from video. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 1851–1858.
  * Zielonka et al. (2022a) Wojciech Zielonka, Timo Bolkart, and Justus Thies. 2022a.  Instant Volumetric Head Avatars.  _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ (2022), 4574–4584.  https://api.semanticscholar.org/CorpusID:253761096
  * Zielonka et al. (2022b) Wojciech Zielonka, Timo Bolkart, and Justus Thies. 2022b.  Towards Metrical Reconstruction of Human Faces. In _European Conference on Computer Vision_.  https://api.semanticscholar.org/CorpusID:248177832