# Solving Navier-Stokes Equation using Helmholtz Decomposition

Roy Burson

###### Abstract.

This work uses Helmholtz decomposition to solve Navier-Stokes equation in any
smooth bounded region of $V\subset\mathbb{R}^{3}$. Specifically, we show that
there is a smooth solution whenever the vorticity vanishes at time $t=0$.
Otherwise there is no solution. To compensate for the failure of these
equations we introduce Einstein energy formula to relate the mass of a fluid
particle to its velocity. In this circumstance we derive a new equation for
Newtonian fluids but do not attempt to derive a solution as it was not the
main objective of our work.

##  1\. Introduction

The Navier-stokes equation was developed from Newtons laws and derived in
attempt to describe the velocity and pressure of a fluid in time and space.
The study of these equations date back to the early 1800’s when they where
first developed by G.G. Stokes in England, and M. Navier in France.
Unfortunately, no one has been able to prove that a smooth solution to the
equation exist and even worse there are hardly any partial results available.
In two dimensions Ladyzhenskaya [6] shows there is a unique solution, but no
one has been able to solve them in higher dimensions. In 1934 Leray [5] showed
that weak-solution exist satisfying the equations in mean value but not point
wise. The work of Elgindi’s [2] and Lorenzo Brandolese, and Fernando Cortez
[7] suggest the Euler Equations might blow up, which are similar to Navier-
stokes equation. In recent research Tao [8] used an averaging type method to
replace the non-linear term suggested that the equations might blow up but
lacks justification for his approach. It almost seems impossible to show that
any type of singularities to his bi linear equations coincide with the blowup
of the Navier-Stokes equation by a simple energy identity. This is because
even a very small change in the derivative of a function can have a big impact
on its behavior. All these works give some intuition or at least suggest that
a solution to these type of equations might not exist. In other words its
possible that people are searching for an answer that does not exist. We show
that if the initial velocity is curl free then for any smooth bounded sub
region $V\subset\mathbb{R}^{3}$ there is a solution that satisfied the Navier-
stokes equation for all time in $V$. Even further, if the initial vorticity is
non-zero then there is no possible solution so there is no need to search for
one. When the vorticity vanishes the solution is smooth and has a unique
representation. The unique solution shows the non-linear term is present in
the pressure and does effect the velocity. In fact, the velocity is the
prodcuct the scalar function $t$ and the gradient of laplace’s solution to the
heat equation. Thus showing the velocity diffuses with respect to space and is
linear in time and does not blow up.

##  2\. Problem Description and Procedure

This work focuses on the Navier-Stokes equation in an arbitrary smooth bounded
sub region $V$ of $\mathbb{R}^{3}$ subjected to curl free external forces:

(1) $\dfrac{\partial}{\partial
t}\textbf{u}+\left(\textbf{u}\cdot\nabla\right)\textbf{u}-\mu\bigtriangleup\textbf{u}=-\dfrac{1}{\rho}\nabla
p+\textbf{f},\indent\nabla\times\textbf{f}=0$

(2) $\nabla\cdot\textbf{u}=0$

(3) $\textbf{u}(x,0)=\textbf{u}^{0}(x).$

Here $\rho>0$ is a scalar which represents the fluids density, $\mu>0$ is a
scalar which denoted the fluids kinetic viscosity, and
$\textbf{f}(x,t)\in\mathbb{R}^{3}\times[0,\infty)$ are the external applied
forces (e.g. gravity) which are known for all time $t\geq 0$ and assumed to be
at least twice differentiable. The term
$\left(\textbf{u}\cdot\nabla\right)\textbf{u}-\mu\bigtriangleup\textbf{u}-\textbf{f}$
will often be denoted as $\text{NL}(\textbf{u})$ for convention to mean the
”non-linear term” and $\nabla\times\textbf{u}$ will be called the vorticity or
the curl of u. These equations are to be solved for a the velocity field
$\textbf{u}(x,t)$ and pressure field $p(x,t)$ as time progresses in all space
and time. For physical reasonable solutions the initial condition and external
forces are restricted to be smooth (infinitly differentiable) and decay

$|\textbf{u}^{0}|_{L^{2}_{x}(\mathbb{R})}\leq\infty,\indent|\textbf{f}|_{L^{\infty},T]\times
L^{2}_{x}(\mathbb{R})}<\infty$

so that $\textbf{u}(x,t)$ does not grow large as $|x|\rightarrow\infty$.
According to Fefferman’s [4] official problem description we say a solution u
and $p$ is reasonable if

(4) $p,\textbf{u}\in
C^{\infty}(\mathbb{R}^{3}\times[0,\infty))\indent\text{and}\indent\int_{\mathbb{R}^{3}}{|\textbf{u}|}<\epsilon,\indent\forall
t>0.$

That is u and $p$ are infinitly differentiable and u has finite energy. To
solve these equations our proof depends on a contradiction. When the vorticity
is present, we show the assumption of a smooth solution leads to something
false. We start by taking advantage of Helmholtz decomposition to rewrite the
non-linear term using its irrotational and solenoid part. The result of this
work depends entirely on this decomposition and the vorticity equation. Hence,
the proof depends entirely on the validity of this decomposition. We need to
analyze the solution in a finite domain since we do not know the general end
behavior of the non-linear term. This finite evaluation is valid because if a
solution to the problem exist for all space and time then the solution exist
in any finite domain (regardless if the domain is smooth or not). We just need
the region to be smooth the apply Helmholtz decomposition. Using this
decomposition we compute the divergence to show the pressure only depends on
the irrotational part of the non linear term and the solution to Laplaces heat
equation allowing us to eliminate the pressure completely and show the
velocity only depends on the solenoidal part of the non-linear component. This
decomposition becomes useful in the since that it allows us to eliminate the
pressure from the equations (or at least show how the velocity is independent
of pressure). Afterwards, we then use the vorticity formula to relate the
expression found to the curl of the velocity field. Coincidentally, the
formula we derive is strong enough to suggest a breakdown when the curl does
not vanish and it also gives us a physical representation when the curl does
vanish. When the initial vorticity vanishes our solution shows the non linear
component affects the pressure and not necessarily the velocity. Meaning the
pressure is not linear. The same procedure can be used to solve many other
similar equations including the Euler equations.

##  3\. Analytic Solution

In this section we present the mathematical derivations necessary to solve the
Navier-Stokes equation. We want to use the second form of Helmholtz theorem
and write the solution over all of $\mathbb{R}^{3}$ but in order to do this we
need a decay assumption on the non-linear term. Hence, this is why we consider
a finite region. When we consider a sub region of $\mathbb{R}^{3}$ we can
apply Helmhotz theorem and decompose our non-linear vector field. We could in
a sense assume that the non-linear term decays and then show the solution we
find actually decays (when the vorticity vanishes) showing that our assumption
was not actually a bad assumption in the first place. To avoid this
technicality we will solve the problem in a smooth sub region instead. Later
on we will show this is actually true.

######  Theorem 1.

let $V\subset\mathbb{R}^{3}$ be any smooth region that is at least twice
differential and consider the surface $S$ that enclosed $V$. If
$\nabla\times\textbf{u}(x,0)=0$ then any solution must take the form

$\textbf{u}(x,t)=-t~{}\nabla\sigma,\indent t>0,$

$p(x,t)=\rho\biggl{[}\sigma-\dfrac{1}{4\pi}\int_{V}{\dfrac{\nabla\cdot\textbf{g}(y,t)}{|x-y|}}dy+\dfrac{1}{4\pi}\oint_{S}\textbf{n}\cdot\dfrac{\textbf{g}(y,t)}{|x-y|}\,ds\biggr{]}$

with $\sigma$ the solution to Laplace’s equation

$\nabla^{2}\sigma=0$

and where g is defined explicitly by

$\textbf{g}(x,t)=t^{2}(\nabla\sigma\cdot\nabla)\nabla\sigma+t\mu\bigtriangleup\nabla\sigma-\textbf{f},$

and if $\nabla\times\textbf{u}(x,0)\neq 0$ then there is no solution to
Eq.(1)-Eq.(3).

###### Proof.

We start by temporarily assuming that there exist a smooth solution u and $p$
to Eq.(1)-Eq.(4) for all time and space
$(x,t)\in\mathbb{R}^{3}\times[0,\infty)$. Next let $V\subset\mathbb{R}^{3}$ be
any smooth region that is at least twice differential and consider the surface
$S$ that enclosed $V$. Since a solution exist for all space and time then a
solution must exist in
$V\times(0,\infty)\subset\mathbb{R}^{3}\times(0.\infty)$. We note that from
Eq.(4) a reasonable solution u is assumed to be infinitely differentiable so
then $\mu\bigtriangleup\textbf{u}$ and $\nabla u_{i}$ must also be infinity
differentiable for each $1\leq i\leq 3$. This means the dot product
$\textbf{u}\cdot\nabla u_{i}$ must also be infinitely differentiable for each
$1\leq i\leq 3$ and so the non-linear vector field
$\left(\textbf{u}\cdot\nabla\right)\textbf{u}-\mu\bigtriangleup\textbf{u}-\textbf{f}$
must be at least twice differentiable. Therefore, in $V$ we may apply
Helmholtz’s decomposition to decompose the non linear vector field
$\text{NL}(\textbf{u})$ into the sum of a irrotational (curl-free) vector
field and a solenoidal (divergence-free) vector field (not necessarily unique)
within $V$;

(5)
$NL(\textbf{u}):=\left(\textbf{u}\cdot\nabla\right)\textbf{u}-\mu\bigtriangleup\textbf{u}-\textbf{f}=\nabla\Phi+\nabla\times\Gamma$

where

$\Phi(x,t)=\dfrac{1}{4\pi}\int_{V}{\dfrac{\nabla\cdot
NL(\textbf{u})\circ(y)}{|x-y|}}dy+\dfrac{1}{4\pi}\oint_{S}\textbf{n}\cdot\dfrac{NL(\textbf{u})\circ(y)}{|x-y|}\,ds$

$\Gamma(x,t)=\dfrac{1}{4\pi}\int_{V}{\dfrac{\nabla\times
NL(\textbf{u})\circ(y)}{|x-y|}}dy-\dfrac{1}{4\pi}\oint_{S}\textbf{n}\times\dfrac{NL(\textbf{u})\circ(y)}{|x-y|}\,ds$

with $(x,t)\in V\times(0,\infty)$ and n is the normal vector to the surface
$S$. Combining Eq.(5) and Eq.(1) one can write the velocity u as a function of
both $\Phi$ and $\Gamma$

(6) $u_{i}=-\int_{0}^{t}{\biggl{[}\dfrac{1}{\rho}(\nabla
p)_{i}+(\nabla\Phi)_{i}+(\nabla\times\Gamma)_{i}\biggr{]}}~{}dt,\indent
i\in\\{1,2,3\\}.$

Computing the divergence of Eq.(6) and comparing the result to Eq.(2) yields

(7)
$\nabla\cdot\textbf{u}=\int_{0}^{t}{\nabla\cdot\biggl{(}\nabla(\dfrac{1}{\rho}p+\Phi)\biggr{)}}~{}dt=0.$

Now by computing the partial derivative of both sides of Eq.(7) with respect
to time one then finds

(8) $\nabla\cdot\biggl{(}\nabla(\dfrac{1}{\rho}~{}p+\Phi)\biggr{)}=0.$

Under the transformation

(9) $\sigma=\dfrac{1}{\rho}p+\Phi$

Eq.(8) writes

$\nabla^{2}\sigma=0$

with boundary conditions on $S$ determined at initial time by the formula

$\begin{split}\sigma(x,0)&=\dfrac{1}{\rho}p(x,0)+\Phi(x,0).\\\ \end{split}$

Where the pressure and irrotational part $\Phi$ depend on $\textbf{u}(x,0)$.
This derivation shows that $\sigma$ must satisfy Laplace’s heat equation, and
the pressure depends only on $\sigma$ and the irrotational part $\Phi$ of the
non-linear vector field
$\left(\textbf{u}\cdot\nabla\right)\textbf{u}-\mu\nabla\textbf{u}-\textbf{f}$.
The solution $\sigma$ of Laplace’s equation is well known and can be solved
using the method of separation of variables (i.e $\sigma=X(x)Y(y)Z(z)$). To
focus on the problem at hand we use the above to write the gradient of
pressure as a function of $\sigma$ (which is known) and $\Phi$ which is a
function of u itself

(10) $-\dfrac{1}{\rho}\nabla p=\nabla\Phi-\nabla\sigma.$

Substituting Eq.(10) back into Navier-Stokes equation Eq.(1) yields

(11) $\dfrac{\partial}{\partial
t}\textbf{u}+\nabla\times\Gamma=-\nabla\sigma.$

Lets remind our self that $\sigma$ is known so the right hand side of Eq.(11)
is known. Secondly, we know $\Gamma$ is a function of the non-linear term
$NL(\textbf{u})=(\textbf{u}\cdot\nabla)\textbf{u}-\mu\bigtriangleup\textbf{u}-\textbf{f}$
which again is just a function of the vector field u. Thus Eq.(11) is
independent of pressure and is a PDE dependent only on the velocity field u.
Substituting the definition of $\Gamma$ into Eq.(11) gives one

(12) $\dfrac{\partial}{\partial
t}\textbf{u}+\nabla\times\biggl{[}\dfrac{1}{4\pi}\int_{V}{\dfrac{\nabla\times\text{NL}(\textbf{u})\circ(y)}{|x-y|}}dV-\dfrac{1}{4\pi}\oint_{S}\textbf{n}\times\dfrac{\text{NL}(\textbf{u})\circ(y)}{|x-y|}\,dS\biggr{]}=-\nabla\sigma.$

Next we compare Eq.(12) to the vorticity equation (which is found by computing
the curl of Eq.(1))

(13) $\dfrac{\partial}{\partial
t}\left(\nabla\times\textbf{u}\right)+\nabla\times NL(\textbf{u})=0.$

Here we notice that since the external forces are not present in the vorticity
equation because they are assumed to be curl free (see Eq.(1)). Substituting
the result of Eq.(13) into Eq.(12) gives one

(14) $\dfrac{\partial}{\partial
t}\textbf{u}(x,t)-\nabla\times\biggl{[}\dfrac{1}{4\pi}\int_{V}{\dfrac{\dfrac{\partial}{\partial
t}\nabla\times\biggl{(}\textbf{u}(y,t)\biggr{)}\circ(y)}{|x-y|}}dV+\dfrac{1}{4\pi}\oint_{S}\textbf{n}\times\dfrac{NL(\textbf{u})\circ(y)}{|x-y|}\,dS\biggr{]}=-~{}\nabla\sigma.$

Now suppose

(15) $\nabla\times\textbf{u}=0,\indent\text{at time $t=t^{\prime}$}$

then

(16)
$\begin{split}\text{NL}(\textbf{u})\circ(x,t^{\prime})&=\nabla\times\biggl{(}\left(\textbf{u}\cdot\nabla\right)\textbf{u}-\mu\bigtriangleup\textbf{u}\biggr{)}\circ(x,t^{\prime})\\\
&=\biggl{[}(\nabla\times\textbf{u}\cdot\nabla)\textbf{u}-\nabla\times\textbf{u}(\nabla\cdot\textbf{u})-\mu\bigtriangleup~{}(\nabla\times\textbf{u})\biggr{]}\circ(x,t^{\prime})\\\
&=\textbf{0}\end{split}$

and so $\dfrac{\partial}{\partial t}\textbf{u}(x,t^{\prime})=0$ which implies
$\nabla\times\textbf{u}(x,t)=0$ for all time $t\geq t^{\prime}$. So if the
vorticity vanishes in some finite time interval it cannot reappear later on.
Which also means its time derivative vanishes for all time $t\geq t^{\prime}$
and even further

$\dfrac{1}{4\pi}\int_{V}{\dfrac{\dfrac{\partial}{\partial
t}\nabla\times\biggl{(}\textbf{u}(y,t)\biggr{)}\circ(y)}{|x-y|}}dV=\textbf{0},\indent
t\geq t^{\prime}.$

Hence, Eq.(14) simplifies and gives one

(17)
$\textbf{u}(x,t)-\nabla\times\dfrac{1}{4\pi}\oint_{S}\textbf{n}\times\dfrac{NL(\textbf{u})\circ(y)}{|x-y|}\,dS=-t~{}\nabla\sigma,\indent
t\geq t^{\prime}.$

Since, Eq.(17) is valid for any arbitrary surface $S$ we then must have

(18)
$\begin{split}\nabla\times\dfrac{1}{4\pi}\oint_{S}\textbf{n}\times\dfrac{NL(\textbf{u})\circ(y)}{|x-y|}\,dS=\textbf{0}.\\\
\end{split}$

Therefore, Eq.(17) reduces and the velocity is provided by

(19) $\textbf{u}(x,t)=-t~{}\nabla\sigma$

and the pressure can then be computed using Eq.(9)

(20)
$\begin{split}p(x,t)&=\rho\biggl{[}\sigma-\dfrac{1}{4\pi}\int_{V}{\dfrac{\nabla\cdot\textbf{g}(y,t)}{|x-y|}}dy+\dfrac{1}{4\pi}\oint_{S}\textbf{n}\cdot\dfrac{\textbf{g}(y,t)}{|x-y|}\,ds\biggr{]}\\\
\end{split}$

where g is defined by

$\textbf{g}(x,t)=\text{NL}(-t\nabla\sigma)=t^{2}(\nabla\sigma\cdot\nabla)\nabla\sigma+t\mu\bigtriangleup\nabla\sigma-\textbf{f}.$

Next assume

$\nabla\times\textbf{u}\neq 0,\indent t^{\prime}\geq t\geq 0.$

In this case we can rewrite the non-linear term using Eq.(1)

$\text{NL}(\textbf{u})=\textbf{f}-\dfrac{1}{\rho}\nabla
p-\dfrac{\partial\textbf{u}}{\partial t}$

and plug in Eq.(14) to get

(21) $\dfrac{\partial}{\partial
t}\textbf{u}(x,t)-\nabla\times\biggl{[}\dfrac{1}{4\pi}\int_{V}{\dfrac{\dfrac{\partial}{\partial
t}\nabla\times\biggl{(}\textbf{u}(y,t)\biggr{)}\circ(y)}{|x-y|}}dV+\dfrac{1}{4\pi}\oint_{S}\textbf{n}\times\dfrac{\biggl{[}\textbf{f}-\dfrac{1}{\rho}\nabla
p-\dfrac{\partial\textbf{u}}{\partial
t}\biggr{]}\circ(y)}{|x-y|}\,dS\biggr{]}=-~{}\nabla\sigma.$

From here one can use Helmholtz decomposition on u (noting its divergence free
so the irrotational part vanishes) to simplify Eq.(21) to get

(22)
$-\int_{0}^{t}{\nabla\times\biggl{[}\dfrac{1}{4\pi}\oint_{S}\textbf{n}\times\dfrac{\biggl{[}\textbf{f}-\dfrac{1}{\rho}\nabla
p\biggr{]}\circ(y)}{|x-y|}\,dS\biggr{]}}=-t\nabla\sigma$

We claim the left hand side vanishes, but we need to prove this. We note that
the left hand side is a function of the surface $S$ but the right hand side is
not. The left hand side changes based on $S$ but the right hand side doesn’t.
Since, $\textbf{f}-\dfrac{1}{\rho}\nabla p$ is not a constant vector field the
only possibility is then

$\int_{0}^{t}{\nabla\times\biggl{[}\dfrac{1}{4\pi}\oint_{S}\textbf{n}\times\dfrac{\biggl{[}\textbf{f}-\dfrac{1}{\rho}\nabla
p\biggr{]}\circ(y)}{|x-y|}\,dS\biggr{]}}=\textbf{0}.$

Thus we have shown

$\nabla\sigma=\textbf{0}.$

We claim this is contradiction because $\sigma$ is not spatially constant for
any initial condition $\textbf{u}^{0}$ satisfying the decay assumption.

∎

######  Theorem 2.

If there is a unique solution to Eq.(1)- Eq.(3) with
$\nabla\times\textbf{u}(x,0)=0$ in all of $\mathbb{R}^{3}$ such that the non-
linear term
$\left(\textbf{u}\cdot\nabla\right)\textbf{u}-\mu\bigtriangleup\textbf{u}$
decays as fast as $\dfrac{1}{r}$ as $x\rightarrow\infty$ then the vector field
u and and scalar function $p$ must take the form

$\textbf{u}(x,t)=-t~{}\nabla\sigma,\indent t>0,$

$p(x,t)=\rho\biggl{[}\sigma-\dfrac{1}{4\pi}\int_{\mathbb{R}^{3}}{\dfrac{\nabla\cdot\textbf{g}(y,t)}{|x-y|}}dy\biggr{]}$

with $\sigma$ the solution to Laplace’s equation

$\nabla^{2}\sigma=0$

and where g is defined explicitly by

$\textbf{g}(x,t)=t^{2}(\nabla\sigma\cdot\nabla)\nabla\sigma+t\mu\bigtriangleup\nabla\sigma-\textbf{f}.$

Otherwise, if $\nabla\times\textbf{u}(x,t)\neq 0$ for all $t\geq 0$ then there
is no solution.

###### Proof.

The proof is similar to that of Theorem 1 we take the same procedure but
change Eq.(5)-Eq.(14). The integrals with a $S$ go to zero and the integrals
with a $V$ we set to $\mathbb{R}^{3}$. The decay assumption is needed so that
Helmholtz decomposition stays valid. Everything else does not change. Thus
instead when you arrive at Eq.(14) it will write

$\dfrac{\partial}{\partial
t}\textbf{u}(x,t)-\nabla\times\biggl{[}\dfrac{1}{4\pi}\int_{\mathbb{R}^{3}}{\dfrac{\dfrac{\partial}{\partial
t}\nabla\times\biggl{(}\textbf{u}(y,t)\biggr{)}\circ(y)}{|x-y|}}dV\biggr{]}=-~{}\nabla\sigma.$

Now if $\nabla\times\textbf{u}\neq 0$ then we notice that since u is
incompressible the term

$\dfrac{1}{4\pi}\int_{\mathbb{R}^{3}}{\dfrac{\dfrac{\partial}{\partial
t}\nabla\times\biggl{(}\textbf{u}(y,t)\biggr{)}\circ(y)}{|x-y|}}dV$

is a vector potential for $\dfrac{d}{dt}\textbf{u}$ which would give us

$\nabla\sigma=0.$

This again is a contradiction because $\sigma$ is not spatially constant. In
the case $\nabla\times\textbf{u}=0$ we get the result

$\textbf{u}(x,t)=-t~{}\nabla\sigma,\indent t>0,$

$p(x,t)=\rho\biggl{[}\sigma-\dfrac{1}{4\pi}\int_{\mathbb{R}^{3}}{\dfrac{\nabla\cdot\textbf{g}(y,t)}{|x-y|}}dy\biggr{]}$

∎

######  Theorem 3.

The non-linear vector field

$t^{2}(\nabla\sigma\cdot\nabla)\nabla\sigma+t\mu\bigtriangleup\nabla\sigma-\textbf{f}.$

decays at least as fast as $\dfrac{1}{r}$ with $x\rightarrow\infty$ and when
the vorticity is zero the solution provided in Theorem 2 is actually a
solution in all of $\mathbb{R}^{3}$.

###### Proof.

We need to prove this vector field decays and then show our solution actually
satisfies the equation Eq.(1)-Eq.(3). ∎

##  4\. A Relative Fluid flow Using Einstein Energy criterion

During the construction of the Navier-stokes equation Eq.(1)-Eq.(3) both
Navier and Stokes considered the mass of a particle constant relative to the
acceleration because the fluid flow is expected to be very small relative to
the speed of light (which should be true for almost all physical situations).
This is just a speculation but somehow it could be greater then the speed of
light mathematically speaking assuming the physical world allows it. According
to Einsteins [1] energy criterion we know that the relative mass $m_{rel}$ is
related to the energy $E$ of the velocity

$E=m_{rel}c^{2}$

which gives

$m_{rel}=\dfrac{m_{0}}{\sqrt{1-\dfrac{|\textbf{u}|^{2}}{c^{2}}}}$

where $c$ is the speed of the light and $m_{0}$ is the rest mass at time $t=0$
and $|\textbf{u}|^{2}$ is the speed of the fluid particle. Recall Newtons
second law states

(23) $m~{}\textbf{a}=\sum{\textbf{F}}$

where $m$ is the mass of a particle a is the acceleration and F is the
external forces acting on a particle. Substituting Einstein formula for mass
and diving throughout by the finite volume $V$ of the particle we obtain

(24)
$\biggl{[}\dfrac{\rho}{\sqrt{1-\dfrac{|\textbf{u}|^{2}}{c^{2}}}}\biggr{]}~{}\textbf{a}=\sum{\dfrac{1}{V}\textbf{F}}=\sum{\textbf{f}}$

using the same derivation (momentum law) the acceleration can be written

(25) $\textbf{a}=\dfrac{\partial\textbf{u}}{\partial
t}+(\textbf{u}\cdot\nabla)\textbf{u}.$

Where $\dfrac{\partial\textbf{u}}{\partial t}$ is called the temporial
acceleration and $(\textbf{u}\cdot\nabla)\textbf{u}$ is called the convective
acceleration. The external forces can be written as

(26) $\sum{\textbf{f}}=-\nabla p+\nabla\cdot\tau+\textbf{g}$

with $p$ the pressure and $\tau$ the stress tensor and g some external applies
force (i.e. gravity). Applying newton law of viscosity we know

(27) $\nabla\cdot\tau=\nu\bigtriangleup\textbf{u},\indent\nu>0.$

Combining Eq.(23), Eq.(24), Eq.(25), Eq.(26), and Eq.(27) and dividing
throughout by $\rho$ one finds

(28)
$\biggl{[}\dfrac{1}{\sqrt{1-\dfrac{|\textbf{u}|^{2}}{c^{2}}}}\biggr{]}\biggl{[}\dfrac{\partial\textbf{u}}{\partial
t}+(\textbf{u}\cdot\nabla)\textbf{u}\biggr{]}=-\dfrac{1}{\rho}\nabla
p+\mu\bigtriangleup\textbf{u}+\dfrac{1}{\rho}\textbf{g},\indent\mu:=\dfrac{\nu}{\rho}$

We leave off here and observe a few things from these equations. One
observation is that the main result of this work does not apply to this newly
developed formula because the curl is different then the original equation and
so none of the results will hold here. Another is that if u is a real vector
field, then the left hand side of Eq.(25) is a complex number when
$|\textbf{u}|$ is greater then the speed of light $c$ but the right hand side
is a real number. So we need $|\textbf{u}|\leq c$ for this equation to make
since. We do not attempt to solve these problem because it was not the main
purpose of our work.

## References

  * [1] Einstein, Albert. ”Does the inertia of a body depend upon its energy-content.” Annalen der physik 18.13 (1905): 639-641.
  * [2] Elgindi, Tarek M. ”Finite-Time Singularity Formation for $C^{1,\Phi}$ Solutions to the Incompressible Euler Equations on $\mathbb{R}^{3}$.” arXiv preprint arXiv:1904.04795 (2019).
  * [3] Evans, Lawrence C. ”Partial differential equations.” Graduate studies in mathematics 19.2 (1998).
  * [4] Fefferman, Charles L. ”Existence and smoothness of the Navier-Stokes equation.” The millennium prize problems 57 (2006): 67..
  * [5] Leray, Jean. ”Sur le mouvement d’un liquide visqueux emplissant tespace.” Acta Math 63 (1934): 193-248.
  * [6] Ladyzhenskaya, Olga Aleksandrovna. ”The Mathematical Theory of Viscous Incompressible Flow, volume 2 of.” Mathematics and its Applications (1969).
  * [7] Brandolese, Lorenzo, and Fernando Cortez. ”Blowup for the nonlinear heat equation with small initial data in scale-invariant Besov norms.” Journal of Functional Analysis 276.8 (2019): 2589-2604.
  * [8] Tao, Terence. ”Finite time blowup for an averaged three-dimensional Navier-Stokes equation.” Journal of the American Mathematical Society 29.3 (2020): 601-674.
  * [9] Tao, Terence. ”Quantitative bounds for critically bounded solutions to the Navier-Stokes equations.” arXiv preprint arXiv:1908.04958 (2020).


11institutetext:  University of California, Berkeley, Berkeley CA 94720, USA
https://www.dekelgalor.com/noise2image **footnotetext: These authors
contributed equally to this work††footnotetext: These authors contributed
equally in their mentorship

# Noise2Image: Noise-Enabled Static Scene Recovery for Event Cameras

Ruiming Cao* 0000-0002-0936-114X Dekel Galor* 0000-0001-9243-6925 Amit Kohli
0000-0002-8328-9465 Jacob L Yates† 0000-0001-8322-5982
and Laura Waller† 0000-0003-1243-2356

###### Abstract

Event cameras capture changes of intensity over time as a stream of ‘events’
and generally cannot measure intensity itself; hence, they are only used for
imaging dynamic scenes. However, fluctuations due to random photon arrival
inevitably trigger _noise events_ , even for static scenes. While previous
efforts have been focused on filtering out these undesirable noise events to
improve signal quality, we find that, in the photon-noise regime, these noise
events are correlated with the static scene intensity. We analyze the noise
event generation and model its relationship to illuminance. Based on this
understanding, we propose a method, called Noise2Image, to leverage the
illuminance-dependent noise characteristics to recover the static parts of a
scene, which are otherwise invisible to event cameras. We experimentally
collect a dataset of noise events on static scenes to train and validate
Noise2Image. Our results show that Noise2Image can robustly recover intensity
images solely from noise events, providing a novel approach for capturing
static scenes in event cameras, without additional hardware.

###### Keywords:

dynamic vision sensor, event-based camera, image reconstruction, noise
characterization

##  1 Introduction

Event cameras [34], also known as neuromorphic cameras or dynamic vision
sensors, are an emerging modality for capturing dynamic scenes. Their ability
to capture data at a much faster rate than conventional cameras have led to
their application in high-speed navigation, augmented reality, and real-time
3D reconstruction [19]. Unlike a conventional CMOS camera that outputs
intensity images at fixed intervals, an event camera detects brightness
changes at each pixel asynchronously. When the brightness change at a pixel
exceeds some threshold, an event is recorded. The output from an event
consists of three elements: a timestamp, the spatial coordinate of the
triggered pixel, and a binary polarity, indicating whether it is an increase
or decrease of brightness.

While this scheme allows event cameras to operate beyond conventional
framerates, it makes them blind to the stationary components of a scene, which
induce no brightness changes over time. This issue is especially prevalent
when the camera is not moving and therefore provides no information about the
static background. Even though event cameras are not designed to capture an
intensity image, it is often needed for downstream applications such as
initializing motion tracking algorithms [2]. To mitigate this issue, some
event cameras include a conventional frame-based sensor in the pixel circuit
to simultaneously image both events and traditional intensity images [9, 7].
Alternatively, when circuit-level modification of the event camera is not
possible, a frame-based camera can be installed in parallel, using either a
beam splitter [24, 58, 57] or additional view registration [51]. Both of these
solutions introduce additional hardware, increasing cost, complexity, size,
and power consumption.

Figure 1: Schematic of the Noise2Image pipeline. Recorded events are first
separated into noise events and signal events using an existing event
denoiser. Signal events are triggered by intensity changes of the scene, which
can be fed into existing event-to-video reconstruction methods. The noise
events are then used to reconstruct the static scene intensity with our
Noise2Image method. This relies on characterizing the relationship between
noise events and using learned priors to resolve ambiguities. Data in this
figure was captured in an outdoor environment late afternoon.

Our work leverages the fact that, even when the scene is static, event cameras
still produce noise events. We focus on low and moderate-brightness regimes
(_e.g_. room light, outdoor sunset) [23, 22, 21] in which the dominant source
of noise is photon noise — random fluctuations in the photon arrival process.
In contrast, the high-brightness regime (_e.g_. outdoor daylight) includes
leakage noise events [36], which we do not model here. While the photon noise
is well-studied in the context of frame-based scenes, those events triggered
by photon noise are commonly deemed as part of the general background noise
activity of the event-based sensor, to be filtered out.

In this work we propose a method called Noise2Image that can reconstruct a
static scene from its event noise statistics, with no hardware modifications
and negligible computational overhead. First, we derive a statistical noise
model describing how noise event generation correlates with scene intensity,
which shows a good correspondence with our experimental measurements. Unlike
in conventional sensors, where photon noise grows with the signal, we find
that for event cameras, the number of events triggered by photon noise is
mostly negatively correlated with the illuminance level due to the logarithmic
sensitivity of the sensor. Imaging the static scene then amounts to inverting
this intensity-to-noise process. However, the mapping is one-to-many, so not
directly invertible; thus, we rely on a learned prior to resolve ambiguities.
To train and validate our method, we experimentally collect a dataset of event
recordings on static scenes. We demonstrate that Noise2Image can recover
photo-realistic images from noise events alone, or can be used to recover the
static parts of scenes with dynamics. In addition to testing on in-
distribution data, we demonstrate the robustness of this approach with out-of-
distribution testing data and live scenes (Fig. 1).

Our contributions are summarized as follows:

  * •

We characterize noise event generation due to photon noise and derive a
mathematical model describing the statistical relationship between noise
events and pixel illuminance.

  * •

We propose the Noise2Image method to recover the intensity image of a static
scene from a recording of noise events using a learned prior.

  * •

We collect a noise-events-to-image (NE2I) dataset with recordings of noise
events paired with the corresponding intensity images to train and validate
our method.

  * •

We show that Noise2Image is complementary to event-to-video reconstruction
methods (E2VID), enabling recovery of both static and dynamic parts of a
scene.

##  2 Related Work

####  2.0.1 Event-to-video reconstruction.

Event-to-video reconstruction (E2VID) is a class of methods that recover high
frame-rate video from event recordings. These methods can capture high-speed
dynamics without the motion blur that plagues traditional frame-based cameras.
Because event cameras only capture scene changes, many E2VID approaches
include one or two traditional frame-based images, using events to deblur [43,
27, 39, 49], synthesize adjacent frames [50], or interpolate temporally [48,
47, 55, 52].

In contrast, a more challenging class of E2VID reconstruction only uses events
for video reconstruction. Because the initial scene intensity is unknown and
only the relative intensity changes are measured, reconstruction requires
either explicit modeling of spatiotemporal relationships [4] or deep neural
networks as a data prior to fill in the missing information [42, 46, 44, 53,
10, 40, 17]. Because it is difficult to collect event recordings paired with
the corresponding frame-based videos at scale, the data prior is obtained
through synthetic data generation [42, 46] using an event camera simulator
[41]. While E2VID methods are effective for recordings with object motion or
camera motion, the reconstruction of static scenes or regions is still out of
reach since no events are thought to be triggered without motion. Our
Noise2Image method can be considered complementary to E2VID; one might use
Noise2Image to recover the static parts of the scene and E2VID to recover the
dynamic parts.

####  2.0.2 Event camera noise characterization.

An event recording often contains a number of events not associated with
intensity changes, which are termed noise events or background activity [12,
22]. Noise events are attributed to two main sources: photon noise and leakage
current [36, 21, 23]. Photon noise is the dominant source of noise in low-
brightness conditions [31, 21], while leakage noise events dominates in high-
brightness conditions [36, 21]. Although the noise model of event cameras is
less studied than traditional sensors, in the lower-light regime, it is
generally believed that noise events become less likely to trigger as
intensity increases [14, 26]. In event camera simulators, the events triggered
by photon noise are modeled as a Poisson process, with the noise event rate
linearly decreasing with intensity [26]. The intensity dependency of noise
events has been experimentally measured in [21], and shows a non-monotonic
relationship with intensity in low light. Our experimental results are
consistent with this trend, and we derive a theoretical noise model which
explains the relationship between intensity and noise events.

####  2.0.3 Event denoising.

While denoising methods for CMOS or CCD sensors often focus on building an
accurate noise model, event camera denoising methods instead emphasize noise
detection by identifying the “signal” events corresponding to changes in the
scene and removing everything else. Because natural scene changes are
inherently spatiotemporal, an event triggered by real signal should be
accompanied by a number of other events at the neighboring spatial and
temporal locations. With this observation, a background activity filter (BAF)
(also called nearest neighbor filter) is used to identify real events by
checking the time difference between a new event and the most recent previous
event in its proximate spatial location and rejecting events when the time
difference passes a threshold [13, 11, 38]. Similarly, the noise events can
also be found through spatiotemporal local plane fitting [6]. BAF is simple to
compute and can be implemented in hardware with limited computational
resources [33, 5]. Guo and Delbruck further extended BAF by reducing its
memory footprint and incorporating structural information from a data-driven
signal-versus-noise classifier [23]. A recent study points out that events are
often triggered simultaneously by noise and signal and therefore treats
denoising as a regression problem to predict the noise likelihood [3]. Here,
we assume access to a good event denoiser [18, 14] that isolates noise events
resulting from static components of a scene.

####  2.0.4 Event camera datasets.

Unlike frame-based cameras, the collection and curation of event camera
datasets is challenging, largely due to the inaccessible hardware and the
scarcity of online data repositories (_e.g_., flickr, Google Images for
framed-based images). Nevertheless, since large datasets are critical for
machine learning [29, 15], there are a handful of recent efforts to
systematically generate large-scale event camera datasets. One common approach
is to display images on a monitor and record them using an event camera [45,
37, 30, 28]. The motion in the scene can then be generated by either moving
the displayed image on the monitor [45, 30] or moving the camera [37, 28].
Alternatively, event datasets can also be generated from frame-based video
datasets by over-sampling in time and feeding into an event camera simulator
[20, 26, 56]. In this work, we generate an NE2I dataset using experimental
measurements and synthetic noise, described in Sec. 3.

##  3 Modeling Noise Event Statistics

First, we develop a model of noise event statistics for later use in our
synthetic dataset generation and image reconstruction steps. An event $e$ is
triggered when a pixel $\left(x,y\right)$ detects a change in logarithmic
intensity, $\log(I)$, greater than the contrast threshold, $\epsilon$, at the
time $t$. The polarity $s=+1$ if the change is positive, and $s=-1$ if the
change is negative. The triggering condition for an event can be written as

$\left(\log(I(x,y,t))-\log(I(x,y,t_{0}))\right)\cdot s>\epsilon,$ (1)

where $t_{0}$ is the timestamp for the most recent event at the same pixel.
The logarithmic sensitivity ensures that this triggering condition is adaptive
to the brightness level, _e.g_. a tenfold increase from 1 lux to 10 lux would
result in a similar response as a tenfold increase from 100 lux to 1000 lux.
As pixels operate asynchronously, we will proceed by considering events at an
arbitrary pixel and drop the $\left(x,y\right)$ notation for simplicity.

By design, there should be no events triggered by a static scene, because
there will be no changes in intensity. However, the inherent randomness in the
photon arrival process means that there will be fluctuations in the detected
intensity and thus noise events triggered. Consider a static scene in which a
pixel sees $n$ photons over a short period of time. $n$ follows a Poisson
distribution with the average photon count $\lambda\propto I$, which can be
further approximated as a Gaussian distribution for the light levels we work
with ($\lambda>10$). We similarly write the photon count during the last event
trigger as $n_{0}$, and both
$n,n_{0}\overset{\text{i.i.d.}}{\sim}\mathcal{N}\left(\lambda,\lambda\right)$.

Figure 2: a. Theoretical noise event probability, $p_{e}$, versus the average
photon count, $\lambda$, at different photoreceptor bias values, $b_{pr}$
($\epsilon$ is the contrast threshold). b. Experimentally measured noise event
rate versus illuminance (proportional to $\lambda$) matches well with our
theoretical model after fitting parameters $\epsilon,b_{pr}$ and $N$.

To trigger a positive event, we need
$\log\left(n\right)-\log\left(n_{0}\right)>\epsilon$, which can be re-written
as $n-n_{0}e^{\epsilon}>0$. Under the Gaussian approximation,
$n-n_{0}e^{\epsilon}\sim\mathcal{N}\left(\lambda\left(1-e^{\epsilon}\right),\lambda\left(1+e^{\epsilon}\right)\right)$.
Thus, we can write the probability of triggering a positive event,

$Pr\left(n-n_{0}e^{\epsilon}>0\right)=\frac{1}{2}-\frac{1}{2}\textit{erf}\left(\frac{\lambda\left(e^{\epsilon}-1\right)}{\sqrt{2\lambda\left(1+e^{\epsilon}\right)}}\right),$
(2)

where erf denotes the error function. The derivation for negative events is
similar. This function is plotted in Fig. 2a (cyan curve) and reveals a
monotonically decreasing trend between noise event probability and
illuminance. In other words, as the illuminance increases, it is less likely
to find $n,n_{0}$ that satisfies the triggering condition. This trend matches
with previous literature [14, 21, 22].

For low-light conditions, where $\lambda$ is relatively small, the relative
signal fluctuation gets stronger, which can lead to more noise events.
Consequently, the analog circuit of an event pixel is designed to have a
photoreceptor bias voltage and source-follower buffer which help stabilize the
photoreceptor’s reading and filter out fluctuations beyond a certain bandwidth
[12, 22]. To account for this in our model, we approximate the filtering
effect by adding a photoreceptor bias term, $b_{pr}$, to the photon count
before the logarithmic operation. As a result, the triggering condition of a
positive event becomes $\log(n+b_{pr})-\log(n_{0}+b_{pr})>\epsilon$, which can
be re-written as $n+b_{pr}-e^{\epsilon}\left(n_{0}+b_{pr}\right)>0$. We can
similarly write the probability of triggering an event, $p_{e}$, as a function
of the average photon count (proportional to the illuminance):

$p_{e}\left(\lambda\right)=\frac{1}{2}-\frac{1}{2}\textit{erf}\left(\frac{\left(\lambda+b_{pr}\right)\left(e^{\epsilon}-1\right)}{\sqrt{2\lambda\left(1+e^{\epsilon}\right)}}\right).$
(3)

With the bias term, the model shows fewer noise events at lower illuminance
levels (Fig. 2a). As average photon count increases, the number of noise
events will increase up to a point and then decrease. While this model is not
monotonic, we will show that we can still recover illuminance from noise event
counts.

To validate our derivation, we acquired experimental measurements of noise
events at different illuminance levels (Fig. 2b). The measured positive event
rate matches to our derived noise model after parameter fitting. Note that our
formulation is a simplified noise model for the event circuit, which
characterizes the events triggered by photon noise. There are more
controllable bias parameters in the actual analog circuit [12, 22] that are
beyond our formulation.

Figure 3: a. Displayed static scene. b-c. Synthetic noise event count sampled
from the Poisson distribution and generalized negative binomial distribution,
respectively. d. Noise event count from a experimental recording. The insets
show the histogram for the noise event count.

####  3.0.1 Sampling Noise Events.

Once we know the probability of observing a single noise event, we can
simulate the number of noise events over a fixed time window given the
intensity of a scene, $I$. Viewing each potential noise event in this window
as a Bernoulli trial with probability $p_{e}$, the noise event count has a
binomial distribution with probability of $p_{e}\left(I\right)$ and $N$
trials. With the refractory period (_i.e_. minimum time between events) much
smaller than the inter-event interval, $N$ will be large, and the binomial
distribution can be approximated by a Poisson distribution with parameter
$N\cdot p_{e}\left(I\right)$.

However, we observe that the empirical noise event count (Fig. 3d) has a
higher variance than the Poisson distribution (Fig. 3b), referred to as
overdispersion. The overdispersion of the experimental data is likely caused
by some inter-pixel variabilities, such as the variation of contrast threshold
values [31]. Thus, we instead sample the noise count from a generalized
negative binomial distribution—which is often used in lieu of Poisson when
there is overdispersion—with a mean of $N\cdot p_{e}\left(I\right)$ and an
illuminance-dependent variance obtained empirically from the calibration data.
The noise count sampled from the negative binomial distribution (Fig. 3c)
resembles the experimental count.

##  4 Reconstructing the Scene

Once we have a model for the mapping between scene intensity and noise event
count at each pixel, we can develop an algorithm for recovering the static
scene. We first estimate the true noise event count $N\cdot
p_{e}\left(I\right)$ from the empirical noise event count that the camera
measures, and then estimate the intensity $I$ by inverting Eq. (3). By doing
this for every pixel, we can form an image for a static scene.

This inverse problem is non-trivial to solve for two reasons. First, the
problem is ill-posed since Eq. (3) is one-to-many, as in Fig. 2b, for nonzero
biases and thus using the noise count alone is insufficient to solve for exact
intensity values. Second, even though the empirical event count is the maximum
likelihood estimator of the true event count, the estimate from the empirical
event counts in a finite time window will have some statistical error leading
to further downstream error after inverting $p_{e}(I)$.

We approach the first challenge by counting the positive and negative
polarities of noise events separately. We found that event cameras often set
different contrast threshold values to trigger positive and negative events,
since the leakage current of the event circuit causes positive events to be
triggered more easily [36]. Because of potentially asymmetric behavior of the
two polarities, the correlation between light intensity and noise events is
also different for each polarity. As shown in Supplementary Fig. 1, the
corresponding light intensity becomes more uniquely defined when counting
positive and negative events separately.

The second issue — inexact event count estimation — can also be mitigated
through the use of data priors in the inverse problem. This will make the
resulting reconstruction robust to small errors in the event rate. Combining
these two, we train a neural network to map the estimated event count directly
to the corresponding intensity image. The network accepts inputs with two
spatial channels for positive and negative event counts. To further reduce the
variance, we perform pixel binning across blocks of $2\times 2$ pixels.

###  4.1 Application on Dynamic Scenes

As in Fig. 1, the static scene reconstruction can also operate in parallel
with the event-to-video reconstruction (E2VID) pipeline on recordings with
both static and dynamic components, _e.g_., scenes with moving foreground and
static background. We first separate signal events triggered by dynamic
changes from other recorded events using an event denoising algorithm [18,
14], and apply an E2VID method to reconstruct the dynamic components. The
remaining events can be considered noise events triggered mostly by photon
fluctuation, which can be fed into the proposed Noise2Image model for static
background reconstruction. For a variable-length recording, we aggregate noise
events using a moving window with a fixed temporal width. Lastly, we stitch
the dynamic and static scene reconstructions together using a motion mask
identified from the signal events.

##  5 Experiments

The existing event camera datasets all have a significant amount of scene
motion and/or camera motion. It is not possible to fully distinguish between
events triggered by intensity changes and noise events [3], and thus they are
not suitable for our goal of static scene reconstruction. Hence, we collect
our own training/validation dataset, termed noise events-to-image (NE2I). NE2I
contains pairs of high-resolution intensity images and noise event recordings
from both experimental acquisition and synthetic noise based on the model
presented in Sec. 3.0.1.

We image a 24.5 inch LCD monitor displaying static images. In order to
calibrate the noise model, we display 256 grayscale values on the monitor and
capture both their event response and light intensity. The illuminance of each
grayscale value is measured using a light meter placed next to the
camera111Since the light meter has optics different from the lens on the event
camera, the measured illuminance value may not be equal to but is proportional
to the actual illuminance level hitting the camera sensor.. Our event camera
is a monochromatic Prophesee Metavision EVK3-HD. The default bias parameters
were used for the camera, and no denoising was performed on the raw data.
While the event camera has a high pixel count ($1280\times 720$ pixels), it
does not have an active pixel sensor (APS) that records frame-based intensity
images. To register the event camera recording to the screen, we first imaged
a standard checkerboard flashing on the monitor to induce events. The
aggregated event count is used to establish a transformation matrix [8], which
is applied to any event recordings to align it spatially with the monitor.

The full NE2I dataset consists of an in-distribution set and an out-of-
distribution test set. The in-distribution data contains 1004 high-resolution
images of artistic human portraits from Unsplash, split into 754 images for
training, 100 for validation, and 150 for testing. The out-of-distribution
test set has 100 high-resolution images from the validation set of DIV2K image
super-resolution dataset [1], aiming to provide a variety of scenes much
beyond the training data distribution. We intentionally chose a confined
distribution for training data and a broad distribution for testing, so that
the evaluation can test if the model learns local correlations instead of an
image-level prior.

Given a sequence of noise events, we first aggregate them into a 2D-matrix of
event count. To account for the positive and negative polarities, we
separately aggregate them and store as two channels. We train a U-net to map
an event count matrix into the corresponding intensity image. A modified U-net
from [25] is used for enhanced performance. The network is trained using
either experimental training data (described above) or synthetic event counts
generated by the statistical noise model in Sec. 3.0.1. For experimental data
training, we augment the input noise count by choosing a random starting time
for the 1-second window over a 10-second event recording. For synthetic data
training, event counts are re-sampled each time on-the-fly.

We use E2VID as a baseline method for the evaluation of static scene recovery
even though it was not trained by such data. We compare to the pre-trained
model implementation [16] for the original E2VID [42], E2VID+ [46], FireNet
[44], FireNet+ [46], ET-Net [53], SPADE-E2VID [10], SSL-E2VID [40], HyperE2VID
[17]. A sequence of events within a 1-second window is fed into each method,
and the last frame of predicted video (5 predicted frames in total) is used
for evaluation.

####  5.0.1 Metrics.

Evaluation of all methods is performed using experimentally-collected testing
data, with the quality of recovered intensity images being calculated for the
in-distribution and out-of-distribution testing datasets using three common
quantitative metrics [42]: peak signal-to-noise ratio (PSNR), structured
similarity (SSIM) and perceptual similarity (LPIPS [54]). LPIPS is computed
using pre-trained AlexNet with image intensity values normalized to [-1, 1].

###  5.1 Results

Methods | In-distribution test |  | Out-of-distribution test
---|---|---|---
PSNR |  SSIM |  LPIPS |  |  PSNR |  SSIM |  LPIPS
Original E2VID [42] |  7.76 |  0.043 |  1.131 |  |  10.1 |  0.055 |  1.032
E2VID+ [46] |  8.72 |  0.148 |  0.872 |  |  9.26 |  0.108 |  0.768
FireNet [44] |  9.35 |  0.089 |  1.047 |  |  10.5 |  0.098 |  0.970
FireNet+ [46] |  9.78 |  0.063 |  0.959 |  |  9.77 |  0.057 |  0.800
ET-Net [53] |  9.13 |  0.166 |  0.805 |  |  10.1 |  0.131 |  0.730
SPADE-E2VID [10] |  7.74 |  0.229 |  0.856 |  |  8.74 |  0.149 |  0.870
SSL-E2VID [40] |  8.81 |  0.133 |  0.764 |  |  8.95 |  0.123 |  0.720
HyperE2VID [17] |  9.09 |  0.159 |  0.858 |  |  0.89 |  0.094 |  0.759
Noise2Image, synthetic |  20.9 |  0.589 |  0.462 |  |  16.3 |  0.468 |  0.583
Noise2Image, experimental |  25.0 |  0.742 |  0.349 |  |  19.3 |  0.552 |  0.509
Table 1: Quantitative results for static scene reconstruction with in-
distribution and out-of-distribution testing data. Our Noise2Image models are
trained with either synthetic or experimental noise data. Pre-trained events-
to-video reconstruction (E2VID) methods333Since E2VID models were trained by
synthetic motion data generated from the MS-COCO dataset [32], both of our
testing sets are considered as out-of-distribution.are used as baselines. We
report peak signal-to-noise ratio (PSNR), structured similarity (SSIM), and
perceptual similarity (LPIPS [54]). Figure 4: Comparison between Noise2Image
and baseline pre-trained event-to-video (E2VID) methods on noise event-to-
intensity reconstruction. The first row shows the input event count (captured
in experiment) aggregated over a 1-second window. Noise2Image is trained using
either synthetic or experimental data as specified. Full comparison with all
baseline methods can be found in Supplementary Fig. 2.

For static-only scenes, the quantitative scores for Noise2Image as well as
baseline methods using NE2I dataset are reported in Table 3. As shown in Fig.
4, the Noise2Image model trained by either experimental or synthetic data
recovers detailed intensity images from the count of noise events. Both
Noise2Image models generalized well to out-of-distribution testing data,
providing a good level of contrast and details, suggesting that Noise2Image
learns the correlation between noise event and intensity. The Noise2Image
model trained by experimental data out-performed the one trained by the
synthetic data by 4.1dB of PSNR. As the synthetic data-trained model often
recovered speckle-like patterns for the uniform image background, we speculate
that this performance margin is caused by the spatial correlation of the noise
events, which is beyond our synthesis model but potentially captured by the
experimental data training. Baseline E2VID methods did not perform well on
static scene reconstruction for two reasons: they were only trained by scenes
with various degrees of motion (not static scenes) and the training data of
E2VID was generated without an intensity-dependent noise model [41].

In addition to testing data, we tested Noise2Image on real-world scenes, both
indoor and outdoor. The Noise2Image model trained by images displayed on a
monitor generalizes well for scenes outside of the laboratory setting, as
shown in Fig. 1 and Supplementary Fig. 3. We have implemented a real-time demo
running on a laptop computer as well.

####  5.1.1 Effect of aggregation duration.

Table 2 shows the effect of varying the aggregation window time for training
and testing the Noise2Image method. As the aggregation duration shortens,
fewer noise events will be triggered, and the estimation of the true event
count becomes less accurate. Table 2 shows that the Noise2Image reconstruction
works even with short aggregation duration (0.1 seconds), although longer
integration will result in better reconstruction quality.

Aggregation duration | In-distribution test |  | Out-of-distribution test
---|---|---|---
PSNR |  SSIM |  LPIPS |  |  PSNR |  SSIM |  LPIPS
0.1 s |  23.8 |  0.714 |  0.391 |  |  18.4 |  0.513 |  0.586
0.25 s |  24.4 |  0.721 |  0.377 |  |  18.8 |  0.522 |  0.564
0.5 s |  24.7 |  0.738 |  0.358 |  |  19.1 |  0.545 |  0.537
1 s |  25.0 |  0.742 |  0.349 |  |  19.3 |  0.552 |  0.509
2 s |  25.4 |  0.765 |  0.324 |  |  19.5 |  0.598 |  0.473
Table 2: Noise2Image performance using noise event counts aggregated over
different time windows. For each aggregation duration, the Noise2Image model
is trained and evaluated using the experimental data with identical training
setting.

####  5.1.2 Reconstructing scenes with dynamic components.

Finally, we demonstrated that Noise2Image is complementary to E2VID
reconstruction when the scene has both static and dynamic components. We
imaged a fan rapidly moving in front of a static scene, and fed signal events
into a pre-trained E2VID model [42]. While the E2VID method performed well on
the moving object, it could not recover the background static scene (Fig. 5c).
To incorporate Noise2Image, we first identified a motion mask at each
timepoint (Fig. 5b) by thresholding signal events. We then aggregated noise
events and fed the event count at pixels without motion to the Noise2Image
model. Stitching together the moving foreground from E2VID and the static
background from Noise2Image, we obtained the final high-quality reconstruction
as in Fig. 5d and Supplementary Video 1.

Figure 5: Dynamic scene reconstruction of a moving fan in front of a static
scene (see Supplementary Video 1). a. Aggregated event count. The first row is
at a timepoint with no motion and only faint noise, and the second row has
motion. b. Motion mask obtained from thresholding the signal events as
determined by the event denoiser. c. E2VID reconstruction using pre-trained
model from [42]. d. The dynamic foreground reconstructed by E2VID and the
static background reconstructed by Noise2Image are stitched together.

##  6 Discussion

Our experiments were done using a single event camera, and there might be
differences in the noise characteristics between different event circuit
designs. However, our finding in Sec. 3 is based on the photon arrival
process, which is generalizable to other event camera hardware. This
correlation between noise events and illuminance level was also independently
reported in previous studies using different event cameras [21, 26, 14, 22].

Denoising and camera bias parameters [35] can also affect this illuminance
dependency. Our data was acquired without denoising and using bias parameters
default to our event camera. We also tested different high-pass filtering bias
values which affect the correlation between noise event rate and illuminance
(Supplementary Fig.4). For this reason, the trained Noise2Image model is bound
to the camera model and bias parameters that were used in the training data
collection. We hope that our modeling of noise events can be improved upon by
incorporating the analog nature of event sampling and modeling additional
event camera bias parameters, _e.g_. using non-parametric models to account
for unknown camera behavior.

One practical constraint for our study is that the dynamic range of
Noise2Image is limited by the monitor we used. As a result, the increase in
noise events at high illuminance settings due to leakage current cannot be
measured using our setup. This issue can likely be alleviated by changing to a
high-dynamic range and high-brightness monitor or projector for data
collection. We hypothesize that our Noise2Image approach will still be valid
for brighter scenes since the statistics of leakage current-induced noise
events is quite distinct [36].

Another future direction is to incorporate the proposed noise event modeling
into existing event camera simulators. This will help synthesize data with
realistic noise events, which can be used for training event-to-video recovery
(E2VID) models. Overall, the model similarities between Noise2Image and E2VID
suggest that in the future, Noise2Image can be incorporated into the E2VID
pipeline via more realistic training data, complementing each other’s goals.

In summary, we demonstrated imaging of static intensity using an event camera
with no additional hardware. This is made possible by the fact that photon-
noise triggers events which are correlated with illuminance. We developed a
statistical model of the noise event generation, and leveraged it to develop a
strategy called Noise2Image, which maps noise events to an intensity image via
a neural network that incorporates local priors. We demonstrate that our
approach works well quantitatively and qualitatively on experimental event
recordings, including ones taken in the wild. We also show that our method is
compatible with scenes that have object dynamics. Our hope is that Noise2Image
reduces the need for event cameras to have additional hardware for measuring
static scenes.

## Acknowledgments

This material is based upon work supported by the National Science Foundation
Graduate Research Fellowship Program under Grant No. DGE-1752814. Any
opinions, findings, and conclusions or recommendations expressed in this
material are those of the author(s) and do not necessarily reflect the views
of the National Science Foundation. Research was supported by the National Eye
Institute of the National Institute of Health under award number R00EY032179.
Ruiming Cao was supported in part by Siebel Scholarship. This work was
supported by Weill Neurohub Investigators Program. This work has been made
possible in part by CZI grant DAF2021-225666 and grant DOI
10.37921/192752jrgbnh from the Chan Zuckerberg Initiative DAF, an advised fund
of Silicon Valley Community Foundation (funder DOI 10.13039/100014989). Laura
Waller is a Chan Zuckerberg Biohub SF Investigator.

## References

  * [1] Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: Dataset and study. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. pp. 126–135 (2017)
  * [2] Angelopoulos, A.N., Martel, J.N., Kohli, A.P., Conradt, J., Wetzstein, G.: Event based, near eye gaze tracking beyond 10,000 hz. IEEE Transactionson Visualizations and Graphics (2021)
  * [3] Baldwin, R.W., Almatrafi, M., Asari, V., Hirakawa, K.: Event probability mask (epm) and event denoising convolutional neural network (edncnn) for neuromorphic cameras. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020)
  * [4] Bardow, P., Davison, A.J., Leutenegger, S.: Simultaneous optical flow and intensity estimation from an event camera. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 884–892 (2016)
  * [5] Barrios-Avilés, J., Rosado-Muñoz, A., Medus, L.D., Bataller-Mompeán, M., Guerrero-Martínez, J.F.: Less data same information for event-based sensors: A bioinspired filtering and data reduction algorithm. Sensors 18(12), 4122 (2018)
  * [6] Benosman, R., Clercq, C., Lagorce, X., Ieng, S.H., Bartolozzi, C.: Event-based visual flow. IEEE transactions on neural networks and learning systems 25(2), 407–417 (2013)
  * [7] Berner, R., Brandli, C., Yang, M., Liu, S.C., Delbruck, T.: A 240$\times$ 180 10mw 12us latency sparse-output vision sensor for mobile applications. In: 2013 Symposium on VLSI Circuits. pp. C186–C187. IEEE (2013)
  * [8] Bradski, G.: The OpenCV Library. Dr. Dobb’s Journal of Software Tools (2000)
  * [9] Brandli, C., Muller, L., Delbruck, T.: Real-time, high-speed video decompression using a frame-and event-based davis sensor. In: 2014 IEEE International Symposium on Circuits and Systems (ISCAS). pp. 686–689. IEEE (2014)
  * [10] Cadena, P.R.G., Qian, Y., Wang, C., Yang, M.: Spade-e2vid: Spatially-adaptive denormalization for event-based video reconstruction. IEEE Transactions on Image Processing 30, 2488–2500 (2021)
  * [11] Czech, D., Orchard, G.: Evaluating noise filtering for event-based asynchronous change detection image sensors. In: 2016 6th IEEE International Conference on Biomedical Robotics and Biomechatronics (BioRob). pp. 19–24. IEEE (2016)
  * [12] Delbruck, T., Mead, C.: Analog vlsi phototransduction by continuous-time, adaptive, logarithmic photoreceptor circuits. Technical report (1995)
  * [13] Delbruck, T., et al.: Frame-free dynamic digital vision. In: Proceedings of Intl. Symp. on Secure-Life Electronics, Advanced Electronics for Quality Life and Society. vol. 1, pp. 21–26. Citeseer (2008)
  * [14] Ding, S., Chen, J., Wang, Y., Kang, Y., Song, W., Cheng, J., Cao, Y.: E-mlb: Multilevel benchmark for event-based camera denoising. IEEE Transactions on Multimedia (2023)
  * [15] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [16] Ercan, B., Eker, O., Erdem, A., Erdem, E.: Evreal: Towards a comprehensive benchmark and analysis suite for event-based video reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3942–3951 (2023)
  * [17] Ercan, B., Eker, O., Saglam, C., Erdem, A., Erdem, E.: Hypere2vid: Improving event-based video reconstruction via hypernetworks. arXiv preprint arXiv:2305.06382 (2023)
  * [18] Feng, Y., Lv, H., Liu, H., Zhang, Y., Xiao, Y., Han, C.: Event density based denoising method for dynamic vision sensor. Applied Sciences (2020)
  * [19] Gallego, G., Delbrück, T., Orchard, G., Bartolozzi, C., Taba, B., Censi, A., Leutenegger, S., Davison, A.J., Conradt, J., Daniilidis, K., et al.: Event-based vision: A survey. IEEE transactions on pattern analysis and machine intelligence 44(1), 154–180 (2020)
  * [20] Gehrig, D., Gehrig, M., Hidalgo-Carrió, J., Scaramuzza, D.: Video to events: Recycling video datasets for event cameras. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3586–3595 (2020)
  * [21] Graca, R., Delbruck, T.: Unraveling the paradox of intensity-dependent dvs pixel noise. arXiv preprint arXiv:2109.08640 (2021)
  * [22] Graça, R., McReynolds, B., Delbruck, T.: Shining light on the dvs pixel: A tutorial and discussion about biasing and optimization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4044–4052 (2023)
  * [23] Guo, S., Delbruck, T.: Low cost and latency event camera background activity denoising. IEEE Transactions on Pattern Analysis and Machine Intelligence 45(1), 785–795 (2022)
  * [24] Hidalgo-Carrió, J., Gallego, G., Scaramuzza, D.: Event-aided direct sparse odometry. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5781–5790 (2022)
  * [25] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems 33, 6840–6851 (2020)
  * [26] Hu, Y., Liu, S.C., Delbruck, T.: v2e: From video frames to realistic dvs events. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1312–1321 (2021)
  * [27] Jiang, Z., Zhang, Y., Zou, D., Ren, J., Lv, J., Liu, Y.: Learning event-based motion deblurring. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3320–3329 (2020)
  * [28] Kim, J., Bae, J., Park, G., Zhang, D., Kim, Y.M.: N-imagenet: Towards robust, fine-grained object recognition with event cameras. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 2146–2156 (2021)
  * [29] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25 (2012)
  * [30] Li, H., Liu, H., Ji, X., Li, G., Shi, L.: Cifar10-dvs: an event-stream dataset for object classification. Frontiers in neuroscience 11, 309 (2017)
  * [31] Lichtsteiner, P., Posch, C., Delbrück, T.: A 128x128 120 db 15mus latency asynchronous temporal contrast vision sensor. IEEE Journal of Solid-State Circuits 43, 566–576 (2008)
  * [32] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pp. 740–755. Springer (2014)
  * [33] Liu, H., Brandli, C., Li, C., Liu, S.C., Delbruck, T.: Design of a spatiotemporal correlation filter for event-based sensors. In: 2015 IEEE International Symposium on Circuits and Systems (ISCAS). pp. 722–725. IEEE (2015)
  * [34] Mahowald, M., Mahowald, M.: The silicon retina. An Analog VLSI System for Stereoscopic Vision pp. 4–65 (1994)
  * [35] McReynolds, B., Graca, R., Oliver, R., Nishiguchi, M., Delbruck, T.: Demystifying event-based sensor biasing to optimize signal to noise for space domain awareness. In: Advanced Maui Optical and Space Surveillance Technologies Conference (AMOS). University of Zurich (2023)
  * [36] Nozaki, Y., Delbruck, T.: Temperature and parasitic photocurrent effects in dynamic vision sensors. IEEE Transactions on Electron Devices 64(8), 3239–3245 (2017)
  * [37] Orchard, G., Jayawant, A., Cohen, G.K., Thakor, N.: Converting static image datasets to spiking neuromorphic datasets using saccades. Frontiers in neuroscience 9, 437 (2015)
  * [38] Padala, V., Basu, A., Orchard, G.: A noise filtering algorithm for event-based asynchronous change detection image sensors on truenorth and its implementation on truenorth. Frontiers in neuroscience 12, 118 (2018)
  * [39] Pan, L., Scheerlinck, C., Yu, X., Hartley, R., Liu, M., Dai, Y.: Bringing a blurry frame alive at high frame-rate with an event camera. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6820–6829 (2019)
  * [40] Paredes-Vallés, F., de Croon, G.C.: Back to event basics: Self-supervised learning of image reconstruction for event cameras via photometric constancy. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3446–3455 (2021)
  * [41] Rebecq, H., Gehrig, D., Scaramuzza, D.: Esim: an open event camera simulator. In: Conference on robot learning. pp. 969–982. PMLR (2018)
  * [42] Rebecq, H., Ranftl, R., Koltun, V., Scaramuzza, D.: High speed and high dynamic range video with an event camera. IEEE transactions on pattern analysis and machine intelligence 43(6), 1964–1980 (2019)
  * [43] Scheerlinck, C., Barnes, N., Mahony, R.: Continuous-time intensity estimation using event cameras. In: Asian Conference on Computer Vision. pp. 308–324. Springer (2018)
  * [44] Scheerlinck, C., Rebecq, H., Gehrig, D., Barnes, N., Mahony, R., Scaramuzza, D.: Fast image reconstruction with an event camera. In: IEEE Winter Conf. Appl. Comput. Vis. (WACV). pp. 156–163 (2020)
  * [45] Serrano-Gotarredona, T., Linares-Barranco, B.: Poker-dvs and mnist-dvs. their history, how they were made, and other details. Frontiers in neuroscience 9, 481 (2015)
  * [46] Stoffregen, T., Scheerlinck, C., Scaramuzza, D., Drummond, T., Barnes, N., Kleeman, L., Mahony, R.: Reducing the sim-to-real gap for event cameras. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVII 16. pp. 534–549. Springer (2020)
  * [47] Tulyakov, S., Bochicchio, A., Gehrig, D., Georgoulis, S., Li, Y., Scaramuzza, D.: Time lens++: Event-based frame interpolation with parametric non-linear flow and multi-scale fusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 17755–17764 (2022)
  * [48] Tulyakov, S., Gehrig, D., Georgoulis, S., Erbach, J., Gehrig, M., Li, Y., Scaramuzza, D.: Time lens: Event-based video frame interpolation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16155–16164 (2021)
  * [49] Wang, B., He, J., Yu, L., Xia, G.S., Yang, W.: Event enhanced high-quality image recovery. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIII 16. pp. 155–171. Springer (2020)
  * [50] Wang, Z.W., Jiang, W., He, K., Shi, B., Katsaggelos, A., Cossairt, O.: Event-driven video frame synthesis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. pp. 0–0 (2019)
  * [51] Wang, Z., Ng, Y., Scheerlinck, C., Mahony, R.: An asynchronous kalman filter for hybrid event cameras. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 448–457 (2021)
  * [52] Wang, Z., Hamann, F., Chaney, K., Jiang, W., Gallego, G., Daniilidis, K.: Event-based continuous color video decompression from single frames. arXiv preprint arXiv:2312.00113 (2023)
  * [53] Weng, W., Zhang, Y., Xiong, Z.: Event-based video reconstruction using transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2563–2572 (2021)
  * [54] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR (2018)
  * [55] Zhang, X., Yu, L.: Unifying motion deblurring and frame interpolation with events. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 17765–17774 (2022)
  * [56] Zhu, A.Z., Wang, Z., Khant, K., Daniilidis, K.: Eventgan: Leveraging large scale image datasets for event cameras. In: 2021 IEEE International Conference on Computational Photography (ICCP). pp. 1–11. IEEE (2021)
  * [57] Zhu, L., Li, J., Wang, X., Huang, T., Tian, Y.: Neuspike-net: High speed video reconstruction via bio-inspired neuromorphic cameras. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2400–2409 (2021)
  * [58] Zou, Y., Zheng, Y., Takatani, T., Fu, Y.: Learning to reconstruct high speed and high dynamic range videos from events. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2024–2033 (2021)

Supplementary Fig. 1: Noise event counts on positive events versus negative
events. The color indicates the corresponding illuminance level.
Supplementary Fig. 2: Additional comparison between Noise2Image and all
baseline event-to-video (E2VID) methods on noise event-to-intensity
reconstruction. The first row shows the input event count (captured in
experiment) aggregated over a 1-second window. Supplementary Fig. 3: Real-
world examples of Noise2Image taken outside of the laboratory setting. The
Noise2Image model trained by synthetic data results in background artifacts on
the in-door scene (second row, second column), presumably caused by the one-
to-many relationship of Eq. 3. In contrast, the Noise2Image trained by
experimental data predicts the background correctly, hinting that there exist
spatial correlations in the experimental noise events, beyond our derived
spatially independent noise event synthesis. The reference images were taken
by an iPhone 12 plus back camera.  Supplementary Fig. 4: Experimentally
captured noise event rate vs. illuminance levels using various sensor high-
pass filter bias values which is called “bias_hpf” in Prophesee Metavision
SDK. The correlation between illuminance and event rate changes with different
bias_hpf values.