# Remarks on the principles of statistical fluid mechanics

Koji Ohkitani
Research Institute for Mathematical Sciences,
Kyoto University, Kyoto 606-8502 Japan

###### Abstract

This is an idiosyncratic survey of statistical fluid mechanics centering on
the Hopf functional differential equation. Using the Burgers equation for
illustration we review several functional integration approaches to theory of
turbulence. We notice in particular that some important contributions have
been brought about by researchers working on wave propagation in random media,
among which Uriel Frisch is not an exception. We also discuss a particular
finite-dimensional approximation for the Burgers equation.

turbulence, Hopf equation, functional integration

##  1 Introduction

In field theory one important task is the following. Given an action integral
$S[\phi]$, e.g. that of the $\phi^{4}$-model [20]

$S[\phi]=\iint
dxdt\left\\{\frac{1}{2}\left(\phi_{t}^{2}-|\nabla\phi|^{2}\right)-\frac{m^{2}}{2}\phi^{2}-\lambda\phi^{4}+J\phi\right\\}$

written with standard notations, we consider the partition function

$Z(J)={\cal N}\int\exp\left(iS[\phi]\right){\cal D}[\phi].$

The objectives are to figure out how to make sense of what the symbol means
mathematically and to extract physically meaningful information from it. In
view of a correspondence between field theory and statistical mechanics, such
a view can be carried over to fluid mechanics. Hence in principle statistical
theory of turbulence should be based on that footing.

We will revisit the formulation of the Hopf functional differential equation
(hereafter FDE)[1, 2] which governs statistics of fluid turbulence. It is not
our intention to give an exhaustive survey of the subject matter. Rather we
will restrict our attention to one of its facets, that is, functional
integration approaches to theory of turbulence. In section 2 we compare some
FDEs for statistical fluid mechanics with historical notes. We discuss
decaying turbulence in Section 3 and forced turbulence in Section 4. In
Section 5

we consider the Hopf equation for a discrete model. Section 6 is a brief
summary. This is basically a survey article, except for Sections 4(b) and 5.

##  2 The birth of the Hopf equation

The primary objective of so-called ergodic theory is to show that the ensemble
average equals the time average for dynamical systems of physical interest. In
1930s there was substantial progress in the field, including the mean ergodic
theorem (strong in $L^{2}$) [3] and the individual ergodic theorem (strong in
$L^{1}$) [4]. Hopf extended von Neumann’s work to weak convergence and
published a monograph [5]. For its brief history, see e.g. [6].

In the development of $L^{2}$-theory an innocent-looking but far-reaching idea
was put forward in [7], that is, the introduction of a kind of composition
operators [8]. That enabled: i) linearisation by considering observables and
ii) spectral analysis due to unitarity of operators. The price we have to pay
is that we need to handle an infinite-dimensional space. For an observable
$f(\cdot)$ and a phase point $P$ the Koopman operator111The same concept was
introduced independently by A.Weil in connection with [9]. See also [10],
p.32. Paying attention to an invariant measure in Hamiltonian systems, Weil
realised that such a system defines a one-parameter group of unitary
transformations in an $L^{2}$-space with respect to the measure. $U_{t}$ is
defined by

$U_{t}f(P)=f(\phi_{t}P),$ (1)

where $\phi_{t}$ denotes a phase flow. In the language of probability theory
the Koopman operator is the adjoint group to the Perron-Frobenius operator and
the process of lifting of a dynamical system to a group of operators is called
Koopmanism [11]. Hopf did not mention explicitly the notion of the Koopman
operator in [1], but his formulation of statistical fluid mechanics clearly
rests on it, as the Hopf-Foias approach can be regarded as such a
manifestation.

Statistical study of the Navier-Stokes equations was initiated on the basis of
a functional differential equation that now bears his name [1, 2]. There the
functional equation was heuristically derived for the characteristic
functional of the velocity field. On the other hand mathematical study of the
Navier-Stokes equations was initiated in [12, 13] on the basis of the
Liouville equation, a functional equation for the probability measure. Further
developments in this direction were summarised in [14], including the so-
called first integral method. As a rule of thumb, Table 1 shows a rough
classification of several representations in mechanics. A related but
different topic which we will not discuss here is the so-called Koopman mode
analysis. There has been significant progress recently therein and interested
readers ought to consult e.g.[15, 16, 17].

| quantum mechanics | fluid mechanics
---|---|---
group of operators (FDE) |  | $\stackrel{{\scriptstyle\mbox{Hopf-Foias formalism}}}{{\mbox{(or, Vishik-Fursikov formalism)}}}$
group of operators (PDE) | $\stackrel{{\scriptstyle\mbox{Heisenberg representation}^{*}}}{{\mbox{(or, Dirac representation)}}}$ |
dynamical system | Schrödinger equation | Navier-Stokes equations
Table 1: Some examples of Koopmanism in mechanics. $*$A rough analogy where
evolution is described by time-dependent observables.

For the sake of simplicity, to illustrate the main ideas in this article we
will make use of the Burgers equation in $\mathbb{R}^{1}$

$\frac{\partial u}{\partial t}+u\frac{\partial u}{\partial
x}=\nu\frac{\partial^{2}u}{\partial x^{2}},$ (2) $u(x,0)=u_{0}(x),$

where $u_{0}(x)$ is a smooth initial condition. They are transposable at least
formally to the Navier-Stokes equations. Thus, when we say turbulence in this
paper we mean a flow of the 1D model.

Hopf considered the characteristic functional of the velocity field
$\Phi[\theta(x),t]=\left<\exp i\int u(x,t)\theta(x)dx\right>,$ where
$\left<\cdot\right>$ denotes an ensemble average with respect to $u_{0}$. The
governing equation he derived in [1] reads

$\frac{\partial\Phi}{\partial
t}=\frac{i}{2}\int\theta(x)\frac{\partial}{\partial
x}\frac{\delta^{2}\Phi}{\delta\theta(x)dx^{2}}dx+\nu\int\theta(x)\frac{\partial^{2}}{\partial
x^{2}}\frac{\delta\Phi}{\delta\theta(x)dx}dx.$ (3)

This is a functional differential equation (FDE).  We recall that when the
functional varies as $\delta\Phi[\theta(x)]\approx\int A(x)\delta\theta(x)dx$
for a small variation $\delta\theta(x),$ the functional derivative
$\frac{\delta\Phi}{\delta\theta(x)dx}$ is defined by $A(x)$.

Let us write down back to back the Liouville equation, which is also known as
the Hopf-Foias equation

$\frac{d}{dt}\int\Psi[u]d\mu_{t}(u)=\int\left(\nu\frac{\partial^{2}u}{\partial
x^{2}}-u\frac{\partial u}{\partial x},\,\frac{\delta\Psi}{\delta
u(x)dx}\right)d\mu_{t}(u).$ (4)

This can be derived as follows[18]. Define a mapping $T:u\longrightarrow
u+(\nu u_{xx}-uu_{x})dt$ and consider an observable $\Psi[u]$. By
$\mu_{t+dt}=\mu_{t}\circ T^{-1}$ (Hopf identity), we have
$\int\Psi[u]d\mu_{t+dt}=\int(\Psi\circ T)d\mu_{t}.$ Now we have

$\mbox{LHS}\approx\int\Psi[u]d\mu_{t}+\frac{d}{dt}\left(\int\Psi[u]d\mu_{t}\right)dt,$

whereas

RHS $\displaystyle\approx$
$\displaystyle\int\Psi\left[u+\left(\nu\frac{\partial^{2}u}{\partial
x^{2}}-u\frac{\partial u}{\partial x}\right)dt\right]d\mu_{t}$
$\displaystyle\approx$
$\displaystyle\int\Psi[u]d\mu_{t}+\int\left(\nu\frac{\partial^{2}u}{\partial
x^{2}}-u\frac{\partial u}{\partial x},\,\frac{\delta\Psi}{\delta
u(x)dx}\right)d\mu_{t}\,dt.$

Hence the Liouville equation follows. $\square$
It should be noted that this equation is valid for any observables. In
particular, to obtain the original form of the Hopf functional equation we may
simply take $\Psi[u]=\exp\left(i(u,\theta)\right),$ where $(u,\theta)$ denotes
an inner-product, $(u,\theta)=u\cdot\theta=\int u(x,t)\theta(x)dx.$

We note in passing that the equation for probability density function
$F=\Pi_{k}\delta(u_{k}-u_{k}(x,t))$ was written down in [19, 20], where the
symbolic product over $k$ denotes the one over all $x$. Adapting to the
Burgers equation, it takes the following form

$\frac{\partial F}{\partial t}=\int\frac{\delta}{\delta
u(x){\color[rgb]{0,0,0}dx}}\left(\nu\frac{\partial^{2}u}{\partial
x^{2}}-u\frac{\partial u}{\partial x}\right)\,Fdx.$ (5)

This is equivalent to the Hopf-Foias equation (4). Anyway in the Liouville-
type equations the underlying PDE appears explicitly. A complaint made by
mathematical analysts about the Hopf equation is that they don’t see PDEs in
it. Actually PDEs do appear in the path integral representation of their
formal solution of the latter.

##  3 Decaying turbulence

###  3.1 Rosen’s action integral

Apparently the first path integral representation of this ilk is due to [21].
We begin considering freely decaying turbulence. A formal and symbolic
solution was constructed to the Hopf equation in [21]. See also [22]. Making
use of the linearity of the Hopf equation we write the solution as

$\Phi[\theta(x),t]=\int\Phi_{0}[\theta_{0}(x)]K[\theta_{0}(x)|\theta(x),t]{\cal
D}[\theta_{0}(x)],$ (6)

for a Green’s function $K[\theta_{0}(x)|\theta(x),t].$ We will make use of the
plane wave expansion of the Dirac mass

$\delta(x)=\frac{1}{2\pi}\int_{-\infty}^{\infty}\exp(ikx)dk,$ (7)

or more precisely, in its functionally generalised form [23]

$\delta[\psi]=\frac{1}{2\pi}\int\exp\left\\{i(\psi\cdot\phi)\right\\}{\cal
D}[\phi].$ (8)

A symbolic expression of the Green’s function $K[\theta_{0}(x)|\theta(x),t]$
was given as follows:

$K[\theta_{0}(x)|\theta(x),t]=C\iint_{\eta(x,0)=\theta_{0}(x)}^{\eta(x,t)=\theta(x)}\exp\left\\{i\int_{0}^{t}d\tau\int\left(\frac{\partial\eta}{\partial\tau}\zeta+\eta
Q[\zeta]\right)dx\right\\}{\cal D}[\eta(x,\tau)]{\cal D}[\zeta(x,\tau)],$ (9)

where $Q[\zeta]=-\zeta\zeta_{x}+\nu\zeta_{xx}$ and $C$ is a formal
normalization constant. In those expressions e.g. ${\cal
D}[\zeta]=\Pi_{x}d\zeta(x,t)$ denotes a fictitious (translation-invariant)
measure.222We may call it ’abstract Feynman measure’. It may be compared with
the classical Wiener measure in $\mathbb{R}^{n}$ and the abstract Wiener
measure in $\mathbb{R}^{\infty},$ both of which are well understood. Abstract
Feynman measure does not exit because Feynman measure does not, see Appendix.

Actually the basic formula (8) is common to all the derivations we review in
this article.

We recall how Rosen’s path integral representation is derived. Because of
linearity it suffices to show that $K$ satisfies the Hopf equation. For small
$\Delta t$ the Green’s function is proportional to

$\displaystyle\int\exp\left\\{i\int\left((\eta-\eta^{\prime})\cdot\zeta+\Delta
t\eta Q[\zeta]\right)dx\right\\}{\cal D}[\zeta]$ $\displaystyle=$
$\displaystyle\int\exp\left\\{i\int\zeta\cdot(\eta-\eta^{\prime})dx\right\\}\exp\left\\{i\Delta
t\int\eta(x)Q[\zeta]dx\right\\}{\cal D}[\zeta]$ $\displaystyle\approx$
$\displaystyle\int\left(1+i\Delta
t\int\eta(x)Q[\zeta]dx\right)\exp\left\\{i\int\zeta\cdot(\eta-\eta^{\prime})dx\right\\}{\cal
D}[\zeta]$ $\displaystyle=$ $\displaystyle\left(1+i\Delta t\int
dx\eta(x)Q\left[\frac{\delta}{i\delta\eta}\right]\right)\int\exp\left\\{i\int\zeta\cdot(\eta-\eta^{\prime})dx\right\\}{\cal
D}[\zeta],$

where we have written $\eta=\eta(x,t+\Delta t),\eta^{\prime}=\eta(x,t).$ Hence
we have by (7)

$K(t|t+\Delta t)=\left(1+i\Delta t\int
dx\eta(x)Q\left[\frac{\delta}{i\delta\eta}\right]\right)\underbrace{\delta[\eta-\eta^{\prime}]}_{=K(t|t)},$

that is,

$\frac{\partial K}{\partial
t}=i\int\eta(x)Q\left[\frac{\delta}{i\delta\eta}\right]Kdx.\;\square$

Out of the double functional integrations, the ${\cal D}[\zeta]$ integration
can be carried out by the method of stationary phase [21], but the ${\cal
D}[\eta]$ integration remains a challenge333If it were possible to carry out
the ${\cal D}[\eta]$ integration, the resultant functional would be termed the
Onsager-Machlup action in analogy with similar results for statistics of
linear dynamical systems. e.g. Section VI.9 of [24]. [26, 25].

For the particular case of the heat equation, at least we know

$C\iint_{\eta(x,0)=\theta_{0}(x)}^{\eta(x,t)=\theta(x)}\exp\left\\{i\int_{0}^{t}d\tau\int\zeta\left(\frac{\partial\eta}{\partial\tau}+\nu\frac{\partial^{2}\eta}{\partial
x^{2}}\right)dx\right\\}{\cal D}[\eta(x,\tau)]{\cal
D}[\zeta(x,\tau)]=\delta[\theta_{0}-\exp(\nu t\triangle)\theta]$

so that we recover $\Phi[\theta,t]=\Phi_{0}[\exp(\nu t\triangle)\theta].$ This
is a solution which represents the final period of decay in turbulence [1, 2].

###  3.2 Alternative derivation for Rosen’s path integral

Rosen’s original derivation of the characteristic functional based on the Hopf
equation is mostly straightforward. We recall that Rosen’s symbolic expression
can be obtained without using the Hopf equation, which is allegedly based on
the idea of Novikov, Section 29.5 of [25]. See also [27], p.252. The rationale
for including this is that the second derivation stands valid when we have
external forcing which is not necessarily white-in-time Gaussian. For
simplicity we illustrate it here using the unforced equation.

We formally rewrite $\frac{\partial u}{\partial t}=Q[u]$ as

$u(x,t)=u_{0}(x)+\int_{0}^{t}Q[u(x,\tau)]d\tau.$

In fact, this virtually trivial recasting leads to a nontrivial result.
Consider

$\exp\left(i\theta(x)\cdot
u(x,t)\right)=\exp\left\\{i\theta(x)\cdot\left(u_{0}(x)+\int_{0}^{t}Q[u(x,\tau)]d\tau\right)\right\\}$
$=\iint\exp\left\\{i\theta(x)\cdot\left(u_{0}(x)+\int_{0}^{t}Q[v(x,\tau)]d\tau\right)\right\\}\exp\left(i\int_{0}^{t}(u(x,\tau)-v(x,\tau))w(x,\tau)d\tau\right){\cal
D}[v]{\cal D}[w],$

by virtue of (8). Plugging
$u(x,\tau)=u_{0}(x)+\int_{0}^{\tau}Q[v(x,\tau^{\prime})]d\tau^{\prime}$ into
the above equation, we have

$\exp\left(i\theta(x)\cdot
u(x,t)\right)=\iint\exp\left\\{i\theta(x)\cdot\left(u_{0}(x)+\int_{0}^{t}Q[v(x,\tau)]d\tau\right)\right\\}$
$\times\exp\left\\{i\int_{0}^{t}\left(u_{0}(x)+\int_{0}^{\tau}Q[v(x,\tau^{\prime})]d\tau^{\prime}-v(x,\tau)\right)w(x,\tau)d\tau\right\\}{\cal
D}[v]{\cal D}[w].$

Integrating by parts we find

$\int_{0}^{t}d\tau\int_{0}^{\tau}Q[v(x,\tau^{\prime})]d\tau^{\prime}w(x,\tau)=\left[\int_{t}^{\tau}w(x,\tau^{\prime})d\tau^{\prime}\int_{0}^{\tau}Q[v(x,\tau^{\prime})]d\tau^{\prime}\right]_{\tau=0}^{t}-\int_{0}^{t}d\tau\int_{t}^{\tau}w(x,\tau^{\prime})d\tau^{\prime}Q[v(x,\tau)]$
$=+\int_{0}^{t}d\tau\int_{\tau}^{t}w(x,\tau^{\prime})d\tau^{\prime}Q[v(x,\tau)],$

and hence deduce

$e^{i(u,\theta)}=\iint\exp\left\\{iu_{0}(x)\cdot\left(\theta(x)+\int_{0}^{t}w(x,\tau^{\prime})d\tau^{\prime}\right)\right\\}\exp\left\\{i\int_{0}^{t}d\tau
Q[v]\cdot\left(\theta(x)+\int_{\tau}^{t}w(x,\tau^{\prime})d\tau^{\prime}\right)\right\\}$
$\times\exp\left\\{-i\int_{0}^{t}v(x,\tau)\cdot w(x,\tau)d\tau\right\\}{\cal
D}[v]{\cal D}[w].$

Averaging over $u_{0}$, we find

$\Phi[\theta(x),t]=\iint\Phi_{0}\left[\theta(x)+\int_{0}^{t}w(x,\tau^{\prime})d\tau^{\prime}\right]$
$\times\exp\left\\{i\int_{0}^{t}d\tau\left(Q[v]\cdot\left(\theta(x)+\int_{\tau}^{t}w(x,\tau^{\prime})d\tau^{\prime}\right)-v(x,\tau)\cdot
w(x,\tau)\right)\right\\}{\cal D}[v]{\cal D}[w].$

We rename $v(x,\tau)=\zeta(x,\tau)$ and
$\theta(x)+\int_{\tau}^{t}w(x,\tau^{\prime})d\tau^{\prime}=\eta(x,\tau)$ with
$\eta(x,t)=\theta(x),\eta(x,0)=\theta(x)+\int_{0}^{t}wd\tau^{\prime}$ and
$d\eta(x,\tau)=-w(x,\tau)d\tau.$ Replacing ${\cal D}[v]={\cal D}[\eta]$ and
${\cal D}[w]\propto{\cal D}[\eta]$ and subsuming extra factors into the
prefactor, we recover (6, 9) with the lower limit of integration unspecified.
$\square$

##  4 Stationary turbulence

###  4.1 Generalised Hopf equation

We now turn our attention to statistically steady turbulent flows driven by an
external forcing. The Burgers equation with the forcing reads

$\frac{\partial u}{\partial t}+u\frac{\partial u}{\partial
x}=\nu\frac{\partial^{2}u}{\partial x^{2}}+f(x,t).$

For simplicity let us consider a white-in-time Gaussian random force $f(x,t)$
[19], which satisfies

$\left<f(x,t)f(y,s)\right>=F(x-y)\delta(t-s).$

Here $F(r)$ denotes the correlation function and $\left<\cdot\right>$ a
statistical average. That way we can decouple the joint probability
distributions of the velocity and the forcing.

The Hopf equation was generalized to accommodate random forcing in [28]. This
is based on the following Furutsu-Novikov-Donsker theorem [28, 29, 30, 31],
which states for any functional $R$

$\left<f(x)R[f]\right>=\int F(x-x^{\prime})\left<\frac{\delta R[f]}{\delta
f(x^{\prime})dx^{\prime}}\right>dx^{\prime}.$ (10)

The generalised Hopf equation turned out to be

$\frac{\partial\Phi}{\partial
t}=\frac{i}{2}\int\theta(x)\frac{\partial}{\partial
x}\frac{\delta^{2}\Phi}{\delta\theta(x)dx^{2}}dx+\nu\int\theta(x)\frac{\partial^{2}}{\partial
x^{2}}\frac{\delta\Phi}{\delta\theta(x)dx}dx-\frac{1}{2}\iint\theta(x)F(x-y)\theta(y)dxdy\,\Phi.$
(11)

For the Hopf equation subject to more general forcing, see [32].

Just like the original Hopf equation, it seems that this equation has not been
put to practical use. Here let us see how we may modify Rosen’s functional
expression to accommodate the forcing and verify that it satisfies the
generalized Hopf equation. To that end the following paragraph by Uriel Frisch
would be instructive, which we quote here verbatim from [33].

We consider first the randomly perturbed heat equation

$\frac{\partial}{\partial
t}\Psi(t,\bm{r})=\triangle\Psi+\mu(\bm{r})\Psi,\;\;\Psi(0,\bm{r})=\delta(\bm{r}),$
(3.44)

where $\mu(\bm{r})$ is a centered random Gaussian function with covariance

$\Gamma(\bm{r},\bm{r}^{\prime})={\cal
E}\left\\{\mu(r)\mu(r^{\prime})\right\\}.$ (3.45)

For any realization of $\mu(\bm{r}),$ Eq.(3.44) can be solved by means of
Kac’s formula [cf. Appendix Eq. (A12)]

$\Psi(t,\bm{r})={\cal
E}_{W}\left\\{\delta(\bm{\rho}(t)-\bm{r})\exp\left\\{\int_{0}^{t}\mu(\bm{\rho}(\tau))d\tau\right\\}\right\\}.$
(3.46)

To find ${\cal E}_{\mu}(\Psi(t,\bm{r}))$ we must average again over $\mu$.
Assuming that the $\mu$ and $W$ averages can be interchanged, we obtain

${\cal E}_{\mu}\\{\Psi(t,\bm{r})\\}={\cal
E}_{W}\left\\{\delta(\bm{\rho}(t)-\bm{r}){\cal
E}_{\mu}\left\\{\exp\left\\{\int_{0}^{t}\mu(\bm{\rho}(\tau))d\tau\right\\}\right\\}\right\\}.$
(3.47)

For a fixed Brownian path $\bm{\rho}(\tau),\;0\leq\tau\leq t$, the random
variable

$\phi=\int_{0}^{t}\mu(\bm{\rho}(\tau))d\tau$ (3.48)

is a linear functional of the Gaussian random function $\mu$; hence

${\cal E}_{\mu}\\{\exp\phi\\}=\exp\left\\{\frac{1}{2}{\cal
E}_{\mu}\\{\phi^{2}\\}\right\\}=\exp\left\\{\frac{1}{2}\int_{0}^{t}\int_{0}^{t}\Gamma(\bm{\rho}(\tau),\bm{\rho}(\tau^{\prime}))d\tau
d\tau^{\prime}\right\\}.$ (3.49)

Eq.(3.47) can now be rewritten as

${\cal E}_{\mu}\\{\Psi(t,\bm{r})\\}={\cal
E}_{W}\left\\{\delta(\bm{\rho}(t)-\bm{r})\exp\left\\{\frac{1}{2}\int_{0}^{t}\int_{0}^{t}\Gamma(\bm{\rho}(\tau),\bm{\rho}(\tau^{\prime}))d\tau
d\tau^{\prime}\right\\}\right\\}.$ (3.50)

###  4.2 Linking Rosen’s action integral to Wyld’s

Note that the action integral for turbulence driven by Gaussian white forcing
is called Wyld functional[34] and appears in the so-called Martin-Siggia-Rose
formalism (hereafter, MSR) [35]. The formalism builds a field theory
associated with statistical mechanics of a given dynamical system. That
appears in a now standard field-theoretic treatment of turbulence,
particularly instanton theory. Such an action integral is sometimes called
after the names of Wyld444The explicit functional form, however, does not
appear explicitly in [36]. and Rosen e.g. [37]. The link is well expected in
the folklore, but best described here explicitly with some justifications.

Before that it is suitable to refer some cornerstone papers on the theory of
turbulence. The space-time characteristic functional was introduced in [38],
the path integral representation for the MSR theory was derived in [39, 40]
and a connection to the Kolmogorov theory was sought in [41]. See also Section
9.5 of [42] for a succinct survey.

We will see that (i) the symbolic expression associated with Rosen’s action
integral subject to random forcing yields with the MSR action and (ii) it is
indeed a solution of Novikov’s generalised Hopf equation. To include forcing
we replace RHS of the Burgers equation $Q[\zeta]\rightarrow Q[\zeta]+f(x,t)$
where $Q[\zeta]=-\zeta\zeta_{x}+\nu\zeta_{xx},$ for each realization of
$f(x,t)$, that is, we regard $f(x,t)$ as deterministic at this stage. We have

$K[\theta_{0}(x)|\theta(x),t]$
$=C\iint_{\eta(x,0)=\theta_{0}(x)}^{\eta(x,t)=\theta(x)}\exp\left\\{i\int_{0}^{t}d\tau\int
dx\,\left(\frac{\partial\eta}{\partial\tau}\zeta+\eta
Q[\zeta]\right)\right\\}\exp\left\\{i\int_{0}^{t}d\tau\int\eta(x)f(x)dx\,\right\\}{\cal
D}[\eta(x,\tau)]{\cal D}[\zeta(x,\tau)].$

Randomise $f(x,t)$ and average over ${\cal D}[f(x,\tau)]$ with the weight
$\exp\left\\{-\frac{1}{2}\int_{0}^{t}d\tau\iint
f(x)\kappa(x-y)f(y)dxdy\right\\}$ where $\kappa(x,y)=\kappa(x-y)$ is an
inverse of $F(x,y)=F(x-y).$ We find

$K=C\iint_{\eta(x,0)=\theta_{0}(x)}^{\eta(x,t)=\theta(x)}\exp\left\\{i\int_{0}^{t}d\tau\int
dx\,\left(\frac{\partial\eta}{\partial\tau}\zeta+\eta
Q[\zeta]\right)-\frac{1}{2}\int_{0}^{t}d\tau\iint\eta(x)F(x-y)\eta(y)dxdy\right\\}{\cal
D}[\eta]{\cal D}[\zeta].$

It can be seen to be equivalent to the Wyld functional in the MSR scheme.
First, integrating by parts under the constraints $\zeta(x,0)=\zeta(x,t)=0$,
we have

$K=C\iint_{\eta(x,0)=\theta_{0}(x)}^{\eta(x,t)=\theta(x)}\exp\left\\{-i\int_{0}^{t}d\tau\int
dx\,\eta\left(\frac{\partial\zeta}{\partial\tau}-Q[\zeta]\right)-\frac{1}{2}\int_{0}^{t}d\tau\iint\eta(x)F(x-y)\eta(y)dxdy\right\\}{\cal
D}[\eta]{\cal D}[\zeta].$

Second, to verify that the expression defined with the above kernel satisfies
the generalised Hopf equation we note

$\frac{\partial}{\partial
t}\exp\left\\{-\frac{1}{2}\int_{0}^{t}d\tau\iint\eta(x,\tau)F(x-y)\eta(y,\tau)dxdy\right\\}$
$=-\frac{1}{2}\iint\eta(x,t)F(x-y)\eta(y,t)dxdy\exp\left\\{-\frac{1}{2}\int_{0}^{t}d\tau\iint\eta(x)F(x-y)\eta(y)dxdy\right\\}.$

The additional contribution to the RHS of the Hopf equation is

$-\frac{C}{2}\iint_{\eta(x,0)=\theta_{0}(x)}^{\eta(x,t)=\theta(x)}\iint\left.\eta(x,t)F(x-y)\eta(y,t)dxdy\right|_{\eta(\cdot,t)=\theta(\cdot)}$
$\times\exp\left\\{-i\int_{0}^{t}d\tau\int
dx\,\eta\left(\frac{\partial\zeta}{\partial\tau}-Q[\zeta]\right)-\frac{1}{2}\int_{0}^{t}d\tau\iint\eta(x)F(x-y)\eta(y)dxdy\right\\}{\cal
D}[\eta]{\cal D}[\zeta].$ $=-\frac{1}{2}\iint\theta(x)F(x-y)\theta(y)dxdy$
$\times
C\iint_{\eta(x,0)=\theta_{0}(x)}^{\eta(x,t)=\theta(x)}\exp\left\\{-i\int_{0}^{t}d\tau\int
dx\,\eta\left(\frac{\partial\zeta}{\partial\tau}-Q[\zeta]\right)-\frac{1}{2}\int_{0}^{t}d\tau\iint\eta(x)F(x-y)\eta(y)dxdy\right\\}{\cal
D}[\eta]{\cal D}[\zeta],$

which indeed yields the final term on the RHS of Novikov’s equation (11).
$\square$

###  4.3 Instanton theory

The functional integrations remain merely symbolic. However, the action
integral that appears in the exponent is just $(n+1)$-dimensional integral and
well-defined $(n=1,2,3)$. Hence it is sensible to consider stationary
conditions by writing down the Euler-Lagrange equations. For Rosen’s action, a
pair of equations consists of the Burgers equation and its adjoint (dual)

$\frac{\partial v}{\partial t}+u\frac{\partial v}{\partial
x}=-\nu\frac{\partial^{2}v}{\partial x^{2}},$

which is linear and runs backward in time. Statistical fluid mechanics has
inherently control-theoretic aspects built-in [45].

Here we will have a look at standard manipulations in instanton theory,
following [43, 44]. See also [45, 46]for the numerical implementations,
including the Navier-Stokes equations.

Consider a formal normalisation ${\cal N}=\int\delta(u_{t}+uu_{x}-\nu
u_{xx}-f){\cal D}[u]$ and the partition function

$Z=\int\exp\left(-\frac{1}{2}\iiint f(x,t)\kappa(x-y)f(y,t)dxdydt\right){\cal
D}[f]$ $=\frac{1}{\cal N}\iint\delta(u_{t}+uu_{x}-\nu
u_{xx}-f)\exp\left(-\frac{1}{2}\iiint
f(x,t)\kappa(x-y)f(y,t)dxdydt\right){\cal D}[f]{\cal D}[u].$

We rewrite this by (8), redefining the normalisation constant

$Z=\frac{1}{\cal N}\iiint\exp\left(i\iint v(u_{t}+uu_{x}-\nu
u_{xx}-f)dxdt-\frac{1}{2}\iiint f(x,t)\kappa(x-y)f(y,t)dxdydt\right){\cal
D}[f]{\cal D}[u]{\cal D}[v]$

$=\frac{1}{\cal N}\iiint\exp\left(i\iint v(u_{t}+uu_{x}-\nu
u_{xx})dxdt\right)$

$\times\exp\left(-i\iint vfdxdt-\frac{1}{2}\iiint
f(x,t)\kappa(x-y)f(y,t)dxdydt\right){\cal D}[f]{\cal D}[u]{\cal D}[v]$
$=\frac{1}{\cal N}\iint\exp\left(i\iint v(u_{t}+uu_{x}-\nu
u_{xx})dxdt\right)\exp\left(-\frac{1}{2}\iiint
v(x,t)F(x-y)v(y,t)dxdydt\right){\cal D}[u]{\cal D}[v],$

where a functional Fourier transform of the Gaussian function has been taken
in the final line.

###  4.4 Yet another form of path integral

A simple conversion of the Burgers equation into an integral equation was
considered in 3(b). By virtue of Duhamel principle we can also convert the
Burgers equation into another integral equation of the following form

$u-G:uu={\hat{u}},$

where ${\hat{u}}=g*u_{0}$ is a heat flow defined with the heat kernel
$g(x,t)=\dfrac{1}{(4\pi\nu t)^{1/2}}\exp\left(-\dfrac{x^{2}}{4\nu t}\right).$
We have also denoted

$G:uu\equiv\int_{0}^{t}ds\int_{-\infty}^{\infty}dyG(x-y,t-s)u(y,s)^{2},$

where $G(x,t)\equiv-\frac{1}{2}\partial_{x}g(x,t).$ On this basis the
following representation can be derived as in [47, 48]

$\Phi[\theta]=\iint\exp\left(i\int\left\\{(\theta-v)\cdot u+v\cdot
G:uu\right\\}dx-A[u]\right){\hat{\Phi}}[v]{\cal D}[v]J{\cal D}[u],$ (12)

where $A[u]$ denotes an unknown functional to account for possible breakdown
of classical solutions and $J={\rm det}\left(\frac{\delta{\hat{u}}}{\delta
u}\right).$

In fact we can confirm that $J\equiv 1$ as in the 3D incompressible case. We
also have $A[u]\equiv 0$ because the Burgers equation is globally well-posed.
Hence we find

$\Phi[\theta]=\iint\exp\left(i\int\left\\{(\theta-v)\cdot u+v\cdot
G:uu\right\\}dx\right){\hat{\Phi}}[v]{\cal D}[v]{\cal D}[u].$ (13)

This formula connects the characteristic functional ${\hat{\Phi}}[v]$ of the
heat equation with that of the Burgers equation $\Phi[\theta]$. If we discard
the second term of the exponent, it is clear that we recover
$\Phi[\theta]={\hat{\Phi}}[v]$ by the property of the Dirac measure (8). This
form of path integral is valid only for a viscous fluid, in contrast to the
previous one which can cover an inviscid fluid as well. It is of interest to
study whether and how the above expression is simplified further when we make
use of the Cole-Hopf transform which linearises the Burgers equation.

##  5 Hopf equation for a discrete Burgers equation

As seen above the Hopf functional equation faces a daunting mathematical
difficulty because of the infinite-dimensional character of functional
differential equation. To alleviate the difficulty at least partially, we
consider a discrete Burgers equation in this section so that its ’Hopf
equation’ is a PDE rather than an FDE.

There are some works in the similar spirit [49, 50, 51]. The master equation
was presented for a discrete Burgers equation, but that does not respect Cole-
Hopf transform. See also [52]. Here we take up a discretisation of Burgers
equation which respects the Cole-Hopf transformation and present asymptotic
analysis using a series expansion in time.

###  5.1 Discrete Burgers equation

The following version of discretisation was proposed in [53] for the Burgers
equation.

$\frac{du(x,t)}{dt}=-\frac{2\nu^{2}}{\epsilon^{2}}\Delta_{x}\left\\{\exp\left(-\frac{\epsilon}{2\nu}u(x,t)\right)+\exp\left(\frac{\epsilon}{2\nu}u(x-\epsilon,t)\right)-2\right\\}$
(14)
$=-\frac{2\nu^{2}}{\epsilon^{3}}\left\\{\exp\left(-\frac{\epsilon}{2\nu}u(x+\epsilon,t)\right)-\exp\left(-\frac{\epsilon}{2\nu}u(x,t)\right)+\exp\left(\frac{\epsilon}{2\nu}u(x,t)\right)-\exp\left(\frac{\epsilon}{2\nu}u(x-\epsilon,t)\right)\right\\},$

where $\epsilon$ denotes spatial spacing and
$\Delta_{x}f(x)=(f(x+\epsilon)-f(x))/\epsilon.$ Writing $u_{n}(t)=u(x_{n},t)$
with $x_{n}=n\epsilon,\;(N=1,\ldots,N),$ it can also be written

$\frac{du_{n}}{dt}=-\frac{2\nu^{2}}{\epsilon^{3}}\Delta_{n}\left\\{\exp\left(-\frac{\epsilon}{2\nu}u_{n}\right)+\exp\left(\frac{\epsilon}{2\nu}u_{n-1}\right)-2\right\\},$
(15)

where $\Delta_{n}f_{n}=f_{n+1}-f_{n}$ and periodic boundary conditions
$f_{N}=f_{0},f_{N+1}=f_{1}$ are assumed.

###  5.2 Its Hopf equation

Consider the characteristic function $\Phi[\theta_{n}]=\left<\exp
i\sum_{n=1}^{N}u_{n}(t)\theta_{n}\right>,$ then its governing equation can be
derived as

$\frac{\partial\Phi}{\partial
t}=-\frac{2\nu^{2}i}{\epsilon^{3}}\sum_{n=1}^{N}\theta_{n}\Delta_{n}\left(\Phi\left[\theta_{1},\ldots,\theta_{n}+\frac{\epsilon
i}{2\nu},\ldots,\theta_{N}\right]+\Phi\left[\theta_{1},\ldots,\theta_{n-1}-\frac{\epsilon
i}{2\nu},\ldots,\theta_{N}\right]-2\Phi\left[\theta_{1},\ldots,\theta_{N}\right]\right).$
(16)

It is readily checked that (16) reduces to (3) in the limit of small
$\epsilon$. Using exponential operators we can recast it as

$\frac{\partial\Phi}{\partial
t}=-\frac{2\nu^{2}i}{\epsilon^{3}}\sum_{n=1}^{N}\theta_{n}\Delta_{n}\left\\{\exp\left(\frac{\epsilon
i}{2\nu}\partial_{n}\right)+\exp\left(-\frac{\epsilon
i}{2\nu}\partial_{n-1}\right)-2\right\\}\Phi,$
$=\frac{2\nu^{2}i}{\epsilon^{2}}\sum_{n=1}^{N}(\widetilde{\Delta_{n}}\theta_{n-1})\left\\{\exp\left(\frac{\epsilon
i}{2\nu}\partial_{n}\right)+\exp\left(-\frac{\epsilon
i}{2\nu}\partial_{n-1}\right)-2\right\\}\Phi,$

where the final line follows from ’summation by parts’ and
$\widetilde{\Delta_{n}}f_{n}=\frac{1}{\epsilon}\Delta_{n}f_{n},\,\theta_{0}=\theta_{N},\theta_{N+1}=\theta_{1}$
(by periodicity). We can then write its formal solution

$\Phi=\exp\left[\frac{2\nu^{2}it}{\epsilon^{2}}\sum_{n=1}^{N}(\widetilde{\Delta_{n}}\theta_{n-1})\left\\{\exp\left(\frac{\epsilon
i}{2\nu}\partial_{n}\right)+\exp\left(-\frac{\epsilon
i}{2\nu}\partial_{n-1}\right)-2\right\\}\right]\Phi_{0}[\theta_{1},\ldots,\theta_{N}].$
(17)

Expanding in a power series in time, we have

$\Phi=\sum_{k=0}^{\infty}\frac{1}{k!}\left(\frac{2\nu^{2}it}{\epsilon^{2}}\right)^{k}\left[\sum_{n=1}^{N}(\widetilde{\Delta_{n}}\theta_{n-1})\left\\{\exp\left(\frac{\epsilon
i}{2\nu}\partial_{n}\right)+\exp\left(-\frac{\epsilon
i}{2\nu}\partial_{n-1}\right)-2\right\\}\right]^{k}\Phi_{0}[\theta_{1},\ldots,\theta_{N}].$

These operators are non-commutative, e.g.

$\exp\left(\frac{\epsilon
i}{2\nu}\partial_{n}\right)(\theta_{n}-\theta_{n-1})=(\theta_{n}-\theta_{n-1})\exp\left(\frac{\epsilon
i}{2\nu}\partial_{n}\right)+\frac{\epsilon i}{2\nu},$

but the non-commutativity appears in higher order in $\epsilon$. Hence, by
multinomial expansion, we find to leading order

$[\ldots]^{k}\Phi_{0}\approx\sum_{\sum_{j=1}^{N}k_{j}=k,\,k_{j}\geq
0}\frac{k!}{k_{1}!k_{2}!\ldots
k_{N}!}\Pi_{j=1}^{N}\left(\widetilde{\Delta_{j}}\theta_{j-1}\right)^{k_{j}}\left\\{\exp\left(\frac{\epsilon
i}{2\nu}\partial_{j}\right)+\exp\left(-\frac{\epsilon
i}{2\nu}\partial_{j-1}\right)-2\right\\}^{k_{j}}\Phi_{0},$

Again, by multinomial expansion

$\left\\{\ldots\right\\}^{k_{j}}\Phi_{0}=\sum_{\sum_{j=1}^{3}m_{j}=m,\;m_{j}\geq
0}\frac{k_{j}!}{m_{1}!m_{2}!m_{3}!}\exp\left(\frac{i\epsilon}{2\nu}m_{1}\partial_{j}\right)\exp\left(-\frac{i\epsilon}{2\nu}m_{2}\partial_{j-1}\right)(-2)^{m_{3}}\Phi_{0}[\theta_{1},\ldots,\theta_{N}]$
$=\sum_{\sum_{j=1}^{3}m_{j}=m,\;m_{j}\geq
0}\frac{k_{j}!(-2)^{m_{3}}}{m_{1}!m_{2}!m_{3}!}\Phi_{0}\left[\theta_{1},\ldots,\theta_{j-1}-\frac{i\epsilon}{2\nu}m_{2},\theta_{j}+\frac{i\epsilon}{2\nu}m_{1},\ldots,\theta_{N}\right].$

All together we obtain

$\Phi\approx\sum_{k=0}^{\infty}\frac{1}{k!}\left(\frac{2\nu^{2}it}{\epsilon^{2}}\right)^{k}\sum_{\sum_{j=1}^{N}k_{j}=k,\,k_{j}\geq
0}\frac{k!}{k_{1}!k_{2}!\ldots
k_{N}!}\Pi_{j=1}^{N}\left(\widetilde{\Delta_{j}}\theta_{j-1}\right)^{k_{j}}$
$\times\sum_{\sum_{j=1}^{3}m_{j}=m,\;m_{j}\geq
0}\frac{k_{j}!(-2)^{m_{3}}}{m_{1}!m_{2}!m_{3}!}\Phi_{0}\left[\theta_{1},\ldots,\theta_{j-1}-\frac{i\epsilon}{2\nu}m_{2},\theta_{j}+\frac{i\epsilon}{2\nu}m_{1},\ldots,\theta_{N}\right].$

Note that in the above expressions $j$-dependence of $m_{1},m_{2},m_{3}$ has
been suppressed for simplicity.

##  6 Summary

After making a note on the Koopman (and Weil) operator, in this survey we
reviewed some functional integration methods for handling the Hopf equation
for turbulence. We have seen a number of different ways of path integral
representations for both decaying turbulence (with Rosen’s action) and forced
turbulence (with Wyld’s action). We have also seen how those two actions are
related via a standard technique known to researchers on wave propagation, as
evidenced by Uriel’s paragraph. The technique is also useful in modern
instanton theory. Common to all those derivations lies the plane wave
expansion of the Dirac delta measure, which is essentially a tool in linear
theory (Fourier analysis).

In the final part we discuss a particular finite-dimensional approximation for
the Burgers equation and construct approximate solutions to its Hopf
characteristic function.

For recent references on statistical theory of turbulence from functional
integration viewpoints, it may be in order to consult e.g. [56, 27, 54, 55,
57]. For the developments of turbulence theory in general, the following
references are also of interest [63, 64].

A very happy birthday, Uriel and best wishes for many more!

##  Appendix A Non-existence of Feynman Measure

This counter-example appeared in the introduction section in [58]. The
following exposition is based on [59]. See also Problem 64 of Chapter X in
[60], [61], or Chapter 6 and Appendix K in [62].

All we need to know is the convolution of two Gaussian distributions

$p_{1}*p_{2}(x)=\frac{1}{\sqrt{2\pi(\sigma_{1}^{2}+\sigma_{2}^{2})}}\exp\left(-\frac{x^{2}}{2(\sigma_{1}^{2}+\sigma_{2}^{2})}\right)$

for

$p_{1}(x)=\frac{1}{\sqrt{2\pi\sigma_{1}^{2}}}\exp\left(-\frac{x^{2}}{2\sigma_{1}^{2}}\right),p_{2}(x)=\frac{1}{\sqrt{2\pi\sigma_{2}^{2}}}\exp\left(-\frac{x^{2}}{2\sigma_{2}^{2}}\right),$

where $\sigma_{1},\sigma_{2}>0.$ For positive $\lambda$ we compute the
$N$-fold convolution products for $(N+1)$ functions by repeated use of it,

$I=\left(\frac{\lambda}{2\pi\epsilon}\right)^{\frac{N+1}{2}}\int\ldots\int\exp\left(-\frac{\lambda}{2\epsilon}\sum_{l=0}^{N}(x_{l+1}-x_{l})^{2}\right)\Pi_{l=1}^{N}dx_{l}=\left(\frac{\lambda}{2\pi
N\epsilon}\right)^{\frac{1}{2}}\exp\left(-\frac{\lambda}{2N\epsilon}(x_{N+1}-x_{0})^{2}\right).$

This also holds for complex-valued $\lambda$ provided that $\Re(\lambda)>0$.
If the above formula remains valid in the limit of $N\to\infty$ while
$N\epsilon$=const, the path integral does make sense. Because absolute
convergence implies the existence of the integral, we are led to estimate the
majorant

$|I|\leq\left(\frac{|\lambda|}{2\pi\epsilon}\right)^{\frac{N+1}{2}}\underbrace{\int\ldots\int\exp\left(-\frac{\Re(\lambda)}{2\epsilon}\sum_{l=0}^{N}(x_{l+1}-x_{l})^{2}\right)\Pi_{l=1}^{N}dx_{l}}_{=J,\;\mbox{say.}}.$

We have

$J=\left(\frac{2\pi\epsilon}{\Re(\lambda)}\right)^{\frac{N+1}{2}}\left(\frac{\Re(\lambda)}{2\pi
N\epsilon}\right)^{\frac{1}{2}}\exp\left(-\frac{\Re(\lambda)}{2N\epsilon}(x_{N+1}-x_{0})^{2}\right)$

by an identity obtained by $\lambda\to\Re(\lambda)$ in $I$ above, and hence

$\left(\frac{\Re(\lambda)}{2\pi\epsilon}\right)^{\frac{N+1}{2}}\int\ldots\int\exp\left(-\frac{\Re(\lambda)}{2\epsilon}\sum_{l=0}^{N}(x_{l+1}-x_{l})^{2}\right)\Pi_{l=1}^{N}dx_{l}=\left(\frac{\Re(\lambda)}{2\pi
N\epsilon}\right)^{\frac{1}{2}}\exp\left(-\frac{\Re(\lambda)}{2N\epsilon}(x_{N+1}-x_{0})^{2}\right).$

Calibrating the prefactors, we find

$|I|\leq\left(\frac{|\lambda|}{2\pi\epsilon}\right)^{\frac{N+1}{2}}\left(\frac{2\pi\epsilon}{\Re(\lambda)}\right)^{\frac{N+1}{2}}\left(\frac{\Re(\lambda)}{2\pi
N\epsilon}\right)^{\frac{1}{2}}\exp\left(-\frac{\Re(\lambda)}{2N\epsilon}(x_{N+1}-x_{0})^{2}\right)$
$=\left(\frac{|\lambda|}{\Re(\lambda)}\right)^{\frac{N}{2}}\left(\frac{|\lambda|}{2\pi
N\epsilon}\right)^{\frac{1}{2}}\exp\left(-\frac{\Re(\lambda)}{2N\epsilon}(x_{N+1}-x_{0})^{2}\right).$

When $\Im(\lambda)\neq 0,$ we have $\frac{|\lambda|}{\Re(\lambda)}>1.$
Therefore the final line diverges in the limit of $N\to\infty$ with
$N\epsilon$ held fixed.

## References

  * [1] Hopf E. 1952 Statistical hydromechanics and functional calculus. J. Rat. Mech. Anal. 1 87-123.
  * [2] Hopf E. and Titt EW. 1953 On certain special solutions of the $\Phi$-equation of statistical hydrodynamics. J. Rat. Mech. Anal. 2 587-591.
  * [3] von Neumann J. 1929 Beweis des Ergodensatzes und des H-Theorems in der neuen Mechanik. Zeit. Phys. 57 30-70.
  * [4] Birkhoff GD. 1931. Proof of the ergodic theorem. Proc. Nat. Acad. Sci. 17 656-660.
  * [5] Hopf E. 1937. Ergodentheorie. Berlin: Springer.
  * [6] Toda M., Kubo R. & Saito N. 2013. Statistical Physics I: Equilibrium Statistical Mechanics. Berlin: Springer.
  * [7] Koopman BO.1931 Hamiltonian systems and transformation in Hilbert space. Proc. Nat. Acad. Sci. 17 315\.
  * [8] Singh RK. and Manhas JS. 2012. Composition Operators on Function Spaces. Amsterdam: North Holland.
  * [9] Weil A. 1932. On systems of curves on a ring-shaped surface. J. Ind. Math. Soc. 19 109–114. Also in André Weil Oeuvres Scientifiques Collected Papers Volume 1: 57–62 & Commentaire (in French) p.522. 1979, Berlin: Springer-Verlag.
  * [10] Rédei M. (ed.) 2005 John Von Neumann: Selected Letters Rhode Island: American Mathematical Society.
  * [11] Applebaum D. 2019. Semigroups of Linear Operators: With Applications to Analysis, Probability and Physics. Cambridge: Cambridge University Press.
  * [12] Foias C. 1972 Statistical study of Navier-Stokes equations I. Rend. Sem. Mat. Univ. Padova 8 219–348.
  * [13] Foias C. 1973 Statistical study of Navier-Stokes equations II. Rend. Sem. Mat. Univ. Padova 49 9–123.
  * [14] Vishik MI. and Fursikov AV. 1988 Mathematical problems of statistical hydromechanics. Dordrecht and Boston: Kluwer.
  * [15] Mezić I. 2013. Analysis of fluid flows via spectral properties of the Koopman operator. Ann. Rev. Fluid Mech. 45 357–378.
  * [16] Rowley CW. & Dawson ST. 2017. Model reduction for flow analysis and control. Ann. Rev. Fluid Mech. 49 387–417.
  * [17] Page J. and Kerswell RR. 2018. Koopman analysis of Burgers equation. Phys. Rev. Fluids 3 071901\.
  * [18] Foias C. 1974 A functional approach to turbulence. Russ. Math. Surv. 29 293\.
  * [19] Edwards SF. 1964 The statistical dynamics of homogeneous turbulence. J. Fluid Mech. 18 239-273.
  * [20] Edwards SF. 1964 A new method of solution for quantum field theory and associated problems. in Proceedings of a conference on the theory and applications of analysis in function space held at Endicott House in Dedham, Massachusetts, June 9-13, 1963, eds. W.T. Martin and I.E. Segal. MIT Press: Cambridge. 31-50
  * [21] Rosen G. 1960 Turbulence Theory and Functional Integration. I. and II. Phys. Fluids 3 519-524 and 525-528.
  * [22] Rosen G. 1969 Formulation of Classical and Quantum Dynamical Theory. New York and London: Academic Press.
  * [23] Novikov EA. 1961 The solution of certain equations with variational derivatives. (in Russian) Uspekhi Mat. Nauk 16 135–141.
  * [24] Ikeda N. and Watanabe S. 2014 Stochastic differential equations and diffusion processes. Amsterdam: Elsevier.
  * [25] Monin AS. and Yaglom AM. 1975 Statistical fluid dynamics: mechanics of turbulence vol.2 MIT Press.
  * [26] Tatarskii VI. 1962 Application of the methods of quantum field theory to the problem of degeneration of homogeneous turbulence. Sov. Phys. JETP 15 961–967.
  * [27] Klyatskin VI. 2015 Stochastic Equations: Theory and Applications in Acoustics, Hydrodynamics, Magnetohydrodynamics, and Radiophysics, Volume 1 & 2. Berlin: Springer.
  * [28] Novikov EA. 1965 Functionals and the random force method in turbulence theory. Sov. Phys. JETP 20 1290–4.
  * [29] Furutsu K. 1963 On the statistical theory of electromagnetic waves in a fluctuating medium: I. J. Res. Natl. Bur. Stand. 67D 303–23.
  * [30] Donsker MD. 1964 On function space integrals. in Proceedings of a conference on the theory and applications of analysis in function space held at Endicott House in Dedham, Massachusetts, June 9-13, 1963, eds. W.T. Martin and I.E. Segal. MIT Press: Cambridge. 17-30.
  * [31] Donsker MD. and Lions JL. 1996 Frechet-Volterra variational equations, boundary value problems, and function space integrals. Acta Math. 108 147-228.
  * [32] Hosokawa I. 1968 A functional treatise on statistical hydromechanics with random force action. J. Phys. Soc. Jpn. 25 271-278.
  * [33] Frisch U. 1968 Wave propagation in random media in Probabilistic Methods in Applied Mathematics: volume 1. A.T. Bharucha-Reid (ed.) New York and London: Academic Press.
  * [34] Zinn-Justin J. 2021. Quantum Field Theory and Critical Phenomena, 5th edition. Oxford: Oxford University Press.
  * [35] Martin PC., Siggia ED. and Rose HA., 1973. Statistical dynamics of classical systems. Phys. Rev. A 8 423–437.
  * [36] Wyld Jr HW. 1961. Formulation of the theory of turbulence in an incompressible fluid. Ann. Phys. 14 143-165.
  * [37] Botelho LC. and da Silva EP 1998 The Caldirola-Kanai Theory from ”Brownian” Path Integral Quantum Mechanics. Mod. Phys. Lett. B 12 569–573.
  * [38] Lewis RM. and Kraichnan RH. 1962 A space-time functional formalism for turbulence. Commun. Pure Appl. Math. 15 397–411.
  * [39] de Dominicis C. 1976 Techniques de renormalisation de la théorie des champs et dynamique des phénomenès critiques. (in French) J. Phys. (Paris), Colloq 1 247–253.
  * [40] Janssen H-K. 1976 On a Lagrangean for classical field dynamics and renormalization group calculations of dynamical critical properties. Z. für Physik B 23 377–380.
  * [41] de Dominicis C. and Martin PC. 1979 Energy spectra of certain randomly-stirred fluids. Phys. Rev. A 19 419–422.
  * [42] Frisch U. 1995 Turbulence: the legacy of AN Kolmogorov Cambridge: Cambridge University Press.
  * [43] Gurarie V. and Migdal A. 1996 Instantons in the Burgers equation. Phys. Rev. E 54 4908\.
  * [44] Falkovich G., Kolokolov I., Lebedev V. and Migdal A. 1996 Instantons and intermittency. Phys. Rev. E 54 4896\.
  * [45] Grafke T., Grauer R. and Schäfer T. 2015. The instanton method and its numerical implementation in fluid mechanics. J. Phys. A: Math. and Theor. 48 333001\.
  * [46] Grafke T., Grauer R., Schäfer T. and Vanden-Eijnden, E. 2015. Relevance of instantons in Burgers turbulence. Europhys. Lett. 109 34003\.
  * [47] Rosen G. 1971. Functional calculus theory for incompressible fluid turbulence. J. Math. Phys. 12 812–820.
  * [48] Testa FJ. and Gerald R. 1974 Theory for incompressible fluid turbulence. J. Franklin Inst. 297 127–133.
  * [49] Breuer HP. and Petruccione F. 1993 Burgers’s turbulence model as a stochastic dynamical system: Master equation and simulation. Phys. Rev. E 47 83\.
  * [50] Breuer HP. and Petruccione F. 1996 Mesoscopic modelling and stochastic simulations of turbulent flows. in Nonlinear Stochastic PDEs: Hydrodynamic Limit and Burgers’ Turbulence (ed.) T. Funaki, W.A. Woyczynski, Berlin: Springer.
  * [51] Breuer HP., Petruccione F. and Weber F. 1996 On a Fourier space master equation for Navier-Stokes turbulence. Z. fur Phys. B 100 461-468.
  * [52] Kollmann W. 1976 Functional integrals in the statistical theory of turbulence and the burgers model equation. J. Stat. Phys. 14 291-303.
  * [53] Hirota R. 1979 Nonlinear Partial Difference Equations. V. Nonlinear Equations Reducible to Linear Equations. J. Phys. Soc. Jpn. 46 312–319.
  * [54] Kollmann W. 2019 Navier-Stokes Turbulence: Theory and Analysis. Berlin: Springer.
  * [55] Venturi D. 2018 The numerical approximation of nonlinear functionals and functional differential equations. Phys. Rep. 732 1-102.
  * [56] Botelho LCL. 2008 Lecture Notes In Applied Differential Equations Of Mathematical Physics. Singapore: World Scientific.
  * [57] Ohkitani K. 2020 Study of the Hopf functional equation for turbulence: Duhamel principle and dynamical scaling. Phys. Rev. E 101 013104-1–15.
  * [58] Cameron RH. 1960 A family of integrals serving to connect the Wiener and Feynman integrals. J. Math. Phys. 39 126–140.
  * [59] Klauder KR. 2003 The Feynman path integral: An historical slice. in A Garden of Quanta: Essays in Honor of Hiroshi Ezawa 55-76.
  * [60] Reed M. and Simon B. 1975 Methods of Modern Mathematical Physics II: Fourier Analysis, Self-Adjointness New York and London: Academic Press.
  * [61] Albeverio S. 1997 Wiener and Feynman-path integrals and their applications. In Proceedings of Symposia in Applied Mathematics, Vol. 52, pp. 163-194. Rhode Island: American Mathematical Society.
  * [62] Strocchi F. 2008 An Introduction to the Mathematical Structure of Quantum Mechanics: A Short Course for Mathematicians. Singapore: World Scientific.
  * [63] Sreenivasan KR. and Eyink GL. 2005 Sam Edwards and the turbulence theory. in Stealing the gold: A celebration of the pioneering physics of Sam Edwards 66-85.
  * [64] Eyink G. and Frisch U. 2011 Robert H. Kraichnan. in A voyage through turbulence. (eds.) Davidson P., Kaneda Y., Moffatt HK. and Sreenivasan KR. Cambridge University Press: Cambridge 329-372.


# Improving generalization with synthetic training data for deep learning
based quality inspection

Antoine Cordier  Scortex, 22 rue Berbier du Mets, Paris, France  Pierre
Gutierrez  Scortex, 22 rue Berbier du Mets, Paris, France  Victoire Plessis
Scortex, 22 rue Berbier du Mets, Paris, France

###### Abstract

Automating quality inspection with computer vision techniques is often a very
data-demanding task. Specifically, supervised deep learning requires a large
amount of annotated images for training. In practice, collecting and
annotating such data is not only costly and laborious, but also inefficient,
given the fact that only a few instances may be available for certain defect
classes. If working with video frames can increase the number of these
instances, it has a major disadvantage: the resulting images will be highly
correlated with one another. As a consequence, models trained under such
constraints are expected to be very sensitive to input distribution changes,
which may be caused in practice by changes in the acquisition system (cameras,
lights), in the parts or in the defects aspect. In this work, we demonstrate
the use of randomly generated synthetic training images can help tackle domain
instability issues, making the trained models more robust to contextual
changes. We detail both our synthetic data generation pipeline and our deep
learning methodology for answering these questions.

###### keywords:

deep learning, computer vision, visual inspection, simulations, 3d rendering,
domain randomization, robustness, sim2real

##  1 Introduction

The lack of availability of training data is one of the biggest challenges
when applying deep supervised learning to quality inspection. Annotating
defective data not only is a costly and difficult task, but also a highly
inefficient one: gathering enough defective data with high variability may in
some cases simply not be possible, given the rarity of some defects. This is
especially true when applying computer vision techniques to video frames,
which are highly correlated with one another. This issue contrasts with more
standard deep learning vision tasks such as ImageNet[1], where each class is
represented with numerous images of great diversity in the training set.
Consequently, the learnt models are often very sensitive to distribution
changes, such as lightning condition or view angle changes. Data augmentation
may be used to improve the models robustness to such changes, but has its own
limitations[2].

To overcome the data availability problem, an encouraging approach is to
automatically generate labeled training data using a synthetic data generation
pipeline[3]. Indeed, generating a wide distribution of randomized scenes with
defective parts may not only help the model learn the defects on the original
real domain better, but should also improve their generalization capability to
unseen changes of distributions: this paradigm is commonly referred to as
domain randomization[4]. As a result, deep learning networks may be trained
more effectively, leading to more robust and adaptive models. This idea has
been effectively used in the autonomous driving industry[5, 6].

Following our work on assessing the usefulness of synthetically generated
training data for deep learning based quality inspection[7], we here aim to
evaluate the impact of the use of synthetically generated data on the
generalization capability of deep convolutional neural networks (CNNs) trained
on the same kind of defect detection task. In this paper, we define four
changes of distributions for our data, and evaluate models trained with or
without simulated data on test sets which reproduce these chosen changes of
distributions. These test sets will be referred to as transfer test sets, and
include an acquisition system (cameras and lights) change, two part aspect
changes, and a defect aspect change. For this study, the parts to be inspected
are car doors from one of our clients, while the only targeted defects are
breaks.

We first review works from the literature that are related to ours in section
2. We then detail our methodology in section 3, for both the simulation data
generation pipeline and the training of our deep learning networks. Our
results are presented in section 4. We demonstrate that while simulated data
may not improve performances on the original real domain significantly, its
role in making the model more robust to unseen distribution changes is
substantial, as shown by the improvement of performance observed on the
transfer test sets. During this work, we experimented with various domain
adaptation techniques (namely DANN[8] and ASS[9]) in order to help bridge the
gap between the real and simulated image domains, but without success.
Consequently, we will not be presenting the results related to our domain
adaptation experiments in this paper.

##  2 Related work

Automated visual inspection via the use of deep learning often relies on
supervised learning[10]. In order to classify and localize defects on visual
data, convolutional neural networks (CNNs) are trained on images, most of the
time using either semantic segmentation networks (such as U-Net [11]), or
object detection architectures (such as YOLO[12] or RetinaNet [13]). However,
one of the drawbacks of deep supervised learning is that it requires large
quantities of data for training. Unfortunately for the quality control
usecase, obtaining images of defective parts can be difficult, since
defectiveness may be a rare event. In other words, collecting enough data to
properly learn to detect defects may not always be possible. To tackle this
issue, one can move from supervised to unsupervised anomaly detection
techniques [14], or make use of simulated data to enrich the training set
[15].

For our literature review on the the use of simulated data for deep learning,
we repeat some elements from our previous work on the topic.[7] The reader may
refer to this paper for a more comprehensive review. Simulated training data
use has been widely studied for object detection in images[3], most
specifically in the fields of robotics [4], as well as for autonomous driving
[16]. They are particularly useful for handling tasks where human labeling is
hard or costly. This includes pose or depth estimation [17, 18, 19], and deep
learning-based rendering [20]. Simulated data does not only lower the cost of
human annotation, but also allows to learn theoretical modes for which no real
data exist. Nevertheless, it is common for models trained on simulated data to
fail at generalizing well on real data. This phenomena is known as the domain
gap problem.

Three techniques help solving this issue: improving photorealism (bringing the
synthetic domain closer to the real one), synthetic domain randomization
(randomizing the synthetic domain enough so that it covers the real one), and
domain adaptation (train the model in a way that the two domains align with
one another). Improved photorealism is essential for learning strong detectors
[21]. It has recently been shown that rendering 3D models in physically
realistic positions can have a significant impact on performances[22] compared
to randomly rendering the same models in random positions on random
backgrounds[23]. However, in order to generate simulated data diverse enough
for learning, photorealism has to be coupled with domain randomization [4, 6].
The idea behind domain randomization [4] was initially to randomize the
simulation process so that the real domain could become a subset of the
simulated one. This has successfully been applied to autonomous driving [3]
and industrial applications [24]. The concept has now been extended to
“automated domain adaptation” [25] (in which the randomization increases over
time in a curriculum learning fashion) and to “structured domain
randomization” [6] (for which domain randomization does not come at the cost
of photorealism). To bridge the domain gap between synthetic and real data
domains, domain adaptation techniques can finally be used. We refer the reader
to the review by Toldo, Marco, et al. [26]. In our experiments, we tried DANN
[8] and ASS [9] methods, but do not exhibit detailed conclusions, since no
significant results were achieved on this subject.

In this article, we focus on combining both real and simulated data at the
same time, a topic already covered in some articles[3, 9]. More particularly,
we are interested in generalization of models trained with simulated data to
new image distributions, e.g. changes of acquisition systems or part
references. This topic has strong ties to generalization under covariate
shifts, as well as to neural networks robustness against natural
perturbations. Several datasets have been proposed to evaluate networks
generalization and transferability to other domains. For example, ImageNet-A
and ImageNet-O [27] are composed of “natural adversarial examples”. On the
contrary, ImageNet-C and Imagenet-P [28] were later created synthetically from
ImageNet, via the use of common pertubations like noise, motion blur, random
contrast, and elastic deformations. Finally, Imagenet-R [29] is composed of
images of ImageNet objects with different renditions (paintings, cartoons,
sculptures, origamis).

Several solutions have been proposed to tackle robustness to natural
perturbations. First, networks architectural modifications. Larger and deeper
networks have been proven to be more robust [28]. Leveraging attention,
through the use of SE and SK blocks [27, 30] has also been proven to be useful
in some cases. Dedicated architectural layers such as BlurPool [31] can be
used to tackle the fact that modern neural networks are not translation
invariant nor equivariant. Second, data augmentation based techniques. Common
data augmentations such as CLAHE [28] or Mixup [30, 32] may help. Stability
training [33] inspired techniques such as Augmix [34] or Auxiliary training
[35] can also work. However, it has been proven data augmentation has trouble
generalizing outside the training data manifold, and thus do not provide
general network invariances. [2]. Close to this idea, self-supervised learning
with rotations prediction [36] can also improve robustness. Last but not
least, transfer learning [37, 38]. Transfer learning should preferably be
applied using deep and large networks trained with very large corpuses of
images coming from external domains.[39].

Studies have shown these techniques are not effective for all cases, and that
there is no such thing as a silver bullet for making CNNs more robust to
perturbations [29, 38]. Therefore, the robustness of neural networks is still
a very open research topic. In this article, we propose improving
generalization via the use of synthetic training images. To some extent, this
can be seen as an extension of data augmentation techniques.

##  3 Materials and methods

###  3.1 Synthetic training data generation

We designed a synthetic image generation pipeline based on Blender[40] and its
physically-based rendering (PBR) engine Cycles. Our software allows us to
randomly generate large batches of realistic images with great variability,
and can be controlled with a simple configuration file. While realism is
ensured via advanced physically-based modelling and rendering options (e.g.
Monte Carlo ray tracing[41], IES lights[42], PBR materials[43, 44], …),
variability is achieved through the randomization of several simulation
parameters (such as position, orientation, and other characteristics for
lights, cameras, parts and defects). Our core idea is to couple photorealism
with domain randomization[4, 6] to generate realist, but diverse enough images
of defective car doors.

####  3.1.1 Environment generation

First, we design a bare environment for our scene, which is a 3D simulated
stage based on the real factory setup: a conveyor belt, over which an ark
supporting nine cameras is placed. Default positions and orientations are
chosen for the opto-electronic elements (cameras and lights) based on the real
setup, which are then placed on the stage. An overview of the default
environment with an additional positioned part can be seen in Figure 1.

For both cameras and lights, pose (i.e. orientation and position) is
randomized. For cameras, we also randomize the focus blur (via the working
distance and aperture). For lights, we additionally randomize light power and
color. Environment randomization is then finalized by randomizing the texture
of some elements of the stage (e.g. conveyor belt).

####  3.1.2 Parts generation

After importing the CAD model for the parts of interest (which were given by
our client), one to several parts are then selected to be positioned on the
stage, which is an horizontal plane. Because a set of known physical
equilibrium positions has been attributed in advance to each part with regards
to this plane, it is possible to randomly pick one of these equilibrium
positions for each selected part, before placing it on the plane. The part is
placed on the plane with an additional random translation and rotation in the
plane to ensure pose randomization. To make sure no unrealistic overlaps
between the parts and the rest of the environment can happen, multiple
positioning attempts are made until a satisfying result is achieved. The part
material and textures are randomized using procedural texturing.

Figure 1: The default simulated environment for our scene, with an additional
positioned part.

####  3.1.3 Defects generation

A set of defect texture maps (normal, height, roughness, and occlusion maps)
is generated in advance with a procedural texture generator. Once generated,
these texture maps can then be combined in Blender for randomly placing
defects on the parts. We define our simulated defects as a combination of up
to three types of textures: deformation, encrustation, and hole textures. The
exact combination and characteristics of these textures will depend on the
desired defect.

Figure 2: The different texture maps used for simulating a break, with the
final rendering on the far right.

  * •

Deformation textures are used for defects which are defined as a physical
deformation of the part, such as shocks or scratches. In these cases, we
create four texture maps (normal, height, roughness maps, as well as an
occlusion map to select the deformed zone) to apply.

  * •

Encrustation textures are used for defect which are defined as a distinct
material change or insertion on the part, such as craters, burns, or marks. In
these cases, a unique texture mask map is created, and a new PBR material is
applied locally using the mask.

  * •

Hole textures are used for defects which are defined as a zone where the
original material from the part is missing, such as holes or cracks. In these
cases, a texture mask map is used for removing the material where desired.

In our case, the defect of interest is a break. To simulate it, we combine all
three approaches mentioned above, as shown in Figure 2. When placing a defect
on a part, both its position on the part mesh and size are randomized. When
randomizing the size of the defect, texture maps of the latter are scaled
accordingly.

####  3.1.4 Rendering scheme

Once parts and defects are placed, a content verification algorithm is ran for
each camera field of view to decide whether the generated scene should be
rendered for the given camera: this allows to only render useful images, since
rendering time represents the main compute cost of our pipeline. The algorithm
consists in verifying the presence of at least one visible part with at least
one visible defect for the given camera. Visibility of a part is decided
following geometrical considerations: at least one vertex of the part model
should fall within the field of view of the considered camera. Visibility of a
defect is decided following a specific rendering of the scene, where only the
selected defect reflects light: this is possible thanks to the available
texture mask attached to each defect, as explained in 3.1.3. The result will
be an image where only the defect is visible, while the rest is dark. After
binarization of this image, a visibility mask is created for the defect, and a
threshold can then be applied to the size of this mask to decide whether the
defect is considered visible or not. Note that this visibility mask will also
serve for generating the label annotation associated to this defect. To avoid
mistakes, if a defect is considered not visible, it will simply be removed
from the part before rendering. Finally, an image will be rendered only if at
least one visible defect is present on the part.

Thereby, we were able to generate large batches of synthetic images of
defective parts. Ensuring that the generated images follow a similar
distribution to that of the real ones may be one of the hardest challenges of
the described approach. In practice, iterating on the simulation parameters
can become a long and tedious process. Additionally, one needs to define
adequate metrics for measuring the proximity between simulated and real
images. While experimenting, we iterated on the size of the synthetic defects,
after understanding that the size of the defects we were initially generating
was too big. Such an iteration had a significant impact on the performance of
the simulated models (data not shown). However, it is currently not possible
to confidently predict the impact a given iteration will have on the
performances of the future model before training the latter. This makes
iterating laborious, even though it probably is a crucial element of the
overall approach.

###  3.2 Defect detection with deep learning

In order to detect defects, we rely on supervised deep learning. The detection
task is framed as a binary segmentation problem for training, but is evaluated
using aggregated boxes for computing recall, precision and mAP, since it is
closer to the manufacturer metrics of interest. Because the parts are rather
large (2 meters long) and the defects may be relatively small (a few
centimeters), we use input images of high resolution (2048 x 2592). No
resizing or cropping is used as inference pre-processing. Because of this, we
rely on custom architectures in order to ensure real-time inference with a
high frame rate. These architectures are inspired from the literature (e.g.
VGG, ResNet, U-Net, or RetinaNet[13]) but have thinner and shallower
backbones.

In order to improve performance and generalization, we apply data augmentation
to all training images, including the synthetic ones. We use random noise,
blurring, contrast, color jittering, cutouts, flips, zooms, translations,
rotations, and shears.

##  4 Experiments

Similarly than in our previous work on synthetic data generation[7], we will
answer the following questions:

  1. (a)

Can we achieve good performances on real data with simulated images alone?

  2. (b)

Can we improve (a) by using additional healthy real images? This is an
important question since healthy parts (hence images) are easily collectable
in large quantities, as opposed to defective parts, which we may not have at
our disposal in case the defect is a rare event.

  3. (c)

Can we leverage simulations as additional inputs to improve over models
trained with real data only?

In all of these experiments, we assess whether simulations can improve
generalization capability over unseen distributions, e.g. robustness against
part and defect variations, or against external condition changes. This is
possible because of the applied domain randomization when generating the
simulated data.

###  4.1 Training and evaluation data

We generate 9413 simulated images, which are split into a training (8871
images) and a test (542 images) set. The CAD model used to generate the
simulated parts corresponds to the same part reference that is used for the
real training images. Each generated image contains at least one defect. It is
worth noting that the generation of this data only took three days, which is
notably faster than what it would have taken for a single person to annotate
an equivalent amount of defective real images (which may not necessarily be
available). Note also that we fine-tune our simulation parameters so that we
generate defects that have comparable size with real ones.

We also collect real images from our client parts directly from the production
line, in the factory: we collect 10000 images for the real training set, each
of which also contains at least one defect. This allows us to have comparable
simulated and real training sets in terms of number of images and defects.
Additionally, 10000 extra real healthy (i.e. without defects) images are
collected, all coming from the same distribution than the defective real
images (same acquisition system, same part reference, but healthy). Note that
the amount of extra healthy images corresponds to the number of images present
in the real training set to ensure comparability of trainings (see next
section).

To evaluate our models in real life conditions, we create a main real test set
of 1520 images (795 defectives, 725 healthy) with different physical parts
than the ones used to design the real training set. To assess generalization
capabilities of our models to unseen distribution shifts, we also evaluate on
four “transfer” test sets: a first one with a change of the acquisition system
(new camera and lightning setup), a second one with a change of part reference
(right door instead of left door), a third one with a second change of part
reference (rear door instead of front door), and a last one with change in
defect aspect (see Figure 3). Our four transfer test sets have respectively
1130 (285 defectives, 845 healthy), 935 (135 defectives, 800 healthy), 1245
(280 defectives, 965 healthy) and 755 (320 defectives, 435 healthy) images.

We evaluate our models on the different test sets by following the precision,
recall and mAP metrics, which are all computed based on the following true
positive matching criterion. Both our ground truth and our model predictions
can be understood as bounding boxes. An inferred box that is intersecting an
annotated ground truth box will be considered as a true positive (but only one
inferred box can be matched per ground truth annotation). Inferred boxes have
associated confidences, which allow to compute precision and recall at
multiple thresholds.

Figure 3: Comparison of some defect close-ups for the different test sets
(main, change of acquisition system, two changes of part reference, change of
defect aspect), as well as for the simulated data.

###  4.2 Training with simulated data

Table 1: Performance results: mAP for each trained model on all test sets after 200 epochs. Transfer test sets are acq system (change in acquisition system), part (change in part reference), part 2 (another change in part reference), and defect (change in defect aspect). | main | acq system | part | part 2 | defect
---|---|---|---|---|---
Real | 0.913 | 0.452 | 0.421 | 0.432 | 0.591
Simulated | 0.316 | 0.183 | 0.402 | 0.446 | 0.099
Real + additional healthy | 0.912 | 0.500 | 0.436 | 0.444 | 0.569
Simulated + additional healthy | 0.563 | 0.337 | 0.489 | 0.447 | 0.367
Real + simulated | 0.913 | 0.664 | 0.622 | 0.560 | 0.619
Real + simulated + additional healthy | 0.922 | 0.699 | 0.663 | 0.590 | 0.610

To answer our questions, we train models using different training data.
Results after convergence are summarized in table 1. Learning curves for mAP,
precision, and recall on all test sets are given in Appendix A (Figure 5 and
6). We first train a baseline model using real data only. As expected, the
baseline model performs very well on the main test set (mAP of 0.913).
However, a significant drop in performance can be noticed when changing the
content of the test set (mAP drops to approximately 0.4-0.6, depending on the
type of transfer). The fact that the model trained on real data only does not
generalize well to unseen distributions is mostly due to the generation of
false positives, as evidenced by the precision and recall learning curves from
Figure 5. Indeed, they indicate a high recall and low precision on all
transfer test sets for the “Real” baseline training. In practice, the
generated false positives are often due to the stamped holes in the door,
which may look like cracks under certain point of views. This could explain
the performance drop for the acquisition system change, but also for the two
part reference changes, which both induce a lot of these stamped holes to
become visible. More broadly, this lack of generalization is not too
surprising, since these real images are expected to be very correlated with
one another.

####  4.2.1 Simulated data only

We first compare our baseline model trained on real data with a model trained
on simulated data only. Unfortunately, using solely simulated images does not
allow to get satisfatory performances on the main test set (mAP after
convergence is only of 0.316). Interestingly, we nevertheless observe that it
is possible to get a performance similar to the real baseline on both part
change transfer test sets, using simulations only (mAP of around 0.4). This
highlights the simulation data transfer capability: simulated data
generalization power is approximately “one reference away”. To nuance, one can
however notice that the simulation model does not transfer well to the defect
aspect change, which thus appears as a harder transfer task: we hypothesize
that this is due to the difference in the defect size distribution between the
simulated data and the change of defect aspect test set. This answers question
(a).

####  4.2.2 Simulated data with additional healthy real images

Second, we aim to assess whether we can improve the simulation data
performance by using the additional real healthy images combined to the
simulated defective images as the training set. Performance is improved on the
main test set by more than 0.20 points (mAP of 0.563), without nonetheless
getting close to the “Real + additional healthy” baseline counterpart (mAP of
0.912). In terms of generalization, performance of the simulated model
improves by a significant margin on the transfer test sets when using the
additional real healthy images (+0.15 points on acquisition system change,
+0.09 points on the first part change, +0.27 points on the defect aspect
change), even beating the “Real” previous baseline transfer capability on the
first part change test set. Even though this model may still not be on par
with the real baselines on the main test set, its performance on both part
change test sets is actually better than the one from the same baselines. This
answers question (b).

####  4.2.3 Simulated data in combination with real data

Finally, using both simulated and real defective images, performance is the
same than using only real defective images on the main test set (mAP of
0.913). However, mAP is improved on all transfer test sets (from +0.03 for
defect aspect change to +0.21 for the second part aspect change). We test
whether this gain is simply due to the quantity of data by comparing this
model to a baseline model trained on the real data plus the additional healthy
real images (hence, number of images is comparable between the two trainings).
The fact that the “Real + additional healthy” model does not generalize as
good as the “Real + simulated” model enable us to conclude that simulations
are actually helping generalization. We also observe an improvement in
generalization when comparing the “Real + simulated + additional healthy”
model to the “Real + additional healthy” baseline. Overall, best performances
are obtained when using all available training data: real, additional healthy
real, and simulated images. This answers question (c).

####  4.2.4 Conclusion: simulated data helps generalizing better

As seen in the previous sections, simulations may not improve performance on
the main test set, but definitely help generalizing better on all tested
unseen distributions, especially if used in combination with real images. We
hypothesize this gain in generalization power to come from two phenomenons.
First, a reduction of the false positives that are induced by the new domains
shifts. Second, an improvement in learning the defect aspect, due to the
randomized simulated defects. In the first case, we should observe a decrease
in false positives (improvement of precision) when using the simulated images,
while in the second case, we should observe a decrease of false negatives
(improvement of recall).

###  4.3 Why is generalization improved with the simulated data?

Figure 4: Precision-recall curves on all four transfer test sets for when training without simulated images (blue), with simulated images (orange), with non-defective simulated images (green), and with additional healthy real images coming from all tested distributions (red). Models are trained until convergence (200 epochs). Table 2: mAP comparison for when training without simulated images, with simulated images, with non-defective simulated images, and with additional healthy real images coming from all tested distributions. Models are trained until convergence (200 epochs). | main | acq system | part | part 2 | defect
---|---|---|---|---|---
Real | 0.913 | 0.452 | 0.421 | 0.432 | 0.591
Real + simulated | 0.913 | 0.664 | 0.622 | 0.560 | 0.619
Real + simulated (healthy) | 0.923 | 0.663 | 0.551 | 0.479 | 0.564
Real + additional healthy (all distributions) | 0.918 | 0.679 | 0.660 | 0.627 | 0.622

We test the above hypotheses by comparing recall and precision on all test
sets for two trainings: “Real” and “Real + simulated”. To test the hypothesis
further, we train and evaluate two supplementary models. The first one is
trained with the real images as well as around the same amount of healthy
simulated images, and should help determine whether the simulated defects
actually help improving the recall or not. The second one is trained with the
real images as well as around the same amount of additional healthy real
images, but this time coming from all tested distributions for which this
makes sense (i.e. acquisition system change and both part changes). This
second training should help assess the ability of the model trained with
simulation data to improve precision. Note that such healthy images are here
relatively easy to collect and annotate, given that the changes in
distribution happened already in production.

Results are recapitulated in Figure 4 for the precision-recall curves, and in
Table 2 for the corresponding mAPs. While precision and recall values are very
similar between the “Real” (blue curves) and the “Real + simulated” (orange
curves) trainings on the main test set, both precision and recalls are
improved when adding the simulated images on the transfer test sets. Replacing
the defective simulated images by healthy ones has a significant negative
impact on performance for most test sets, as seen in Table 2. This decrease in
mAP can be explained by a loss in both precision and recall, depending on the
test set (see Figure 4). Additionally, the model trained on the real data as
well as the additional healthy data coming from all transfer distributions
performs only slightly better than our “Real + simulated” model. Most of the
improvement here is coming from an improvement of precision, as shown from the
precision-recall curves for the two part changes test sets (see Figure 4).

As a conclusion, simulated data mostly reduces the amount of false positives
on unseen distributions, but also improve recall on these distributions thanks
to the randomization of the simulated defects variability. Importantly, the
main advantage of using simulations compared to using diverse healthy images
for false positive reduction is that thanks to domain randomization, it is
possible to prevent domain shift before it actually happens in production. In
comparison, collecting many diverse healthy images from the production to try
cover various types of shifts in advance is not expected to be an easy task,
not to mention that this may be time consuming as well.

###  4.4 Impact of the network size and pre-training

Because the capacity of our custom network is limited, we decide to make it
deeper in order to assess the potential role of the network size on transfer
capabilities, with and without simulated images. It has indeed experimentally
be shown that the network size has a role in robustness to common
perturbations[28], and thus in generalizing better. As a result, the number of
trainable parameters for our network is roughly doubled. This bigger network
is pre-trained on the ImageNet dataset classification task, and fine-tuned
using either “Real + additional healthy”, or “Real + simulated + additional
healthy” images for training. Results are presented in Table 3. Learning
curves for mAP, precision, and recall on all test sets are available in
Appendix A (Figure 7).

Table 3: mAP comparison between the original and the deeper network architectures, with and without the use of simulated data. Each model is trained for 200 epochs and reaches convergence. | main | acq system | part | part 2 | defect
---|---|---|---|---|---
Baseline network |  |  |  |  |
Real + additional healthy | 0.912 | 0.500 | 0.436 | 0.444 | 0.569
Real + simulated + additional healthy | 0.922 | 0.699 | 0.663 | 0.590 | 0.610
Deeper network |  |  |  |  |
Real + additional healthy | 0.957 | 0.650 | 0.607 | 0.538 | 0.526
Real + simulated + additional healthy | 0.943 | 0.814 | 0.717 | 0.560 | 0.620

When comparing the effect of a deeper network on real data alone (without
simulations), one can already notice a significant gain in mAP (from +0.04 to
+0.17) on all test sets, save for the change in defect aspect test set. This
confirms the low capacity of our original baseline network architecture, as
well as the hypothesis that bigger architectures may help for better
generalization. Similarly to what was discussed in 4.2, although simulated
images do not profit to the main test set, they still improve generalization
on all transfer test sets when using the deeper network (gains varying from
0.02 to 0.16 points of mAP). One should note that the gains coming from the
use of simulated data are comparable to the ones obtained via the architecture
improvement alone. Interestingly, both types of gains are cumulative.

We also investigate the impact of the ImageNet pre-training on the deeper
network, but find that the latter is actually null on mAP (pre-training only
improves convergence – data not shown).

##  5 Conclusion

In this work, we used a generic pipeline to massively generate synthetic
training data and to assess the quality of industrial parts, via deep
learning. It is first important to emphasize that iterating on the generation
of simulated data is critical, but a very difficult thing to do in practice,
for two reasons. First, it is almost impossible to predict a priori how good
the generated defects will be until an actual model is trained on them.
Second, understanding a posteriori why a given model fails on the test data is
also in itself challenging.

We showed that while the use of simulation images alone is not sufficient
(this was also the conclusion of our previous work[7]), its combination with
real images can be beneficial. This is especially true when testing the model
on unseen distributions: the combined use of simulated and real images then
allows better generalization almost systematically. We demonstrate that this
improvement mostly comes from a reduction in false positives, even though we
also observe a better learning of the real defects. Furthermore, the
generalization power brought by the simulated data is approximately “one part
reference away”.

As a conclusion, simulated data could therefore be used as a safeguard against
changes of input image distributions, which arise regularly in production. In
the future, we plan to further work on finding simpler ways to iterate on the
simulated data. Simulations could also help tackling deep learning tasks for
which annotation is more tedious, such as part segmentation or part pose
estimation.

## References

  * [1] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L., “Imagenet: A large-scale hierarchical image database,” in [2009 IEEE Conference on Computer Vision and Pattern Recognition ], 248–255 (2009).
  * [2] Azulay, A. and Weiss, Y., “Why do deep convolutional networks generalize so poorly to small image transformations?,” arXiv preprint arXiv:1805.12177 (2018).
  * [3] Tremblay, J., Prakash, A., Acuna, D., Brophy, M., Jampani, V., Anil, C., To, T., Cameracci, E., Boochoon, S., and Birchfield, S., “Training deep networks with synthetic data: Bridging the reality gap by domain randomization,” in [Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops ], 969–977 (2018).
  * [4] Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., and Abbeel, P., “Domain randomization for transferring deep neural networks from simulation to the real world,” in [2017 IEEE/RSJ international conference on intelligent robots and systems (IROS) ], 23–30, IEEE (2017).
  * [5] Pouyanfar, S., Saleem, M., George, N., and Chen, S.-C., “Roads: Randomization for obstacle avoidance and driving in simulation,” in [Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops ], (June 2019).
  * [6] Prakash, A., Boochoon, S., Brophy, M., Acuna, D., Cameracci, E., State, G., Shapira, O., and Birchfield, S., “Structured domain randomization: Bridging the reality gap by context-aware synthetic data,” in [2019 International Conference on Robotics and Automation (ICRA) ], 7249–7255, IEEE (2019).
  * [7] Gutierrez, P., Luschkova, M., Cordier, A., Shukor, M., Schappert, M., and Dahmen, T., “Synthetic training data generation for deep learning based quality inspection,” CoRR abs/2104.02980 (2021).
  * [8] Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., and Lempitsky, V., “Domain-adversarial training of neural networks,” The Journal of Machine Learning Research 17(1), 2096–2030 (2016).
  * [9] Wang, Z., Wei, Y., Feris, R., Xiong, J., Hwu, W.-M., Huang, T. S., and Shi, H., “Alleviating semantic-level shift: A semi-supervised domain adaptation method for semantic segmentation,” in [Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops ], 936–937 (2020).
  * [10] Dong, H., Song, K., He, Y., Xu, J., Yan, Y., and Meng, Q., “Pga-net: Pyramid feature fusion and global context attention network for automated surface defect detection,” IEEE Transactions on Industrial Informatics 16(12), 7448–7458 (2019).
  * [11] Ronneberger, O., Fischer, P., and Brox, T., “U-net: Convolutional networks for biomedical image segmentation,” in [International Conference on Medical image computing and computer-assisted intervention ], 234–241, Springer (2015).
  * [12] Redmon, J., Divvala, S., Girshick, R., and Farhadi, A., “You only look once: Unified, real-time object detection,” in [Proceedings of the IEEE conference on computer vision and pattern recognition ], 779–788 (2016).
  * [13] Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P., “Focal loss for dense object detection,” in [Proceedings of the IEEE international conference on computer vision ], 2980–2988 (2017).
  * [14] Bergmann, P., Fauser, M., Sattlegger, D., and Steger, C., “Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection,” in [Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ], 9592–9600 (2019).
  * [15] Zambal, S., Heindl, C., Eitzinger, C., and Scharinger, J., “End-to-end defect detection in automated fiber placement based on artificially generated data,” in [Fourteenth International Conference on Quality Control by Artificial Vision ], 11172, 111721G, International Society for Optics and Photonics (2019).
  * [16] Li, W., Pan, C., Zhang, R., Ren, J., Ma, Y., Fang, J., Yan, F., Geng, Q., Huang, X., Gong, H., et al., “Aads: Augmented autonomous driving simulation using data-driven algorithms,” Science robotics 4(28) (2019).
  * [17] Chen, W., Wang, H., Li, Y., Su, H., Wang, Z., Tu, C., Lischinski, D., Cohen-Or, D., and Chen, B., “Synthesizing training images for boosting human 3d pose estimation,” in [2016 Fourth International Conference on 3D Vision (3DV) ], 479–488, IEEE (2016).
  * [18] Sundermeyer, M., Puang, E. Y., Marton, Z.-C., Durner, M., and Triebel, R., “Learning implicit representations of 3d object orientations from rgb,” (2018).
  * [19] Langlois, J., Mouchère, H., Normand, N., and Viard-Gaudin, C., “On the fly generated data for industrial part orientation estimation with deep neural networks,” in [Fourteenth International Conference on Quality Control by Artificial Vision ], 11172, 111720I, International Society for Optics and Photonics (2019).
  * [20] Deschaintre, V., Aittala, M., Durand, F., Drettakis, G., and Bousseau, A., “Single-image svbrdf capture with a rendering-aware deep network,” ACM Transactions on Graphics (ToG) 37(4), 1–15 (2018).
  * [21] Movshovitz-Attias, Y., Kanade, T., and Sheikh, Y., “How useful is photo-realistic rendering for visual learning?,” in [European Conference on Computer Vision ], 202–217, Springer (2016).
  * [22] Hodaň, T., Vineet, V., Gal, R., Shalev, E., Hanzelka, J., Connell, T., Urbina, P., Sinha, S. N., and Guenter, B., “Photorealistic image synthesis for object instance detection,” in [2019 IEEE International Conference on Image Processing (ICIP) ], 66–70, IEEE (2019).
  * [23] Dwibedi, D., Misra, I., and Hebert, M., “Cut, paste and learn: Surprisingly easy synthesis for instance detection,” in [Proceedings of the IEEE International Conference on Computer Vision ], 1301–1310 (2017).
  * [24] Lee, Y.-H., Chuang, C.-C., Lai, S.-H., and Jhang, Z.-J., “Automatic generation of photorealistic training data for detection of industrial components,” in [2019 IEEE International Conference on Image Processing (ICIP) ], 2751–2755, IEEE (2019).
  * [25] Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., et al., “Solving rubik’s cube with a robot hand,” arXiv preprint arXiv:1910.07113 (2019).
  * [26] Toldo, M., Maracani, A., Michieli, U., and Zanuttigh, P., “Unsupervised domain adaptation in semantic segmentation: a review,” Technologies 8(2), 35 (2020).
  * [27] Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D., “Natural adversarial examples,” in [Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ], 15262–15271 (2021).
  * [28] Hendrycks, D. and Dietterich, T., “Benchmarking neural network robustness to common corruptions and perturbations,” arXiv preprint arXiv:1903.12261 (2019).
  * [29] Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al., “The many faces of robustness: A critical analysis of out-of-distribution generalization,” arXiv preprint arXiv:2006.16241 (2020).
  * [30] Lee, J., Won, T., Lee, T. K., Lee, H., Gu, G., and Hong, K., “Compounding the performance improvements of assembled techniques in a convolutional neural network,” arXiv preprint arXiv:2001.06268 (2020).
  * [31] Zhang, R., “Making convolutional networks shift-invariant again,” in [International Conference on Machine Learning ], 7324–7334, PMLR (2019).
  * [32] Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D., “mixup: Beyond empirical risk minimization,” arXiv preprint arXiv:1710.09412 (2017).
  * [33] Zheng, S., Song, Y., Leung, T., and Goodfellow, I., “Improving the robustness of deep neural networks via stability training,” in [Proceedings of the ieee conference on computer vision and pattern recognition ], 4480–4488 (2016).
  * [34] Hendrycks, D., Mu, N., Cubuk, E. D., Zoph, B., Gilmer, J., and Lakshminarayanan, B., “Augmix: A simple data processing method to improve robustness and uncertainty,” arXiv preprint arXiv:1912.02781 (2019).
  * [35] Zhang, L., Yu, M., Chen, T., Shi, Z., Bao, C., and Ma, K., “Auxiliary training: Towards accurate and robust models,” in [Proceedings of the IEEE/CVF conference on computer vision and pattern recognition ], 372–381 (2020).
  * [36] Hendrycks, D., Mazeika, M., Kadavath, S., and Song, D., “Using self-supervised learning can improve model robustness and uncertainty,” arXiv preprint arXiv:1906.12340 (2019).
  * [37] Hendrycks, D., Lee, K., and Mazeika, M., “Using pre-training can improve model robustness and uncertainty,” in [International Conference on Machine Learning ], 2712–2721, PMLR (2019).
  * [38] Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., and Schmidt, L., “Measuring robustness to natural distribution shifts in image classification,” arXiv preprint arXiv:2007.00644 (2020).
  * [39] Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N., “Big transfer (bit): General visual representation learning,” arXiv preprint arXiv:1912.11370 6(2), 8 (2019).
  * [40] Community, B. O., Blender - a 3D modelling and rendering package.  Blender Foundation, Stichting Blender Foundation, Amsterdam (2018).
  * [41] Lafortune, E., “Mathematical Models and Monte Carlo Algorithms for Physically Based Rendering,” (1996).
  * [42] “Illuminating Engineering Society – The Lighting Authority.”
  * [43] Cook, R. L., “A Reflectance Model for Computer Graphics,” ACM Transactions on Graphics 1(1) (1981).
  * [44] Burley, B., “Physically Based Shading at Disney,” (2012).

##  Appendix A Learning curves

Figure 5: mAP, precision, and recall on all test sets over training epochs for
trainings “Real”, “Simulated”, “Simulated + additional healthy”, and “Real +
simulated”. Figure 6: mAP, precision, and recall on all test sets over
training epochs for trainings “Real + additional healthy”, “Simulated +
additional healthy”, and “Real + simulated + additional healthy”. Figure 7:
mAP, precision, and recall on all test sets over training epochs for trainings
“Real + additional healthy” and “Real + simulated + additional healthy” for
both the baseline and the deeper network architecture.