$\displaystyle\gamma_{i}^{\prime}(t)={\mathfrak{r}}^{-1}+\operatorname{\mathcal{O}}(|w_{t}(\gamma_{i}(t))-w_{t_{c}}(x_{c})|),\quad\widetilde{\gamma}_{i}^{\prime}(t)={\mathfrak{r}}^{-1}+\operatorname{\mathcal{O}}(|\widetilde{w}_{t}(\widetilde{\gamma}_{i}(t))-w_{t_{c}}(x_{c})|),$
(7.68)
$\displaystyle\gamma_{i}^{\prime}(t)-\widetilde{\gamma}_{i}^{\prime}(t)\lesssim
f_{t}(\gamma_{i}(t))-\widetilde{f}_{t}(\widetilde{\gamma}_{i}(t))\lesssim
w_{t}(\gamma_{i}(t))-\widetilde{w}_{t}(\widetilde{\gamma}_{i}(t)).$

We next complete the proofs of 7.2 and 7.9 using (7.63)–(7.68).

####  7.6.1. Proof of 7.2

At $t=t_{c}$, we have $\gamma_{0}(t_{c})=x_{c}$. Then, from (7.63) and (7.64),
we obtain that

$(t-t_{c})\varpi_{t}(\gamma_{0}(t))+\frac{a}{3}\varpi_{t}(\gamma_{0}(t))^{3}+\cal
E(\varpi_{t}(\gamma_{0}(t)))=\gamma_{0}(t)-c(t)=\int_{t_{c}}^{t}(\gamma_{0}^{\prime}(t^{\prime})-{\mathfrak{r}}^{-1}){\rm
d}t^{\prime}\lesssim\int_{t_{c}}^{t}|\varpi_{t}(\gamma_{0}(t^{\prime}))|{\rm
d}t^{\prime},$

where we used the fact that $t/{\mathfrak{r}}=c(t)$ (since
$x_{c}=t_{c}/{\mathfrak{r}}$) and applied (7.67) for the last inequality. We
can rewrite the above equation as (7.38), with
$\varpi=\varpi(t)=\varpi_{t}(\gamma_{0}(t))$, $|\varepsilon(\varpi)|\leq
C_{1}|\varpi|^{4}$, $\tau=(t-t_{c})/a$, and

$y(\varpi):=\frac{3}{2a}\int_{t_{c}}^{t}\gamma_{0}^{\prime}(t^{\prime})-{\mathfrak{r}}^{-1}{\rm
d}t^{\prime},\quad|y(\varpi)|\leq
C_{2}\int_{t_{c}}^{t}|\varpi(t^{\prime})|{\rm d}t^{\prime},$

for some constants $C_{1},C_{2}>0$. Then, we have $\varpi=q_{t}(\varpi)$,
where the function $q_{t}(\varpi)$ is defined in terms of $\tau$,
$\varepsilon(\varpi)$, and $y(\varpi)$ as the right-hand side of (7.39). Thus,

$|q_{t}(\varpi)|\leq
4|y(\varpi)|^{1/3}+4|\varepsilon(\varpi)|^{1/3}+2\tau^{1/2}\leq
4C_{2}^{1/3}|t-t_{c}|^{1/3}\sup_{t^{\prime}\in[t_{c},t]}|\varpi(t^{\prime})|^{1/3}+4C_{1}^{1/3}|\varpi(t)|^{4/3}+2\sqrt{\frac{t-t_{c}}{a}}.$

With this bound, we can check that there exists a constant $\cal A>0$
depending on $a,C_{1},C_{2}$ such that if $|\varpi(t^{\prime})|\leq\cal
A|t-t_{c}|^{1/2}$ for all $t^{\prime}\in[t_{c},t]$, then we have
$|\varpi(t)|\leq\cal A|t-t_{c}|^{1/2}/2$.

Combining the above fact with the continuity of $\varpi(t)$, we can conclude
that $|\varpi(t)|\leq\cal A|t-t_{c}|^{1/2}/2$ for all $t\in[t_{c},t_{1}]$.
More precisely, we first notice that $\varpi(t)$ is Hölder-$1/3$ continuous in
$t$ (since the right-hand side of (7.39) is Hölder-$1/3$ continuous in both
$t$ and $x$ and $\gamma_{0}(t)$ is Lipschitz continuous in $t$). Suppose
$|\varpi(t^{\prime})|\leq\cal A|t^{\prime}-t_{c}|^{1/2}/2\leq\cal
A|t-t_{c}|^{1/2}/2$ for all $t^{\prime}\in[t_{c},t]$. Then, for a sufficiently
small $\varepsilon$, $|\varpi(t^{\prime})|\leq\cal A|t^{\prime}-t_{c}|^{1/2}$
for all $t^{\prime}\in[t_{c},t+\varepsilon]$, from which we get that
$|\varpi(t^{\prime})|=|q_{t^{\prime}}(\varpi)|\leq\cal
A|t^{\prime}-t_{c}|^{1/2}/2$ for all $t^{\prime}\in[t_{c},t+\varepsilon]$. In
this way, we can extend the estimate $|\varpi(t)|\leq\cal A|t-t_{c}|^{1/2}/2$
at $t=t_{c}$ all the way to $t_{1}$.

Finally, plugging the estimate $|\varpi(t)|\leq\cal A|t-t_{c}|^{1/2}/2$ into
(7.64), we conclude the proof.

####  7.6.2. Proof of (7.60)

To bound $|\gamma_{L}(t)-\widetilde{\gamma}_{L}(t)|$, we now bound the right-
hand side of (7.68) for $i=L$.

By 4.1 and (7.23) (recall that $x_{c}=t_{c}/{\mathfrak{r}}$ and
$\widetilde{\gamma}_{i}(0)=\gamma_{i}(0)$ for $i\in\llbracket-M,N\rrbracket$),
we have

(7.69) $\gamma_{L}(0)=\widetilde{\gamma}_{L}(0)\asymp n^{3\delta/4}\Delta
t^{3/2}.$

For $t\in[0,t_{1}]$, we get from (7.63)–(7.65) (and using (7.50)) that

(7.70) $\displaystyle\gamma_{L}(t)-t/{\mathfrak{r}}$
$\displaystyle=F_{0}(\varpi_{t}(\gamma_{L}(t)))+t\varpi_{t}(\gamma_{L}(t))$
$\displaystyle=(t-t_{c})\varpi_{t}(\gamma_{L}(t))+\frac{a}{3}\varpi_{t}(\gamma_{L}(t))^{3}+\cal
E(\varpi_{t}(\gamma_{L}(t))),$ (7.71)
$\displaystyle\widetilde{\gamma}_{L}(t)-t/{\mathfrak{r}}$
$\displaystyle=F_{0}(\widetilde{\varpi}_{t}(\widetilde{\gamma}_{L}(t)))+t(\widetilde{\varpi}_{t}(\widetilde{\gamma}_{L}(t))+\Delta
w_{0}({\widetilde{u}_{t}}(\widetilde{\gamma}_{L}(t))))$
$\displaystyle=(t-t_{c})\widetilde{\varpi}_{t}(\widetilde{\gamma}_{L}(t))+\frac{a}{3}\widetilde{\varpi}_{t}(\widetilde{\gamma}_{L}(t))^{3}+\cal
E(\widetilde{\varpi}_{t}(\widetilde{\gamma}_{L}(t)))+\operatorname{\mathcal{O}}(|{\widetilde{u}_{t}}(\widetilde{\gamma}_{L}(t))|\Delta
t).$

We first assume the following a priori bound:

(7.72) $|\gamma_{L}(t)-t/{\mathfrak{r}}|\leq n^{\delta}\Delta
t^{3/2},\quad|\widetilde{\gamma}_{L}(t)-t/{\mathfrak{r}}|\leq n^{\delta}\Delta
t^{3/2},\quad\forall t\in[0,t_{1}].$

Under (7.72), using (7.70) and (7.71), we can check that for any
$t\in[0,t_{1}]$,

$\begin{split}\varpi_{t}(\gamma_{L}(t))=\operatorname{\mathcal{O}}(n^{\delta/3}\Delta
t^{1/2}),\quad&u_{t}(\gamma_{L}(t))=F_{0}(w(\gamma_{L}(t)))=\operatorname{\mathcal{O}}(n^{\delta}\Delta
t^{3/2}),\\\
\widetilde{\varpi}_{t}(\widetilde{\gamma}_{L}(t))=\operatorname{\mathcal{O}}(n^{\delta/3}\Delta
t^{1/2}),\quad&{\widetilde{u}_{t}}(\widetilde{\gamma}_{L}(t))=F_{0}(\widetilde{\varpi}_{t}(\widetilde{\gamma}_{L}(t)))=\operatorname{\mathcal{O}}(n^{\delta}\Delta
t^{3/2}),\end{split}$

which imply that

(7.73)
$\begin{split}&|w_{t}(\gamma_{L}(t))-w_{t_{c}}(x_{c})|=|\varpi_{t}(\gamma_{L}(t))|=\operatorname{\mathcal{O}}(n^{\delta/3}\Delta
t^{1/2}),\\\
&|\widetilde{w}_{t}(\widetilde{\gamma}_{L}(t))-w_{t_{c}}(x_{c})|=|\widetilde{\varpi}_{t}(\widetilde{\gamma}_{L}(t))|+\operatorname{\mathcal{O}}(|{\widetilde{u}_{t}}(\widetilde{\gamma}_{L}(t))|)=\operatorname{\mathcal{O}}(n^{\delta/3}\Delta
t^{1/2}).\end{split}$

Plugging these estimates into (7.67) for $i=L$ yields that for
$t\in[0,t_{1}]$,

(7.74)
$\displaystyle\gamma_{L}^{\prime}(t)={\mathfrak{r}}^{-1}+\operatorname{\mathcal{O}}(n^{\delta/3}\Delta
t^{1/2}),\quad\widetilde{\gamma}_{L}^{\prime}(t)={\mathfrak{r}}^{-1}+\operatorname{\mathcal{O}}(n^{\delta/3}\Delta
t^{1/2}).$

By integrating them, we obtain that

(7.75)
$\gamma_{L}(t)-\gamma_{L}(0)-t/{\mathfrak{r}}=\operatorname{\mathcal{O}}(n^{\delta/3}\Delta
t^{3/2}),\quad\widetilde{\gamma}_{L}(t)-\gamma_{L}(0)-t/{\mathfrak{r}}=\operatorname{\mathcal{O}}(n^{\delta/3}\Delta
t^{3/2}),$

under (7.72). Note that (7.69) and (7.75) together imply (7.72). Thus, to show
(7.75) without assuming (7.72), we only need to consider an $n^{-10}t_{1}$-net
of $[0,t_{1}]$ and use a simple induction argument. More precisely, we define
a sequence of times ${\mathfrak{t}}_{k}:=kn^{-10}t_{1}$,
$k=0,1,\ldots,n^{10}$. First, the estimates (7.72) and (7.75) hold at
$t={\mathfrak{t}}_{0}$ by (7.69). Second, suppose (7.75) holds at some
${\mathfrak{t}}_{k}$. With (7.63)–(7.65), we can check that $\gamma_{L}(t)$
and $\widetilde{\gamma}_{L}(t)$ are Hölder-$1/3$ continuous in $t$. Thus, from
(7.69) and (7.75) at $t={\mathfrak{t}}_{k}$, we obtain that (7.72) holds
uniformly for all $t\in[{\mathfrak{t}}_{k},{\mathfrak{t}}_{k+1}]$. The
arguments above then imply that (7.75) holds at $t={\mathfrak{t}}_{k+1}$. With
mathematical induction in $k$, we conclude (7.75) for all $t\in[0,t_{1}]$.

Now, with (7.69) and (7.75), we get that

(7.76)
$\gamma_{L}(t)-t/{\mathfrak{r}}=(1+\operatorname{o}(1))(\widetilde{\gamma}_{L}(t)-t/{\mathfrak{r}})=(1+\operatorname{o}(1))\gamma_{L}(0)\asymp
n^{3\delta/4}\Delta t^{3/2}.$

Applying it to equations (7.70) and (7.71), we obtain that

(7.77)
$\begin{split}&\varpi_{t}(\gamma_{L}(t))=(1+\operatorname{o}(1))\widetilde{\varpi}_{t}(\widetilde{\gamma}_{L}(t))\asymp
n^{\delta/4}\Delta t^{1/2},\\\
&u_{t}(\gamma_{L}(t))=(1+\operatorname{o}(1)){\widetilde{u}_{t}}(\widetilde{\gamma}_{L}(t))=(1+\operatorname{o}(1))\gamma_{L}(0)\asymp
n^{3\delta/4}\Delta t^{3/2}.\end{split}$

Subtracting the equation (7.70) from (7.71) and applying (7.77) yield that

$\displaystyle|\varpi_{t}(\gamma_{L}(t))-\widetilde{\varpi}_{t}(\widetilde{\gamma}_{L}(t))|\cdot|\varpi_{t}(\gamma_{L}(t))|^{2}\lesssim\frac{a}{3}|\varpi_{t}(\gamma_{L}(t))^{3}-\widetilde{\varpi}_{t}(\widetilde{\gamma}_{L}(t))^{3}|\leq|\gamma_{L}(t)-\widetilde{\gamma}_{L}(t)|$
$\displaystyle\quad+|t-t_{c}||\varpi_{t}(\gamma_{L}(t))-\widetilde{\varpi}_{t}(\widetilde{\gamma}_{L}(t))|+|\cal
E(\varpi_{t}(\gamma_{L}(t)))-\cal
E(\widetilde{\varpi}_{t}(\widetilde{\gamma}_{L}(t)))|+\operatorname{\mathcal{O}}(|{\widetilde{u}_{t}}(\widetilde{\gamma}_{L}(t))|\Delta
t)$ (7.78)
$\displaystyle\lesssim|\gamma_{L}(t)-\widetilde{\gamma}_{L}(t)|+(\Delta
t+|\varpi_{t}(\gamma_{L}(t))|^{3})|\varpi_{t}(\gamma_{L}(t))-\widetilde{\varpi}_{t}(\widetilde{\gamma}_{L}(t))|+n^{3\delta/4}\Delta
t^{5/2}.$

Thus, we obtain that

$\displaystyle|\varpi_{t}(\gamma_{L}(t))-\widetilde{\varpi}_{t}(\widetilde{\gamma}_{L}(t))|$
$\displaystyle\lesssim\frac{n^{3\delta/4}\Delta
t^{5/2}+|\gamma_{L}(t)-\widetilde{\gamma}_{L}(t)|}{|\varpi_{t}(\gamma_{L}(t))|^{2}-\operatorname{\mathcal{O}}(\Delta
t+|\varpi_{t}(\gamma_{L}(t))|^{3})}$ (7.79) $\displaystyle\lesssim
n^{\delta/4}\Delta t^{3/2}+n^{-\delta/2}\Delta
t^{-1}|\gamma_{L}(t)-\widetilde{\gamma}_{L}(t)|,$

which, together with (7.68) for $i=L$, implies that

$\displaystyle|\gamma_{L}^{\prime}(t)-\widetilde{\gamma}_{L}^{\prime}(t)|\lesssim
n^{\delta/4}\Delta t^{3/2}+n^{-\delta/2}\Delta
t^{-1}|\gamma_{L}(t)-\widetilde{\gamma}_{L}(t)|.$

Finally, an application of the Grönwall’s inequality gives that

$\max_{0\leq t\leq t_{1}}|\gamma_{L}(t)-\widetilde{\gamma}_{L}(t)|\leq
n^{\delta/2}\Delta t^{5/2}.$

The proof for the bound on $|\gamma_{-L}(t)-\widetilde{\gamma}_{-L}(t)|$ is
similar.

####  7.6.3. Proof of (7.61)

To be concise, we abuse the notations and abbreviate $u(\xi):=u_{t_{1}}(\xi)$,
$\varpi(\xi):=\varpi_{t_{1}}(\xi)$ and
$\widetilde{u}(\xi):=\widetilde{u}_{t_{1}}(\xi)$,
$\widetilde{\varpi}(\xi):=\widetilde{\varpi}_{t_{1}}(\xi)$. By (7.63)–(7.65)
and (7.50), for any fixed
$\xi\in[\gamma_{-L}(t_{1})\wedge\widetilde{\gamma}_{-L}(t_{1}),\gamma_{L}(t_{1})\vee\widetilde{\gamma}_{L}(t_{1})]$,
$\varpi(\xi)$ and $\widetilde{\varpi}(\xi)$ satisfy the equations

(7.80) $\displaystyle(t_{1}-t_{c})\varpi+\frac{a}{3}\varpi^{3}+\cal
E(\varpi)=\xi-t_{1}/{\mathfrak{r}},$ (7.81)
$\displaystyle(t_{1}-t_{c})\widetilde{\varpi}+\frac{a}{3}\widetilde{\varpi}^{3}+\cal
E(\widetilde{\varpi})=\xi-
t_{1}/{\mathfrak{r}}+\operatorname{\mathcal{O}}(\Delta
t|F_{0}(\widetilde{\varpi})|).$

By equation (7.76), the chosen $\xi$ satisfies $|\xi-
t_{1}/{\mathfrak{r}}|\lesssim n^{3\delta/4}\Delta t^{3/2}$. Then, from (7.80),
(7.81) and (7.64), (7.65), we obtain that

(7.82) $|\varpi(\xi)|+|\widetilde{\varpi}(\xi)|\lesssim n^{\delta/4}\Delta
t^{1/2},\quad|u(\xi)|+|\widetilde{u}(\xi)|\lesssim n^{3\delta/4}\Delta
t^{3/2}.$

First, consider the case where $|\xi-t_{1}/{\mathfrak{r}}|>C\Delta t^{3/2}$
for a large enough constant $C>0$, so that the $\frac{a}{3}\varpi^{3}$ and
$\frac{a}{3}\widetilde{\varpi}^{3}$ terms dominate in (7.80) and (7.81). In
particular, we can choose $C$ such that

(7.83) $\varpi\asymp\widetilde{\varpi}\asymp|\xi-
t_{1}/{\mathfrak{r}}|^{1/3},\quad\frac{a}{3}|\varpi^{3}-\widetilde{\varpi}^{3}|>2(t_{1}-t_{c})|\varpi-\widetilde{\varpi}|.$

Subtracting the equations (7.80) and (7.81) and using a similar argument as in
(7.78) and (7.79), we get

(7.84) $|\varpi-\widetilde{\varpi}|\lesssim\frac{\Delta
t|F_{0}(\widetilde{\varpi})|}{|\varpi|^{2}-\operatorname{\mathcal{O}}(|\varpi|^{3})}\lesssim\frac{\Delta
t|\widetilde{\varpi}|^{3}+\Delta
t^{2}|\widetilde{\varpi}|}{|\varpi|^{2}}\lesssim n^{\delta/4}\Delta t^{3/2},$

where we used (7.82) and (7.83) in the last step.

We next consider the case where $|\xi-t_{1}/{\mathfrak{r}}|\leq C\Delta
t^{3/2}$. From (7.80) and (7.81), we get

(7.85) $|\varpi|\lesssim\Delta t^{1/2},\quad|\widetilde{\varpi}|\lesssim\Delta
t^{1/2}.$

Moreover, we can write (7.80) and (7.81) as

$\displaystyle\varpi^{3}+3\tau\varpi-2(y+\varepsilon_{1})=0,\quad\widetilde{\varpi}^{3}+3\tau\widetilde{\varpi}-2(y+\varepsilon_{2})=0,$

where $\tau:=(t_{1}-t_{c})/a$, $y:=3(\xi-t_{1}/{\mathfrak{r}})/(2a)$,
$\varepsilon_{1}=\operatorname{\mathcal{O}}(|\varpi|^{4})$ and
$\varepsilon_{2}=\operatorname{\mathcal{O}}(|\widetilde{\varpi}|^{4}+\Delta
t|F_{0}(\widetilde{\varpi})|)$. Using the general cubic formula, we obtain
that

(7.86)
$\begin{split}&\varpi=\alpha\left[(y+\varepsilon_{1})+\sqrt{(y+\varepsilon_{1})^{2}+\tau^{3}}\right]^{1/3}+\alpha^{-1}\left[(y+\varepsilon_{1})-\sqrt{(y+\varepsilon_{1})^{2}+\tau^{3}}\right]^{1/3},\\\
&\widetilde{\varpi}=\alpha\left[(y+\varepsilon_{2})+\sqrt{(y+\varepsilon_{2})^{2}+\tau^{3}}\right]^{1/3}+\alpha^{-1}\left[(y+\varepsilon_{2})-\sqrt{(y+\varepsilon_{2})^{2}+\tau^{3}}\right]^{1/3},\end{split}$

where $\alpha$ is a primitive cube root of unity chosen such that
$\mathrm{Im}\,\varpi<0$ and $\mathrm{Im}\,\widetilde{\varpi}<0$. With the
estimates $\tau\gtrsim\Delta t$, $|y|\lesssim\Delta t^{3/2}$, and
$\varepsilon_{1},\varepsilon_{2}=\operatorname{\mathcal{O}}(\Delta t^{2})$ by
(7.85), it is easy to check that

$\displaystyle\left|\left[(y+\varepsilon_{1})\pm\sqrt{(y+\varepsilon_{1})^{2}+\tau^{3}}\right]^{1/3}-\left[(y+\varepsilon_{2})\pm\sqrt{(y+\varepsilon_{2})^{2}+\tau^{3}}\right]^{1/3}\right|\lesssim\frac{|\varepsilon_{1}|+|\varepsilon_{2}|}{\tau}\lesssim\Delta
t,$

thereby giving that $|\varpi-\widetilde{\varpi}|\lesssim\Delta t$. Combining
this and (7.84), we obtain that for
$\xi\in[\gamma_{-L}(t_{1})\wedge\widetilde{\gamma}_{-L}(t_{1}),\gamma_{L}(t_{1})\vee\widetilde{\gamma}_{L}(t_{1})]$,

(7.87)
$|f_{t_{1}}(\xi)-\widetilde{f}_{t_{1}}(\xi)|\lesssim|w_{t_{1}}(\xi)-\widetilde{w}_{t_{1}}(\xi)|\lesssim|\varpi-\widetilde{\varpi}|+\operatorname{\mathcal{O}}(|\widetilde{z}|)\lesssim\Delta
t,$

where we also used $n^{3\delta/4}\Delta t^{1/2}\ll 1$ since
$0<\delta<\omega/2$. Since $f_{t_{1}}$ and $\widetilde{f}_{t_{1}}$ are bounded
away from $0$, we have

(7.88)
$|\rho_{t_{1}}(\xi)-\widetilde{\rho}_{t_{1}}(\xi)|=\frac{1}{\pi}\left|\arg^{*}f_{t_{1}}(\xi)-\arg^{*}\widetilde{f}_{t_{1}}(\xi)\right|\lesssim|f_{t_{1}}(\xi)-\widetilde{f}_{t_{1}}(\xi)|\lesssim\Delta
t.$

On the other hand, by Item 2 of 7.1 (whose proof also applies to
$\widetilde{\rho}_{t_{1}}$), we have

(7.89)
$\rho_{t_{1}}(\xi)\asymp\widetilde{\rho}_{t_{1}}(\xi)\asymp\begin{cases}\Delta
t^{1/2},&|\xi-{t_{1}}/{\mathfrak{r}}|\leq C\Delta t^{3/2}\\\
|\xi-{t_{1}}/{\mathfrak{r}}|^{1/3},&C\Delta
t^{3/2}\leq|\xi-{t_{1}}/{\mathfrak{r}}|\leq Cn^{3\delta/4}\Delta
t^{3/2}\end{cases}.$

Now, we compare the quantiles $\gamma_{i}({t_{1}})$ and
$\widetilde{\gamma}_{i}({t_{1}})$ through the following equation:

(7.90) $\int_{\gamma_{i}({t_{1}})}^{\gamma_{L}({t_{1}})}\rho_{t_{1}}(\xi){\rm
d}\xi=\frac{L-i}{n}=\int_{\widetilde{\gamma}_{i}({t_{1}})}^{\widetilde{\gamma}_{L}({t_{1}})}\widetilde{\rho}_{t_{1}}(\xi){\rm
d}\xi,\quad i\in[\\![{-L,L}]\\!].$

With (7.88), we can write

$\displaystyle\int_{\widetilde{\gamma}_{i}({t_{1}})}^{\widetilde{\gamma}_{L}({t_{1}})}\widetilde{\rho}_{t_{1}}(\xi){\rm
d}\xi$
$\displaystyle=\int_{\widetilde{\gamma}_{i}({t_{1}})}^{\widetilde{\gamma}_{L}({t_{1}})}\rho_{t_{1}}(\xi){\rm
d}\xi+\operatorname{\mathcal{O}}\left(\Delta
t|\widetilde{\gamma}_{L}({t_{1}})-\widetilde{\gamma}_{i}({t_{1}})|\right)$
$\displaystyle=\int_{\widetilde{\gamma}_{i}({t_{1}})}^{\gamma_{L}({t_{1}})}\rho_{t_{1}}(\xi){\rm
d}\xi+\operatorname{\mathcal{O}}\left(n^{3\delta/4}\Delta
t^{5/2}+n^{\delta/4}\Delta
t^{1/2}\left|\widetilde{\gamma}_{L}({t_{1}})-\gamma_{L}({t_{1}})\right|\right)$
$\displaystyle=\int_{\widetilde{\gamma}_{i}({t_{1}})}^{\gamma_{L}({t_{1}})}\rho_{t_{1}}(\xi){\rm
d}\xi+\operatorname{\mathcal{O}}(n^{3\delta/4}\Delta t^{5/2}),$

where in the second step we used (7.89) and the estimates $|\xi-
t_{1}/{\mathfrak{r}}|\lesssim n^{3\delta/4}\Delta t^{3/2}$,
$|\widetilde{\gamma}_{L}(t_{1})-\widetilde{\gamma}_{i}(t_{1})|\lesssim
n^{3\delta/4}\Delta t^{3/2}$ by (7.76), and in the third step we used (7.60).
Plugging it into (7.90) then gives

$\int_{\widetilde{\gamma}_{i}({t_{1}})\wedge\gamma_{i}({t_{1}})}^{\widetilde{\gamma}_{i}({t_{1}})\vee\gamma_{i}({t_{1}})}\rho_{t_{1}}(\xi){\rm
d}\xi\lesssim n^{3\delta/4}\Delta t^{5/2}.$

Combining this equation with (7.89), we get that

$\Delta t^{1/2}|\widetilde{\gamma}_{i}({t_{1}})-\gamma_{i}({t_{1}})|\lesssim
n^{3\delta/4}\Delta t^{5/2},$

which concludes (7.61).

## References

  * [ACVM11] Mark Adler, Mattia Cafasso, and Pierre Van Moerbeke.  From the Pearcey to the Airy process.  Electron. J. Probab., 16:1048–1064, 2011.
  * [ADPZ20] Kari Astala, Erik Duse, István Prause, and Xiong Zhong.  Dimer models and conformal structures.  Preprint arXiv:2004.02599, 2020.
  * [AFvM10] Mark Adler, Patrik L. Ferrari, and Pierre van Moerbeke.  Airy processes with wanderers and new universality classes.  Ann. Probab., 38(2):714 – 769, 2010.
  * [AG22] Amol Aggarwal and Vadim Gorin.  Gaussian unitary ensemble in random lozenge tilings.  Probab. Theory Related Fields, 184(3-4):1139–1166, 2022.
  * [Agg19] Amol Aggarwal.  Universality for lozenge tilings local statistics.  Preprint arXiv:1907.09991, 2019.
  * [AH21] Amol Aggarwal and Jiaoyang Huang.  Edge statistics for lozenge tilings of polygons, II: Airy line ensemble.  Preprint arXiv:2108.12874, 2021.
  * [AIM] AimPL: Kardar-Parisi-Zhang equation and universality class.  available at http://aimpl.org/kpzuniversality.
  * [AJv14] Mark Adler, Kurt Johansson, and Pierre van Moerbeke.  Double Aztec diamonds and the tacnode process.  Adv. Math., 252:518–571, 2014.
  * [AJvM18a] Mark Adler, Kurt Johansson, and Pierre van Moerbeke.  Lozenge tilings of hexagons with cuts and asymptotic fluctuations: a new universality class.  Math. Phys. Anal. Geom., 21(1):Paper No. 9, 53, 2018.
  * [AJvM18b] Mark Adler, Kurt Johansson, and Pierre van Moerbeke.  Tilings of non-convex polygons, skew-Young tableaux and determinantal processes.  Comm. Math. Phys., 364(1):287–342, 2018.
  * [AL22] Arka Adhikari and Benjamin Landon.  Local law and rigidity for unitary Brownian motion.  Preprint arXiv:2202.06714, 2022.
  * [AOVM10] Mark Adler, Nicolas Orantin, and Pierre Van Moerbeke.  Universality for the Pearcey process.  Phys. D, 239(12):924–941, 2010.
  * [AvM07] Mark Adler and Pierre van Moerbeke.  PDEs for the Gaussian ensemble with external source and the Pearcey distribution.  Comm. Pure Appl. Math., 60(9):1261–1292, 2007.
  * [AvM23] Mark Adler and Pierre van Moerbeke.  Double interlacing in random tiling models.  J. Math. Phys., 64(3):033509, 2023.
  * [BCJ22] Vincent Beffara, Sunil Chhita, and Kurt Johansson.  Local geometry of the rough-smooth interface in the two-periodic Aztec diamond.  Ann. App. Probab., 32(2):974 – 1017, 2022.
  * [BD11] Alexei Borodin and Maurice Duits.  Limits of determinantal processes near a tacnode.  Ann. Inst. Henri Poincaré Probab. Stat., 47(1):243–258, 2011\.
  * [BD19] Tomas Berggren and Maurice Duits.  Correlation functions for determinantal processes defined by infinite block Toeplitz minors.  Adv. Math., 356:106766, 2019.
  * [BF15] Alexei Borodin and Patrik L. Ferrari.  Random tilings and Markov chains for interlacing particles.  Preprint arXiv:1506.03910, 2015.
  * [BG12] Alexei Borodin and Vadim Gorin.  Lectures on integrable probability.  Preprint arXiv:1212.3351, 2012.
  * [BGR10] Alexei Borodin, Vadim Gorin, and Eric M. Rains.  $q$-distributions on boxed plane partitions.  Sel. Math., 16(4):731–789, 2010.
  * [BH98a] Edouard Brézin and Shinobu Hikami.  Level spacing of random matrices in an external source.  Phys. Rev. E, 58:7176–7185, Dec 1998.
  * [BH98b] Edouard Brézin and Shinobu Hikami.  Universal singularity at the closure of a gap in a random matrix theory.  Phys. Rev. E, 57(4):4140, 1998.
  * [BK08] Alexei Borodin and Jeffrey Kuan.  Asymptotics of Plancherel measures for the infinite-dimensional unitary group.  Adv. Math., 219(3):894–931, 2008.
  * [CDKL20] Christophe Charlier, Maurice Duits, Arno B.J. Kuijlaars, and Jonatan Lenells.  A periodic hexagon tiling model and non-Hermitian orthogonal polynomials.  Comm. Math. Phys., 378(1):401–466, 2020.
  * [CEKS19] Giorgio Cipolloni, László Erdős, Torben Krüger, and Dominik Schröder.  Cusp universality for random matrices, II: The real symmetric case.  Pure Appl. Anal., 1(4):615 – 707, 2019.
  * [CEP96] Henry Cohn, Noam Elkies, and James Propp.  Local statistics for random domino tilings of the Aztec diamond.  Duke Math. J., 85(1):117–166, 1996.
  * [Cer06] Raphaël Cerf.  The Wulff Crystal in Ising and Percolation Models, volume 1878 of Lecture Notes in Mathematics.  Springer Berlin, Heidelberg, 2006.
  * [CH14] Ivan Corwin and Alan Hammond.  Brownian Gibbs property for Airy line ensembles.  Invent. Math., 195(2):441–508, 2014.
  * [CH16] Ivan Corwin and Alan Hammond.  KPZ line ensemble.  Probab. Theory Related Fields, 166:67–185, 2016.
  * [Cha20] Christophe Charlier.  Doubly periodic lozenge tilings of a hexagon and matrix valued orthogonal polynomials.  Preprint arXiv:2001.11095, 2020.
  * [CK01] Raphaël Cerf and Richard Kenyon.  The low-temperature expansion of the Wulff crystal in the 3D Ising model.  Comm. Math. Phys., 222(1):147–179, 2001.
  * [CKP01] Henry Cohn, Richard Kenyon, and James Propp.  A variational principle for domino tilings.  J. Amer. Math. Soc., 14(2):297–346, 2001.
  * [CLP98] Henry Cohn, Michael Larsen, and James Propp.  The shape of a typical boxed plane partition.  New York J. Math., 4:137–165, 1998.
  * [CP16] Mireille Capitaine and Sandrine Péché.  Fluctuations at the edges of the spectrum of the full rank deformed GUE.  Probab. Theory Related Fields, 165(1):117–161, Jun 2016.
  * [DJM16] Erik Duse, Kurt Johansson, and Anthony Metcalfe.  The cusp-Airy process.  Electron. J. Probab., 21:Paper No. 57, 50, 2016.
  * [DK20] Maurice Duits and Arno B.J. Kuijlaars.  The two-periodic Aztec diamond and matrix valued orthogonal polynomials.  J. Eur. Math. Soc., 23(4):1075–1131, 2020.
  * [DKS92] Roland Dobrushin, Roman Kotecký, and Senya Shlosman.  Wulff Construction: A Global Shape from Local Interaction, volume 104 of Translations of Mathematical Monographs.  American Mathematical Society, 1992.
  * [DM18] Erik Duse and Anthony Metcalfe.  Universal edge fluctuations of discrete interlaced particle systems.  Ann. Math. Blaise Pascal, 25(1):75–197, 2018.
  * [DM21] Evgeni Dimitrov and Konstantin Matetski.  Characterization of Brownian Gibbsian line ensembles.  Ann. Probab., 49(5):2477–2529, 2021.
  * [DNV19] Duncan Dauvergne, Mihai Nica, and Bálint Virág.  Uniform convergence to the Airy line ensemble.  Preprint arXiv:1907.10160, 2019.
  * [DSS10] Daniela De Silva and Ovidiu Savin.  Minimizers of convex functionals arising in random surfaces.  Duke Math. J., 151(3):487–532, 2010.
  * [EKS20] László Erdős, Torben Krüger, and Dominik Schröder.  Cusp universality for random matrices I: Local law and the complex Hermitian case.  Comm. Math. Phys., 378(2):1203–1278, 2020.
  * [Gor17] Vadim Gorin.  Bulk universality for random lozenge tilings near straight boundaries and for tensor products.  Comm. Math. Phys., 354(1):317–344, 2017.
  * [Gor21] Vadim Gorin.  Lectures on random lozenge tilings, volume 193 of Cambridge Stud. Adv. Math. Cambridge Univ. Press, 2021.
  * [GP19] Vadim Gorin and Leonid Petrov.  Universality of local statistics for noncolliding random walks.  Ann. Probab., 47(5):2686–2753, 2019.
  * [HHN16] Walid Hachem, Adrien Hardy, and Jamal Najim.  Large complex correlated Wishart matrices: the Pearcey kernel and expansion at the hard edge.  Electron. J. Probab., 21(none):1 – 36, 2016.
  * [Hua21] Jiaoyang Huang.  Edge statistics for lozenge tilings of polygons, I: Concentration of height function on strip domains.  Preprint arXiv:2108.12872, 2021.
  * [Joh18] Kurt Johansson.  Edge fluctuations of limit shapes.  In Current Developments in Mathematics 2016, pages 47–110. Int. Press, Somerville, MA, 2018.
  * [JPS98] William Jockusch, James Propp, and Peter Shor.  Random domino tilings and the arctic circle theorem.  arXiv:9801068, 1998.
  * [Ken08] Richard Kenyon.  Height fluctuations in the honeycomb dimer model.  Comm. Math. Phys., 281(3):675–709, 2008.
  * [KO07] Richard Kenyon and Andrei Okounkov.  Limit shapes and the complex Burgers equation.  Acta Math., 199(2):263–302, 2007.
  * [KOR02] Wolfgang König, Neil O’Connell, and Sébastien Roch.  Non-colliding random walks, tandem queues, and discrete orthogonal polynomial ensembles.  Electron. J. Probab., 7(none):1 – 24, 2002.
  * [KOS06] Richard Kenyon, Andrei Okounkov, and Scott Sheffield.  Dimers and amoebae.  Ann. of Math. (2), 163:1019–1056, 2006.
  * [Las19] Benoit Laslier.  Local limits of lozenge tilings are stable under bounded boundary height perturbations.  Probab. Theory Related Fields, 173(3-4):1243–1264, 2019.
  * [Mkr21] Sevak Mkrtchyan.  Turning Point Processes in Plane Partitions with Periodic Weights of Arbitrary Period, pages 497–527.  Springer International Publishing, Cham, 2021.
  * [Oko02] Andrei Okounkov.  Symmetric functions and random partitions.  In Symmetric functions 2001: surveys of developments and perspectives, pages 223–252. Springer, 2002.
  * [OR07] Andrei Okounkov and Nicolai Reshetikhin.  Random skew plane partitions and the Pearcey process.  Comm. Math. Phys., 269(3):571–609, 2007.
  * [Pet14] Leonid Petrov.  Asymptotics of random lozenge tilings via Gelfand–Tsetlin schemes.  Probab. Theory Relat. Fields, 160(3-4):429–487, 2014.
  * [She05] Scott Sheffield.  Random surfaces.  Astérisque, (304), 2005.
  * [TW06] Craig A. Tracy and Harold Widom.  The Pearcey process.  Comm. Math. Phys., 263(2):381–400, Apr 2006.
  * [Wu21] Xuan Wu.  The Bessel line ensemble.  Preprint arXiv:2109.09070, 2021.


Ultra-chaos: an insurmountable objective obstacle of reproducibility and
replicability

Shijun Liao1,2,3 ***The corresponding author, email address:
<EMAIL_ADDRESS>and Shijie Qin 2

1 State Key Laboratory of Ocean Engineering, Shanghai 200240, China

2 Center of Marine Numerical Experiment, School of Naval Architecture, Ocean
and Civil Engineering, Shanghai Jiaotong University, China

3 School of Physics and Astronomy, Shanghai Jiaotong University, China

Abstract In this paper, a new concept, i.e. ultra-chaos, is proposed for the
first time. Unlike a normal-chaos, statistical properties such as the
probability density functions (PDF) of an ultra-chaos are sensitive to tiny
disturbances. We illustrate that ultra-chaos is widely existed and thus has
general scientific meanings. It is found that statistical non-reproducibility
is an inherent property of an ultra-chaos so that an ultra-chaos is at a
higher-level of disorder than a normal-chaos. Thus, it is impossible in
practice to replicate experimental/numerical results of an ultra-chaos even in
statistical meanings, since random environmental noises always exist and are
out of control. Thus, the ultra-chaos should be an insurmountable obstacle of
reproducibility and replicability. Similar to Gödel’s incompleteness theorem,
such kind of “incompleteness of reproducibility” reveals a limitation of our
traditional scientific paradigm based on reproducible experiments, which can
be traced back to Galileo. The ultra-chaos opens a new door and possibility to
study chaos theory, turbulence theory, computational fluid dynamics (CFD), the
statistical significance, reproducibility crisis, and so on.

Keywords Chaos, statistical stability, reproducibility and replication.

AMS 35N05, 65P20

##  1 Introduction

The chaos theory [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] is widely regarded as the
third greatest scientific revolution in physics in 20th century, comparable to
Einstein’s theory of relativity and the quantum mechanics. It is Poincaré [1]
who first discovered “the sensitivity dependence on initial conditions” (SDIC)
of chaotic systems, which was rediscovered by Lorenz [2] with a popular name
“butterfly-effect”: the exact time of formation and the exact path of a
tornado might be influenced by tiny disturbances such as a distant butterfly
that flapped its wings several weeks earlier. In addition, Lorenz [11] further
discovered “the sensitivity dependence on numerical algorithms” (SDNA) of
chaotic systems: computer-generated chaotic numerical simulations given by
different algorithms in single/double precision quickly depart from each other
with distinct deviations. Naturally, such kind of non-replicability of chaotic
trajectory leaded to some heated debates on the credence of numerical
simulations of chaos, and brought a crisis of confidence: some even made a
rather pessimistic conclusion that “for chaotic systems, numerical convergence
cannot be guaranteed forever” [12].

In order to gain a reproducible/reliable numerical simulations of chaotic
trajectory, Liao [13] suggested a numerical strategy, namely the “Clean
Numerical Simulation” (CNS). In the frame of the CNS [13, 14, 15, 16, 17, 18,
19, 20, 21, 22, 23, 24, 25, 26], the temporal/spatial truncation errors are
reduced to a required tiny level by means of a high enough order of Taylor
expansion in time and a fine enough spatial discretization with spatial
Fourier expansion, respectively. Besides, the round-off error is reduced to a
required tiny level by means of reserving a large enough number of significant
digits for all physical/numerical variables and parameters in multiple
precision [27]. Furthermore, an additional simulation with even smaller
numerical noises is needed to determine the so-called “critical predictable
time” $T_{c}$ by comparing these two simulations, so that the numerical noise
is negligible and thus the computer-generated result is reproducible/reliable
within the whole spatial domain in the time interval $t\in[0,T_{c}]$.

Figure 1: Influence of tiny noises to the statistics of a normal-chaos. (a)
The probability density function (PDF) of $x(t)$; (b) The autocorrelation
functions (ACF) of $x(t)$. The statistic results are based on the chaotic
simulations $x(t)$ in $0\leq t\leq 10000$ governed by the Lorenz equations
(1)-(4) (with one positive Lyapunov exponents), given by the CNS (red line)
and the Runge-Kutta algorithms (symbols) with double-precision (RKwD) or
single-precision (RKwS) using different time-step $\Delta t$.

Figure 2: Influence of tiny noises to the statistics of a hyper-chaos. (a) The
probability density function (PDF) of $x(t)$; (b) The autocorrelation
functions (ACF) of $x(t)$. The statistic results are based on the simulations
$x(t)$ in $0\leq t\leq 10000$ governed by the four-dimensional Rössler system
(5)-(9) (with two positive Lyapunov exponents), given by the CNS (red line)
and the Runge-Kutta algorithms (symbols) with double-precision (RKwD) or
single-precision (RKwS) using different time-step $\Delta t$.

The CNS has been successfully applied to gain reproducible/reliable
simulations of trajectories of many chaotic systems, such as Lorenz equation
[16], Rayleigh-Bénard turbulent flows [18], and some spatiotemporal chaotic
systems related to the complex Ginzburg-Landau equation [22], the damped
driven sine-Gordon equation [23] and the chaotic motion of a free fall desk
[24]. Especially, more than 2000 new periodic orbits of three-body system have
been found by means of the CNS [19, 20, 21], which were reported twice by the
popular magazine New Scientist [28, 29], because only three families of
periodic orbits of the three-body problem have been reported in three hundred
years since Newton mentioned the famous problem in 1687. All of these
illustrate the validity of the CNS for chaos.

It should be emphasized that such a reproducible/reliable result given by the
CNS provides us a “true” solution, and more importantly, a kind of
reproducibility and replicability of chaotic trajectory. Such kind of
trajectory reproducibility and replicability can remain in a long enough
interval of time $t\in[0,T_{c}]$ as long as the background numerical noises
could be globally reduced to a rather tiny level [22, 23, 24]. This kind of
strict reproducibility and replicability of chaotic trajectory provides us a
confidence of credence/reliability and especially a benchmark solution. For
example, by means of an algorithm based on the CNS, Liao and Wang [16] gained
a computer-generated chaotic simulation of Lorenz equation, which is
reproducible/reliable in a quite long interval of time $0\leq t\leq 10000$
(Lyapunov unit) and thus provides us a true result and a benchmark solution of
the Lorenz equation in $t\in[0,10000]$, while numerical simulations given by
other algorithms in single/double precision are reproducible/reliable only in
a quite smaller interval $0\leq t\leq 32$ (Lyapunov unit) and has a distinct
deviation from the benchmark solution for $t>32$. Having the benchmark
solution of the Lorenz equation in such a long enough interval of time
$t\in[0,10000]$, it is possible to investigate the influence of numerical
noise to statistics of computer-generated chaotic simulations given by other
algorithms in single/double precision.

Note that, it has been widely believed (or assumed) that statistics of a chaos
should be “stable” even if its trajectory is sensitive to small disturbances.
Mostly, this is indeed true, such as the chaos governed by the Lorenz
equations (1)-(4) [13, 16] with one positive Lyapunov exponent, as shown in
Figure 1, and the so-called hyper-chaos governed by the four-dimensional
Rössler system (5)-(9) with two positive Lyapunov exponents [30], as shown in
Figure 1, respectively. Note that Figure 1 is given by Lorenz equations (with
one positive Lyapunov exponents)

$\displaystyle\dot{x}=\sigma\,(y-x),$ (1) $\displaystyle\dot{y}=R\,x-y-x\,z,$
(2) $\displaystyle\dot{z}=x\,y+B\,z$ (3)

in the case of $\sigma=10$, $R=28$, and $B=-8/3$, with the initial condition

$x(0)=-15.8,\;\;y(0)=-17.48,\;\;z(0)=35.64.$ (4)

Liao & Wang [16] gained a trajectory-replicable chaotic simulation of the
above-mentioned Lorenz equations in a quite long interval $t\in[0,10000]$
(Lorenz unit time) by means of a parallel algorithm of the CNS using the
$3500$th-order Taylor expansion and the $4180$-digit multiple-precision for
all physical/numerical parameters and variables, which is used here as the
benchmark solution (marked by CNS). For comparison, we also solve the Lorenz
equations (1)-(4) by means of the $4$th-order Runge-Kutta’s method with
double-precision (RKwD) or single-precision (RKwS) using different time-step
$\Delta t$, corresponding to different levels of numerical/environmental
noises.

Figure 2 is given by the hyper-chaotic Rossler system (with two positive
Lyapunov exponents) [30]

$\displaystyle\dot{x}=-\,y-z,$ (5) $\displaystyle\dot{y}=x+a\,y+w,$ (6)
$\displaystyle\dot{z}=b+x\,z,$ (7) $\displaystyle\dot{w}=-\,c\,z+d\,w$ (8)

in the case of $a=0.25$, $b=3$, $c=0.5$ and $d=0.05$, with the initial
condition

$x(0)=-20,\;\;y(0)=z(0)=0,\;\;w(0)=15.$ (9)

We gained a trajectory-replicable chaotic simulation of this hyper-chaotic
Rossler system in a quite long interval $t\in[0,10000]$ by means of a parallel
algorithm of the CNS with the $200$th-order Taylor expansion and $500$-digit
multiple-precision for all physical/numerical parameters and variables using
the time-step $\Delta t=0.001$, which is used here as the benchmark solution
(marked by CNS). For comparison, we also solve this hyper-chaotic Rossler
system by means of the $4$th-order Runge-Kutta’s method with double-precision
(RKwD) or single-precision (RKwS) using different time-step $\Delta t$,
corresponding to different levels of numerical/environmental noises.

It should be emphasized that, although the chaotic trajectories of the above-
mentioned two chaotic systems are quite sensitive to small disturbances, their
PDFs are not sensitive to numerical noises. However, it has been reported [22,
23] that even statistics (such as probability density function) of some
chaotic systems are sensitive to numerical noises. This is quite unusual,
which reveals the existence of chaotic systems that might be at a higher level
of disorder. We thought hard about this surprising result for a long time and
realized that this kind of phenomena are so important that some new concepts
should be proposed, since a new concept might be more fundamental than lots of
examples. Thus, we define here the following new concepts:

  * 1.

normal-chaos: a chaotic system, whose statistics are not sensitive to small
disturbances.

  * 2.

ultra-chaos: a chaotic system, whose statistics are sensitive to small
disturbances.

Note that Figure 2 clearly illustrates that the well-known “hyper-chaos” is
just a normal-chaos, i.e. its statistics is not sensitive to the
numerical/environmental noises. This reveals the novelty of ultra-chaos, a new
concept we proposed in this paper.

In this paper, using the CNS as a powerful tool to investigate the influence
of tiny external disturbances, we illustrate that the so-called ultra-chaos
indeed widely exist. This fact should have important scientific meanings, as
discussed in § 3.

##  2 An illustrative example of ultra-chaos

Note that, by means of the CNS, the artificial background numerical noises can
be globally reduced to a required tiny level much smaller than the objective
environmental noises, so that we can accurately study the influence of the
environmental disturbances to the statistics of some chaos. For example, let
us consider here the damped driven sine-Gordon equation [31, 32, 33], but with
a Gaussian white noise $\epsilon(x,t)$:

$u_{tt}=u_{xx}-\sin(u)-\alpha
u_{t}+\Gamma\sin(\omega\,t-\lambda\,x)+\epsilon(x,t)$ (10)

subject to a periodic boundary condition

$u(x+l,t)=u(x,t),$ (11)

where the subscript denotes the derivative, $x$ and $t$ are the spatial and
temporal variables, $\alpha$ is a constant related to the damped friction,
$\Gamma$ is a constant related to the external force, $\omega$ is the temporal
frequency, $\lambda=2\pi/l$ is the spatial frequency with $l$ being the total
length of the system, respectively. Here, $\epsilon(x,t)$ corresponds to the
tiny environmental noises. Without loss of generality, we follow Chacón et al.
[31] to choose the following values of these physical parameters

$\omega=\frac{3}{5},\;\alpha=\frac{1}{10},\;\Gamma=\frac{461}{500},\;l=500,\;\lambda=\frac{2\pi}{l},$
(12)

subject to the zero initial condition

$u(x,0)=0,\hskip 14.22636ptu_{t}(x,0)=0.$ (13)

As reported by Qin & Liao [23], when there are no environmental noises, i.e.
$\epsilon(x,t)=0$, the above model corresponds to a spatiotemporal chaos,
whose statistics are sensitive to numerical noises, and thus is an ultra-
chaos. However, numerical noises are artificial and thus have no physical
meanings. So, in this paper, using the CNS as a powerful tool, we investigate
the influence of tiny external disturbances to its statistical properties.
This has important meanings in physics, since tiny external disturbances are
unavoidable in practice. Let $\sigma_{n}$ denote its standard deviation of
external disturbances $\epsilon(x,t)$ in Gaussian white noise. We use the same
CNS algorithm [23] to gain reproducible/reliable numerical simulations in the
three cases $\sigma_{n}=0$, $\sigma_{n}=10^{-18}$ and $\sigma_{n}=10^{-20}$,
respectively, where $\sigma_{n}=0$ corresponds to the benchmark solution given
by the CNS without any objective environmental noise, the others correspond to
the reproducible/reliable simulations with the influence of the corresponding
tiny environmental noises. Unlike Qin & Liao [23] who used a CNS algorithm
with $N=16384$ and $N_{s}=60$ to gain reliable simulations in $t\in[0,900]$,
we discretized the spatial domain here much better by $N=2^{16}=65536$
equidistant points, used the multiple precision with $N_{s}=230$ significant
digits for all variables and parameters, and applied the variable stepsize
scheme in the temporal dimension with a given allowed tolerance
$tol=10^{-230}$ of the governing equations. In this way, the corresponding
background numerical noises (related to truncation error and round-off error)
are much less than the tiny external disturbances. Besides, according to Qin &
Liao [23], it holds

$T_{c}\approx\min\left\\{0.0558N+10.7,16.5N_{s}-87.8\right\\}.$

Thus, we can gain the reproducible/reliable chaotic simulations within the
whole spatial domain in a much larger temporal interval $t\in[0,3600]$ in the
above-mentioned three cases, which is long enough for our statistical analysis
reported in this paper.

Figure 3: Influence of small disturbances to the total spectrum energy
$E_{s}(t)$ of an ultra-chaos. The curves are based on the CNS results in
$t\in[0,3600]$ of an ultra-chaos governed by the damped driven sine-Gordon
equations (10)-(13) in case of $\sigma_{n}=0$ (red), $\sigma_{n}=10^{-18}$
(black) and $\sigma_{n}=10^{-20}$ (blue), respectively, with the maximum
relative error 10880%, where $\sigma_{n}$ denotes the standard deviation of
Gaussian white noise.

Note that $\sigma_{n}=0$ corresponds to no external disturbance. Using the
reproducible/reliable result in case of $\sigma_{n}=0$ as the benchmark
solution, we can accurately investigate the influence of the tiny
environmental noises in cases of $\sigma_{n}=10^{-18}$ and
$\sigma_{n}=10^{-20}$, respectively. It is found that the tiny environmental
noises are equivalent to the numerical noises in essence: the small
disturbances indeed lead to huge and distinct deviations of chaotic
trajectories and many physical quantities from the benchmark solution (in case
of $\sigma_{n}=0$) in $t>360$ when the tiny external disturbances are enlarged
up to the macroscopic level. For example, the total spectrum energy $E_{s}(t)$
in case of $\sigma_{n}=10^{-18}$ and $\sigma_{n}=10^{-20}$ have huge
deviations from that of the benchmark solution (in case of $\sigma_{n}=0$)
with the maximum relative error 10880%, as shown in Figure 5. It indicates
that, due to the existence of the tiny environmental noises, the ultra-chaotic
system under consideration might contain much larger energy than that without
any environmental noises.

Figure 4: Influence of small disturbances to the spatial statistics of an
ultra-chaos. (a) The spatial mean $\mu(t)$; (b) The spatial Root-Mean-Square
(RMS) $u_{rms}(t)$. The curves are based on the CNS results in $t\in[0,3600]$
of an ultra-chaos governed by the damped driven sine-Gordon equations
(10)-(13) in case of $\sigma_{n}=0$ (red), $\sigma_{n}=10^{-18}$ (black) and
$\sigma_{n}=10^{-20}$ (blue), with the maximum absolute error $67.21$ for
$\mu(t)$ and the maximum relative error $675\%$ for $u_{rms}(t)$,
respectively, where $\sigma_{n}$ denotes the standard deviation of Gaussian
white noise.

Figure 5: Influence of small disturbances to the temporal statistics of an
ultra-chaos. (a) The temporal mean $\mu(x)$; (b) The temporal Root-Mean-Square
(RMS) $u_{rms}(x)$. The curves are based on the CNS results in $t\in[0,3600]$
of an ultra-chaos governed by the damped driven sine-Gordon equations
(10)-(13) in case of $\sigma_{n}=0$ (red), $\sigma_{n}=10^{-18}$ (black) and
$\sigma_{n}=10^{-20}$ (blue), with the maximum absolute error $33.98$ for
$\mu(x)$ and the maximum relative error $187\%$ for $u_{rms}(x)$,
respectively, where $\sigma_{n}$ denotes the standard deviation of Gaussian
white noise.

How about their statistic properties? Figure 4 shows the influence of small
disturbances to the spatial statistics, i.e. the spatial mean $\mu(t)$ and the
spatial root-mean-square (RMS) $u_{rms}(t)$, in case of $\sigma_{n}=0$,
$\sigma_{n}=10^{-18}$ and $\sigma_{n}=10^{-20}$, respectively, with the
maximum absolute error $67.21$ for $\mu(t)$ and the maximum relative error
$675\%$ for $u_{rms}(t)$. Note that, when there exist no environmental noises
($\sigma_{n}=0$), the spatial mean $\mu(t)$ is always zero and $u_{rms}(t)$ is
always less than 20. However, the tiny environmental noise leads to a
gradually increasing deviation of $\mu(t)$ from zero and results in
$u_{rms}(t)>20$ in $t\in[2200,3600]$. Figure 5 shows the influence of the
small disturbances to the temporal statistics, i.e. the temporal mean $\mu(x)$
and the temporal root-mean-square (RMS) $u_{rms}(x)$, in case of
$\sigma_{n}=0$, $\sigma_{n}=10^{-18}$ and $\sigma_{n}=10^{-20}$, respectively,
with the maximum absolute error $33.98$ for $\mu(x)$ and the maximum relative
error $187\%$ for $u_{rms}(x)$. Note that, when there exist no environmental
noises ($\sigma_{n}=0$), we always have $|\mu(x)|<10$ and $u_{rms}(x)<15$.
However, due to the tiny environmental noise, it mostly holds $|\mu(x)|>10$
and $u_{rms}(x)>15$. Note that, when there exist no environmental noises
($\sigma_{n}=0$), the global mean $\mu$ is always zero, but the tiny
environmental noise leads to the huge deviations of the global mean
$\mu=14.12$ in the case of $\sigma_{n}=10^{-18}$ and $\mu=-17.64$ in the case
of $\sigma_{n}=10^{-20}$, respectively.

Figure 6: Influence of small disturbances to the probability density functions
(PDFs) of an ultra-chaos. The PDFs of $u(x,t)$ are based on the CNS results in
$t\in[0,3600]$ of an ultra-chaos governed by the damped driven sine-Gordon
equations (10)-(13) in case of $\sigma_{n}=0$ (red), $\sigma_{n}=10^{-18}$
(black) and $\sigma_{n}=10^{-20}$ (blue), respectively, where the relative
error between the PDFs of $u=0$ given by $\sigma_{n}=0$ and
$\sigma_{n}=10^{-20}$ reaches $62\%$, and $\sigma_{n}$ denotes the standard
deviation of Gaussian white noise.

Figure 7: Influence of small disturbances to the autocorrelation functions
(ACFs) of an ultra-chaos. (a) ACF of $u(0,t)$; (b) ACF of $Re(a_{1}(t))$. The
curves are based on the CNS results $u(x,t)$ in $t\in[0,3600]$ of an ultra-
chaos governed by the damped driven sine-Gordon equations (10)-(13) in case of
$\sigma_{n}=0$ (red), $\sigma_{n}=10^{-18}$ (black) and $\sigma_{n}=10^{-20}$
(blue), respectively, where $Re(a_{1})$ is the real part of the coefficient
$a_{1}(t)$ of Fourier series (14) and $\sigma_{n}$ denotes the standard
deviation of Gaussian white noise.

Figure 6 shows the influence of small disturbances to the probability density
functions (PDFs) of the chaotic simulation $u(x,t)$ in case of $\sigma_{n}=0$,
$\sigma_{n}=10^{-18}$ and $\sigma_{n}=10^{-20}$, respectively, where the
relative error of the PDFs at $u=0$ in case of $\sigma_{n}=0$ and
$\sigma_{n}=10^{-20}$ reaches $62\%$. Note that, when there exist no
environmental noises ($\sigma_{n}=0$), the PDF has a kind of symmetry about
$u=0$. However, such kind of symmetry is lost due to appearance of the
environmental noises. In other words, the environmental noises result in the
symmetry breaking of the PDFs of $u(x,t)$. Besides, the tiny difference
between the external environmental noises (such as $\sigma_{n}=10^{-18}$ and
$\sigma_{n}=10^{-20}$) leads to huge deviation of their PDFs of $u(x,t)$, as
shown in Figure 6.

Figure 7(a) shows the influence of small external disturbances to the
autocorrelation functions (ACFs) of the time series $u(0,t)$ in case of
$\sigma_{n}=0$, $\sigma_{n}=10^{-18}$ and $\sigma_{n}=10^{-20}$. Obviously,
the ACF of $u(0,t)$ has sensitivity dependance on the environmental noises
(SDEN). Especially, the ACF of $u(0,t)$ in case of $\sigma_{n}=10^{-18}$ has
huge deviations from those in case of $\sigma_{n}=10^{-20}$ and
$\sigma_{n}=0$. Besides, expand $u(x,t)$ in Fourier series

$u(x,t)=\sum_{k=0}^{+\infty}a_{k}(t)\exp(k\lambda x{\bf i}),$ (14)

where $a_{k}(t)$ denotes the amplitude and ${\bf i}=\sqrt{-1}$. It is found
that the autocorrelation functions (ACFs) of the amplitude $a_{k}(t)$ of the
Fourier series of $u(x,t)$ also have sensitivity dependence on the
environmental noises, too, as shown in Figure 7(b) for the comparison of the
ACFs of the real part of $a_{1}(t)$ in case of $\sigma_{n}=0$,
$\sigma_{n}=10^{-18}$ and $\sigma_{n}=10^{-20}$.

Figure 8: Phase plot of chaos of the damped driven sine-Gordon equation (10),
(11) and (13). Red domain: ultra-chaos; Blue domain: normal-chaos; Gray
domain: non-chaos (the state without any chaos). This phase plot is given by
different values of the amplitude $\Gamma$ and temporal frequency $\omega$ of
external force, with the constant $\alpha=1/10$ related to the damped
friction, the total length $l=500$ of the system and the spatial frequency
$\lambda=2\pi/l$.

Figure 9:  Influence of tiny external noises to the statistics of an ultra-
chaos and a normal-chaos. (a) and (b): the total spectrum energy $E_{s}(t)$;
(c) and (d): the probability density function (PDF). The curves are based on
the CNS results in $t\in[0,3600]$ of an ultra-chaos (a) and (c) in case of
$\omega=0.97$ (with the maximum Lyapunov exponent $\lambda_{max}=0.13$) and a
normal-chaos (b) and (d) in case of $\omega=0.98$ (with the maximum Lyapunov
exponent $\lambda_{max}=0.016$) governed by the damped driven sine-Gordon
equations (10), (11) and (13) under the two tiny external disturbances with
the standard deviation $\sigma_{n}=10^{-18}$ (black) and $\sigma_{n}=10^{-20}$
(blue) of Gaussian white noises, respectively, when $\Gamma=0.6$ and the other
parameters are the same as (12).

Therefore, the tiny external disturbances indeed lead to the great deviations
of chaotic simulations of the damped driven sine-Gordon equation not only in
trajectories but also in statistics. In other words, statistical results of an
ultra-chaos are sensitive to small external disturbances.

Does such kind of ultra-chaos widely exist? Note that the above-mentioned
ultra-chaos is found by using the physical parameters (12). Using various
amplitude $\Gamma$ and temporal frequency $\omega$ of external force, while
keeping other physical parameters unchanged, we find that the so-called ultra-
chaos widely exists for the damped driven sine-Gordon equation (10), as shown
in Figure 8. Note that, when $\Gamma=0.6$ and $\omega$ increases from 0 to
1.4, the system transfers from non-chaos to ultra-chaos, then from ultra-chaos
to normal-chaos, and final from normal-chaos to non-chaos. For example, let us
consider the two cases of $\omega=0.97$ and $\omega=0.98$ when $\Gamma=0.6$,
where other physical parameters are the same as (12). Note that, when
$\omega=0.98$, the total spectrum energy $E_{s(t)}$ and the PDFs are not
sensitive to the tiny external disturbances, corresponding to a normal-chaos
(with the maximum Lyapunov exponent $\lambda_{max}=0.016$), as shown in Figure
9. However, when $\omega=0.97$, the total spectrum energy $E_{s(t)}$ and
especially the PDFs are rather sensitive to the very tiny external
disturbances, corresponding to an ultra-chaos (with the maximum Lyapunov
exponent $\lambda_{max}=0.13$). Thus, as $\omega$ decreases from 0.98 to 0.97,
the system suddenly changes from a normal-chaos to an ultra-chaos. This
example illustrates that an ultra-chaos is indeed quite different from a
normal-chaos in essence. All of these also illustrate that the so-called
ultra-chaos indeed widely exist and thus has general meanings.

This is a surprising result. Such kind of chaos with statistical sensitivity
on tiny external disturbances has never been reported, to the best of our
knowledge. Note that the four-dimensional Rössler equations [30] with two
positive Lyapunov exponents belongs to the so-called hyper-chaos [4, 34, 35,
36, 30], i.e. a chaos with at least two positive Lyapunov exponents. But, this
kind of hyper-chaos is not an ultra-chaos: it belongs to normal-chaos. This
reveals that the ultra-chaos is essentially different from the previously
reported types of chaos and thus is indeed a new concept.

It is worth noting that the environmental noises considered in this article
are at a quite tiny level, i.e. $10^{-18}$ and $10^{-20}$, which is at least
two and four orders of magnitude smaller than the round-off error of
traditional algorithms in double precision. So, if traditional algorithms in
single/double precision are used, the tiny environmental disturbances are
submerged by the man-made numerical noises so that we can not investigate the
influence of these tiny objective disturbances at all. Note that the
background numerical error (i.e. the round-off error and the truncation error)
of the CNS algorithm used here for the damped driven sine-Gordon equations
(10)-(13) is at the level of $10^{-230}$, which is at least 210 orders of
magnitude smaller than the tiny environmental noises! Note that any results
given by the CNS are not absolutely “clean”, since they also contain numerical
noises. However, in a finite interval $t\in[0,T_{c}]$ of time, the numerical
noises of CNS results are much smaller than all physical variables under
considerations and thus are negligible, where $T_{c}$ is the so-called
critical predictable time. This is similar to the drinking water from
waterworks: although it always contains few bacteria, it is safe to the health
of human-being as long as the number of bacteria (per unit volume) is less
than a standard. Thus, all results given the clean numerical simulation (CNS)
are relatively “clean”. This reveals the importance of the critical
predictable time $T_{c}$ as a key concept in the frame of the CNS. So, it is
the CNS which provides us a powerful tool to accurately investigate the
influence of such tiny environmental noises to an ultra-chaos.

Figure 10: Influence of tiny numerical noises to the probability density
functions (PDFs) of the ultra-chaos governed by (15)-(18) in case of $A=1$,
$B=0.5$ and $C=0.5$. (a) The PDFs of $x(t)$; (b) The PDFs of $y(t)$. The PDFs
are based on the numerical results in $t\in[0,10000]$ of Eqs. (15)-(18), given
by the 4th-order Runge-Kutta’s method with $40$-digit multiple-precision for
all parameters and variables using two different time-steps $\Delta t=10^{-5}$
and $\Delta t=10^{-6}$, respectively.

To show the wide existence of the so-called ultra-extra, let us further
consider the following ODEs:

$\displaystyle\dot{x}=A\,\sin(z)+C\,\cos(y),$ (15)
$\displaystyle\dot{y}=B\,\sin(x)\,\tanh^{2}(x)+A\,\cos(z),$ (16)
$\displaystyle\dot{z}=C\,\sin(y)+B\,\cos(x)$ (17)

with the initial condition

$x(0)=y(0)=z(0)=0.$ (18)

In case of $A=1$, $B=0.5$ and $C=0.5$, the above equations have chaotic
simulations with one positive Lyapunov exponent $\lambda=0.04$. We gain the
numerical simulations of (15)-(18) in a quite long interval $t\in[0,10000]$ by
means of the 4th-order Runge-Kutta’s method with $40$-digit multiple-precision
for all parameters and variables but using two different time-steps $\Delta
t=10^{-5}$ and $\Delta t=10^{-6}$, respectively. Figure 10 shows the influence
of tiny numerical noises to the probability density functions (PDFs) of the
chaotic simulation $x(t)$ and $y(t)$ in case of $\Delta t=10^{-5}$ and $\Delta
t=10^{-6}$ (whose numerical noises are at the level of $10^{-20}$ and
$10^{-24}$), respectively. Obviously, the PDFs of $x(t)$ and $y(t)$ are rather
sensitive to the tiny numerical noises. Note that these numerical noises are
equivalent to tiny external noises. Thus, Eqs. (15)-(18) in case of $A=1$,
$B=0.5$ and $C=0.5$ correspond to an ultra-chaos. This illustrates once again
that ultra-chaos should exist widely.

In summary, the above-mentioned examples illustrate that small disturbances
can indeed lead to the huge deviations of an ultra-chaos, not only in
trajectory but also in statistics. In other words, statistical properties of
an ultra-chaotic system have sensitivity dependance on small disturbances.
Therefore, even in statistical meanings, small disturbances could result in
the loss of the reproducibility and replicability of an ultra-chaos, no matter
what and how we do. Obviously, an ultra-chaos is at the higher-level of
disorder than a normal-chaos. It should be emphasized that small external
disturbances always exist in practice, and are out of control. Thus,
theoretically speaking, the reproducibility and replicability for an ultra-
chaos are essentially impossible in practice, forever. Therefore, ultra-chaos
is indeed an insurmountable objective obstacle of reproducibility and
replicability.

##  3 Concluding remarks and discussions

In this paper, a new concept, i.e. ultra-chaos, is proposed for the first
time. Unlike a normal-chaos, statistical properties such as the probability
density functions (PDF) of an ultra-chaos are sensitive to tiny disturbances.
We illustrate that ultra-chaos is widely existed and thus has general
scientific meanings. Obviously, for an ultra-chaotic system whose statistics
are sensitive to small disturbances, it is impossible in practice to replicate
any discoveries based on “statistically significant” findings, even in the
statistical meanings. Obviously, an ultra-chaos is indeed an insurmountable
obstacle to reproducibility and replicability. The ultra-chaos as a new
concept might open a new door and possibility to study chaos theory,
turbulence theory, computational fluid dynamics (CFD), the statistical
significance, reproducibility crisis, and so on.

First, given a chaotic system, we must judge whether it is an ultra-chaos or
not. For a normal-chaos, it is even unnecessary to use a very fine grid to
gain its numerical simulations, since its statistics are not sensitive to
numerical noises. Besides, it is also unnecessary to consider the influence of
small environmental noises to a normal-chaos, since environmental noises are
essentially equivalent to numerical noises in mathematics. However, for an
ultra-chaos, its statistics are sensitive to tiny environmental noises, as
illustrated here. Note that every dynamical system has environmental noises,
which are random, unavoidable, and out of control. Therefore, from practical
viewpoint, it is impossible to replicate an ultra-chaotic system even in the
statistical meanings. Theoretically speaking, it is impossible to make any
reliable/replicable predictions about such kind of ultra-chaos, forever, even
in statistical meanings.

Secondly, since turbulence has a close relationship with spatiotemporal chaos
[37], it is important to know whether a turbulent flow belongs to ultra-chaos
or not. If a turbulent flow is a kind of normal-chaos, which we designate as
“normal-turbulence”, it is unnecessary to use a very fine grid of
discretization for numerical simulations, since its statistics are not
sensitive to numerical noises. In this case, numerical noise itself is not a
trouble at all. However, If a turbulent flow is a kind of ultra-chaos, which
we designate as “ultra-turbulence”, it should be impossible in practice to
gain reproducible/replicable results even in statistics, no matter how fine
grid of discretization for numerical simulation or how advanced apparatus with
very high precision for experimental measures are used. This also reveals the
importance of ultra-chaos as a new concept, since turbulence is a very
important field in science and engineering.

Thirdly, the ultra-chaos as a new concept should be of benefit to deepen our
understanding about the so-called “crisis of reproducibility”. According to
Nature’s survey of 1,576 researchers in physics & engineering, chemistry,
earth & environment, biology, medicine and so on, more than half of these
surveyed have failed to reproduce their own experiments and agree that there
indeed exists a significant “crisis of reproducibility” [38]. In addition, in
a current investigation on evaluating replicability of laboratory experiments
in economics, the replication ratio is only 66% on average [39]. The so-called
“crisis of reproducibility” widely rises in numerous fields of science and
engineering. Currently, computational science has led to exciting advances in
many fields, such as the Computational Fluid Dynamics (CFD). The
reproducibility and replicability in the CFD serve as a minimum standard to
judge scientific claims and discoveries [40]. However, as reported by Mesnard
and Barba [41], “completing a full replication study of previously published
findings on bluff-body aerodynamics is harder than it looks, even when using
good reproducible-research practices and sharing code and data openly”. In a
survey about the reproducible researches [40], the replication ratio in
computational science is only about 25%. There might be lots of reasons that
could lead to irreproducible researches, such as poor experimental design, low
statistical power, poor analysis, insufficient oversight/mentoring, selective
reporting, raw data not available from original lab, code & computational mesh
unavailable, insufficient peer review, fraud, and so on [38, 41]. The non-
reproducibility of many scientific researches has leaded to growing worry
about the reliability of claims of new discoveries that are based on
“statistically significant” findings [42]. For an ultra-chaos, the derivations
caused by either the artificial disturbances or objective environmental noises
are macroscopic, and more importantly, the corresponding statistics are
sensitive to these disturbances/noises. Note that the statistical significance
and $p$-values are based on reliable/replicable results of statistics.
Naturally, for an ultra-chaotic system whose statistics are sensitive to small
disturbances, it should be impossible in practice to replicate new discoveries
based on “statistically significant” findings. Without doubt, an ultra-chaos
is indeed an insurmountable obstacle to reproducibility and replicability, and
is really an objective cause to “crisis of reproducibility”.

What we would like to emphasize here is that, even if all man-made
disturbances could be smoothed out, the reproducibility and replicability
might be still essentially impossible even in statistical meanings for some
dynamic systems such as an ultra-chaos. Thus, in theory, the ultra-chaos as a
new concept reveals a kind of incompleteness. Similar to Gödel’s
incompleteness theorem, such kind of “incompleteness of reproducibility and
replicability” reveals a limitation of our scientific paradigm based on
reproducible/repeatable experiments (at least in statistics), which can be
traced back to Galileo. Note that scientific findings are based on the
reproducibility and replicability of their main supporting evidences at least
in statistics: such kind of the reproducibility and replicability is a
cornerstone of the modern science. It is an open question whether or not this
“incompleteness of reproducibility and replicability” might shake the
cornerstone of our modern science.

Unlike the so-called hyper-chaos [4, 34, 35, 36, 30], i.e. a chaotic system
with at least two positive Lyapunov exponents, the characteristic of an ultra-
chaos is the sensitivity dependence of statistics on small disturbance. Note
that the hyper-chaos governed by the four-dimensional Rössler equation [30] is
just only a normal-chaos. This reveals that ultra-chaos is essentially
different from the previously reported types of chaos and thus is indeed a
totally new concept.

Finally, we emphasize that scientific claims should gain credence due to the
reproducibility and replicability of their main supporting evidences.
Unfortunately, we can not gain such kind of credence for an ultra-chaos! Would
this “incompleteness of reproducibility” shake the cornerstone of the modern
science? Would the ultra-chaos lead to a crisis of confidence in scientific
researches? How should we understand and interpret numerical/experimental
results of such kind of ultra-chaotic systems? Should we regard random
environmental noises as an inherent part of solution of an ultra-chaos? How
should we define the “truth” of an ultra-chaos? What kind of “truth” could an
ultra-chaos tell us? Do there exist some ultra-turbulence in practice?
Certainly, there are still lots of works to do in future.

## Acknowledgments

This work is partly supported by the National Natural Science Foundation of
China (No. 91752104).

## References

  * [1] J. H. Poincaré, Sur le probléme des trois corps et les équations de la dynamique. Divergence des séries de m. Lindstedt, Acta Mathematica 13 (1890) 1–270.
  * [2] E. N. Lorenz, Deterministic nonperiodic flow, Journal of the Atmospheric Sciences 20 (2) (1963) 130–141.
  * [3] T. Y. Li, J. A. Yorke, Period three implies chaos, American Mathematical Monthly 82 (10) (1975) 985–992.
  * [4] O. E. Rössler, An equation for hyperchaos, Physics Letters A 71 (2–3) (1979) 155 – 157.
  * [5] J. P. Eckmann, Ergodic theory of chaos and strange attractors, Reviews of Modern Physics 57 (3) (1985) 617 – 656.
  * [6] E. N. Lorenz, The Essence of Chaos, University of Washington Press, 1993.
  * [7] P. Smith, Explaining Chaos, Cambridge University Press, 1998.
  * [8] J. C. Sprott, Chaos and Time-Series Analysis, Oxford University Press, 2003.
  * [9] J. C. Sprott, Elegant Chaos: Algebraically Simple Chaotic Flows, World Scientific, Singapore, 2010.
  * [10] C. Letellier, L. F. Olsen, S. Mangiarotti, Chaos: from theory to applications for the 80th birthday of Otto E. Rössler, Chaos 31 (2021) 060402.
  * [11] E. N. Lorenz, Computational periodicity as observed in a simple system, Tellus Series A-dynamic Meteorology & Oceanography 58 (5) (2006) 549–557.
  * [12] J. Teixeira, C. A. Reynolds, K. Judd, Time step sensitivity of nonlinear atmospheric models: Numerical convergence, truncation error growth, and ensemble design, Journal of the Atmospheric Sciences 64 (1) (2007) 175.
  * [13] S. Liao, On the reliability of computed chaotic solutions of nonlinear differential equations, Tellus Series A-dynamic Meteorology & Oceanography 61 (4) (2009) 550–564.
  * [14] S. Liao, On the numerical simulation of propagation of micro-level inherent uncertainty for chaotic dynamic systems, Chaos, Solitons & Fractals 47 (2013) 1–12.
  * [15] S. Liao, Physical limit of prediction for chaotic motion of three-body problem, Communications in Nonlinear Science and Numerical Simulation 19 (3) (2014) 601–616.
  * [16] S. Liao, P. Wang, On the mathematically reliable long-term simulation of chaotic solutions of Lorenz equation in the interval [0, 10000], Science China Physics, Mechanics & Astronomy 57 (2) (2014) 330–335.
  * [17] X. Li, S. Liao, On the stability of the three classes of Newtonian three-body planar periodic orbits, Science China Physics, Mechanics & Astronomy 57 (11) (2014) 2121–2126.
  * [18] Z. Lin, L. Wang, S. Liao, On the origin of intrinsic randomness of Rayleigh-Bénard turbulence, Science China Physics, Mechanics & Astronomy 60 (1) (2017) 014712.
  * [19] X. Li, S. Liao, More than six hundred new families of Newtonian periodic planar collisionless three-body orbits, Science China Physics, Mechanics & Astronomy 60 (12) (2017) 129511.
  * [20] X. Li, Y. Jing, S. Liao, Over a thousand new periodic orbits of a planar three-body system with unequal masses, Publications of the Astronomical Society of Japan 70 (4) (2018) 64.
  * [21] X. Li, S. Liao, Collisionless periodic orbits in the free-fall three-body problem, New Astronomy 70 (2019) 22–26.
  * [22] T. Hu, S. Liao, On the risks of using double precision in numerical simulations of spatiotemporal chaos, Journal of Computational Physics 418 (2020) 109629.
  * [23] S. Qin, S. Liao, Influence of numerical noises on computer-generated simulation of spatiotemporal chaos, Chaos, Solitons & Fractals 136 (2020) 109790.
  * [24] T. Xu, S. Liao, Accurate predictions of chaotic motion of a free fall disk, Physics of Fluids 33 (2021) 037111.
  * [25] X. Li, X. Li, S. Liao, One family of 13315 stable periodic orbits of non-hierarchical unequal-mass triple systems, SCIENCE CHINA Physics, Mechanics & Astronomy 64 (1) (2021) 1–6.
  * [26] S. Liao, S. Qin, Ultra-chaos: an insurmountable objective obstacle of reproducibility and replication, arXiv Preprint 2110.00143.
  * [27] P. Oyanarte, MP-a multiple precision package, Computer Physics Communications 59 (2) (1990) 345–358.
  * [28] L. Crane, Infamous three-body problem has over a thousand new solutions, New Scientist(https://www.newscientist.com/article/2148074-infamous-three-body-problem-has-over-a-thousand-new-solutions/).
  * [29] C. Whyte, Watch the weird new solutions to the baffling three-body problem, New Scientist(https://www.newscientist.com/article/2170161-watch-the-weird-new-solutions-to-the-baffling-three-body-problem/).
  * [30] N. Stankevich, A. Kazakov, S. Gonchenko, Scenarios of hyperchaos occurrence in 4D Rössler system, Chaos 30 (2020) 123129.
  * [31] R. Chacón, A. Bellorín, L. Guerrero, J. González, Spatiotemporal chaos in sine-Gordon systems subjected to wave fields: Onset and suppression, Physical Review E 77 (4) (2008) 046212.
  * [32] J. B. Keller, D. W. Mclaughlin, G. C. Papanicolaou, Surveys in Applied Mathematics, Plenum Press, 1995.
  * [33] M. A. Ferré, M. G. Clerc, S. Coulibally, R. G. Rojas, M. Tlidi, Localized structures and spatiotemporal chaos: comparison between the driven damped sine-Gordon and the Lugiato-Lefever model, The European Physical Journal D 71 (6) (2017) 172.
  * [34] M. Eiswirth, T. M. Kruel, G. Ertl, F. W. Schneider, Hyperchaos in a chemical reaction, Chemical Physics Letters 193 (4) (1992) 305.
  * [35] T. Kapitaniak, K. E. Thylwe, I. Cohen, J. Wjewoda, Chaos - hyperchaos transition, Chaos, Solitons & Fractals 5 (10) (1995) 2003 – 2011.
  * [36] G. Baier, S. Sahle, Design of hyperchaotic flows, Physical Review E 51 (1995) 2712 – 2714.
  * [37] R. N. Henriksen, Compressible turbulence or spatiotemporal chaos?, Astrophysics and Space Science 221 (1994) 25 – 39.
  * [38] M. Baker, D. Penny, Is there a reproducibility crisis?, Nature 533.
  * [39] F. C. Camerer, et al., Evaluating replicability of laboratory experiments in economics, Science 351 (6280) (2016) 1433.
  * [40] R. D. Peng, Reproducible research in computational science, Science 334 (2011) 1226\.
  * [41] O. Mesnard, L. A. Barba, Reproducible and replicable computational fluid dynamics: It is harder than you think, Computing in Science & Engineering 19 (4) (2017) 44 – 55.
  * [42] D. J. Benjamin, et al., Redefine statistical significance, Nature Human Behaviour 2 (2018) 6 – 10.


# Learning Across Decentralized Multi-Modal Remote Sensing Archives with
Federated Learning

###### Abstract

The development of federated learning (FL) methods, which aim to learn from
distributed databases (i.e., clients) without accessing data on clients, has
recently attracted great attention. Most of these methods assume that the
clients are associated with the same data modality. However, remote sensing
(RS) images in different clients can be associated with different data
modalities that can improve the classification performance when jointly used.
To address this problem, in this paper we introduce a novel multi-modal FL
framework that aims to learn from decentralized multi-modal RS image archives
for RS image classification problems. The proposed framework is made up of
three modules: 1) multi-modal fusion (MF); 2) feature whitening (FW); and 3)
mutual information maximization (MIM). The MF module performs iterative model
averaging to learn without accessing data on clients in the case that clients
are associated with different data modalities. The FW module aligns the
representations learned among the different clients. The MIM module maximizes
the similarity of images from different modalities. Experimental results show
the effectiveness of the proposed framework compared to iterative model
averaging, which is a widely used algorithm in FL. The code of the proposed
framework is publicly available at https://git.tu-berlin.de/rsim/MM-FL.

Index Terms—  Remote sensing, federated learning, multi-modal image
classification.

##  1 Introduction

Remote sensing (RS) image archives can be stored under different databases due
to their growth in size and the data storage limitations of gathering all the
data in a centralized server. In addition, some RS archives of data providers
(e.g., commercial providers) may not be directly accessible due to commercial
concerns, legal regulations, etc. Legal restrictions, such as privacy laws and
national security concerns, may also prohibit public access to sensitive
information present in the RS image archives [1]. However, most of the deep
learning (DL) based approaches require full access to data, while learning the
model parameters of deep neural networks (DNNs) during training. To overcome
these challenges, federated learning (FL) can be used when there is no access
to data on decentralized RS image archives. FL aims to learn DNN models on
distributed databases (i.e., clients) and to find the optimal model parameters
in a global server (i.e., global model) without accessing data on clients. As
one of the first FL studies, the federated averaging (FedAvg) algorithm is
introduced in [2] to learn a global model by iterative model averaging. In
this algorithm, a local model is trained on each client and then its
parameters are sent to the global server, in which the parameters of all local
models are iteratively averaged and sent back to clients. Although FL is
highly studied in computer vision [2, 3, 4, 5], it is seldom considered in RS
[6]. As an example, in [6] an FL algorithm is proposed to learn several global
models through hierarchical clustering (denoted as FedPHC) when RS images are
non-independently and identically distributed among clients. FedPHC assumes
that RS images in the clients are associated with the same data modality.
However, RS images on different clients can be associated with different data
modalities. In addition, multi-modal images associated with the same
geographical area allow for a rich characterization of RS images when jointly
considered, and thus improve the effectiveness of the considered image
analysis task. To jointly exploit multi-modal RS images, the development of
image classification methods has attracted great attention in RS [7, 8, 9].
However, these methods assume that all the multi-modal RS images are
accessible during training and thus cannot be directly utilized in the
framework of FL. In addition, the adaptation of well-known FL algorithms for
the cases, where RS images in different clients are associated with different
data modalities, may not be always feasible. To adapt iterative model
averaging for such cases, one could operate a dedicated FedAvg algorithm for
each modality during training and then employ late fusion during inference
(denoted as MSFedAvg). In this way, there is one global model learnt for each
modality. Fig. 1 shows the inference phase of MSFedAvg. As one can see from
the figure, the images on the same geographical area are fed to the global
models based on their modality. Then, the prediction is made by averaging the
resulting class probabilities. However, MSFedAvg does not extract and exploit
the complementary information among the different modalities during the
training stage. This may result in limited image classification performance.
We would like to note that most of the existing FL algorithms assume that the
same DNN architecture is considered in each client. However, different DNN
architectures can be required for RS images with different data modalities.

Fig. 1: An illustration of MSFedAvg during inference, while the clients are
assumed to separately include Sentinel-1 (S1) and Sentinel-2 (S2) images for
the sake of simplicity.

To address these issues, in this paper we propose a novel multi-modal FL
framework that aims to accurately learn model parameters from decentralized
multi-modal RS image archives without having direct access to images in the
clients for RS image classification problems.

##  2 Proposed Multi-Modal Federated Learning Framework for RS
Image Classification

Let $K$ be the total number of clients and $C^{i}$ be the $i$th client, where
$1\leq i\leq K$. Each client $C^{i}$ locally holds the corresponding training
set
$D_{i}=\\{(\boldsymbol{x}_{z}^{i},\boldsymbol{y}_{z}^{i})\\}_{z=1}^{M^{i}}$
including $M^{i}$ samples, where $\boldsymbol{x}_{z}^{i}$ is the $z$th RS
image of the $i$th client, and $\boldsymbol{y}_{z}^{i}$ is the corresponding
class label. Let $\phi^{i}$ and $\mu^{i}$ be the image encoder and classifier,
respectively, which are employed for local training on $D^{i}$. We assume that
data modalities can be different between clients (i.e., data modalities
associated with $\boldsymbol{x}_{z}^{i}$ and $\boldsymbol{x}_{z}^{j}$ can be
different for $i\neq j\\!\\!\\!\\!\quad\forall z$). The proposed framework
aims to learn DNNs on a central server for RS image classification problems
without accessing data on clients, while different clients can contain RS
images associated with different data modalities. To this end, our framework
includes three modules: 1) multi-modal fusion (MF); 2) feature whitening (FW);
and 3) mutual information maximization (MIM). Fig. 2 shows an illustration of
the local training stage of the proposed framework, which is explained in
detail in the following.

(a)

(b)

Fig. 2: An illustration of the local training stage of our framework when two
clients (a) and (b) are considered and include images associated with
Sentinel-1 (S1) and Sentinel-2 (S2), respectively, for the sake of simplicity.

The MF module aims to employ DNNs for iterative model averaging when clients
are associated with different data modalities. To this end, instead of
employing a single DNN, we define modality-specific backbones and a common
classifier on the server side. Let $P$ be the total number of modalities
associated with clients. Let $\psi_{m}$ be a modality-specific backbone while
$1\leq m\leq P\leq K$ and $\theta$ be a common classifier. After defining DNNs
on the server side, the model parameters are shared with the clients. Then,
$\mu^{i}$ is updated with $\theta$ for the $i$th client and $\phi^{i}$ is
updated with the corresponding modality-specific backbone. To jointly utilize
information from different modalities during inference, the feature vectors
extracted by all modality-specific backbones $\\{\psi_{m}\\}_{m=1}^{P}$ are
fused through concatenation. However, during local training, a pseudo fusion
via concatenation with the zero vector $\boldsymbol{v}$ = $\vec{0}$ is applied
since there is no access to data from different clients. Then, the resulting
vector is fed into $\mu^{i}$. MF module allows defining different
architectures for different modalities and also jointly utilizing and
extracting information from multiple modalities.

It is worth noting that the distributions of feature vectors may differ
between clients because of the modality differences that may lead to sub-
optimal parameters of the aggregated models. To reduce such distribution
differences, the FW module aims to project the feature vectors of the images
with different modalities that are extracted by local models into a common
distribution. This is achieved by using batch whitening layers instead of
batch normalization layers inspired by [10]. The batch whitening layer $BW$
for $\boldsymbol{x}_{z,p}^{i}$ is defined as follows:

$\begin{split}\boldsymbol{\hat{x}}^{i}_{z}&=W_{B}(\boldsymbol{x}^{i}_{z}-\mu_{B}),\\\
BW({\boldsymbol{x}}_{z,p}^{i})&=\gamma_{p}\boldsymbol{\hat{x}}^{i}_{z,p}+\beta_{p},\end{split}$
(1)

where ${\boldsymbol{x}}_{z,p}^{i}$ is the $p$th element of
${\boldsymbol{x}}_{z}^{i}$, $\mu_{B}$ is the mean vector, $\gamma_{p}$ is the
scaling parameter and $\beta_{p}$ is the shifting parameter. This module
aligns the data distributions between clients using covariance matrices. It
uses domain-specific alignment layers, which compute domain-specific
covariance matrices of intermediate features.

The MIM module aims to model mutual-information between different modalities
by maximizing the similarity of images, which are acquired on the same
geographical area and associated to different modalities. To achieve this, we
employ the NT-Xent loss function $\mathcal{L}_{NTX}$ [11] for modelling
similarity through the feature vectors of a local model and that of aggregated
models from different modalities. To this end, the feature vectors are
extracted by feeding images into the local and global models. Since the global
models are not trained with images that have the same modality as the images
in local client, it is not possible to directly feed into the global model.
Therefore, images are fed to the first convolutional layer of the local model
and the output is fed to the global models. Then, the $\mathcal{L}_{NTX}$ for
a given mini-batch $\mathbfcal{B}$ is calculated using two feature vectors as
follows:

$\mathcal{L}_{NTX}(\mathbfcal{B})\\!=\\!\\!-\\!\\!\\!\sum_{\boldsymbol{x}_{z}^{i}\in\mathbfcal{B}}\\!\\!\log\frac{e^{S(\phi^{i}(\boldsymbol{x}_{z}^{i}),\psi_{m}(\boldsymbol{x}_{z}^{i}))/\tau}}{\sum\limits_{\boldsymbol{x}_{t}^{i}\in\mathbfcal{B}}\mathbbm{1}_{[z\neq
t]}e^{S(\phi^{i}(\boldsymbol{x}_{z}^{i}),\psi_{m}(\boldsymbol{x}_{t}^{i}))/\tau}},$
(2)

where $\tau$ is a temperature parameter, $\mathbbm{1}$ is the indicator
function, $S(.,.)$ measures cosine similarity and $\mathbin{\|}$ is the
concatenation operator. Accordingly, we define the local objective
$\mathcal{L}$ based on $\mathcal{L}_{NTX}$ and cross-entropy loss
$\mathcal{L}_{BCE}$ as follows:

$\mathcal{L}(\mathbfcal{B})\\!=\\!\mathcal{L}_{NTX}\\!(\mathbfcal{B})\\!+\\!\\!\\!\\!\\!\\!\sum_{\boldsymbol{x}_{z}^{i},\boldsymbol{y}_{z}^{i}\in\mathbfcal{B}}\\!\\!\\!\\!\\!\mathcal{L}_{BCE}(\theta(\phi^{i}(\boldsymbol{x}_{z}^{i})\\!\\!\mathbin{\|}\\!\boldsymbol{v}),\boldsymbol{y}_{z}^{i}).$
(3)

Once the local training procedure of our framework is completed, the
considered local parameters are sent back to the server and aggregated on the
server side.

##  3 Experimental Results

The experiments were conducted on the BigEarthNet-MM archive [12]. It includes
590,326 multi-modal image pairs, each of which includes Sentinel-1 and
Sentinel-2 images acquired on the same geographical area. Each pair in
BigEarthNet-MM is annotated with multi-labels. The multi-modal image pairs
acquired in summer were used for experiments. We defined six clients, in which
three clients contain Sentinel-1 images and the remaining clients contain
Sentinel-2 images. All six clients participate in each round of the local
training. We defined two different scenarios: 1) the images are randomly
distributed to the clients; and 2) the images acquired in the same country are
present in the same client. We utilized the ResNet-50 CNN architecture [13] as
the backbone of the proposed framework. We used the Adam optimizer with the
learning rate of 0.001 and the momentum of 0.9. We trained our framework for
40 epochs with the mini-bath size of 256. We evaluate the performance of our
method in terms of classification accuracy (in $F\textsubscript{1}$-Score) and
local training complexity (in seconds).

Table 1: $F_{1}$-scores (%) associated with the different combinations of the modules of the proposed framework. Module |  $\boldsymbol{F_{1}}$-Score
---|---
MF | FW | MIM | Scenario 1 | Scenario 2
✓ | ✗ | ✗ | 74.4 | 61.7
✓ | ✗ | ✓ | 74.2 | 67.1
✓ | ✓ | ✗ | 75.5 | 68.5
✓ | ✓ | ✓ | 76.7 | 69.3

In the first set of trials, we analyze the effectiveness of each module of our
framework. Table 1 shows the corresponding results. One can see from the table
that jointly using three modules achieves the highest results than other
combinations for both scenarios. This shows that FW and MIM modules reduce the
data distribution differences between clients, which leads to accurate
characterization of multi-modal RS images during training. The lowest accuracy
is obtained by the joint use of MF and MIM modules for Scenario 1, which is
2.5% lower than that obtained by the joint use of all modules. Moreover, the
lowest accuracy is obtained by using only MF module for Scenario 2, which is
7.6% lower than the joint use of all modules. This indicates that the FW
module increases the performance more than the MIM module. One can also
observe from the table that the results achieved with Scenario 1 are higher
than those achieved with Scenario 2. As an example, using only the MF module
provides 74.4% and 61.7% accuracies for Scenario 1 and Scenario 2,
respectively. This is due to the higher non-IID level of Scenario 2 compared
Scenario 1.

Table 2: $F_{1}$-scores (%) and local training complexity (in seconds)
obtained by MSFedAvg and the proposed framework.

Method |  $\boldsymbol{F_{1}}$-Score |  Local Training Complexity
---|---|---
Scenario 1 | Scenario 2
MSFedAvg [2] | 67.2 | 58.9 | 10.5
| Proposed
---
Framework
76.7 | 69.3 | 11.6

In the second set of trials, we compare the proposed method with MSFedAvg in
terms of accuracy and local training complexity under both scenarios. The
local training complexity refers to the average computational cost for
training the considered DNN for each client. Table 2 shows the corresponding
results. One can observe from the table that our framework achieves the
highest $F\textsubscript{1}$-scores under both scenarios compared to MSFedAvg.
The proposed framework outperforms MSFedAvg by 9.5% and 10.4% for Scenario 1
and Scenario 2, respectively. This shows the effectiveness of our framework
compared to MSFedAvg. One can also see from the table that the proposed
framework leads to higher accuracies than MSFedAvg at the cost of a slight
increase in local training complexity. The average completion time of a
training round on one client increases only 10% when the proposed framework is
used instead of MSFedAvg. This increase can be compensated by using higher
mini-batch sizes to reduce the local training complexity of our framework.

##  4 Conclusion

In this paper, we have introduced a novel framework, which is capable of
learning DNN model parameters from decentralized multi-modal RS image archives
without accessing data on clients in the context of RS image classification.
Our framework includes: i) multi-modal fusion module to perform iterative
model averaging when the images in different clients are associated to
different data modalities; ii) feature whitening module to align the
representations learned among different clients; and iii) mutual information
maximization module to maximize the similarity of images from different
modalities. Experimental results show the success of the proposed framework
compared to MSFedAvg [2].

We would like to note that the proposed framework is independent from the
number of clients and also the modality-specific backbones being selected for
the considered modalities. Thus, any DNN architecture specifically designed
for each modality (e.g., attention-based CNNs for very high resolution aerial
images) can be utilized in our framework. This can allow to accurately
describe modality-specific information content, and thus lead to higher RS
image classification performance. As a future work, we plan to investigate the
joint use of RS images with socio-economic data (e.g., demographic data,
household surveys, etc.) in the framework of FL in RS.

##  5 ACKNOWLEDGEMENTS

This work is supported by the European Research Council (ERC) through the
ERC-2017-STG BigEarth Project under Grant 759764 and by the German Research
Foundation through the IDEAL-VGI project under Grant 424966858.

## References

  * [1] B. Zhang, Y. Wu, B. Zhao, J. Chanussot, D. Hong, J. Yao, and L. Gao, “Progress and challenges in intelligent remote sensing satellite systems,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , 2022\.
  * [2] H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. Arcas, “Communication-efficient learning of deep networks from decentralized data,” _International Conference on Artificial Intelligence and Statistics_ , pp. 1273–1282, 2017.
  * [3] Q. Li, B. He, and D. Song, “Model-contrastive federated learning,” _IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 10 713–10 722, 2021\.
  * [4] X. Ma, J. Zhang, S. Guo, and W. Xu, “Layer-wised model aggregation for personalized federated learning,” _IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 10 092–10 101, 2022.
  * [5] L. Gao, H. Fu, L. Li, Y. Chen, M. Xu, and C. Xu, “FedDC: Federated learning with non-iid data via local drift decoupling and correction,” _IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 10 112–10 121, 2022\.
  * [6] B. Zhang, X. Zhang, M. Pun, and M. Liu, “Prototype-based clustered federated learning for semantic segmentation of aerial images,” _IEEE International Geoscience and Remote Sensing Symposium_ , pp. 2227–2230, 2022.
  * [7] D. Hong, L. Gao, N. Yokoya, J. Yao, J. Chanussot, Q. Du, and B. Zhang, “More diverse means better: Multimodal deep learning meets remote-sensing imagery classification,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 59, no. 5, pp. 4340–4354, 2020.
  * [8] X. Wu, D. Hong, and J. Chanussot, “Convolutional neural networks for multimodal remote sensing data classification,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 60, pp. 1–10, 2021.
  * [9] G. Sumbul, M. Müller, and B. Demir, “A novel self-supervised cross-modal image retrieval method in remote sensing,” _IEEE International Conference on Image Processing_ , pp. 2426–2430, 2022.
  * [10] S. Roy, A. Siarohin, E. Sangineto, S. R. Bulo, N. Sebe, and E. Ricci, “Unsupervised domain adaptation using feature-whitening and consensus loss,” _IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 9471–9480, 2019.
  * [11] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” _International Conference on Machine Learning_ , pp. 1597–1607, 2020.
  * [12] G. Sumbul, A. D. Wall, T. Kreuziger, F. Marcelino, H. Costa, P. Benevides, M. Caetano, B. Demir, and V. Markl, “BigEarthNet-MM: A large-scale, multimodal, multilabel benchmark archive for remote sensing image classification and retrieval,” _IEEE Geoscience and Remote Sensing Magazine_ , vol. 9, no. 3, pp. 174–180, 2021.
  * [13] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 770–778.