Let $(\lambda_{1},\dots,\lambda_{d})$ are the eigenvalues of $H$ such that
there exists at least one which is strictly negative. If
$\rho>-\frac{1}{\lambda_{*}}$ where $\lambda_{*}$ is the largest negative
eigenvalue of H, then the stationary distribution of Eq. (136) is
$\displaystyle
P\left(x,\infty\mid\rho\right)=\sqrt{\frac{\lambda_{i}}{\pi\eta\varsigma^{2}}\frac{1}{1+\rho\lambda_{i}}}\exp\left[-\frac{\lambda_{i}}{\eta\varsigma^{2}}\frac{1}{1+\rho\lambda_{i}}x^{2}\right]$
(138)

###### Proof of Theorem C.3.

Note that Eq. (136) is a linear SDE, and that drift and diffusion matrices are
co-diagonalizable: Let $H=U\Lambda U^{\top}$ be one eigenvalue decomposition
of $H$, with $\Lambda=\text{diag}(\lambda_{1},\dots,\lambda_{d})$. If we plug
this in, we get

$dX_{t}=-U\left(\Lambda+\rho\Lambda^{2}\right)U^{\top}X_{t}dt+U\left[(I_{d}+\rho\Lambda)\sqrt{\eta}\varsigma\right]U^{\top}dW_{t}.$

Let us multiply the LHS with $U^{\top}$, then

$d(U^{\top}X_{t})=-\left(\Lambda+\rho\Lambda^{2}\right)(U^{\top}X_{t})dt+\left[(I_{d}+\rho\Lambda)\sqrt{\eta}\varsigma\right]U^{\top}dW_{t}.$

Finally, note that $U^{\top}dW_{t}=dW_{t}$ in law, so we can write

$d(U^{\top}X_{t})=-\left(\Lambda+\rho\Lambda^{2}\right)(U^{\top}X_{t})dt+\left[(I_{d}+\rho\Lambda)\sqrt{\eta}\varsigma\right]dW_{t}.$

This means that the coordinates of the vector $Y=U^{\top}X$ evolve
independently

$dY_{t}=-\left(\Lambda+\rho\Lambda^{2}\right)Y_{t}dt+\left[(I_{d}+\rho\Lambda)\sqrt{\eta}\varsigma\right]dW_{t},$

since $\Lambda$ is diagonal. Therefore for the $i$-th component $Y_{i}$ we can
write

$dY_{i,t}=-\left(\lambda_{i}+\rho\lambda_{i}^{2}\right)Y_{i,t}dt+\left[(1+\rho\lambda_{i})\sqrt{\eta}\varsigma\right]dW_{i,t}.$
(139)

Note that this is a simple one-dimensional Ornstein–Uhlenbeck process
$dY_{t}=-\theta Y_{t}dt+\sigma dW_{t}$ ($\theta>0,\sigma\neq 0$) with
parameters

$\theta=\lambda_{i}\left(1+\rho\lambda_{i}\right)>0\quad\text{and}\quad\sigma=(1+\rho\lambda_{i})\sqrt{\eta}\varsigma>0$
(140)

Therefore, from Section 4.4.4 of (Gardiner et al., 1985), we get that

${\mathbb{E}}[Y_{t}]=e^{-\theta
t}Y_{0},\quad\text{Var}(Y_{t})=\frac{\sigma^{2}}{2\theta}(1-e^{-2\theta t}).$
(141)

In our case we have that

${\mathbb{E}}[Y_{t}]=e^{-\theta t}Y_{0}\rightarrow
0\quad\text{and}\quad\text{Var}(Y_{t})=\frac{\sigma^{2}}{2\theta}(1-e^{-2\theta
t})\rightarrow\frac{\sigma^{2}}{2\theta}=\frac{\eta\varsigma^{2}}{2\lambda_{i}}(1+\rho\lambda_{i}).$
(142)

Additionally, using the Fokker–Planck equation, see Section 5.3 of (Risken,
1996), we have the following formula for the stationary distribution of each
eigendirection. Indeed, let us recall that for $D:=\frac{\sigma^{2}}{2}$, the
probability density function is

$P\left(x,t\mid x^{\prime},t^{\prime},\rho\right)=\sqrt{\frac{\theta}{2\pi
D\left(1-e^{-2\theta\left(t-t^{\prime}\right)}\right)}}\exp\left[-\frac{\theta}{2D}\frac{\left(x-x^{\prime}e^{-\theta\left(t-t^{\prime}\right)}\right)^{2}}{1-e^{-2\theta\left(t-t^{\prime}\right)}}\right].$
(143)

Therefore, the stationary distribution is

$\displaystyle P\left(x,\infty\mid\rho\right)$
$\displaystyle=\sqrt{\frac{\theta}{2\pi
D}}\exp\left[-\frac{\theta}{2D}x^{2}\right]$
$\displaystyle=\sqrt{\frac{\theta}{\pi\sigma^{2}}}\exp\left[-\frac{\theta}{\sigma^{2}}x^{2}\right]$
$\displaystyle=\sqrt{\frac{\lambda_{i}}{\pi\eta\varsigma^{2}}\frac{1}{1+\rho\lambda_{i}}}\exp\left[-\frac{\lambda_{i}}{\eta\varsigma^{2}}\frac{1}{1+\rho\lambda_{i}}x^{2}\right].$
(144)

To conclude,

$Y_{i,\infty}\sim\mathcal{N}\left(0,\frac{\eta\varsigma^{2}}{\lambda_{i}}(1+\rho\lambda_{i})\right).$
(145)

Since all of the eigenvalues are positive, this distribution has more variance
than SGD on each direction. ∎

Since the proof of Theorem C.4 is perfectly similar to that of Theorem C.3, we
skip it. Additionally, a very analogous result holds true even if all the
eigenvalues are strictly negative and thus the quadratic has a single maximum
as a critical point. From these results, we understand that under certain
circumstances, USAM might be attracted not only by the minimum but possibly
also by a saddle or a maximum. This is fully consistent with the results
derived for the ODE of USAM in Lemma C.1 and Lemma C.2.

######  Observation C.5 (Suboptimality under the Stationary Distribution –
comparison to SGD).

In the special case where the stochastic process has reached stationarity, one
can approximate the loss landscape with a quadratic loss (Jastrzebski et al.,
2018). By further assuming that $\Sigma^{SGD}(x)=H$ (see e.g.(Sagun et al.,
2018; Zhu et al., 2019)), Theorem (A.4) implies that for USAM
$dX_{t}=-H\left(I_{d}+\rho H\right)X_{t}dt+\left[(I_{d}+\rho
H)\sqrt{\eta}\sqrt{H}\right]dW_{t}.$ (146) Up to a change of variable, we
assume $H$ to be diagonal and therefore
${\mathbb{E}}_{USAM}\left[f\right]=\frac{1}{2}\sum_{i=1}^{d}\lambda_{i}{\mathbb{E}}[X_{i}^{2}]=\frac{\eta}{4}\sum_{i=1}^{d}\lambda_{i}(1+\rho\lambda_{i})^{2}=\frac{\eta}{4}\left(Tr(H)+2\rho
Tr(H^{2})+\rho^{2}Tr(H^{3})\right)\gg{\mathbb{E}}_{SGD}\left[f\right],$ (147)
where subscripts indicate that $f$ is being optimized with SGD and USAM,
respectively. Regarding DNSAM, Theorem (A.9) implies that
$dX_{t}=-H\left(I_{d}+\frac{\rho H}{\lVert
HX_{t}\rVert}\right)X_{t}dt+\sqrt{\eta}\sqrt{H}\left(I_{d}+\frac{\rho
H}{\lVert HX_{t}\rVert}\right)dW_{t}.$ (148) Therefore, we argue that DNSAM
has to have a suboptimality with respect to SGD which is even larger than that
of USAM. Intuitively, when $\lVert HX_{t}\rVert<1$, the variance of DNSAM is
larger than that of USAM. Therefore, its suboptimality has to be larger as
well. Finally, the behavior of SAM is close to that of DNSAM, but less
pronounced because the denominator can never get too close to $0$ due to the
noise injection.

###  C.3 ODE SAM

W.l.o.g, we take $H$ to be diagonal and if it has negative eigenvalues, we
denote the largest negative eigenvalue with $\lambda_{*}$. Let us recall that
the ODE of SAM for the quadratic loss is given by

$dX_{t}=-H\left(I_{d}+\frac{\rho H}{\lVert HX_{t}\rVert}\right)X_{t}dt$ (149)

######  Lemma C.6.

For all $\rho>0$, if $H$ is PSD, the origin is (locally) asymptotically
stable. Additionally, if $H$ is not PSD, if $\lVert
HX_{t}\rVert\leq-\rho\lambda_{*}$, then the origin is still (locally)
asymptotically stable.

###### Proof of Lemma C.6.

Let $V(x):=\frac{x^{\top}Kx}{2}$ be the Lyapunov function, where $K$ is a
diagonal matrix with positive eigenvalues $\left(k_{1},\cdots,k_{d}\right)$.
Therefore, we have

$V(X_{t})=\frac{1}{2}\sum_{i=1}^{d}k_{i}\left(X_{t}^{i}\right)^{2}>0$ (150)

and

$\dot{V}(X_{t})=\sum_{i=1}^{d}k_{i}X_{t}^{i}\dot{X}_{t}^{i}=\sum_{i=1}^{d}k_{i}(-\lambda_{i})\left(1+\frac{\rho\lambda_{i}}{\lVert
HX_{t}\rVert}\right)X_{t}^{i}X_{t}^{i}dt=-\sum_{i=1}^{d}k_{i}\lambda_{i}\left(1+\frac{\rho\lambda_{i}}{\lVert
HX_{t}\rVert}\right)\left(X_{t}^{i}\right)^{2}dt.$ (151)

Let us analyze the terms

$k_{i}\lambda_{i}\left(1+\frac{\rho\lambda_{i}}{\lVert
HX_{t}\rVert}\right)\left(X_{t}^{i}\right)^{2}.$

When $\lambda_{i}>0$, these quantities are all positive and the proof is
concluded. However, if there exists $\lambda_{i}<0$, these quantities are
positive only if $\left(1+\frac{\rho\lambda_{i}}{\lVert
HX_{t}\rVert}\right)\leq 0$, that is if $\lVert
HX_{t}\rVert\leq-\rho\lambda_{i}$. Therefore, a sufficient condition for
$\dot{V}(X_{t})\leq 0$ is that

$\lVert HX_{t}\rVert\leq-\rho\lambda_{i},\quad\forall i\text{ s.t.
}\lambda_{i}<0.$

Based on Theorem 1.1 of (Mao, 2007), we conclude that if $\lVert
HX_{t}\rVert\leq-\rho$ where $\lambda_{*}$ is the largest negative eigenvalue
of H, $V(X_{t})>0$ and $\dot{V}(X_{t})\leq 0$, and that therefore the dynamics
of $X_{t}$ is bounded inside this compact set and cannot diverge. ∎

From this result, we understand that the dynamics of the ODE of USAM might
converge to a saddle or even a maximum if it gets too close to it.

###  C.4 SDE DNSAM

W.l.o.g, we take $H$ to be diagonal and if it has negative eigenvalues, we
denote the largest negative eigenvalue with $\lambda_{*}$. Based on Eq. (152)
and in the case where $\Sigma^{\text{SGD}}=\varsigma^{2}I_{d}$, the SDE of
DNSAM for the quadratic loss is given by

$dX_{t}=-H\left(I_{d}+\frac{\rho H}{\lVert
HX_{t}\rVert}\right)X_{t}dt+\sqrt{\eta}\varsigma\left(I_{d}+\frac{\rho
H}{\lVert HX_{t}\rVert}\right)dW_{t}$ (152)

######  Observation C.7.

For all $\rho>0$, there exists an $\epsilon>0$ such that if $\lVert
HX_{t}\rVert\in\left(\epsilon,-\rho\lambda_{*}\right)$, the dynamics of
$X_{t}$ is attracted towards the origin. If the eigenvalues are all positive,
the condition is $\lVert HX_{t}\rVert\in\left(\epsilon,\infty\right)$. On the
contrary, if $\lVert HX_{t}\rVert<\epsilon$, then the dynamics is pushed away
from the origin.

###### Formal calculations to support Observation C.7.

Let $V(t,x):=e^{-t}\frac{x^{\top}Kx}{2}$ be the Lyapunov function, where $K$
is a diagonal matrix with strictly positive eigenvalues
$\left(h_{1},\cdots,h_{d}\right)$. Therefore, we have

$V(X_{t})=e^{-t}\frac{1}{2}\sum_{i=1}^{d}k_{i}\left(X_{t}^{i}\right)^{2}>0$
(153)

and

$\displaystyle LV(t,X_{t})$
$\displaystyle=-e^{-t}\frac{1}{2}\sum_{i=1}^{d}k_{i}\left(X_{t}^{i}\right)^{2}+e^{-t}\sum_{i=1}^{d}k_{i}(-\lambda_{i})\left(1+\frac{\rho\lambda_{i}}{\lVert
HX_{t}\rVert}\right)\left(X_{t}^{i}\right)^{2}+e^{-t}\frac{\eta\varsigma^{2}}{2}\sum_{i=1}^{d}k_{i}\left(1+\frac{\rho\lambda_{i}}{\lVert
HX_{t}\rVert}\right)^{2}$
$\displaystyle=-e^{-t}\left(\frac{1}{2}\sum_{i=1}^{d}k_{i}\left(X_{t}^{i}\right)^{2}+\sum_{i=1}^{d}k_{i}\lambda_{i}\left(1+\frac{\rho\lambda_{i}}{\lVert
HX_{t}\rVert}\right)\left(X_{t}^{i}\right)^{2}-\frac{\eta\varsigma^{2}}{2}\sum_{i=1}^{d}k_{i}\left(1+\frac{\rho\lambda_{i}}{\lVert
HX_{t}\rVert}\right)^{2}\right)$ (154)

Let us analyze the terms

$k_{i}\lambda_{i}\left(1+\frac{\rho\lambda_{i}}{\lVert
HX_{t}\rVert}\right)\left(X_{t}^{i}\right)^{2}.$

When $\lambda_{i}>0$, these quantities are all positive. When $\lambda_{i}<0$,
these quantities are positive only if $\left(1+\frac{\rho\lambda_{i}}{\lVert
HX_{t}\rVert}\right)\leq 0$, that is if $\lVert
HX_{t}\rVert\leq-\rho\lambda_{i}$. Let us now assume that

$\lVert HX_{t}\rVert\leq-\rho\lambda_{i},\quad\forall i\text{ s.t.
}\lambda_{i}<0.$

that is, $\lVert HX_{t}\rVert\leq-\rho\lambda_{*}$ such that $\lambda_{*}$.
Then, we observe that

  * •

If $\lVert HX_{t}\rVert\rightarrow 0$, $LV(t,X_{t})\geq 0$

  * •

If $\varsigma$ is small enough, for $\lVert
HX_{t}\rVert\approx-\rho\lambda_{*}$, $LV(t,X_{t})\leq 0$

Given that all functions and functionals involved are continuous, there exists
$\epsilon>0$ such that

  * •

If $\lVert HX_{t}\rVert<\epsilon$, $LV(t,X_{t})\geq 0$

  * •

If $\varsigma$ is small enough, for $\lVert
HX_{t}\rVert\in(\epsilon,-\rho\lambda_{*})$, $LV(t,X_{t})\leq 0$

∎

Based on Theorem 2.2 of (Mao, 2007), we understand that if the dynamics is
sufficiently close to the origin, it gets pulled towards it, but if gets too
close, it gets repulsed from it. If there is no negative eigenvalue, the same
happens but the dynamics can never get close to the minimum.

##  Appendix D Experiments

In this section, we provide additional details regarding the validation that
the SDEs we proposed indeed weakly approximate the respective algorithms. We
do so on a quadratic landscape, on a classification task with a deep linear
model, a binary classification task with a deep nonlinear model, and a
gregression task with a teacher-student model. Since our SDEs prescribe the
calculation of the Hessian of the whole neural network at each iteration step,
this precludes us from testing our theory on large-scale models.

###  D.1 SDE Validation

##### Quadratic

In this paragraph, we provide the details of the Quadratic experiment. We
optimize the loss function $f(x)=\frac{1}{2}x^{\top}Hx$ of dimension $d=20$.
The Hessian $H$ is a random SPD matrix generated using the standard Gaussian
matrix $A\in\mathbb{R}^{d\times 2d}$ as $H=AA^{\top}/(2d)$. The noise used to
perturb the gradients is $Z\sim\mathcal{N}(0,\Sigma)$ where $\Sigma=\sigma
I_{d}$ and $\sigma=0.01$. We use $\eta=0.01$,
$\rho\in\\{0.001,0.01,0.1,0.5\\}$. The results are averaged over $3$
experiments.

##### Deep Linear Classification

In this paragraph, we provide the details of the Deep Linear Classification
experiment. This is a classification task on the Iris Database (Dua & Graff,
2017). The model is a Linear MLP with 1 hidden layer with a width equal to the
number of features and we optimize the cross-entropy loss function. The noise
used to perturb the gradients is $Z\sim\mathcal{N}(0,\Sigma)$ where
$\Sigma=\sigma I_{d}$ and $\sigma=0.01$. We use $\eta=0.01$,
$\rho\in\\{0.001,0.01,0.1,0.2\\}$. The results are averaged over $3$
experiments.

##### Deep Nonlinear Classification

In this paragraph, we provide the details of the Deep Nonlinear Classification
experiment. This is a binary classification task on the Breast Cancer Database
(Dua & Graff, 2017). The model is a Nonlinear MLP with 1 hidden layer with a
width equal to the number of features, sigmoid activation function, and we
optimize the $\ell^{2}$-regularized logistic loss with parameter
$\lambda=0.1$. The noise used to perturb the gradients is
$Z\sim\mathcal{N}(0,\Sigma)$ where $\Sigma=\sigma I_{d}$ and $\sigma=0.01$. We
use $\eta=0.01$, $\rho\in\\{0.001,0.01,0.1,0.5\\}$. The results are averaged
over $3$ experiments.

##### Deep Teacher-Student Model

In this paragraph, we provide the details of the Teacher-Student experiment.
This is a regression task where the database is generated by the Teacher model
based on random inputs in $\mathbb{R}^{5}$ and output in $\mathbb{R}$. The
Teacher model is a deep linear MLP with $20$ hidden layers with $10$ nodes,
while the Student is a deep nonlinear MLP with $20$ hidden layers and $10$
nodes. We optimize the MSE loss. The noise used to perturb the gradients is
$Z\sim\mathcal{N}(0,\Sigma)$ where $\Sigma=\sigma I_{d}$ and $\sigma=0.001$.
We use $\eta=0.001$, $\rho\in\\{0.0001,0.001,0.03,0.05\\}$. The results are
averaged over $3$ experiments.

####  D.1.1 Importance of the Additional Noise.

In this section, we empirically test the importance of using the correct
diffusion terms the USAM SDE Eq. (9) and the DNSAM SDE Eq. (10). Let us
introduce two new SDEs where the diffusion term is the one of the SGD SDE in
Eq. (18) rather than that from the correct SDEs:

$dX_{t}=-\nabla\tilde{f}^{\text{USAM}}(X_{t})dt+\sqrt{\eta\left(\Sigma^{\text{SGD}}(X_{t})\right)}dW_{t},\quad\text{where}\quad\tilde{f}^{\text{USAM}}(x):=f(x)+\frac{\rho}{2}\lVert\nabla
f(x)\rVert^{2}_{2}.$ (155)

$dX_{t}=-\nabla\tilde{f}^{\text{DNSAM}}(X_{t})dt+\sqrt{\eta\left(\Sigma^{\text{SGD}}(X_{t})\right)}dW_{t},\quad\text{where}\quad\tilde{f}^{\text{DNSAM}}(x):=f(x)+\rho\lVert\nabla
f(x)\rVert_{2}.$ (156)

In Figure 6 we observe how approximating USAM with the SGD SDE (Eq. (18))
brings a large error in all four cases. Introducing the correct drift but
excluding the correct covariance, i.e. using Eq. (155), reduces the error, but
the best performer is the complete USAM SDE Eq. (7). From Figure 7, the same
observations hold for DNSAM.

From these experiments, we understand that naively adding noise to the ODEs of
USAM and SAM does not provide SDEs with sufficient approximation power. This
is consistent with our proofs.

(a)  (b)  (c)  (d)

Figure 6: USAM - Comparison in terms of $g_{1}(x)$ with respect to $\rho$ \-
Quadratic (left); Teacher-Student (center-left); Deep linear class (center-
right); Deep Nonlinear class (right).

(a)  (b)  (c)  (d)

Figure 7: DNSAM - Comparison in terms of $g_{1}(x)$ with respect to $\rho$ \-
Quadratic (left); Teacher-Student (center-left); Deep linear class (center-
right); Deep Nonlinear class (right).

###  D.2 Quadratic Landscape

##### Interplay Between Hessian, $\rho$, and the Noise.

In this paragraph, we provide additional details regarding the interplay
between the Hessian, $\rho$, and the noise. In the first experiment
represented in Figure 8, we fix $\rho=\sqrt{\eta}$, where $\eta=0.001$ is the
learning rate. Then, we fix the Hessian $H\in\mathbb{R}^{100\times 100}$ to be
diagonal with random positive eigenvalues. Then, we select the scaling factors
$\sigma\in\\{1,2,4\\}$. For each value of $\sigma$, we optimize the quadratic
loss with SGD and DNSAM where the hessian is scaled up by a factor $\sigma$.
The starting point is $x_{0}=(0.02,\cdots,0.02)$ and the number of iterations
is $20000$. The results are averaged over $5$ runs.

In the second experiment represented in Figure 11, we fix the Hessian $H$ with
random positive eigenvalues. then, we select $\rho=\sqrt{\eta}$, where
$\eta=0.001$ is the learning rate. Then, we select the scaling factors
$\sigma\in\\{1,2,4\\}$. For each value of $\sigma$, we optimize the quadratic
loss with SGD and DNSAM where the hessian is fixed and $\rho$ is scaled up by
a factor $\sigma$. The starting point is $x_{0}=(0.02,\cdots,0.02)$ and the
number of iterations is $20000$. The results are averaged over $5$ runs.

The very same setup holds for the experiments carried out for USAM and is
represented in Figure 9 and Figure 13.

The very same setup holds for the experiments carried out for SAM and is
represented in Figure 10 and Figure 12.

(a)  (b)

Figure 8: Role of the Hessian - Left: Comparison between SGD and DNSAM for
fixed rho and larger Hessians. Right: Ratio between the Loss of DNSAM for
different scaling of the Hessian by the loss of the unscaled case of SGD.

(a)  (b)

Figure 9: Role of the Hessian - Left: Comparison between SGD and USAM for
fixed rho and larger Hessians. Right: Ratio between the Loss of USAM for
different scaling of the Hessian by the loss of the unscaled case of SGD.

(a)  (b)

Figure 10: Role of the Hessian - Left: Comparison between SGD and SAM for
fixed rho and larger Hessians. Right: Ratio between the Loss of PSAM for
different scaling of the Hessian by the loss of the unscaled case of SGD.

(a)  (b)  (c)

Figure 11: Role of $\rho$ \- Left: Comparison between SGD and DNSAM for fixed
hessian and larger $\rho$ values. Center: Zoom at convergence. Right: Ratio
between the Loss of DNSAM for different scaling of the Hessian by the loss of
the unscaled case of DNSAM.

(a)  (b)  (c)

Figure 12: Role of $\rho$ \- Left: Comparison between SGD and PSAM for fixed
hessian and larger $\rho$ values. Center: Zoom at convergence. Right: Ratio
between the Loss of PSAM for different scaling of the Hessian by the loss of
the unscaled case of PSAM.

(a)  (b)  (c)

Figure 13: Role of $\rho$ \- Left: Comparison between SGD and USAM for fixed
hessian and larger $\rho$ values. Center: Zoom at convergence. Right: Ratio
between the Loss of USAM for different scaling of the Hessian by the loss of
the unscaled case of USAM.

##### Stationary Distribution Convex Case

In this paragraph, we provide the details of the experiment about the dynamics
of the SDE of DNSAM in the quadratic convex case of dimension $2$. The hessian
is diagonal with both eigenvalues equal to $1$. We select $\rho=\sqrt{\eta}$,
where $\eta=0.001$ is the learning rate. In the first image on the left of
Figure 14, we show the distribution of $10^{5}$ trajectories all starting at
$(0.02,0.02)$ after $5\cdot 10^{4}$ iterations. In the second image, we plot
the number of trajectories that at a certain time are inside a ball of radius
$0.007$, e.g. close to the origin. As we can see in greater detail in the
third image, all of them are initialized outside such a ball, then they get
attracted inside, and around the $600$-th iteration they get repulsed out of
it. We highlight that the proportion of points inside/outside the ball is
relatively stable. In the fourth image, we count the number of trajectories
that are jumping in or out of such a ball. All of the trajectories enter the
ball between the $400$-th and $500$-th iteration, and then they start jumping
in and out after the iteration $600$. We conclude that this experimental
evidence are supporting the claim that the origin attracts the dynamics, but
repulses it at the moment that the trajectories get too close to it.

(a)  (b)  (c)  (d)

Figure 14: Convex Quadratic - Left: Distribution points around the origin is
scarcer near to the origin; Center-Left: Number of trajectories outside a
small ball around the origin increases over time; Center-Right: All the
trajectories eventually enter the ball and then start exiting it; Right: There
is a constant oscillation of points in and out of the ball.

##### Stationary Distribution Saddle Case

In this paragraph, we provide the details of the experiment about the dynamics
of the SDE of DNSAM in the quadratic saddle case of dimension $2$. The hessian
is diagonal with eigenvalues equal to $1$ and $-1$. We select
$\rho=\sqrt{\eta}$, where $\eta=0.001$ is the learning rate. In the first
image on the left of Figure 4, we show the distribution of $10^{5}$
trajectories all starting at $(0.02,0.02)$ after $5\cdot 10^{4}$ iterations.
In the second image, we plot the number of trajectories that at a certain time
are inside a ball of radius $0.007$, e.g. close to the origin. As we can see
in greater detail in the third image, all of them are initialized outside such
a ball, then they get attracted inside, and around the $1200$-th iteration
they get repulsed out of it. We highlight that the proportion of points
outside the ball is stably increasing, meaning that the trajectories are
slowly escaping from the saddle. In the fourth image, we count the number of
trajectories that are jumping in or out of such a ball. All of the
trajectories enter the ball between the $950$-th and $1000$-th iteration, and
then they start jumping in and out after the iteration $1200$. We conclude
that this experimental evidence are supporting the claim that the origin
attracts the dynamics, but repulses it at the moment that the trajectories get
too close to it, even when this is a saddle.

##### Escaping the Saddle - Low Dimensional

In this paragraph, we provide details for the Escaping the Saddle experiment
in dimension $d=2$. As in the previous experiment, the saddle is a quadratic
of dimension $2$ and its hessian is diagonal with eigenvalues equal to $1$ and
$-1$. We select $\rho=\sqrt{\eta}$, where $\eta=0.001$ is the learning rate.
We initialize the GD, USAM, SAM, SGD, PUSAM, DNSAM, and PSAM in the point
$x_{0}=(0,0.01)$, e.g. in the direction of the fastest escape from the saddle.
In the left of Figure 15, we observe that GD and USAM manage to escape the
saddle while SAM remains stuck. We highlight that running for more iterations
would not change this as SAM is oscillating across the origin. In the second
figure, we observe that the stochastic optimizers escape the saddle quicker
than their deterministic counterpart and even PSAM and DNSAM manage to escape.
Results are averaged over $3$ runs.

##### Escaping the Saddle - High Dimensional

In this paragraph, we provide details for the Escaping the Saddle experiment
in dimension $d=400$. We fix the Hessian $H\in\mathbb{R}^{400\times 400}$ to
be diagonal with random positive eigenvalues. To simulate a saddle, we flip
the sign of the smallest $10$ eigenvalues. We select $\rho=\sqrt{\eta}$, where
$\eta=0.001$ is the learning rate. We study the optimization dynamics of SAM,
PSAM, and DNPSAM as we initialize the process closer and closer to the saddle
in the origin. The starting point $x_{0}=(1,\cdots,1)$ is scaled with factors
$\sigma\in\\{10^{0},10^{-4},10^{-8}\\}$ and we notice that the one scaled with
$\sigma=1$ escapes slowly from the saddle. The one scaled with
$\sigma=10^{-8}$ experiences a sharp spike in volatility and jumps away from
the origin and ends up escaping the saddle faster than the previous case.
Finally, the one scaled with $\sigma=10^{-4}$ stays trapped in the saddle.
Results are represented in Figure 15. Results are averaged over $3$ runs.

(a)  (b)  (c)  (d)

(e)  (f)  (g)  (h)

Figure 15: Escaping the Saddle - Top-Left: Comparison between GD, SAM, and
USAM at escaping from a quadratic saddle. Top-Center-Left: Comparison between
PGD, PSAM, DNSAM, and PUSAM at escaping from a quadratic saddle. Top-Center-
Right: If close enough to the origin, DNSAM escapes from the origin
immediately due to a volatility spike. Top-Right: The DNSAM initialized far
from the origin starts escaping from it and the PSAM which jumped away from it
efficiently escapes the saddle.
Bottom: Both SAM and PSAM get stuck in the origin if initialized too close to
it.

###  D.3 Linear Autoencoder

In this paragraph, we provide additional details regarding the Linear
Autoencoder experiment. In this experiment, we approximate the Identity matrix
of dimension 20 as the product of two square matrices $W1$ and $W2$. As
described in (Kunin et al., 2019), there is a saddle of the loss function
around the origin. Inspired by the results obtained for the quadratic
landscape, we test if SAM and its variants struggle to escape this saddle as
well. To do this, we initialize the two matrices with entries normally
distributed. We select $\rho=\sqrt{\eta}$, where $\eta=0.001$ is the learning
rate. Then, we study the dynamics of the optimization process in case we scale
the matrices by a factor $\sigma\in\\{10^{-2},10^{-3},5\cdot
10^{-3},10^{-4},10^{-5}\\}$. As we can see from the first image of Figure 5,
initializing SAM far from the origin, that is $\sigma=0.01$, allows SAM to
optimize the loss. Decreasing $\sigma$ implies that SAM becomes slower and
slower at escaping the loss up to not being able to escape it anymore. The
second image shows that the very same happens if we use DNSAM. However, if the
process is initialized extremely close to the origin, that is
$\sigma=10^{-5}$, then the process enjoys a volatility spike that pushes it
away from the origin. This allows the process to escape the saddle quickly and
effectively. In the third image, we observe that similarly to SAM, PSAM
becomes slower at escaping if $\sigma$ is lower. In the fourth image, we
compare the behavior of GD, USAM, SAM, PGD, PUSAM, DNSAM, and PSAM where
$\sigma=10^{-5}$. We observe that DNSAM is the fastest algorithm to escape the
saddle, followed by the others. As expected, SAM does not escape. Results are
averaged over $3$ runs.

###  D.4 Embedded Saddle

In this paragraph, we provide additional details regarding the Embedded Saddle
experiment. In this experiment, we optimize a regularized quadratic
$d$-dimensional landscape
$L(x)=\frac{1}{2}x^{\top}Hx+\lambda\sum_{i=1}^{d}x_{i}^{4}$. As described in
(Lucchi et al., 2022), if $H$ is not PSD, there is a saddle of the loss
function around the origin and local minima away from it. We fix the Hessian
$H\in\mathbb{R}^{400\times 400}$ to be diagonal with random positive
eigenvalues. To simulate a saddle, we flip the sign of the smallest $10$
eigenvalues. The regularization parameter is fixed at $\lambda=0.001$. We use
$\eta=0.005$, $\rho=\sqrt{\eta}$, run for $200000$ and average over $3$ runs.
In Figure 16 we see the very same observations we had for the Autoencoder.

(a)  (b)  (c)  (d)

Figure 16: Embedded Saddle - Left: SAM does not escape the saddle if it is too close to it. Center-Left: DNSAM always escapes it, but more slowly if initialized closer to the origin. If extremely close, it recovers speed thanks to a volatility spike. Center-Right: Similarly to SAM, PSAM gets progressively slower the closer it gets initialized to the origin. Right: SAM is stuck while the other optimizers manage to escape.  Name | Algorithm | Theorem for SDE
---|---|---
SGD | $x_{k+1}=x_{k}-\eta\nabla f_{\gamma_{k}}(x_{k})$ | Theorem 1 (Li et al., 2017)
SAM | $x_{k+1}=x_{k}-\eta\nabla f_{\gamma_{k}}\left(x_{k}+\rho\frac{\nabla f_{\gamma_{k}}(x_{k})}{\lVert\nabla f_{\gamma_{k}}(x_{k})\rVert}\right)$ | Theorem 3.5
USAM | $x_{k+1}=x_{k}-\eta\nabla f_{\gamma_{k}}\left(x_{k}+\rho\nabla f_{\gamma_{k}}(x_{k})\right)$ | Theorem 3.2
DNSAM | $x_{k+1}=x_{k}-\eta\nabla f_{\gamma_{k}}\left(x_{k}+\rho\frac{\nabla f_{\gamma_{k}}(x_{k})}{\lVert\nabla f_{\gamma_{k}}(x_{k})\rVert}\right)$ | Not Available
PGD | $x_{k+1}=x_{k}-\eta\nabla f(x_{k})+\eta Z$ | Theorem 1 (Li et al., 2017)
PSAM | $x_{k+1}=x_{k}-\eta\nabla f\left(x_{k}+\rho\frac{\nabla f(x_{k})+Z}{\lVert\nabla f(x_{k})+Z\rVert}\right)+\eta Z$ | Theorem 3.6
PUSAM | $x_{k+1}=x_{k}-\eta\nabla f\left(x_{k}+\rho\nabla f(x_{k})+Z\right)+\eta Z$ | Theorem 3.3
PDNSAM | $x_{k+1}=x_{k}-\eta\nabla f\left(x_{k}+\rho\frac{\nabla f(x_{k})+Z}{\lVert\nabla f(x_{k})\rVert}\right)+\eta Z$ | Theorem 3.4
Table 1: Comparison of algorithms for methods analyzed in the paper. The learning rate is $\eta$, the radius is $\rho$, and $Z\sim\mathcal{N}(0,\Sigma)$ is the injected noise. Name | Drift Term
---|---
SGD | $-\nabla f(x)$
SAM | $-\nabla\left(f(x)+\rho\mathbb{E}\left[\lVert\nabla f_{\gamma}(x)\rVert_{2}\right]\right)$
USAM | $-\nabla\left(f(x)+\frac{\rho}{2}\mathbb{E}\left[\lVert\nabla f_{\gamma}(x)\rVert^{2}_{2}\right]\right)$
PGD | $-\nabla f(x)$
PSAM | $-\nabla\left(f(x)+\rho\mathbb{E}\left[\lVert\nabla f_{\gamma}(x)\rVert_{2}\right]\right)$
PUSAM | $-\nabla\left(f(x)+\frac{\rho}{2}\lVert\nabla f(x)\rVert^{2}_{2}\right)$
(P)DNSAM | $-\nabla\left(f(x)+\rho\lVert\nabla f(x)\rVert_{2}\right)$
Table 2: Comparison of the drift terms of the SDEs for methods analyzed in the paper.  Name | Diffusion Term | $\tilde{\Sigma}(x)$
---|---|---
SGD | $\sqrt{\eta}\left(\Sigma(x)\right)^{\frac{1}{2}}$ |
SAM | $\sqrt{\eta\left(\Sigma(x)+\rho\left(\tilde{\Sigma}(x)+\tilde{\Sigma}(x)^{\top}\right)\right)}$ | $\mathbb{E}\left[\left(\nabla f\left(x\right)-\nabla f_{\gamma}\left(x\right)\right)\cdot\left({\mathbb{E}}\left[\frac{H_{\gamma}(x)\nabla f_{\gamma}(x)}{\lVert\nabla f_{\gamma}(x)\rVert_{2}}\right]-\frac{H_{\gamma}(x)\nabla f_{\gamma}(x)}{\lVert\nabla f_{\gamma}(x)\rVert_{2}}\right)^{\top}\right]$
USAM | $\sqrt{\eta\left(\Sigma(x)+\rho\left(\tilde{\Sigma}(x)+\tilde{\Sigma}(x)^{\top}\right)\right)}$ | $\mathbb{E}\left[\left(\nabla f\left(x\right)-\nabla f_{\gamma}\left(x\right)\right)\left(\mathbb{E}\left[H_{\gamma}(x)\nabla f_{\gamma}(x)\right]-H_{\gamma}(x)\nabla f_{\gamma}(x)\right)^{\top}\right]$
PGD | $\sqrt{\eta\Sigma(x)}$ |
PSAM | $\sqrt{\eta\left(\Sigma(x)+\rho\left(\bar{\Sigma}(x)+\bar{\Sigma}(x)^{\top}\right)\right)}$ | $H(x)\mathbb{E}\left[\left(\nabla f\left(x\right)-\nabla f_{\gamma}\left(x\right)\right)\cdot\left({\mathbb{E}}\left[\frac{\nabla f_{\gamma}(x)}{\lVert\nabla f_{\gamma}(x)\rVert_{2}}\right]-\frac{\nabla f_{\gamma}(x)}{\lVert\nabla f_{\gamma}(x)\rVert_{2}}\right)^{\top}\right]$
PUSAM | $\left(I_{d}+\rho H(x)\right)\left(\eta\Sigma\left(x\right)\right)^{1/2}$ |
(P)DNSAM | $\left(I_{d}+\rho\frac{H(x)}{\lVert\nabla f(x)\rVert_{2}}\right)\left(\eta\Sigma(x)\right)^{\frac{1}{2}}$ |
Table 3: Comparison of the diffusion terms of SDEs for methods analyzed in the
paper. The matrix $\Sigma(x)$ is equal to $\Sigma(x)^{\text{SGD}}$ and
$H(x)=\nabla^{2}f(x)$.


# Three dimensional near-horizon metrics that are Einstein-Weyl

Matthew Randall  Department of Mathematics and Statistics
Faculty of Science, Masaryk University
Kotlářská 2, 611 37 Brno
Czech Republic<EMAIL_ADDRESS>
###### Abstract.

We investigate which three dimensional near-horizon metrics $g_{NH}$ admit a
compatible 1-form $X$ such that $(X,[g_{NH}])$ defines an Einstein-Weyl
structure. We find explicit examples and see that some of the solutions give
rise to Einstein-Weyl structures of dispersionless KP type and dispersionless
Hirota (aka hyperCR) type.

###### 2010 Mathematics Subject Classification:

53B15, 53B30 (primary), and 83C57 (secondary)

This work is supported by the Grant agency of the Czech Republic P201/12/G028.

##  1\. Introduction

Let $M^{3}$ be a three dimensional smooth manifold equipped with a Lorentzian
metric. A near-horizon metric on $M^{3}$ is a Lorentzian metric of the form

(1) $g_{NH}=2{\rm d}\nu\left({\rm d}r+rh(x){\rm d}x+\frac{r^{2}}{2}F(x){\rm
d}\nu\right)+{\rm d}x{\rm d}x,$

where $x$, $\nu$ and $r$ are local coordinates and $h(x)$, $F(x)$ are
arbitrary functions of $x$. Near-horizon geometries in higher dimensions are
studied in relation to the existence of extremal black holes [5], [14], [16],
[17], [18].

The near-horizon metric (1) is derived as follows. For a smooth null
hypersurface $\Sigma^{2}$ in $M^{3}$, there exists an adapted coordinate
system called Gaussian null coordinates valid in any neighbourhood of
$\Sigma^{2}$. Imposing that the normal vector field $N^{a}$ be Killing in
$M^{3}$ implies that $\Sigma^{2}$ is a Killing horizon. We further require
that the Killing horizon $\Sigma^{2}$ be degenerate. This means that the flow
of $N^{a}$ is along affinely parameterised null geodesics on $\Sigma^{2}$, or
equivalently that the surface gravity of $\Sigma^{2}$ is zero. The scaling
limit of the degenerate Killing horizon is called a near-horizon geometry, and
the metric is of the form (1).

In dimensions 4 or more, imposing the vacuum Einstein equations on the near-
horizon geometry metric $g_{NH}$ give rise to the near-horizon geometry
equations on the spatial section of the degenerate Killing horizon. Solutions
to this overdetermined system of equations on a compact cross-section are
related to the existence of extremal black holes. For further details, see
[13], [14] and [16].

Similarly, we can require that a 3 dimensional near-horizon geometry metric
satisfies Einstein’s equations. This has been done in [14]. In this case, the
near-horizon geometry equations reduce to a pair of first order ODEs that can
be integrated explicitly. It is also found that global and periodic solutions
for $h(x)$ exist on a 1-dimensional cross-section $\mathcal{H}$ which has the
topology of the circle; this is the 1-dimensional analogue of an extremal
black hole horizon.

Instead of asking Einstein’s equations to be satisfied, we can ask for a
natural generalisation which is to impose the Einstein-Weyl equations instead.
This requires an additional structure of a 1-form, to be explained in the next
section. Here we investigate whether there exists a 1-form $X$ compatible with
(1) such that $(X,g_{NH})$ satisfies the Einstein-Weyl equations.

In the 37th winter school in geometry and physics, held in Srní, Czech
Republic in 2017, plenary lectures on non-expanding horizons including near
horizon geometries were presented by Jerzy Lewandowski and lectures on
Einstein-Weyl geometry and dispersionless integrable systems were presented by
Maciej Dunajski. The latter topics were also covered in the lectures by Evgeny
Ferapontov. We hope that attendees of the school would find these topics to be
interesting and delve into the rich and steadily expanding field that relates
integrable PDEs to general relativity and classical differential geometry.

##  2\. Three dimensional Einstein-Weyl geometries

Einstein-Weyl geometries in 3 dimensions play an important role because of its
relationship with the geometry of third order ODEs (see [20] and [21]),
twistor theory (see [12] and [15]) and integrable systems (see [1], [2], [4]
and [11]). Let $(M^{3},[g])$ be a smooth conformal manifold equipped with a
conformal class of metrics $[g]$ of (pseudo)-Riemannian signature. In 3
dimensions, this is either Riemannian or Lorentzian. Any 2 representative
metric $g$, $\tilde{g}\in[g]$ are related via

$\tilde{g}=\Omega^{2}g$

for some smooth positive function $\Omega$. A Weyl structure on $(M^{3},[g])$
is a torsion-free connection $D$ that preserves the conformal class of
metrics. Equivalently,

$D_{a}g_{bc}=2X_{a}g_{bc}$

for $g_{ab}$ a representative in the conformal class and $X_{a}$ a 1-form that
is not necessarily closed. The Einstein-Weyl equations are the system of
equations obtained by requiring that that the symmetric part of the Ricci
tensor of the Weyl connection is pure trace, i.e.

$R^{D}_{(ab)}=sg_{ab}$

for some function $s$. This system of equations is conformally invariant.
Writing $R^{D}_{(ab)}$ in terms of the Levi-Civita connection $\nabla$ for the
metric $g_{ab}$ and 1-form $X_{a}$, we get

$\nabla_{(a}X_{b)}+X_{a}X_{b}+\mathrm{P}_{ab}=\Lambda g_{ab},$

where round brackets denote symmetrisation over indices, $\mathrm{P}_{ab}$ is
the Schouten tensor of $g_{ab}$ and the trace term

$\Lambda=\frac{1}{3}\left(\nabla_{a}X^{b}+X_{a}X^{b}+\mathrm{P}\right)$

is a function (more appropriately, a section of a density valued line bundle
and we refer to [9] and [10] for more details).

In [3], it is shown that a 3-dimensional Lorentzian Einstein-Weyl structure
corresponds to a 2 parameter family of totally-geodesic null hypersurfaces in
$M^{3}$. A twistor description is developed in [12], where it is shown that
real-analytic Lorentzian 3-dimensional Einstein-Weyl structures arise
precisely from the Kodaira deformation space of rational normal curves with
normal bundle $\mathcal{O}(2)$ in a 2-dimensional complex manifold. The 2-form
${\rm d}X$ is called the Faraday 2-form of the Weyl structure. If ${\rm
d}X=0$, then $X$ is locally exact and $g$ can be conformally rescaled to a
metric of constant curvature in 3 dimensions. In particular, it implies that
$g$ is locally conformally flat. A computation of the Cotton tensor of the
near-horizon metric $g_{NH}$ shows that

######  Proposition 2.1.

The metric of the form (1) is locally conformally flat iff

(2) $F^{\prime}(x)=F(x)h(x).$

###### Proof.

Condition (2) implies that the Cotton tensor of $g_{NH}$ is zero, and
conversely so. ∎

##  3\. Results

We shall assume that $(M^{3},[g_{NH}])$ is smooth. We consider an ansatz for
$X$ of the form

$X=ch(x){\rm d}x+X_{2}{\rm d}r+X_{3}{\rm d}v,$

where $c$ is a constant and the functions $X_{2}=X_{2}(x,\nu,r)$,
$X_{3}=X_{3}(x,\nu,r)$ are to be determined. We also require that
$X|_{r=0}=ch(x){\rm d}x$, so that the 1-form $X$ is determined up to a
constant multiple $c$ by its value on the spatial section of the degenerate
Killing horizon. In the following analysis, it turns out that there is a
degenerate case when $c=-\frac{1}{2}$. We have the following:

######  Theorem 3.1.

A 3 dimensional near-horizon geometry metric (1) on $M^{3}$ of the form

$\displaystyle g_{NH}=$ $\displaystyle 2{\rm d}\nu\left({\rm d}r+rh(x){\rm
d}x+\frac{r^{2}}{2}e^{{\rmintop\nolimits h(x)dx}}\wp\left(\rmintop\nolimits
e^{{\frac{1}{2}\rmintop\nolimits h(x)dx}}dx+a;0,b\right){\rm d}\nu\right)$ (3)
$\displaystyle+{\rm d}x{\rm d}x$

where $\wp(z;g_{2},g_{3})$ is the Weierstrass elliptic function, $a$, $b$ are
constants, and a Weyl connection $X$ of the form

$X=-\frac{1}{2}h(x){\rm d}x-2re^{{\rmintop\nolimits
h(x)dx}}\wp\left(\rmintop\nolimits e^{{\frac{1}{2}\rmintop\nolimits
h(x)dx}}dx+a;0,b\right){\rm d}\nu$

defines an Einstein-Weyl structure $([g_{NH}],X)$ on $M^{3}$. This depends on
$1$ free function of one variable.

In particular, taking $h(x)$ to be globally defined and periodic allows
$\mathcal{H}$ to have the topology of a circle. In the case that $h(x)=0$, the
metric simplifies to

$g=2{\rm d}\nu\left({\rm d}r+\frac{r^{2}}{2}\wp(x+a;0,b){\rm d}\nu\right)+{\rm
d}x{\rm d}x$

and the 1-form $X$ is given by

$X=-2r\wp(x+a;0,b){\rm d}\nu.$

We recognise that the function

$u(x,\nu,r)=-\frac{r^{2}}{2}\wp(x+a;0,b)$

satisfies the dispersionless Kadomtsev-Petviashvili (dKP) equation given by

$2(u_{\nu}-uu_{r})_{r}=u_{xx}$

and the Einstein-Weyl structure corresponds to one of dKP type. For further
details about such Einstein-Weyl structures we refer to [8]. For generic
values of $c$, which also allows for $c=-\frac{1}{2}$, we have

######  Theorem 3.2.

A 3 dimensional near-horizon geometry metric (1) on $M^{3}$ and a Weyl
connection $X$ of the form

$X=ch(x){\rm d}x+r((2c+1)h^{\prime}+c(2c+1)h^{2}-2F(x)){\rm d}v$

defines an Einstein-Weyl structure if and only if

$F(x)=\frac{h^{\prime\prime}+4chh^{\prime}+2c^{2}h^{3}}{2h}$

and $h(x)$ satisfies the $4^{\rm th}$ order ODE

$\displaystyle
h^{3}(h^{\prime})^{2}(c-1)^{2}-\frac{1}{2}(c-1)^{2}h^{4}h^{\prime\prime}+\frac{9}{4}(c-1)h^{2}h^{\prime}h^{\prime\prime}$
(4)
$\displaystyle-\frac{3}{4}(c-1)h^{3}h^{\prime\prime\prime}-\frac{1}{2}(h^{\prime})^{2}h^{\prime\prime}+\frac{1}{2}hh^{\prime}h^{\prime\prime\prime}+h(h^{\prime\prime})^{2}-\frac{1}{4}h^{2}h^{\prime\prime\prime\prime}=0.$

##  4\. Proof

###### Proof of Theorems 3.1 and 3.2.

We start with an ansatz of the form

$X=ch(x){\rm d}x+X_{2}{\rm d}r+X_{3}{\rm d}\nu,$

where the functions $X_{2}=X_{2}(x,\nu,r)$, $X_{3}=X_{3}(x,\nu,r)$ are to be
determined, and we require that $X|_{r=0}=ch(x)dx$, so that the 1-form $X$ is
determined by its value on the spatial section of the degenerate Killing
horizon. We find that substituting this ansatz for $X$ into the Einstein-Weyl
equations, the ${\rm d}r{\rm d}r$ component gives

$\partial_{r}X_{2}+X_{2}^{2}=0,$

which has solutions

$X_{2}=\frac{1}{r+f_{1}(x,\nu)}\qquad\mbox{or}\qquad X_{2}=0.$

Since the first solution does not restrict to zero on $\\{r=0\\}$, we take
$X_{2}=0$ instead. The ${\rm d}x{\rm d}x$ component gives now

$-2F-\partial_{r}X_{3}+c(2c+1)h^{2}+(2c+1)h^{\prime}=0,$

from which we obtain

$X_{3}=(-2F+c(2c+1)h^{2}+(2c+1)h^{\prime})r+f_{2}(x,\nu).$

Once again requiring that $X_{3}|_{r=0}=0$ implies $f_{2}(x,\nu)=0$. With
this, only the ${\rm d}x{\rm d}\nu$ and ${\rm d}\nu{\rm d}\nu$ components
remain to be solved in the Einstein-Weyl equations. The ${\rm d}x{\rm d}\nu$
component gives

$\frac{r}{2}(2c+1)\left(h^{\prime\prime}-2hF+4chh^{\prime}+2c^{2}h^{3}\right)=0.$

This vanishes identically when $c=-\frac{1}{2}$. Otherwise, we have

$F(x)=\frac{h^{\prime\prime}+4chh^{\prime}+2c^{2}h^{3}}{2h}.$

In the first case where $c=-\frac{1}{2}$, the remaining equation in the ${\rm
d}\nu{\rm d}\nu$ component is

$-3Fh^{2}+5hF^{\prime}+2Fh^{\prime}+12F^{2}-2F^{\prime\prime}=0,$

which has solutions

$F(x)=\wp\left(\rmintop\nolimits\exp\left({\frac{1}{2}\rmintop\nolimits
h(x)dx}\right)dx+a,0,b\right)\exp\left({\rmintop\nolimits h(x)dx}\right)$

whatever $h(x)$ is. This proves Theorem 3.1. For the other case, substituting

$F(x)=\frac{h^{\prime\prime}+4chh^{\prime}+2c^{2}h^{3}}{2h}$

gives

$X_{3}=\frac{ch^{3}+(1-2c)hh^{\prime}-h^{\prime\prime}}{h}r$

and the only remaining ${\rm d}\nu{\rm d}\nu$ component of the Einstein-Weyl
equations gives the $4^{\rm th}$ order ODE

$\displaystyle
h^{3}(h^{\prime})^{2}(c-1)^{2}-\frac{1}{2}(c-1)^{2}h^{4}h^{\prime\prime}+\frac{9}{4}(c-1)h^{2}h^{\prime}h^{\prime\prime}$
$\displaystyle-\frac{3}{4}(c-1)h^{3}h^{\prime\prime\prime}-\frac{1}{2}(h^{\prime})^{2}h^{\prime\prime}+\frac{1}{2}hh^{\prime}h^{\prime\prime\prime}+h(h^{\prime\prime})^{2}-\frac{1}{4}h^{2}h^{\prime\prime\prime\prime}=0.$

Note that this formula is still well-defined for $c=-\frac{1}{2}$. This proves
Theorem 3.2. ∎

##  5\. Explicit solutions

In this section, we find interesting families of solutions to the 4th order
ODE (3.2). Consider the following second order nonlinear ODE

(5) $\displaystyle h^{\prime\prime}=\alpha hh^{\prime}+\beta h^{3}$

with $\alpha$, $\beta$ constant. We find, upon substituting (5) and its
derivatives into (3.2), that we obtain

$\displaystyle-\frac{h^{3}}{4}\big{(}2(c-1)^{2}+3\alpha(c-1)+\alpha^{2}-\beta\big{)}\left(\beta
h^{4}+\alpha h^{2}h^{\prime}-2(h^{\prime})^{2}\right)=0.$

Thus solutions to (5) automatically satisfy (3.2) provided

(6) $\beta=2(c-1)^{2}+3\alpha(c-1)+\alpha^{2}.$

Switching independent and dependent variables $(x,h(x))\mapsto(h,x(h))$, the
non-linear second order ODE (5) is dual to

$\displaystyle x^{\prime\prime}=-\beta h^{3}(x^{\prime})^{3}-\alpha
h(x^{\prime})^{2},$

which upon setting $y(h)=h^{2}x^{\prime}(h)$, gives an Abel differential
equation of the first kind:

$\displaystyle y^{\prime}=\frac{1}{h}\left(-\beta y^{3}-\alpha
y^{2}+2y\right).$

This has solutions given by

(7) $\displaystyle
h=\frac{\gamma\sqrt{y}\exp\left(\frac{\alpha}{2\sqrt{\alpha^{2}+8\beta}}\tanh^{-1}\left(\frac{2\beta
y+\alpha}{\sqrt{\alpha^{2}+8\beta}}\right)\right)}{(\beta y^{2}+\alpha
y-2)^{\frac{1}{4}}},$

and consequently $x$ viewed as a function of $y$ is given by

(8) $\displaystyle
x(y)=-\frac{1}{\gamma}\rmintop\nolimits\frac{\exp\left(\frac{-\alpha}{2\sqrt{\alpha^{2}+8\beta}}\tanh^{-1}\left(\frac{2\beta
y+\alpha}{\sqrt{\alpha^{2}+8\beta}}\right)\right)}{\sqrt{y}(\beta y^{2}+\alpha
y-2)^{\frac{3}{4}}}{\rm d}y.$

When $\alpha=0$, we obtain $\beta=2(c-1)^{2}$ from (6). In this case,
solutions to the nonlinear ODE (5)

$h^{\prime\prime}=\beta h^{3}=2(c-1)^{2}h^{3}$

are given by the Jacobi elliptic function

$h(x)=a~{}{\rm
sn}\left(a\left(\frac{\sqrt{-2\beta}}{2}x+b\right),i\right)=a~{}{\rm
sn}\left(a(i(c-1)x+b),i\right),$

with $a$, $b$ the constants of integration. Here the elliptic modulus is given
by $i=\sqrt{-1}$. Alternatively, we obtain

$\displaystyle h=\frac{\gamma\sqrt{y}}{(\beta y^{2}-2)^{\frac{1}{4}}}$

from (7) and

$\displaystyle x=-\frac{1}{\gamma}\rmintop\nolimits\frac{1}{\sqrt{y}(\beta
y^{2}-2)^{\frac{3}{4}}}{\rm d}y$

from (8). Passing to $y=\left(\frac{2}{\beta z}\right)^{\frac{1}{2}}$, this
gives the expression in terms of hypergeometric functions

$\displaystyle
h=\frac{\gamma}{\beta^{\frac{1}{4}}(1-z)^{\frac{1}{4}}}\quad\text{and}\quad
x=\frac{\sqrt{2z}}{2\gamma\beta^{\frac{1}{4}}}{}_{2}F_{1}\left(\frac{1}{2},\frac{3}{4};\frac{3}{2};z\right).$

When $\beta=0$, we have

$(2c-2+\alpha)(c-1+\alpha)=0$

from (6) and therefore $\alpha=1-c$ or $\alpha=2-2c$. For $\alpha\neq 0$,
which implies $c\neq 1$, solutions to (5) are given by

$h=\frac{1}{\alpha}\tan\left(\frac{1}{2}\sqrt{2\ell\alpha}(x+b)\right)\sqrt{2\ell\alpha}.$

This is periodic but not globally defined. For $\alpha=0$, or equivalently
$c=1$, solutions to (5) are given by

$h=\ell x+b$

with $b$, $\ell$ the constants of integration. The degenerate Killing horizon
in this case has the topology of the real line and the metric is not
conformally flat. If we do not assume either $\alpha$ or $\beta$ is zero, and
consider the case when $c=1$, then equation (5) is

(9) $\displaystyle h^{\prime\prime}=\alpha hh^{\prime}+\alpha^{2}h^{3}.$

From (7), we obtain

$\displaystyle h=\frac{\gamma\sqrt{y}}{(-1)^{\frac{1}{4}}(\alpha
y+2)^{\frac{1}{6}}(1-\alpha y)^{\frac{1}{3}}}.$

We consider real solutions by taking $\gamma=(-1)^{\frac{1}{4}}\varepsilon$.
This gives

$\displaystyle h^{6}=\frac{\varepsilon y^{3}}{(\alpha y+2)(1-\alpha y)^{2}},$

which implies that $y$ is a solution of the cubic equation

$\varepsilon y^{3}=h^{6}(\alpha y+2)(1-\alpha y)^{2}.$

In the limit $\varepsilon\to 0$, we have $y=-\frac{2}{\alpha}$ or
$y=\frac{1}{\alpha}$. In these cases we obtain

$x(h)=-\rmintop\nolimits\frac{2}{\alpha h^{2}}{\rm d}h+b=\frac{2}{\alpha
h}+b\mbox{~{}and~{}}x(h)=\rmintop\nolimits\frac{1}{\alpha h^{2}}{\rm
d}h+b=-\frac{1}{\alpha h}+b.$

Hence

$h(x)=\frac{2}{\alpha(x-b)}\mbox{~{}and~{}}h(x)=-\frac{1}{\alpha(x-b)}$

satisfy (3.2) with the parameter $c=1$. For these solutions $h(x)$ is singular
on the degenerate Killing horizon. These solutions give conformally flat
metrics when $\alpha$ is chosen so that $h(x)=\frac{-2}{x-b}$ or
$\frac{1}{x-b}$. In fact when $c=1$, (3.2) can be integrated to give

(10)
$\displaystyle-\frac{1}{4}h^{2}h^{\prime\prime\prime}+hh^{\prime}h^{\prime\prime}-\frac{1}{2}(h^{\prime})^{3}=0.$

Passing to $h(x)={\rm e}^{f(x)}$, we see that (10) is satisfied iff $f(x)$
satisfies the non-linear $3^{\rm rd}$ order ODE

(11) $f^{\prime\prime\prime}-f^{\prime}f^{\prime\prime}-(f^{\prime})^{3}=0.$

This ODE is none other than (9) with $f^{\prime}$ replacing $h$ and
$\alpha=1$. This gives the following additional solutions to (3.2)

$h(x)=(x-b)^{2}\mbox{~{}and~{}}h(x)=\frac{1}{x-b}.$

The conformal structure for the first $h(x)$ is not flat and $h(x)$ is
globally defined on the line horizon, while the conformal structure for the
second is flat, which can be rescaled to a metric of constant scalar
curvature.

Interestingly, we can relate the solutions to (5) with $c=-1$, $\alpha=2$,
$\beta=0$ to dispersionless Hirota type or hyperCR Einstein-Weyl structures.
For a review of such structures see [6], [7]. Recall that a metric of the form

$\displaystyle g=$ $\displaystyle({\rm d}x+H_{r}d\nu)^{2}-4\left({\rm
d}r-H_{x}{\rm d}v\right){\rm d}\nu$ $\displaystyle=$ $\displaystyle 2{\rm
d}\nu\left(-2{\rm d}r+H_{r}{\rm d}x+(2H_{x}+\frac{H_{r}^{2}}{2}){\rm
d}\nu\right)+{\rm d}x{\rm d}x$

where $H=H(x,\nu,r)$ and a Weyl connection of the form

$X=\frac{1}{2}H_{rr}{\rm d}x+\frac{1}{2}(H_{r}H_{rr}+2H_{xr}){\rm d}\nu$

defines a hyperCR Einstein-Weyl structure iff the hyperCR equation

(12) $\displaystyle H_{x}H_{rr}-H_{r}H_{xr}-H_{xx}+H_{r\nu}=0$

is satisfied. A particular family of solutions to this equation can be found:

$H(x,\nu,r)=j\tanh^{3}\left(\frac{a^{2}}{b}r+b\nu+ax+e\right)+k\tanh\left(\frac{a^{2}}{b}r+b\nu+ax+e\right)+l$

is a $6$ parameter family of solutions satisfying (12) depending on constants
$a$, $b$, $e$, $j$, $k$, $l$. Aligning the 1-form in the hyperCR case with our
ansatz for $X$, we require that

$X|_{r=0}=\frac{1}{2}H_{rr}{\rm d}x+\frac{1}{2}(H_{r}H_{rr}+2H_{xr}){\rm
d}\nu|_{r=0}=ch(x){\rm d}x.$

Hence

$H=ch(x)r^{2}+f_{2}(x,\nu)$

for some function $h(x)$ and $f_{2}(x,\nu)$. Additionally, to align the
hyperCR metric with the near-horizon metric (1), we require $f_{2}(x,\nu)=0$.
Plugging this solution for $H$ back into the metric and 1-form $X$, we
discover that the Einstein-Weyl equations are satisfied iff

$\displaystyle h^{\prime\prime}=$ $\displaystyle-2chh^{\prime}.$

This has solutions given by

$h(x)=\frac{\sqrt{c\ell}}{c}\tanh\left(\sqrt{c\ell}(x+b)\right)$

for $c\neq 0$. Redefining coordinates $r\mapsto\tilde{r}=-2r$ and renaming
$r$, we have

######  Proposition 5.1.

A 3 dimensional Lorentzian metric on $M^{3}$ of the form

(13) $g=2{\rm d}\nu\left({\rm d}r-chr{\rm
d}x+\frac{r^{2}}{2}(ch^{\prime}+c^{2}h^{2}){\rm d}\nu\right)+{\rm d}x{\rm d}x$

where $c\neq 0$ with

$h(x)=\frac{\sqrt{c\ell}}{c}\tanh\left(\sqrt{c\ell}(x+b)\right)$

satisfying the ODE

$h^{\prime\prime}=-2chh^{\prime}$

and a Weyl connection $X$ of the form

$\displaystyle X=$ $\displaystyle ch{\rm d}x-cr(ch^{2}+h^{\prime}){\rm d}\nu$

defines a hyperCR Einstein-Weyl structure on $M^{3}$. Comparing the metric of
the form (13) to the near horizon metric (1), we see that setting $c=-1$ gives
the near-horizon metric

$g_{NH}=2{\rm d}\nu\left({\rm d}r+hr{\rm
d}x+\frac{r^{2}}{2}(-h^{\prime}+h^{2}){\rm d}\nu\right)+{\rm d}x{\rm d}x$

and Weyl connection

$\displaystyle X=$ $\displaystyle-h{\rm d}x+r(-h^{2}+h^{\prime}){\rm d}\nu.$

In particular, for $c=-1$, the ODE $h^{\prime\prime}=2hh^{\prime}$ that $h(x)$
satisfies agrees with the solution (5) to (3.2) with the parameters $\beta=0$,
$c=-1$, $\alpha=1-c=2$.

##  6\. Further remarks and outlook

It is curious that solving for the Einstein-Weyl equations for a near-horizon
metric (1) give rise to dKP Einstein-Weyl structures for $c=-\frac{1}{2}$ as
in Theorem 3.1 and hyperCR Einstein-Weyl structures for $c=-1$ as in Theorem
3.2 and Proposition 5.1. Do other parameters of $c$ give rise to other
interesting EW structures? The metric in Gaussian null coordinates on $M^{3}$
with a smooth Killing horizon (see [14]) is given by

(14) $g=2{\rm d}\nu\left({\rm d}r+rh(x,r){\rm d}x+\frac{r}{2}F(x,r){\rm
d}\nu\right)+\gamma(x,r)^{2}{\rm d}x{\rm d}x.$

In [19], the authors investigated the case where (14) is Einstein with non-
zero cosmological constant. Similarly, we can investigate whether the metric
of the form (14) admits a compatible 1-form $X_{a}$ such that $([g],X)$ is
Einstein-Weyl. Some computations have been made but the calculations are
considerably more involved than those presented here.

## References

  * [1] B. Doubrov, E. V. Ferapontov, B. Kruglikov and V. Novikov, On the integrability in Grassmann geometries: integrable systems associated with fourfolds $Gr(3,5)$, arXiv:1503.02274.
  * [2] D. M. J. Calderbank and B. Kruglikov, Integrability via geometry: dispersionless differential equations in three and four dimensions, arXiv:1612.02753.
  * [3] E. Cartan, Sur une classe d’espaces de Weyl, Ann. Sci. École Norm. Sup. (3) 60 (1943) 1–16.
  * [4] M. Dunajski, E.V. Ferapontov and B. Kruglikov, On the Einstein-Weyl and conformal self-duality equations, Journal of Mathematical Physics 56 (8) (2015).
  * [5] M. Dunajski, J. Gutowski, W. Sabra, Einstein-Weyl Spaces and Near-Horizon Geometry, arXiv:1610.08953.
  * [6] M. Dunajski, W. Kryński, Einstein-Weyl geometry, dispersionless Hirota equation and Veronese webs, Math. Proc. Cambridge Philos. Soc. 157 1 (2014), 139–150.
  * [7] M. Dunajski, W. Kryński, Point invariants of third-order ODEs and hyper-CR Einstein-Weyl structures, J. Geom. Phys. 86 (2014), 296–302.
  * [8] M. Dunajski, L.J. Mason, and K.P. Tod, Einstein-Weyl geometry, the dKP equation and twistor theory, J. Geom. Phys. 37 (2001), 63–93.
  * [9] M. G. Eastwood, Notes on conformal differential geometry, Supp. Rend. Circ. Matem. Palermo 43 (1996), 57–76.
  * [10] M. G. Eastwood and K. P. Tod, Local constraints on Einstein-Weyl geometries, J. Reine Angew. Math., 491 (1997), 183–198.
  * [11] E. V. Ferapontov and B. Kruglikov, Dispersionless integrable systems in 3D and Einstein-Weyl geometry, J. Diff. Geom. 97 (2014), 215–254.
  * [12] N. J. Hitchin, Complex manifolds and Einstein’s equations, Twistor geometry and nonlinear systems (Primorsko, 1980), 73–99, Lecture Notes in Math. 970, Springer, Berlin-New York (1982).
  * [13] H. K. Kunduri and J. Lucietti, A classification of near-horizon geometries of extremal vacuum black holes, Journal of Mathematical Physics 50, 082502, 2009, 41pp.
  * [14] H. K. Kunduri and J. Lucietti, Classification of Near-Horizon Geometries of Extremal Black Holes, Living Rev. Relativity 16, 8 , 2013.
  * [15] C. LeBrun and L. J. Mason, The Einstein-Weyl equations, scattering maps, and holomorphic disks, Math. Res. Lett. 16, (2009), 291–301.
  * [16] J. Lewandowski, T. Pawlowski, Extremal Isolated Horizons: A Local Uniqueness Theorem, Class. Quantum Grav. 20 (2003), 587–-606, arXiv:gr-qc/0208032.
  * [17] J. Lewandowski, I. Racz, A. Szereszeweski, Near Horizon Geometries and Black Hole Holograph, arXiv:1701.01704.
  * [18] J. Lewandowski, A. Szereszeweski, P. Waluk, When Isolated Horizons met Near Horizon Geometries, 2nd LeCosPA Symposium Proceedings, “Everything about Gravity” celebrating the centenary of Einstein’s General Relativity, December 14-18 (2015), Taipei, arXiv:1602.01158.
  * [19] C. Li and J. Lucietti, Three-dimensional black holes and descendants, Physics Letters B 738, (2014), 48–54.
  * [20] P. Nurowski, Differential equations and conformal structures, J. Geom. Phys. 55, no. 1 (2005), 19–49.
  * [21] K. P. Tod, Einstein-Weyl spaces and third-order differential equations, J. Math. Phys. 41 (2000), 5572–5581.