Fisher zeroes and the fluctuations of the spectral form factor of chaotic
systems

Guy Bunin1, Laura Foini2*, Jorge Kurchan3

1 Department of Physics, Technion-Israel Institute of Technology, Haifa 32000,
Israel Department of Physics, Technion-Israel Institute of Technology, Haifa
32000, Israel

2 IPhT, CNRS, CEA, Université Paris Saclay, 91191 Gif-sur-Yvette, France

3 Laboratoire de Physique de l’École Normale Supérieure, ENS, Université PSL,
CNRS, Sorbonne Université, Université de Paris, F-75005 Paris, France

*<EMAIL_ADDRESS>
## Abstract

The spectral form factor of quantum chaotic systems has the familiar ‘ramp $+$
plateau’ form. Techniques to determine its form in the semiclassical [1] or
the thermodynamic [2] limit have been devised, in both cases based on the
average over an energy range or an ensemble of systems. For a single instance,
fluctuations are large, do not go away in the limit, and depend on the element
of the ensemble itself, thus seeming to question the whole procedure.
Considered as the modulus of a partition function in complex inverse
temperature $\beta_{R}+i\beta_{I}$ ($\beta_{I}\equiv\tau$ the time), the
spectral factor has regions of Fisher zeroes, the analogue of Yang-Lee zeroes
for the complex temperature plane. The large spikes in the spectral factor are
in fact a consequence of near-misses of the line parametrized by $\beta_{I}$
to these zeroes. The largest spikes are indeed extensive and extremely
sensitive to details, but we show that they are both exponentially rare and
exponentially thin. Motivated by this, and inspired by the work of Derrida on
the Random Energy Model, we study here a modified model of random energy
levels in which we introduce level repulsion. We also check that the mechanism
giving rise to spikes is the same in the SYK model.

###### Contents

  1. 1 Introduction
  2. 2 Fisher zeroes and near-missed zeros
  3. 3 The Random Energy Model
  4. 4 Model with level repulsion
  5. 5 Conclusions

##  1 Introduction

The spectral factor (SFF) of a quantum system is the energy level density two-
point correlation. It has been widely studied in Random Matrix Theory, and in
generic quantum chaotic systems [4, 5, 6, 3, 7, 8, 9, 10, 11, 12]. More
recently [13, 14, 15, 16] (see also [17]) it has been revisited in the context
of the Sachdev-Ye-Kitaev (SYK) model [18, 19], which is thought to provide a
‘toy’ version of holography and gravity. The SYK model has random
interactions, like a spin glass. Just as with spin glasses, some quantities
are self-averaging in the large size limit, their sample-to-sample
fluctuations vanishing with some inverse power of the size $N$. It may seem
then surprising that, when one plots the spectral factor of a single
realization over an exponentially long range of times, one finds that it
contains fluctuations (spikes) that do not vanish upon increasing the system’s
size. The spikes are extremely sensitive to a perturbation in the couplings,
and thus an averaged result seems to be blind to them. Moreover, rather
surprisingly, these fluctuations do not seem to contribute to the sample-
averaged result, in spite of them being large.

In this paper we show that (at least some of) such fluctuations are the
manifestation of the Fisher zeroes of the partition function in complex
temperature, each spike is an extremely rare near-miss of the
$\beta_{R}+i\beta_{I}$ line to one of these zeroes. This mechanism explains
why, even if spikes seem large, their averaged contribution to the spectral
factor is in fact exponentially small in $N$. The average density of Fisher
zeroes is readily available in an averaged, large $N$ computation, and gives
us direct access to the density of spikes. Note also that although sometimes
these spikes are attributed to the discreteness of the spectrum, regions with
finite densities of zeros are found in classical statistical models with
continuous degrees of freedom [20].

Some thirty years ago, Derrida [21] computed the partition function in complex
temperature and the distribution of Fisher zeroes for the simplest spin glass,
the Random Energy Model (REM) [22], a system of Poisson-distributed,
independent energy levels. The spectral factor turns out to have a ‘slope’,
and a ‘plateau’, but no ramp. The ‘plateau’ occurs in a region of the complex
temperature plane where there is a surface density of zeroes. In that phase,
random fluctuations destroy the delicate coherence needed for analytic
continuation to work. This is the interesting phase for our purposes here.

Our main task will be to make the analog of the REM but with level-repulsion,
just by transing the spectrum obtained from a random matrix to any desired
level density, or equivalently, of entropy $s(e)$. Such a model plays the role
of a ‘null model’, having just the bare minimum level structure compatible
with it being chaotic.

The structure of the manuscript is as follows: in Section 2 we discuss the
general properties of Fisher zeros and their effect on the SFF. In Section 3
we review the phase diagram of the REM in complex temperature. In Section 4 we
modify the REM to include level repulsion and we compute the SFF of this
model.

##  2 Fisher zeroes and near-missed zeros

The spectral factor of a Hamiltonian at $\beta=\beta_{R}+i\beta_{I}$ (here
$\beta_{I}\equiv\tau$, the time), is defined as $|Z(\beta)|^{2}$, where and
$Z(\beta)$ is the partition function at complex temperature $\beta$:

$Z(\beta)=\sum_{i}e^{-\beta E_{i}}$ (1)

$E_{i}=Ne_{i}$, we use lower case for intensive quantities. For finite number
$N$ of degrees of freedom, $Z$ is an analytic function of $\beta$, with
isolated zeroes. The set of zeroes in complex temperature, analogous to the
Yang-Lee zeroes in complex magnetic field, are referred to as Fisher zeroes
[23, 24, 25, 24]. As we go to the thermodynamic limit, these zeroes may
assemble into lines with a finite linear density, or areas of constant surface
density, or even fractal distributions [26, 27, 21, 28, 20]. The density of
zeroes is obtained directly from $\ln|Z|$ as:

$\Omega(\beta_{R},\beta_{I})=\frac{1}{2\pi}\left(\frac{\partial^{2}}{\partial\beta_{R}^{2}}+\frac{\partial^{2}}{\partial\beta_{I}^{2}}\right)\frac{1}{N}\ln|Z|$
(2)

In Eq. (2) we can interpret $\frac{1}{N}\ln|Z|$ as the electrostatic potential
in two dimensions generated by the distribution of charges
$\Omega(\beta_{R},\beta_{I})$.

It may seem paradoxical that $\frac{1}{N}\ln|Z|$ can have a smooth
thermodynamic limit, and at the same time have a density of infinitely deep
spikes in the zeroes of $Z$. The situation is in fact best understood by
analogy with a charged surface. We know that there is meaningful smooth
electric potential on the surface and outside, and yet we also know that near
each charge the potential has a spike. Somehow the coarse-grained limit
recovers the smoothness, and makes the spikes go away. This is possible
because the regions in the surface around charges where the discontinuity is
appreciable is small compared to the total surface.

Figure 1: Upper panel: logarithmic plot of $|Z(\beta_{R}+i\beta_{I})|^{2}$ vs
$\beta_{I}$ in the plateau regime for a model of random energy levels with
level repulsion and Gaussian density of states. Lower panel: zooming near a
spike, and identifying it as a near-miss to a zero by changing the value of
$\beta_{R}$ slightly Figure 2: The large-deviation function of spike depths,
$d$, along a line with constant $\beta_{R}=0.1$ and varying $\beta_{I}$ inside
the plateau, in a simulation of the model with level repulsion and $N=16$. The
dashed trend line is exponential, $\exp(-d)$.

In the models we shall discuss below, something similar happens for the
complex temperature plane. We shall show that one has regions with an $O(N)$
number of zeroes per element ${\rm d}\beta_{R}{\rm d}\beta_{I}$, and an
averaged, smooth extensive value for $\ln|Z(\beta)|$. There are, however,
small regions around each zero where $\ln|Z|$ has a spike. As an example, in
Figure 1 we choose a large spike, and locate explicitly the nearby pole
responsible for it.

Consider one such region and compute $\ln|Z(\beta_{R}+i\beta_{I})|$ along a
trajectory of fixed $\beta_{R}$ and a large range of $\beta_{I}$ traversing
it. The probability that the trajectory actually hits a zero is either zero or
becomes zero upon small perturbations of the Hamiltonian 111An exception to
this is when there is a symmetry (for example along the imaginary axis), then
one may expect a density of zeroes lying exactly on the invariant lines. See
[29].) . A near-miss along the line $\beta_{R}=const$ of a zero just beside
it, at $(\beta_{R}+e^{-aN},\tau_{0})$ gives222To see this properly one should
study the Weierstrass product development of $Z(\beta)$ in the vicinity of one
zero.

$|Z(\beta)|=\sqrt{e^{-2Na}+(\beta_{I}-\tau_{0})^{2}}\;e^{NG_{\infty}(\beta)}$
(3)

and has a spike of negative dip $=-a$ in the density $\log|Z(\beta)|/N$. The
largest spike in the factor along a trajectory that is exponentially long in
$\tau=\beta_{I}$ is estimated from the closest near miss. Let us say we look
at time-intervals of order $\Delta\beta_{I}=e^{bN}$ for spikes of magnitude
$\frac{1}{N}\ln|Z|\sim a$.

If the average surface density of zeroes along such a line is $\bar{\rho}$
(which may be even $1/N$), then the number of such spikes is:
${\cal{N}}(a)\sim\bar{\rho}e^{(b-a)N}$ and the large deviation function of
spike sizes is:

$\frac{1}{N}\log{\cal{N}}(a)\sim b-a+\frac{1}{N}\log\bar{\rho}$ (4)

The largest spike found in $\Delta\beta_{I}$ is thus of order $a\sim b$.
Figure 2 confirms this asymptotic law for a model with level repulsion we
shall discuss below.

Are the positions of spikes truly independent? This depends on the
correlations of the zeroes involved. We know that generically the zeroes of a
random polynomial repel one another [29], but they could be, for non-random
situations, some other correlations. On the other hand, asking for the closest
misses of a trajectory could force these to be far away, much in the same way
that the Grad limit of Boltzmann’s equation makes collisions independent [30].

If these spikes associated to near misses of zeroes are the only ones that
survive in the thermodynamic limit, then the fact that they are exponentially
rare in time $\tau\equiv\beta_{I}$ means that their averaged contribution to
the logarithm of the spectral form factor will be negligible to all orders in
$1/N$.

In Fig. 3 we show that the same mechanism is at play in the SYK model. In Fig.
3 we show a zoom of the SFF for the $q=4$ SYK which aims at highlighting the
presence of a near missed zero upon varying $\beta_{R}$.

Figure 3: Same as Fig. 1 for a fluctuation of $q=4$ SYK model. Here too, the
presence of a nearby Fisher zero is seen as the cause of the big spike. Figure
4: Complex temperature phase diagram of the (Poissonian) Random Energy Model.
Phase II, the spin glass phase, does not exist for models like SYK, and is
mostly irrelevant here.  Figure 5: A plot along a vertical line through the
gray region (phase III) of Fig. 4 of the Poissonian REM model. There is the
‘slope’ regime and the ‘plateau’ regime, but no ‘ramp”. The fluctuations in
the plateau are large. Here $N=14$ and $\beta_{R}=0.1$.

##  3 The Random Energy Model

The structure of the zeroes is very well understood in the case of the Random
Energy Model (REM) studied by Derrida [21] years ago. The REM is characterized
by $2^{N}$ independent energy levels drawn from a Gaussian distribution with
mean zero and variance $\langle E^{2}\rangle=\frac{N}{2}$. The average density
of energy levels is given by:

$\langle\rho(E)\rangle\propto e^{N\left[\log
2-\left(\frac{E}{N}\right)^{2}\right]}=e^{Ns(e)}$ (5)

(the normalization is such that $\int{\rm d}e\,\langle\rho(Ne)\rangle\simeq
2^{N}$) and this should be used to compute the partition function

$Z(\beta)\simeq\int{\rm d}e\,\rho(Ne)e^{-N\beta e}\ .$ (6)

in the regime $|e|=|\frac{E}{N}|\geq\sqrt{\log 2}=-e_{c}$. Beyond this value
there are no levels and the density in (6) should be taken as zero.

In the complex temperature plane we can recognize three different phases:

Analytic (liquid) phase I. The average of $Z$ itself is well-defined for large
$N$:

$\langle Z(\beta)\rangle\simeq\int de\;\langle\rho(Ne)\rangle e^{-N\beta
e}=\int_{-e_{c}}^{e_{c}}de\;e^{N\left[\ln 2-e^{2}-\beta e\right]}$ (7)

This may be evaluated for $|\beta|=O(1)$ by saddle point (the liquid phase),
$2E_{sp}=-N\beta$ and we get:

$\langle Z(\beta)\rangle\sim
2^{N}e^{N\frac{\beta^{2}}{4}}\quad\quad\frac{1}{N}\log|Z_{typ}(\beta)|\sim\log
2+\frac{\beta_{R}^{2}-\beta_{I}^{2}}{4}$ (8)

and $Z_{typ}$ describes the typical value of a random set of energy levels.

Analytic (glass) phase II. Here too the limit
$\lim_{N\rightarrow\infty}\frac{1}{N}\ln Z$ is well defined. However, at
sufficiently low (real) temperatures, there is no saddle point in a region
with positive density of levels, so we conclude that the system gets frozen
around the lowest energy density:

$\frac{1}{N}\log Z_{typ}(\beta)=\beta e_{c}$ (9)

at the critical temperature at which

$\beta_{c}=\left.\frac{{\rm d}s}{{\rm d}e}\right|_{e_{c}}$ (10)

Note that in a model with $\lim_{e\rightarrow 0}s^{\prime}(e)=\infty$, the
transition happens at zero temperature, as in SYK.

Although we have argued it for real $\beta$, there is a region in the complex
plane (to be determined) where the solution may be continued.

Non-analytic phase III. Here a new phenomenon appears. As we analytically
continue solutions to larger and larger $\beta_{I}$ the integrand becomes more
oscillating and its value becomes exponentially smaller due to precise
cancellations of semi-cycles of $e^{-i\beta_{I}E}$. These cancellations are
however imperfect, because the density of levels is in fact a random function.
For large enough $\beta_{I}$ the fluctuations of the integrand destroy the
coherence of integral, and fluctuations themselves dominate. To estimate this,
one has to compute [21] the fluctuating part of the partition function. This
can be inferred from correlation between energy levels:

$\begin{array}[]{l}\displaystyle\langle\delta\rho(E)\delta\rho(E^{\prime})\rangle=\langle\rho(E)\rho(E^{\prime})\rangle-\langle\rho(E)\rangle\langle\rho(E^{\prime})\rangle\\\
\vspace{-0.2cm}\\\
\displaystyle\qquad\qquad=\displaystyle\langle\rho(E)\rangle\delta(E-E^{\prime})\end{array}$
(11)

as befits a Poissonian distribution.

Putting $Ne_{+}=\frac{1}{2}(E+E^{\prime})$ and $Ne_{-}=E-E^{\prime}$ one has

$\begin{array}[]{l}\displaystyle\langle|Z(\beta)|^{2}\rangle=\int{\rm
d}e\;\;{\rm
d}e^{\prime}\;\;\langle\delta\rho(Ne^{\prime})\delta\rho(Ne)\rangle
e^{-\beta_{R}N(e+e^{\prime})-i\beta_{i}N(e-e^{\prime})}\\\ \vspace{-0.2cm}\\\
\displaystyle=\int_{-e_{c}}^{e_{c}}{\rm
d}e_{+}\;\;{\rho(Ne_{+})e^{-N2\beta_{R}e_{+}}\int{\rm
d}e_{-}\delta(Ne_{-})e^{-iN\beta_{i}e_{-}}}\end{array}$

The saddle point evaluation gives $e_{sp}=-\beta_{R}$ and

$\begin{array}[]{l}\displaystyle\langle|Z(\beta)|^{2}\rangle\sim e^{N\ln
2+N\beta_{R}^{2}}=\langle|Z(2\beta_{R})|\rangle\\\ \vspace{-0.2cm}\\\
\displaystyle\qquad\frac{1}{N}\log|Z(\beta)|_{typ}\sim\frac{\log
2}{2}+\frac{1}{2}\beta_{R}^{2}\end{array}$ (12)

We are thus left with the task of comparing three $|Z_{typ}|$, the transition
lines are obtained equating Eqs. (8) (9) and (12):

$\begin{array}[]{lll}\frac{1}{N}\log|Z_{I}|&=&\log
2+\frac{\beta_{R}^{2}-\beta_{I}^{2}}{4}\\\ \vspace{-0.2cm}\\\
\displaystyle\frac{1}{N}\log|Z_{II}|&=&\beta_{R}e_{c}\\\ \vspace{-0.2cm}\\\
\displaystyle\frac{1}{N}\log|Z_{III}|&=&\frac{\log
2}{2}+\frac{1}{2}\beta_{R}^{2}\end{array}$

The phase diagram obtained in Ref [21] is shown in Figure 4.

The density of zeroes, normalized with $N$ is zero in phases I and II, as is
to be expected, since the partition function is analytic there. In phase III
on the contrary we get:

$\Omega(\beta_{R},\beta_{I})=\frac{1}{2\pi}$ (13)

i.e. we have a constant surface density of zeroes. There are also linear
densities in the boundaries I-II and I-III. In Figure 5 we show a plot of
$\ln|Z|$ along a line of constant $\beta_{R}$: there is a ‘slope’ region,
corresponding to phase I, and a ‘plateau’ region (phase III) with negative
spikes associated with near-misses of zeroes.

### Comments on annealed and quenched averages

In the three cases above, our task is to compute the large $N$ limit of

(a) $\frac{1}{N}\langle\log Z(\beta)\rangle$ if the large $N$ limit exists

(b) $\frac{1}{N}\langle\log|Z(\beta)|\rangle=\lim_{n\rightarrow
0}\frac{1}{2}\frac{d}{dn}\langle|Z(\beta)|^{2n}\rangle$

These are quenched averages. Sometimes, it happens that quenched and annealed
averages to leading order coincide $\langle Z^{n}\rangle=\langle Z\rangle^{n}$
in the large $N$ limit. This means that replicas $1,...,n$ are uncoupled, but
in our case it also means that we may neglect the fluctuations of level
densities: only average densities matter. It may be also true that, again, to
leading order $\langle ZZ^{\ast}\rangle=\langle Z\rangle\;\langle
Z^{\ast}\rangle$: this is the situation in phase I.

In phase II the situation is the same, provided we are content with the
evaluation of $\log Z$ or the energy density. The latter is frozen at the
lowest value $-e_{c}$. If we wish instead to get into the details of the
actual distribution of overlaps, we need to understand how the measure is
split between the infinity of levels having energy density $-e_{c}+O(1/N)$, in
particular the two lowest. This information, which we shall not discuss here,
needs [22] that we consider the quenched average, break replica symmetry, and
for this use the correlation between levels.

In phase III the situation is hybrid: the limit $\frac{1}{N}\langle\log
Z(\beta)\rangle$ simply does not exist. On the other hand,
$\frac{1}{N}\langle\log|Z(\beta)|\rangle$ does, and furthermore
$\frac{1}{N}\langle|Z(\beta)|^{2n}\rangle=\frac{1}{N}\langle|Z(\beta)|^{2}\rangle^{n}$.
Here the replicas corresponding to $Z$ and $Z^{*}$ are coupled, but different
pairs may or may not be, in this case they are not. At any rate, the
correlation of level densities enters into the solution.

Figure 6: Random energy model zeroes without (left) and with (right) level-
repulsion. Dark areas show the locations of zeros in a numerical calculation
with $N=14$. The position of zeros was obtained by partitioning the complex
$(\beta_{R},\beta_{I})$ plane into small rectangles, of a size small compared
to the typical distance between zeros at the simulated $N$. A zero inside a
rectangle was identified by the residue theorem, by integrating over $1/Z$
(equivalently, checking if the complex angle of $\log(Z)$ changed when
traversing the boundary). Figure 7: Spectral form factor of the model with a
Gaussian distribution of levels like the REM, but with level repulsion.

##  4 Model with level repulsion

In Figure 6 we show the empirical density of zeroes obtained in simulations
with the REM (left), and those of the same density but with level repulsion
added (right). Let us discuss this case in detail.

We start with a set set of $2^{N}$ levels $u_{i}$, with level repulsion,
distributed uniformly over $(-1,1)$. This is done by first generating a sample
from a spectrum of a GOE random matrix (sampled efficiently as described in
[31]). A function $\lambda\rightarrow g(\lambda)$ is applied to the
eigenvalues, chosen to obtain a uniform density. We then transform again, to
obtain any desired energy density.

$E_{i}=\Phi^{-1}(u_{i})\qquad;\qquad u_{i}=\Phi(E_{i})$ (14)

Two features concerning the density of levels
$\langle\rho(Ne)\rangle=e^{Ns(e)}$ of the new model are important:

  * •

Zero-temperature entropy finite or zero: $\lim_{T\rightarrow
0}\lim_{N\rightarrow\infty}s(e)=s_{o}$

  * •

Freezing transition temperature: $\frac{1}{T_{c}}=\beta_{c}=\lim_{e\rightarrow
e_{c}}\lim_{N\rightarrow\infty}\frac{\partial s}{\partial e}$ finite or
infinite.

If we wish to mimic the SYK model we need $\beta_{c}=\infty$ and $s_{o}>0$,
while a ‘quantum REM’ (and other spin-glasses) would have $\beta_{c}$ finite
and $s_{o}=0$. For the latter case, $\Phi$ is the error function, so we have a
REM level density with level repulsion.

$\begin{array}[]{l}\displaystyle\langle\tilde{\rho}(E)\rangle{\rm
d}E={\mbox{uniform}}\times\frac{{\rm d}u}{{\rm d}E}{\rm
d}E={\cal{N}}e^{-\frac{E^{2}}{N}}\\\ \vspace{-0.2cm}\\\
\displaystyle\qquad\qquad\frac{{\rm d}\Phi}{{\rm
d}E}=\langle\tilde{\rho}(E)\rangle\end{array}$ (15)

We define $\rho(E)=2^{N}\tilde{\rho}(E)$ normalized as before, $\int{\rm
d}\langle\rho(E)\rangle=2^{N}$. The effect of the level repulsion in manifest
in the connected correlation between energy levels:

$\begin{array}[]{l}\displaystyle\langle\delta\rho(u)\delta\rho(u^{\prime})\rangle=\langle\rho(u)\rho(u^{\prime})\rangle-\langle\rho(u)\rangle\langle\rho(u^{\prime})\rangle\\\
\vspace{-0.2cm}\\\
\displaystyle\qquad\qquad=2^{2N}\left\\{R\left(2^{N}(u-u^{\prime})\right)-1\right\\}\end{array}$
(16)

with

$R(s)-1=\delta(s)-\frac{\sin^{2}(\pi s)}{(\pi s)^{2}}$ (17)

Here and in what follows for the purposes of analytic computations we will use
the GUE statistics, which is slightly simpler and does not change
significantly the discussion.

We shall need the Fourier version:

$\int{\rm d}s\;e^{ius}\;[R(s)-1]=g(u)$ (18)

and

$\displaystyle g(u)=\begin{cases}|u|&\text{for $|u|<2\pi$}\\\ &\\\ 1&\text{for
$|u|>2\pi$}\end{cases}$ (19)

We now make the non-linear transformation (14). The density of levels
transform in the following way:

$\rho(u)=\sum_{i}\delta(u-u_{i})=2^{N}\langle\rho(E)\rangle^{-1}\rho(E)$ (20)

and the expression for the correlations becomes:

$\begin{array}[]{l}\displaystyle\langle\delta\rho(E)\delta\rho(E^{\prime})\rangle=\displaystyle\langle\rho(E)\rho(E^{\prime})\rangle-\langle\rho(E)\rangle\langle\rho(E^{\prime})\rangle\\\
\vspace{-0.2cm}\\\
\qquad\displaystyle=\displaystyle\langle\rho(E)\rangle\langle\rho(E^{\prime})\rangle\left[R(2^{N}(\Phi(E)-\Phi(E^{\prime}))-1\right]\end{array}$

We can evaluate the contribution to the SFF which comes from connected
correlations. The computation is very close to the one in [2].

$\begin{array}[]{l}\displaystyle\langle|Z(\beta)|^{2}\rangle=\int{\rm d}w{\rm
d}e{\rm d}e^{\prime}g(w)\;\langle\rho(Ne)\rangle\\\ \vspace{-0.2cm}\\\
\displaystyle e^{iw2^{N}\Phi\left(Ne\right)-\beta
Ne}\langle\rho(Ne^{\prime})\rangle\;e^{-iw2^{N}\Phi\left(Ne^{\prime}\right)-\beta^{\ast}Ne^{\prime}}\\\
\vspace{-0.2cm}\\\ \displaystyle\qquad=\int{\rm d}w{\rm d}e{\rm
d}e^{\prime}g(w)\;\langle\rho(Ne)\rangle\langle\rho(Ne^{\prime})\rangle\\\
\vspace{-0.2cm}\\\ \displaystyle
e^{iw2^{N}(\Phi\left(Ne\right)-\Phi\left(Ne^{\prime}\right))-\beta_{R}N(e+e^{\prime})-i\beta_{I}N(e-e^{\prime})}\end{array}$
(21)

We define $e_{+}=\frac{1}{2}(e+e^{\prime})$ and $e_{-}=(e-e^{\prime})$ and we
make the following expansion:

$2^{N}(\Phi(E)-\Phi(E^{\prime}))=\frac{N}{2}2^{N}\partial_{E}\Phi(E)|_{E=E_{+}=Ne_{+}}(e-e^{\prime})$
(22)

Considering first the exponentially large terms, we have

$\begin{array}[]{l}\displaystyle\int{\rm
d}e_{-}\,e^{iN\left(\frac{1}{2}w2^{N}\partial_{E}\Phi(E)|_{E=E_{+}=Ne_{+}}-\beta_{I}\right)e_{-}}\\\
\vspace{-0.2cm}\\\
\qquad\qquad\rightarrow\;\delta\left(\frac{w}{2}\langle\rho(Ne_{+})\rangle-\beta_{I}\right)\end{array}$
(23)

where we used that
$2^{N}\partial_{E}\Phi(E)|_{E=E_{+}=Ne_{+}}=\langle\rho(Ne_{+})\rangle$.
Plugging this in Eq. (21) we have:

$\begin{array}[]{l}\displaystyle\displaystyle\langle|Z(\beta)|^{2}\rangle=\\\
\vspace{-0.2cm}\\\ \displaystyle\displaystyle=\int{\rm d}w{\rm
d}e_{+}g(w)\delta\left(\frac{w}{2}\langle\rho(Ne_{+})\rangle-\beta_{I}\right)\;\langle\rho(Ne_{+})\rangle^{2}e^{-2\beta_{R}Ne_{+}}\\\
\vspace{-0.2cm}\\\ \displaystyle=\int{\rm
d}e_{+}g\left(\frac{2\beta_{I}}{\langle\rho(Ne_{+})\rangle}\right)\;\langle\rho(Ne_{+})\rangle
e^{-2\beta_{R}Ne_{+}}\end{array}$ (24)

This integral represents an effective problem with density of states
$g\left(\frac{2\beta_{I}}{\langle\rho(Ne_{+})\rangle}\right)\;\langle\rho(Ne_{+})\rangle$
at inverse temperature $2\beta_{R}$. The situation is represented in Figures 8
and 9. Below we discuss the result of the integral for different values of
$\beta=\beta_{R}+i\beta_{I}$.

### Glassy phase

This phase corresponds to phase II of the Poissonian REM, and does not exist
in a model like SYK which has a transition only at infinite $\beta_{R}$. Above
$\beta_{R}>\sqrt{\log 2}$ (and for sufficiently large $\beta_{I}$) for the
modified REM the solution of the plateau reads:

$\begin{array}[]{l}\displaystyle\frac{1}{N}\log|Z(\beta)|=\beta_{R}e_{c}\end{array}$
(25)

This is the same free energy as in the glassy phase.

### Liquid phase (‘the slope’)

There is a liquid phase which is the same as phase I of the Poissonian REM,
since this depends only on the level density and not the correlations. It
corresponds to the ‘slope’ part of the SFF and its free energy reads:

$f_{I}(\beta_{R},\beta_{I})=\lim_{N\to\infty}\frac{1}{N}\log|Z|=\log
2+\frac{1}{4}(\beta_{R}^{2}-\beta_{I}^{2})$ (26)

### The regions with zeroes

Just as in the case of the REM, we need to compare the analytic continuations
to the ‘incoherent’ part, by calculating $\langle|Z(\beta)|^{2}\rangle$. There
are several possibilities:

#### (i) Plateau regime

If $\beta_{I}>\pi\langle\rho(E=0)\rangle$, above the bell shape in Figs 8 and
9, then the entire integral is with $w$ in the plateau value, we have in fact
$g(w)=1$ for all energies and the integral becomes

$\displaystyle\langle|Z(\beta)|^{2}\rangle\displaystyle=\int{\rm
d}e_{+}\,\langle\rho(e_{+})\rangle\,e^{-2\beta_{R}Ne_{+}}=Z(2\beta_{R})$ (27)

The value of $e_{+}$ that dominates is $e(2\beta_{R})$ . This is the same
result as in the REM, and (for sufficiently low $\beta_{R}$) we have a finite
density of zeros. It is easy to convince oneself that in a plateau regime all
models should have Fisher zeros, unless the dependence on $\beta_{R}$ is
linear.

#### (ii) Ramp I regime

Here $\beta_{I}$ is such that the cycle of $e^{i\beta_{I}E}$ is longer than
the level spacing at the edge of the spectrum - the largest of all level
spacings - $\beta_{I}<\pi\langle\rho(e_{c})\rangle$ 333This condition is quite
different if there is non-zero entropy at zero temperature or not. The largest
level spacing in the REM is of order one in energy, while for SYK it is much
smaller. In this regime we have $g(w)=w$ for all energies and the integral
becomes

$\displaystyle\langle|Z|^{2}\rangle\displaystyle=\beta_{I}\int_{-e_{c}}^{e_{c}}{\rm
d}e_{+}\,e^{-2\beta_{R}Ne_{+}}=\beta_{I}\frac{1}{N\beta_{R}}\sinh(2N\beta_{R}e_{c})$
(28)
$\begin{array}[]{l}\displaystyle\frac{1}{N}\ln|Z(\beta_{R},\beta_{I})|\displaystyle=\frac{1}{2N}\ln\beta_{I}+\frac{1}{2N}\ln|\sinh(2N\beta_{R}e_{c})|\\\
\vspace{-0.2cm}\\\
\displaystyle\qquad\qquad-\frac{1}{2N}\ln|\beta_{R}|-\frac{1}{2N}\ln
N\end{array}$ (29)

#### (iii) Ramp II regime

The transition between the ramp and the plateau occurs in the intermediate
regime where
$\pi\langle\rho(E_{c})\rangle<\beta_{I}<\pi\langle\rho(E=0)\rangle$ (see Figs
8 and 9).

We must integrate two regimes of $w$: i) $\beta_{I}<\pi\langle\rho(E)\rangle$
then $|w|<2\pi$ and ii) $\beta_{I}>\pi\langle\rho(E)\rangle$ and then
$|w|>2\pi$.

$\begin{array}[]{l}\displaystyle\int_{-e_{c}}^{-e^{\ast}}\;{\rm
d}e_{+}\;\;\langle\rho(e_{+})\rangle\,e^{-2\beta_{R}Ne_{+}}+\int_{-e^{\ast}}^{e^{\ast}}\;{\rm
d}e_{+}\;w\;\langle\rho(e_{+})\rangle\,e^{-2\beta_{R}Ne_{+}}\\\
\vspace{-0.2cm}\\\ \displaystyle\qquad\qquad+\int_{e^{\ast}}^{e_{c}}\;{\rm
d}e_{+}\;\langle\rho(e_{+})\rangle\,e^{-2\beta_{R}Ne_{+}}\end{array}$ (30)

where $\pi\langle\rho(\pm e^{\ast})\rangle=\beta_{I}$ are the $w=2\pi$
intercepts. This gives:

$\begin{array}[]{l}\displaystyle\int_{-e_{c}}^{-e^{\ast}}\;{\rm
d}e\;\;\,\langle\rho(e)\rangle
e^{-2\beta_{R}Ne_{+}}+\int_{e^{\ast}}^{e_{c}}\;{\rm
d}e\,\langle\rho(e)\rangle\;e^{-2\beta_{R}Ne_{+}}\\\ \vspace{-0.2cm}\\\
\displaystyle\qquad\qquad+\beta_{I}\int_{-e^{\ast}}^{e^{\ast}}\;{\rm
d}e\;\,e^{-2\beta_{R}Ne_{+}}\end{array}$ (31)

Eq (31) may be understood better by rewriting it as:

$\displaystyle\int_{-e_{c}}^{e_{c}}\;{\rm
d}e\,e^{N{\cal{s}}(e)}\;e^{-2\beta_{R}Ne}$ (32)

where

${\cal{s}}(e)=\left\\{\begin{array}[]{l
l}\frac{1}{N}\ln\rho(e)&\qquad\pi\rho(e)<\beta_{I}\\\
\frac{1}{N}\ln\beta_{I}&\qquad\pi\rho(e)>\beta_{I}\\\ \end{array}\right..$
(33)

For every value of $\beta_{I}$ the saddle point evaluation may be viewed in
Figs 8 and 9.

  * •

for $2\beta_{R}>\beta_{R}^{max}$ the integral is dominated by the lowest
energy $e_{+}=-e_{c}$.

  * •

for $\beta_{R}^{max}>2\beta_{R}>\beta_{R}^{min}$ the integral is dominated by
the saddle $e_{+}=e(2\beta_{R})$, that is the same as it would be if there
would not be a truncation, i.e. for large $\beta_{I}$. This regime corresponds
to the plateau.

  * •

For $2\beta_{R}<\beta_{R}^{min}$ the integral is frozen around the
intersection with the curve $w=2\pi$. Its value describes the ramp and it is
given by the following contributions:

$\begin{array}[]{l}\displaystyle\langle|Z(\beta)|^{2}\rangle\simeq
I_{1}+I_{2}\simeq\beta_{I}\int de\\\ \vspace{-0.2cm}\\\
\displaystyle\underbrace{\left[\theta(-e^{*}-e)e^{N\beta_{min}(e+e^{\ast})}+\theta(e+e^{*})\right]e^{-N2\beta_{R}e}}_{e^{\mu(e)}}\end{array}$
(34)

where $\theta(x)$ is the step function, so that:

$\begin{array}[]{l}\displaystyle I_{1}=\int_{-e_{c}}^{-e^{\ast}}\;{\rm
d}e_{+}\;\,\langle\rho(Ne_{+})\rangle e^{-2\beta_{R}Ne_{+}}\displaystyle\\\
\vspace{-0.2cm}\\\ \displaystyle\simeq
e^{Ns(-e^{\ast})+2\beta_{R}Ne^{\ast}}\int_{-e_{c}}^{-e^{\ast}}{\rm
d}e\,e^{N\left(\beta_{R}^{min}(e+e^{\ast})-2\beta_{R}(e+e^{\ast})\right)}\\\
\vspace{-0.2cm}\\\
\displaystyle\qquad\qquad\simeq\beta_{I}e^{2\beta_{R}Ne^{\ast}}\frac{1}{N}\frac{1}{\beta_{R}^{\min}-2\beta_{R}}\end{array}$
(35)

where we used that $e^{Ns(-e^{\ast})}=\beta_{I}$ and

$\begin{array}[]{l}\displaystyle
I_{2}=\beta_{I}\int_{-e^{\ast}}^{e_{\ast}}\;{\rm
d}e_{+}\;\,e^{-2\beta_{R}Ne_{+}}\displaystyle\\\ \vspace{-0.2cm}\\\
\displaystyle=\beta_{I}\frac{1}{2\beta_{R}N}\sinh(2\beta_{R}Ne^{\ast})\simeq\beta_{I}\frac{1}{N}\frac{1}{4\beta_{R}}e^{2\beta_{R}Ne^{\ast}}\end{array}$
(36)

Note that $\beta_{R}^{\min}$ is a function of $\beta_{I}$. Altogether this
regime describes the transition between the ramp and the plateau because
increasing $\beta_{I}$ the singularity of the effective entropy ${\cal{s}}(e)$
moves at higher energies (and thus lower $\beta_{R}^{min}$) and one crosses
over from the situation where the saddle point is frozen at the singularity
(ramp) to the one in which it occurs at the energy $e(2\beta_{R})$ (plateau),
see Fig. 9.

The regime in which Eq. (35) holds is for $2\beta_{R}<\beta_{R}^{min}$. The
point where $2\beta_{R}=\beta_{R}^{min}$ where there is an apparent divergence
should be treated separately and represents the trasition point between ramp
and plateau.

### Transition

The transition between these regimes is obtained by comparing the magnitude of
free energies. Since $e^{*}\sim e_{c}$ in this regime, we may estimate:

$\lim_{N\to\infty}\frac{1}{N}\log|Z|=\beta_{R}e_{c}\,$ (37)

namely the same free energy as in the glassy phase. This should be therefore
compared with $f_{I}$:

$\beta_{I}=\sqrt{\beta_{R}^{2}-4\beta_{R}\sqrt{\log 2}+4\log 2}=2\sqrt{\log
2}-\beta_{R}$ (38)

where we used $e_{c}=\sqrt{\log 2}$. This is the line we see in Figure 6
(left) that crosses the real axes at $\beta_{c}=2\sqrt{\log 2}$, the critical
temperature of the REM.

### Density of zeroes

In the plateau regime, $2\ln|Z(\beta)|$ is the same as one would obtain for
real temperature $2\beta_{R}$.

The density of zeroes is then:

$\begin{array}[]{ll}\displaystyle\Omega(\beta_{R},\beta_{I})&\displaystyle=\displaystyle\frac{1}{4\pi
N}\frac{\partial^{2}\ln Z(2\beta_{R})}{\partial\beta_{R}^{2}}\\\
\vspace{-0.2cm}\\\
\displaystyle&\displaystyle=\displaystyle\frac{N}{\pi}\left[\langle
e^{2}\rangle_{2\beta_{R}}-\langle
e\rangle_{2\beta_{R}}^{2}\right]=O(1)>0\end{array}$ (39)

where averages are over a real Gibbs measure.

In the Ramp I regime the density of zeros is:

$\begin{array}[]{l}\displaystyle\Omega\propto\nabla^{2}\frac{1}{N}\log|Z|=\frac{1}{2N}\left(\frac{1}{\beta_{R}^{2}}-\frac{1}{\beta_{I}^{2}}\right)-\frac{2e_{c}^{2}N}{\sinh^{2}(2N\beta_{R}e_{c})}\\\
\vspace{-0.2cm}\\\
\displaystyle=2e_{c}\times\underbrace{Ne_{c}\left\\{\left(\frac{1}{(2Ne_{c}\beta_{R})^{2}}\right)-\frac{1}{\sinh^{2}(2N\beta_{R}e_{c})}\right\\}}-\frac{1}{2N\beta_{I}^{2}}\end{array}$
(40)

the underlined term has integral $O(1)$ and ‘fat’ tails
$\propto\frac{1}{N\beta_{R}^{2}}$.

In the Ramp II regime we need to expand around the intersection point of the
$S(-e^{\ast})=\ln\beta_{I}$. This leads to:

$\begin{array}[]{l}\displaystyle\Omega(\beta_{R},\beta_{I})\displaystyle=\displaystyle\frac{1}{2\pi
N}\left[\frac{\partial^{2}\ln|Z|}{\partial\beta_{R}^{2}}+\frac{\partial^{2}\ln|Z|}{\partial\beta_{I}^{2}}\right]\\\
\vspace{-0.2cm}\\\ \displaystyle=\displaystyle\frac{1}{2\pi}\left[4N(\langle
e^{2}\rangle_{\mu}-\langle
e\rangle_{\mu}^{2})+\left\langle\frac{\partial^{2}\mu}{\partial\beta_{I}^{2}}+\left(\frac{\partial\mu}{\partial\beta_{I}}\right)^{2}\right\rangle_{\mu}\right.\\\
\vspace{-0.2cm}\\\
\displaystyle\qquad\qquad\qquad\qquad\left.-\left\langle\frac{\partial\mu}{\partial\beta_{I}}\right\rangle_{\mu}^{2}-\frac{1}{N}\frac{1}{\beta_{I}^{2}}\right]\\\
\vspace{-0.2cm}\\\
\qquad\qquad\displaystyle\sim\displaystyle\frac{2N}{\pi}(\langle
e^{2}\rangle_{\mu}-\langle e\rangle_{\mu}^{2})\sim
O\left(\frac{1}{N}\right)\end{array}$ (41)

where $\langle\bullet\rangle_{\mu}=\frac{\int{\rm
d}e\;\bullet\;e^{\mu(e)}}{\int{\rm d}e\;e^{\mu(e)}}$, and we have neglected
the derivatives with respect to $\beta_{I}$ because this regime concerns
exponentially large $\beta_{I}$.

Note that the ramp has a smaller density of zeroes than the plateau (cf also
Fig 6). This however affects only logarithmically the large deviation function
for the largest spike.

Figure 8: Equation (31) may be seen as the partition function associated to an
entropy with this shape: a truncated density level, with truncation of height
$\ln\beta_{I}$.
$\bullet$ If $\ln\beta_{I}<S(e\rightarrow 0)$ the saddle point equation
determines two temperatures: $\beta_{R}^{max}$ where the saddle freezes in the
lower energy (and finite entropy, if there is one), and $\beta_{R}^{min}$ (a
function of $\beta_{I}$) where the saddle freezes on the intersection of the
bell shape with the flat part. In the latter, the value of the integral is
proportional to $\beta_{I}$. The value $\beta_{c}$ may be infinite, as is in
SYK.  Figure 9: The transition from ramp to plateau upon increasing the level
of $\ln\beta_{I}$, indicated with a dashed line. Given a value of $\beta_{R}$,
we have the saddle point where the gradient is $2\beta_{R}$. (a) If
$\ln\beta_{I}$ is above this value, then the saddle dominates and the result
is independent of $\beta_{I}$ : this is the plateau. (b) If it is below this
value, the integral is dominated by the dashed line level : this is the ramp.
(c) For lower values of $\ln\beta_{I}$, the lowest energy density states
dominate.

##  5 Conclusions

The problem of the spectral form factor is one of computing a free energy in
the complex temperature plane, with the very important caveat that, unlike the
situation in thermodynamics, one goes to complex inverse temperatures
$|\beta|$ which do not remain constant in the thermodynamic limit. The
computation of fluctuations of the free energy sampled over ranges of
$|\beta|$ that are exponential in the size $N$ is thus a problem of Large
Deviation theory. We have given an estimate of the size of the largest
deviations (spikes) assuming that the system may be considered for these
purposes to be featureless, i.e. to have level correlations given only by the
standard level repulsion. That this assumption is valid for all large
fluctuations of SYK is plausible, but not given.

The ‘gravity’ side of SYK would give us the probability density of large
spikes of a single sample at a given time through the density of Fisher zeroes
obtained through the Laplacian in temperature, but the actual position would
require exponential precision in $N$. Here we have only discussed the largest
spikes. It seems possible that a more detailed distribution of fluctuations
may be obtained directly from the theory of random polynomials.

## Acknowledgements

This work was supported by ‘Investissements d’Avenir’ LabEx PALM
(ANR-10-LABX-0039-PALM) (EquiDystant project, L. Foini) and Simons Foundation
Grant No 454943 (J. Kurchan).

## References

  * [1] O. Agam, B. L. Altshuler and A. V. Andreev,  _Spectral statistics: From disordered to chaotic systems_ ,  Phys. Rev. Lett. 75, 4389 (1995),  10.1103/PhysRevLett.75.4389.
  * [2] J. S. Cotler, G. Gur-Ari, M. Hanada, J. Polchinski, P. Saad, S. H. Shenker, D. Stanford, A. Streicher and M. Tezuka,  _Black holes and random matrices_ ,  Journal of High Energy Physics 2017(5), 1 (2017).
  * [3] R. Prange,  _The spectral form factor is not self-averaging_ ,  Physical Review Letters 78(12), 2280 (1997).
  * [4] M. L. Mehta,  _Random matrices_ ,  Elsevier (2004).
  * [5] V. Kravtsov,  _Random matrix theory: Wigner-dyson statistics and beyond.(Lecture notes of a course given at SISSA (Trieste, Italy))_ ,  arXiv:0911.0639 (2009).
  * [6] E. Brézin and S. Hikami,  _Spectral form factor in a random matrix theory_ ,  Physical Review E 55(4), 4067 (1997).
  * [7] S. Müller, S. Heusler, P. Braun, F. Haake and A. Altland,  _Semiclassical foundation of universality in quantum chaos_ ,  Physical Review Letters 93(1), 014103 (2004).
  * [8] A. Chan, A. De Luca and J. T. Chalker,  _Solution of a minimal model for many-body quantum chaos_ ,  Physical Review X 8(4), 041019 (2018).
  * [9] A. Chan, A. De Luca and J. Chalker,  _Spectral statistics in spatially extended chaotic quantum many-body systems_ ,  Physical Review Letters 121(6), 060601 (2018).
  * [10] B. Bertini, P. Kos and T. Prosen,  _Exact spectral form factor in a minimal model of many-body quantum chaos_ ,  Physical Review Letters 121(26), 264101 (2018).
  * [11] P. Kos, M. Ljubotina and T. Prosen,  _Many-body quantum chaos: Analytic connection to random matrix theory_ ,  Physical Review X 8(2), 021062 (2018).
  * [12] S. Moudgalya, A. Prem, D. A. Huse and A. Chan,  _Spectral statistics in constrained many-body quantum chaotic systems_ ,  Physical Review Research 3(2), 023176 (2021).
  * [13] P. Saad, S. H. Shenker and D. Stanford,  _A semiclassical ramp in SYK and in gravity_ ,  arXiv preprint arXiv:1806.06840 (2018).
  * [14] A. Altland and D. Bagrets,  _Quantum ergodicity in the syk model_ ,  Nuclear Physics B 930, 45 (2018).
  * [15] A. Altland, D. Bagrets, P. Nayak, J. Sonner and M. Vielma,  _From operator statistics to wormholes_ ,  Physical Review Research 3(3), 033259 (2021).
  * [16] A. Altland and J. Sonner,  _Late time physics of holographic quantum chaos_ ,  SciPost Physics 11(2), 034 (2021).
  * [17] Y. Chen,  _Spectral form factor for free large $n$ gauge theory and strings_,  arXiv preprint arXiv:2202.04741 (2022).
  * [18] S. Sachdev and J. Ye,  _Gapless spin-fluid ground state in a random quantum Heisenberg magnet_ ,  Physical Review Letters 70, 3339 (1993),  10.1103/PhysRevLett.70.3339,  cond-mat/9212030.
  * [19] J. Maldacena and D. Stanford,  _Remarks on the sachdev-ye-kitaev model_ ,  Physical Review D 94, 106002 (2016),  10.1103/PhysRevD.94.106002,  1604.07818.
  * [20] T. Obuchi and K. Takahashi,  _Partition-function zeros of spherical spin glasses and their relevance to chaos_ ,  Journal of Physics A: Mathematical and Theoretical 45(12), 125003 (2012).
  * [21] B. Derrida,  _The zeroes of the partition function of the random energy model_ ,  Physica A: Statistical Mechanics and its Applications 177(1-3), 31 (1991).
  * [22] B. Derrida,  _Random-energy model: An exactly solvable model of disordered systems_ ,  Physical Review B 24(5), 2613 (1981).
  * [23] C.-N. Yang and T.-D. Lee,  _Statistical theory of equations of state and phase transitions. i. theory of condensation_ ,  Physical Review 87(3), 404 (1952).
  * [24] M. Fisher,  _The nature of critical points (lectures in theoretical physics vol 7c)_ (1965).
  * [25] M. E. Fisher,  _Yang-lee edge singularity and $\phi$ 3 field theory_,  Physical Review Letters 40(25), 1610 (1978).
  * [26] S. Katsura, Y. Abe and K. Ohkohchi,  _Distribution of zeros of the partition function for the slater model of ferroelectricity_ ,  Journal of the Physical Society of Japan 29(4), 845 (1970).
  * [27] H. Brascamp and H. Kunz,  _Zeroes of the partition function for the ising model in the complex temperature plane_ ,  Journal of Mathematical Physics 15(1), 65 (1974).
  * [28] C. Moukarzel and N. Parga,  _Numerical complex zeros of the random energy model_ ,  Physica A: Statistical Mechanics and its Applications 177(1-3), 24 (1991).
  * [29] E. Bogomolny, O. Bohigas and P. Leboeuf,  _Quantum chaotic dynamics and random polynomials_ ,  Journal of Statistical Physics 85(5), 639 (1996).
  * [30] C. Cercignani, V. Gerasimenko and D. Y. Petrina,  _Many-particle dynamics and kinetic equations_ , vol. 420,  Springer Science & Business Media (1997).
  * [31] I. Dumitriu and A. Edelman,  _Matrix models for beta ensembles_ ,  Journal of Mathematical Physics 43(11), 5830 (2002).


# Detecting Cloud Presence in Satellite Images
Using the RGB-based CLIP Vision-Language Model

###### Abstract

This work explores capabilities of the pre-trained CLIP vision-language model
to identify satellite images affected by clouds. Several approaches to using
the model to perform cloud presence detection are proposed and evaluated,
including a purely zero-shot operation with text prompts and several fine-
tuning approaches. Furthermore, the transferability of the methods across
different datasets and sensor types (Sentinel-2 and Landsat-8) is tested. The
results that CLIP can achieve non-trivial performance on the cloud presence
detection task with apparent capability to generalise across sensing
modalities and sensing bands. It is also found that a low-cost fine-tuning
stage leads to a strong increase in true negative rate. The results
demonstrate that the representations learned by the CLIP model can be useful
for satellite image processing tasks involving clouds.

###### Index Terms:

Cloud Processing, Zero-Shot Learning, Classification

##  I Introduction

The text medium has long begun to play a prominent role in the processing of
visual data over the last years, such as images [1], or videos [2]. The use of
language allows human users to easily adapt the computer vision models to
their needs, which has prominently been used for purely creative purposes [3,
4, 5], but also for zero-shot classification [1]. Vision-language foundations
models could pave the way for many remote sensing applications that can be
defined upon inference, without the need for extensive training or any
training at all. This has been looked into in some works on visual question
answering [6, 7, 8], but the application to cloud presence detection remains
unexplored.

At the core of many text-based vision solutions stands CLIP, a vision-language
model designed for measuring alignment between text and image inputs [1]. In
this work, the capability of the CLIP model to recognize cloud-affected
satellite images is investigated. The CLIP model operates on RGB images, while
a typical solution to detect clouds in satellite imagery involves more than
the RGB visible bands, such as infrared, and is often sensor-specific. Some
approaches have explored the potential of an RGB-only cloud detection model
[9], but the task is considered significantly more challenging. Furthermore,
the CLIP model has been trained on the general WebImageText dataset [1], so it
is not immediately clear how well it could perform with a task as specific as
classification of cloud-affected satellite imagery.

|
---|---
1\. Text Prompts | 2\. Linear Probe
|
3\. CoOp [10] | 4\. Radar
Fig. 1: Diagrams of the proposed cloud presence detection methods based on the
CLIP model

In this work, the capability of the original pre-trained CLIP model (ViT-B/32
backbone) is tested. There are two important insights gained here: it allows
to estimate the utility of representations learned by CLIP for cloud-oriented
tasks (which can potentially lead to more complex uses such as segmentation or
removal), and further, it can act as a tool for filtering datasets based on
the presence of clouds.

##  II Method

The CLIP model [1] has been designed for zero-shot classification of images
where labels can be supplied (and hence, specified) as text upon inference.
The CLIP model consists of separate encoders for text (transformer-based [11])
and images (either a Vision Transformer [12] or a ResNet [13]) input, with
jointly learned embedding space. A relative measure of alignment between a
given text-image pair can be obtained by computing the cosine similarity
between the encodings. This way, pair-wise similarity between any collection
of images and text can be computed and compared, which enables use cases such
as classification by extracting the label with the highest similarity to the
input image.

This work explores four variants of using CLIP for cloud presence detection,
shown in Figure 1, one (fully zero-shot) based on text prompts (1), and
(2)-(4) based on minor (1,000 gradient steps with batch size of 10, on only
the training subset) fine-tuning of the high-level classifier module. The text
prompts for method (1) were arbitrarily selected as “This is a satellite image
with clouds” and “This is a satellite image with clear sky” with no attempt to
improve them. In the case of (2), a linear layer is attached to the features
encoded by the frozen image encoder. In the case of (3), a CoOp approach is
employed, as described in [10]. Finally, the Radar (4) approach applies a
linear classifier to the image encodings of both RGB data and a false-color
composite of the SAR Data (VV, VH, and mean of the two channels are encoded as
3 input channels).

##  III Evaluation

The approach is tested on two benchmark datasets: (i) CloudSEN12 [14],
containing Sentinel-2 and Sentinel-1 data and (ii) SPARCS [15], containing
Landsat-8 imagery. Both datasets contain examples of cloudy images as well as
images with no clouds present, representing the two classes in the cloud
presence detection problem. This provides insight on the transferability of
the tested methods. Since CLIP has never been trained to perform the task of
cloud presence detection, it is not clear whether any CLIP-based method works
well with any specific image modality. The ability of these techniques to
transfer across sensing representations can indicate how powerful, or in other
words, how generalisable a given solution is.

The source sensor is not the only potential varying factor. Since the CLIP
model is designed for RGB data, it accepts input with precisely 3 channels.
Sentinel-2 and Landsat-8 contain multispectral data with more than 3 channels,
including the channels approximating the RGB visible bands. Extracting the RGB
channels would likely be the most straightforward approach of applying a pre-
trained CLIP-based model to multi-spectral data. However, other bands are
often useful too for the cloud presence detection task. For example, the
annotators of the SPARCS dataset, while labeling the images, have been shown
false-color images with bands B6 (SWIR), B5 (NIR), and B4 (Red) assigned to
RGB channels, respectively [15]. While these images do not correspond to the
visible RGB data, they can still be interpreted by the CLIP model. To
understand whether the same approach can be used on non-RGB images, two
versions of the SPARCS dataset are tested here, one with the RGB bands and one
with the false-color images observed by the annotators.

The achieved performance is reported in Table I as true positive rate (TPR),
the fraction of all cloudy images detected as cloudy; true negative rate
(TNR), the fraction of all cloud-free images detected as cloud-free; and F1
score, a harmonic mean between the ratio of correct predictions among all
cloudy samples and the ratio of correct predictions from all samples
classified as cloudy.

Table I: Performance on cloud presence detection for the tested datasets and detection methods. The reported metrics include true positive rate (TPR), true negative rate (TNR) and F1 Score. Test Dataset | CloudSEN12 | SPARCS
---|---|---
Modality | S2/RGB | L8/RGB | L8/B6-B4
| TPR | TNR | F1 | TPR | TNR | F1 | TPR | TNR | F1
1\. Text Prompts | 0.929 | 0.638 | 0.919 | 0.922 | 0.737 | 0.907 | 0.900 | 0.737 | 0.895
Trained on: | S2/RGB
2a. Linear Probe | 0.924 | 0.975 | 0.957 | 0.856 | 1.000 | 0.922 | 0.822 | 1.000 | 0.902
3a. CoOp | 0.936 | 0.980 | 0.964 | 0.878 | 0.921 | 0.919 | 0.822 | 0.974 | 0.897
4a. Radar | 0.930 | 0.960 | 0.959 | N/A | N/A | N/A | N/A | N/A | N/A
Trained on: | L8/B6-B4 | L8/RGB | L8/B6-B4
2b. Linear Probe | 0.961 | 0.759 | 0.950 | 0.811 | 1.000 | 0.896 | 0.811 | 1.000 | 0.896
3b. CoOp | 0.988 | 0.578 | 0.943 | 0.789 | 1.000 | 0.882 | 0.844 | 0.974 | 0.910

Three types of test data are used (corresponding to three sets of three
columns in the table), starting with CloudSEN12 data with RGB Sentinel-2
input, and then Landsat-8 data from the SPARCS dataset with either RGB bands
or B6-B4 false colour composite bands. The rows in the table correspond to
different methods proposed in this work, including (1) zero-shot
classification using text prompts, (2) linear probe fine-tuning, (3) CoOp
[10], and (4) Radar-based input. Furthermore, the variants (2) and (3), fine-
tuned on one type of 3-channel input can be tested on another source of
3-channel data from a different sensor. This is indicated by an additional
letter, where (a) signifies models fine-tuned on Sentinel-2 RGB data, and (b)
for models optimized on Landsat-8 data.

The results in the first row with the zero-shot text prompt performance
indicate that the CLIP-based model combined with the employed text prompts can
achieve a high performance of at least 0.9 true positive rate, which means
that the model can be quite reliable at picking up the cloudy samples.
However, consistently across all three test datasets, the true negative rate
is considerably lower, with values of 0.638 for Sentinel-2 data and 0.737 for
Landsat-8 data (regardless of the representation), which means that more
cloud-free images are classified as cloudy and that each technique exhibits a
bias towards the positive label.

The true negative rate is considerably improved by fine-tuning. For the
CloudSEN12 dataset, the true negative rate increases to 0.975 for the linear
probe approach (2a), 0.980 for the CoOp approach (3a) and 0.960 for the radar-
based variant (4a). The true positive rate is consistently lower, meaning that
some of the true positives are in result traded off for true negatives.

For the models fine-tuned on the Landsat-8 data (2b. and 3b.), a similar
effect is observed, with a very high true negative rate, and the true positive
rate decreasing considerably from the level achieved in the fully zero-shot
setting.

The transferability is tested by applying the models from Sentinel-2 (2a-3a)
to the SPARCS dataset and the models trained on Landsat-8 (2b-3b) to the
Sentinel-2 images. In this case, the Sentinel-2 models appear to transfer
better than the Landsat-8 models. However, a decrease in performance is
observed upon transfer across modalities, especially in the case of
transferring from Landsat-8 B6-B4 onto Sentinel-2 RGB data. This could mean
that the discriminative relationships of the CLIP encodings for the false-
colour data are quite different from the RGB data and do not transfer as well
and this would need to be confirmed through further experimentation.

##  IV Conclusion

The results presented herein demonstrate the potential of harnessing the
general vision-language model of CLIP for processing clouds in satellite
imagery with minimal training requirements.

The CLIP model used in a zero-shot setting has been found to consistently
struggle with detecting cloud-free images resulting in lower true negative
rate than true positive rate. This weakness can be reduced by a low-cost
training stage of only 1,000 optimization steps of a single linear layer
trained on the CLIP image encodings. This approach leads to a high performance
and observed transferability across tested sensor types and spectral bands.

Lastly, an approach performing CLIP-based classification based on optical and
radar data has been proposed and found to work at a comparable level to the
optical-only approaches.

## References

  * [1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever,  “Learning transferable visual models from natural language supervision,”  in International Conference on Machine Learning, 2021.
  * [2] Hu Xu, Gargi Ghosh, Po Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer,  “VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding,”  EMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings, pp. 6787–6800, 2021.
  * [3] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever,  “Zero-shot text-to-image generation,”  in Proceedings of the 38th International Conference on Machine Learning, Marina Meila and Tong Zhang, Eds. 18–24 Jul 2021, vol. 139 of Proceedings of Machine Learning Research, pp. 8821–8831, PMLR.
  * [4] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer,  “High-resolution image synthesis with latent diffusion models,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 10684–10695.
  * [5] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi,  “Photorealistic text-to-image diffusion models with deep language understanding,”  in Advances in Neural Information Processing Systems, Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, Eds., 2022.
  * [6] Yakoub Bazi, Mohamad Mahmoud Al Rahhal, Mohamed Lamine Mekhalfi, Mansour Abdulaziz Al Zuair, and Farid Melgani,  “Bi-Modal Transformer-Based Approach for Visual Question Answering in Remote Sensing Imagery,”  IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–11, 2022.
  * [7] Sylvain Lobry, Diego Marcos, Jesse Murray, and Devis Tuia,  “RSVQA: Visual Question Answering for Remote Sensing Data,”  IEEE Transactions on Geoscience and Remote Sensing, vol. 58, no. 12, pp. 8555–8566, 2020.
  * [8] Tingting Wei, Weilin Yuan, Junren Luo, Wanpeng Zhang, and Lina Lu,  “VLCA: vision-language aligning model with cross-modal attention for bilingual remote sensing image captioning,”  Journal of Systems Engineering and Electronics, vol. 34, no. 1, pp. 9–18, 2023.
  * [9] Savas Ozkan, Mehmet Efendioglu, and Caner Demirpolat,  “Cloud detection from RGB color remote sensing images with deep pyramid networks,”  International Geoscience and Remote Sensing Symposium (IGARSS), vol. 2018-July, pp. 6939–6942, 2018.
  * [10] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu,  “Learning to Prompt for Vision-Language Models,”  International Journal of Computer Vision, vol. 130, no. 9, pp. 2337–2348, 2022.
  * [11] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is All you Need,”  in Advances in Neural Information Processing Systems, I Guyon, U Von Luxburg, S Bengio, H Wallach, R Fergus, S Vishwanathan, and R Garnett, Eds. 2017, vol. 30, Curran Associates, Inc.
  * [12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby,  “An image is worth 16x16 words: Transformers for image recognition at scale,”  in International Conference on Learning Representations, 2021.
  * [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2016-Decem, pp. 770–778, 2016.
  * [14] Cesar Aybar, Luis Ysuhuaylas, Karen Gonzales, Jhomira Loja, Fernando Herrera, Lesly Bautista, Angie Flores, Roy Yali, Lissette Diaz, Nicole Cuenca, Fernando Prudencio, David Montero, Martin Sudmanns, Dirk Tiede, Gonzalo Mateo-garc, and G Luis,  “CloudSEN12 - a global dataset for semantic understanding of cloud and cloud shadow in satellite imagery,”  EarthArXiv, 2022.
  * [15] M. Joseph Hughes and Robert Kennedy,  “High-quality cloud masking of landsat 8 imagery using convolutional neural networks,”  Remote Sensing, vol. 11, no. 21, 2019.