# Robust Confidence Intervals in Stereo Matching using Possibility Theory

Roman Malinowski
CNES, CS, UTC
18 avenue E. Belin, Toulouse, France
<EMAIL_ADDRESS>Emmanuelle Sarrazin
Centre National d’Etudes Spatiales (CNES)
18 avenue E. Belin, Toulouse, France
<EMAIL_ADDRESS>Loïc Dumas
CS
5 rue Brindejonc des Moulinais, Toulouse, France
<EMAIL_ADDRESS>Emmanuel Dubois
Centre National d’Etudes Spatiales (CNES)
18 avenue E. Belin, Toulouse, France
<EMAIL_ADDRESS>Sébastien Destercke
UTC
Université de Technologie de Compiègne (UTC)
Avenue de Landshut, Compiègne, France
<EMAIL_ADDRESS>
###### Abstract

We propose a method for estimating disparity confidence intervals in stereo
matching problems. Confidence intervals provide complementary information to
usual confidence measures. To the best of our knowledge, this is the first
method creating disparity confidence intervals based on the cost volume. This
method relies on possibility distributions to interpret the epistemic
uncertainty of the cost volume. Our method has the benefit of having a white-
box nature, differing in this respect from current state-of-the-art deep
neural networks approaches. The accuracy and size of confidence intervals are
validated using the Middlebury stereo datasets as well as a dataset of
satellite images. This contribution is freely available on GitHub.

(a) Left epipolar image

(b) Confidence Intervals

Figure 1: Example of intervals on a image of the city of Montpellier, France.
Fig. 1(a) presents the left image, colored pixels indicate wrong interval
locations. Fig. 1(b) contains confidence intervals along a section of the
dashed line in Fig. 1(a)

##  1 Introduction

Stereo matching is used as a mean to estimate the depth of a scene in numerous
applications, ranging from autonomous driving to Earth observation [11, 19].
With the growing availability of satellite imagery [22], many stereo
algorithms have been proposed to perform 3D reconstruction from remote sensing
images [32, 10, 26, 36]. All those algorithms contain a dense matching step,
which consists in determining the displacement of every pixel, called
disparity, between the different images. Such algorithms usually start by
computing the similarity or sets of features between pixels in the form of a
cost volume, from which the disparity can then be deduced [30, 18]. Point
clouds are retrieved from the disparity, which can themselves be converted
into a mesh or a digital surface model.

Estimating the confidence in the disparity estimation is crucial in many
applications, and can lead to the improvement of overall results [17, 27]. In
the context of 3D reconstruction from satellite imagery, it can even be
propagated to be provided as the confidence related to the final 3D product.
As such, it has become an important research topic [16, 25]. There are two
aspects to the uncertainty regarding disparity computation: how confident we
are in the disparity prediction, and what would be the magnitude of the
potential error. Although state-of-the-art confidence measures reliably
indicate how likely a predicted disparity is to be correct, they do not
indicate the extent of the potential error. Those two notions are linked, but
are not the same: it could be that a prediction is made with high confidence
but would have a large associated error if wrong (meaning there would be a
great gap between the predicted and the true disparity in case of an error).
Similarly, a prediction could be made with low confidence, but the set of
possible disparities is restricted to few values close to the predicted
disparity. Estimating the magnitude of the error and providing sets of
possible disparities bring additional information that could help users to
improve the disparity map, similarly to current work with classical confidence
measures [33, 24].

In this article, we present a method for creating robust disparity confidence
intervals on the disparity estimation. The intervals will be propagated in
future applications to produce confidence intervals on digital surface models,
but this lies outside the scope of this paper and thus will not be covered
here. We design our method so that it can be fully integrated in classical 3D
pipelines [32, 10, 26, 36] using a cost-volume based stereo matching algorithm
depicted in Scharstein _et al_. [30]. To the best of our knowledge, this is
the first approach providing disparity confidence intervals for stereo
matching problems. For each pixel of the reference image, we give a lower and
upper displacement of its position in the target image as in Fig. 1. We aim
for an accuracy of $90\%$, using robust uncertainty models called possibility
distributions. We think this additional information about the magnitude of the
error gives a deeper understanding of the uncertainty in stereo matching
algorithms. No training is required to produce confidence intervals, also
sparing the method from classical criticisms regarding black-box aspects, in
the sense that all processing can be followed and monitored. Additionally, we
detail precautions that must be taken when applying some post-processing steps
[30] to the disparity map so that it stays consistent with the confidence
intervals. Our method for creating confidence intervals can be summarized as
follows:

  1. 1.

Computation of the matching cost volume and confidence measure

  2. 2.

Transformation of matching cost curves into possibility distributions

  3. 3.

Deduction of disparity intervals by taking $\alpha$-cuts on the possibility
distributions

  4. 4.

Filtering of intervals while maintaining consistency with the disparity map

  5. 5.

Statistical regularization of intervals in low-confidence zones

Section 2 contains an overview of current work regarding stereo algorithms,
confidence measures, and uncertainty models used in this paper. Section 3
details the method for constructing confidence intervals. Finally, the
confidence intervals robustness and size are validated in Section 4 using
images from the Middlebury dataset and from a dataset of high-resolution
optical satellite images of various landscapes around the city of Montpellier.
The code is freely available on GitHub: https://github.com/CNES/Pandora.

##  2 Related Works

###  2.1 Stereo matching

Our method is designed for classical 3D reconstruction pipelines from remote
sensing imagery [32, 10, 26, 36]. Those pipelines retrieve stereo information
by means of dense matching algorithms, mainly falling into two main
categories: classical approaches following the steps established by Scharstein
_et al_. [30], and full deep-learning approaches. Classical approaches usually
contain the following steps: matching cost computation, cost aggregation,
disparity computation and disparity refinement. On the other hand, deep-
learning approaches provide strong results (see [18] for details), but are
prone to generalization issues when using images differing from the training
dataset, especially in the context of satellite imagery [20]. Obtaining ground
truth data of various landscape can prove difficult, we therefore focus here
on classical approaches as we aim to produce a general and robust method for
creating confidence intervals. We consider two similarity functions, the
Census cost function [37], and the MC-CNN cost function [38] learned using
convolutional neural networks. Those similarity functions are minimitive,
meaning that a low value indicates a strong similarity of the compared
patches. Both handcrafted and learned similarity functions are considered here
to highlight the generic nature of our method for creating intervals. The cost
volume obtained using those functions is regularized using the semi-global
matching (SGM) methods [15, 9], used in other state-of-the-art methods [3].

Figure 2: Example of three MC-CNN cost curves with different confidences from
ambiguity.

Numerous confidence measures have been proposed regarding the disparity [16],
handcrafted using the properties from the reference image, the cost volume or
the disparity map itself. Learning-based methods constitute the majority of
state of the art confidence measures [25]. We can mention for instance deep
learning methods estimating confidence using CNNs on the cost volume [21] or
on the disparity map [24], and random forests on handcrafted confidence
measures [12]. We refer to [25, 16] for more in-depth details. Estimating the
confidence supporting the disparity map can lead to new strategies to improve
the overall results [8, 34, 17]. In this work, we use use a confidence measure
computed from the matching cost volume, called ambiguity [27].

In [4], authors estimate the magnitude of the absolute symmetric error using a
MLP. Their work demonstrate the interest in estimating the magnitude of the
error as a complement to confident measures, and we push this idea even
further. Indeed, estimating the absolute error provides valuable information
on the magnitude of the error, but does not indicate where the correct
disparity should be, i.e. if it is probably higher or lower than the predicted
one.

We therefore propose to answer this question using confidence intervals on the
disparity. Our method also differs as it can be plugged on a vast range of
cost-volume based stereo algorithms with winner-takes-all strategy.
Additionally, our method does not rely on the accuracy of the algorithm used
for the disparity prediction. In contrast, the network in [4] specifically
takes as input $4$ disparities at various resolutions estimated by a 3D CNN
[13], limiting its applicability. To the best of our knowledge, this is the
only work that estimates the disparity error in a manner similar to ours. The
restricted setting in which they work also means that we cannot compare our
two approaches.

###  2.2 Possibility distributions

To create confidence intervals, we consider using possibility distributions,
closely related fuzzy sets [6], as uncertainty models. This allows to
correctly represent epistemic uncertainty, i.e., due to the partial nature of
available information [1]. In stereo matching problems, errors are mostly due
to epistemic uncertainty. Indeed, similarity functions evaluate how much two
patches are alike based on given or learned features, and there exists some
uncertainty regarding how well this similarity indicates a match between
corresponding pixels. Using possibility distributions aims to address the
downsides of probability distributions regarding epistemic uncertainty [35].
Possibility distributions are well-suited to model an expert’s opinion on the
uncertainty of an imprecise observation, for instance in the context of
groundwater contamination [2, 1]. In the context of stereo matching, a
regularized cost curve using SGM can be seen as an expert evaluating if two
patches are homologous or not, based on their features and global properties
of the cost volume. The use of possibility distributions allows to benefit
from the advanced state of knowledge of IP towards robust estimation of
uncertainty.

Formally, a possibility distribution is defined as a mapping
$\pi:\Omega\rightarrow[0,1]$ verifying:

$\exists\omega\in\Omega,\pi(\omega)=1$ (1)

where $\Omega$ is the set of possible observed states. $\pi$ represents the
degree of possibility of an event $\omega$, $\pi(\omega)=1$ meaning that
$\omega$ is fully possible, and $\pi(\omega)=0$ meaning that $\omega$ is
impossible. Possibility distributions can be used to define the envelope of a
convex set of probability distributions $\mathbb{P}$ [7]:

$\mathbb{P}=\\{P:2^{\Omega}\rightarrow[0,1]~{}|~{}P(E)\leqslant\sup_{\omega\in
E}\pi(\omega)\\}$ (2)

where $P$ is a probability distribution on the power set $2^{\Omega}$ of
$\Omega$.

Alongside possibility distributions are often defined $\alpha$-cuts
$\mathcal{C}^{\pi}_{\alpha}$, which will be used for constructing the
confidence intervals:

$C_{\alpha}^{\pi}=\\{\omega\in\Omega~{}|~{}\pi(\omega)\geqslant\alpha\\}$ (3)

$\alpha$-cuts are the maximal sets whose possibility is at least equal to
$\alpha$ for every $\omega\in\Omega$. From a probabilistic point of view,
$\alpha$-cuts are composed of all $\omega\in\Omega$ for which there is a
$P\in\mathbb{P}$ from Eq. 2 such that $P(\omega)\geqslant\alpha$:

$C_{\alpha}^{\pi}=\\{\omega~{}|~{}\exists
P\in\mathbb{P},~{}P(\omega)\geqslant\alpha\\}$ (4)

##  3 Creation of Disparity Confidence Intervals

In the following, we consider that the images have been resampled in epipolar
geometry so that the displacement of a pixel between left $I_{L}$ and right
$I_{R}$ images can only occur horizontally in a given disparity range
$\mathcal{D}=[d_{min},d_{max}]$. In this setting, a pixel of the left image
$p=(i,j)$ with a disparity $d\in\mathcal{D}$ is matched to the pixel
$q=(i,j+d)$ of the right image with a cost $C_{V}(i,j,d)$.

###  3.1 From Cost Curves to Possibility Distributions

We use possibility distributions to model the epistemic uncertainty associated
with similarity functions, in the same way that an expert would state an
opinion for every pixel on which disparities are more likely to be the correct
ones.

In order to construct a possibility distribution consistent with a cost curve,
it is first necessary to normalize the cost curve to ensure its values lies in
$[0,1]$. Normalization can be done using the minimum and maximum values of the
matching cost volume after SGM regularization. Given a pixel $p=(i,j)$, its
cost curve is normalized as follows:

$\displaystyle f_{i,j}^{norm}(d)=\frac{C_{V}(i,j,d)-\max C_{V}}{\min
C_{V}-\max C_{V}}$ (5)

The $\min$ and $\max$ operators of standard normalization are reversed as the
possibility needs to be maximal when the cost function is minimal.
$f_{i,j}^{norm}$ is not yet a possibility distribution as it does not
necessarily verify Eq. 1. As noticed in Eq. 1, we can interpret it as a form
of inconsistency, in the sense that there is no disparity value such that the
two patches of pixels perfectly match. Assuming $p=(i,j)$ has an homologous
pixel in the right image, we can restore consistency through normalisation
[23] , resulting in a possibility distribution
$\pi_{i,j}(d):\mathcal{D}\rightarrow[0,1]$:

$\pi_{i,j}(d)=f_{i,j}^{norm}(d)+1-\max_{d\in\mathcal{D}}f_{i,j}^{norm}(d)$ (6)

Examples of cost curved transformed into possibility distributions are
presented in Figs. 2 and 3. Another way of verifying Eq. 1 would have been to
obtain $f_{i,j}^{norm}$ using the $\min$ and $\max$ operators on the cost
curve instead of the whole cost volume. Doing so would have artificially
accentuated the differences between the matching costs [23]. Using Eq. 6
instead keeps the curvature of the cost curve.

Figure 3: The possibility distributions obtained from the cost curves of Fig.
2. The arrows and vertical lines indicate the disparity intervals obtained
with $\alpha=0.9$.

###  3.2 Deducing Intervals from Alpha-cuts

Having defined possibility distributions, we now look to define a set of
possible disparities verifying our $90\%$ confidence objective. Every
possibility distribution defines a set of probability distributions
$\mathbb{P}$ using Eq. 2. Disparities $d$ for which every probability measure
in $\mathbb{P}$ evaluated on $d$ are lower than $0.9$ are deemed to be
unlikely. According to Eq. 4, considering disparities $d$ for which there is a
probability $P\in\mathbb{P}$ such that $P(d)\geqslant 0.9$ is equivalent to
consider the $\alpha$-cut $C^{\pi_{i,j}}_{\alpha}$ with $\alpha=0.9$.
Different values of $\alpha$ reflect different levels of confidence. We
compared different values of $\alpha$ in the ablation study presented in Tab.
2.

In general, $\alpha$-cuts are sets and not intervals. We are able to define a
single confidence interval $I_{\alpha}(i,j)$ from an $\alpha$-cut by taking
its extrema:

$I_{\alpha}(i,j)=[\min C^{\pi_{i,j}}_{\alpha},\max C^{\pi_{i,j}}_{\alpha}]$
(7)

Switching from $C^{\pi_{i,j}}_{\alpha}$ to the interval $I_{\alpha}(i,j)$
reduces the amount of information available by adding disparities with low
possibilities to our considered set of disparities. However, as only two
interval bounds need to be considered instead of every disparity of
$C^{\pi_{i,j}}_{\alpha}$, our solution consumes less memory, facilitates
further processing and is easier to understand for users. The level of
confidence of $I_{\alpha}(i,j)$ is also guaranteed to be at least equal to
that of $C^{\pi_{i,j}}_{\alpha}$ as $C^{\pi_{i,j}}_{\alpha}\subseteq
I_{\alpha}(i,j)$. Examples of confidence intervals on disparities are
presented in Fig. 3.

###  3.3 Refinement and Filtering with Intervals

In most stereo pipelines, the output disparity map deduced from the matching
cost volume is being post-processed. Namely, sub-pixel refinement and
filtering steps are usually applied to improve overall results. The confidence
intervals need to be processed accordingly to maintain their coherence with
the disparity map.

Sub-pixel refinement is taking into account by slightly extending the
confidence intervals in the case where the predicted disparity $d_{i,j}$ is
one of the interval bounds. For clarity, we refer to the lower and upper
bounds of an interval $I$ as $\underline{I}$ and $\overline{I}$ respectively.
Confidence intervals are modified as follows:

$\displaystyle\text{if }d_{i,j}=\min
C^{\pi_{i,j}}_{\alpha},\underline{I}_{\alpha}(i,j)=\min
C^{\pi_{i,j}}_{\alpha}-1$ (8) $\displaystyle\text{if }d_{i,j}=\max
C^{\pi_{i,j}}_{\alpha},\overline{I}_{\alpha}(i,j)=\max
C^{\pi_{i,j}}_{\alpha}+1$ (9)

This interval extension is coherent with different methods of interpolation,
like parabolic-fit for instance. V-fit refinement [14] is used in our
experiments.

Similarly, multiple filtering of the disparity map can be considered. We use a
median filter in our experiments, which is a popular method in many stereo
algorithms [30, 9]. Using a median filter modifies the disparity map and might
create inconsistencies with the confidence intervals. However, it is possible
to demonstrate that for every set of pixels $\\{p(i,j)\\}$ and confidence
intervals $\\{I_{\alpha}(i,j)\\}$ verifying $\forall(i,j),p(i,j)\in
I_{\alpha}(i,j)$ then:

$\displaystyle median(\\{\underline{I}_{\alpha}(i,j)\\})\leqslant
median(\\{p(i,j)\\})$ (10) $\displaystyle median(\\{p(i,j)\\})\leqslant
median(\\{\overline{I}_{\alpha}(i,j)\\})$ (11)

Previous equations mean that the median filter can be applied independently to
the disparity map and the confidence interval bounds while still maintaining
their coherence.

###  3.4 Regularization in Low Confidence Areas

Despite running post-processing steps, confidence interval performances
heavily depend on the quality of the similarity function used. Near surface
discontinuities, SGM algorithm sometimes struggle to correctly detect
disparity changes due to the continuity constraint. A shift between the
predicted and the true disparity can be observed, which induces biases in the
cost curve and challenges the interpretation of the cost curve as an expert’s
opinion. To overcome this limitation, confidence intervals are processed with
a more pessimistic approach in those areas.

Low confidence areas are detected using confidence measures. As such, our
approach is complementary to classical confidence estimation approaches. We
use the confidence from ambiguity measure [27] as it presents the advantage of
being both explainable and efficient for this task. This confidence measure
aims to represent the difficulty to single out a disparity value as the
correct disparity. The higher the confidence from ambiguity, the easier it is
easy to identify the correct disparity, whereas a value near $0$ indicates
that numerous patches present the same locally minimal similarity. Examples of
cost curves with different values of ambiguities can be found in Fig. 2. The
ambiguity measure is computed pixel-wise, and thus can present strong spatial
variations in low confidence zones. To compensate this effect, we first smooth
the ambiguity map using a $1\times 5$ $\min$ convolution kernel, and pixels
whose ambiguity is under a threshold $\tau$ are considered to be in low
confidence zones:

$\displaystyle amb_{smooth}(i,j)=\min_{-2\leqslant k\leqslant 2}amb(i,j+k)$
(12) $\displaystyle\text{Low confidence if }~{}amb_{smooth}(i,j)\leqslant\tau$
(13)

We fix empirically $\tau=0.6$ for our experiments with ambiguity, as it does
not seem to depend on the similarity function used.

Low confidence areas usually correspond to regions which possess a strong
disparity variation. The intervals in low confidence areas might be extended
to the minimal and maximal bounds found in the area, but experiments show this
approach is too pessimistic. Indeed, a single unnecessary large confidence
interval would penalize the whole area. Instead, we advocate for a statistical
approach by extending the intervals using quantiles instead of the the minimal
and maximal bounds. This approach has the advantage of being more robust to
outliers.

Figure 4: Low confidence areas with $l=2$.

For each low confidence pixel, we determine the maximal set $A$ of contiguous
low confidence pixels within $l$ lines above and bellow, as presented for
$l=2$ in Fig. 4. Experiments show that $l=2$ allows the set $A$ to contain
enough pixels to be statistically relevant, while maintaining a relatively low
computation time. Using values higher than $2$ does not improve the results.
Once $A$ has been established, the lower interval bound of the low confidence
pixel is set to the $10^{th}$ quantile of the lower interval bounds in $A$.
The same procedure is applied to the upper bounds with the $90^{th}$ quantile.
Examples of intervals with and without regularization along a row of a scene
are presented in Fig. 5. When no regularization is applied, errors occur for
intervals in low confidence areas.

(a) Intervals without regularization

(b) Interval with regularization

Figure 5: Intervals without (Fig. 5(a)) and with (Fig. 5(b)) regularization
from Middlebury’s $cones$. CENSUS cost function is used. Areas with low
confidence are indicated in gray.

##  4 Evaluation

| Global  | High confidence areas  | Low confidence areas
---|---|---|---
| Accuracy $\uparrow$ | Relative Size $\downarrow$ | Accuracy $\uparrow$ | Relative Size $\downarrow$ | Accuracy $\uparrow$ | Relative Size $\downarrow$ | Overestimation $\downarrow$
Dataset  | CENSUS | MCCNN  | CENSUS | MCCNN  | CENSUS | MCCNN  | CENSUS | MCCNN  | CENSUS | MCCNN  | CENSUS | MCCNN  | CENSUS | MCCNN
$2003$ | 0.973 |  $0.954$ | 0.033 |  0.033 | 0.983 |  $0.968$ | 0.033 |  0.033 | 0.942 |  $0.89$ | 0.183 |  $0.233$ | 0.165 |  $0.182$
$2005$ | $0.963$ |  0.971 | 0.026 |  $0.038$ | $0.969$ |  0.973 | 0.026 |  0.026 | $0.95$ |  0.969 | 0.218 |  $0.256$ | 0.152 |  $0.228$
$2006$ | 0.989 |  0.989 | 0.026 |  $0.038$ | 0.993 |  $0.992$ | 0.026 |  0.026 | $0.98$ |  0.985 | $0.569$ |  0.397 | 0.109 |  $0.268$
$2014$ | $0.957$ |  0.983 | $0.063$ |  0.029 | $0.912$ |  0.972 | 0.007 |  $0.013$ | $0.991$ |  0.996 | 0.872 |  $0.993$ | 0.12 |  $0.339$
$2021$ | $0.936$ |  0.991 | 0.594 |  $1.0$ | $0.818$ |  0.969 | 0.012 |  $0.026$ | $0.987$ |  0.999 | 0.859 |  $1.0$ | 0.168 |  $0.314$
Rural  | $0.904$ |  0.975 | 0.100 |  $0.250$ | $0.867$ |  0.967 | 0.083 |  $0.222$ | $0.952$ |  0.998 | 0.286 |  $1.0$ | 0.360 |  $0.713$
Urban  | $0.926$ |  0.986 | 0.100 |  $0.263$ | $0.894$ |  0.981 | 0.091 |  $0.238$ | $0.965$ |  0.999 | 0.286 |  $1.0$ | 0.367 |  $0.722$
Table 1: Accuracy $Acc$, Relative size $S_{rel}$ and Relative overestimation
$O_{rel}$ for different Middlebury and satellite datasets. “Global” column
consider every intervals in the dataset, while “High confidence areas” and
“Low confidence areas” separate the intervals based on the confidence measure
(Eq. 13). Two cost functions are compared: CENSUS and MC-CNN. The best results
for each dataset appear in bold font.

We define different metrics in order to simultaneously evaluate intervals
reliability and size. The metrics are adapted to our future objective of
propagating the disparity intervals into height intervals for 3D
reconstruction from satellite imagery. As such, intervals must be both
reliable and small. The metrics are first assessed globally by considering
every intervals for each dataset. Then, they are measured separately for high
confidence and low confidence areas. The distinction between both regions is
performed using a threshold on the ambiguity measure, as in Eq. 13.

###  4.1 Evaluation Metrics

The confidence intervals are evaluated following different criteria:

  * •

Their global accuracy. An interval is considered accurate if it contains the
true disparity. The accuracy is computed as:

$Acc=\frac{\\#accutate~{}intervals}{\\#intervals}$ (14)

where $\\#$ refers to the number of elements of a set.

  * •

The relative size of the intervals compared to the disparity range. The
relative size is computed over a scene or a whole dataset as:

$S_{rel}=median\left(\frac{\overline{I}-\underline{I}}{d_{max}-d_{min}}\right)$
(15)

This criterion is important as one could achieve $100\%$ accuracy by simply
setting every interval to $[d_{min},d_{max}]$.

  * •

$S_{rel}$ is not adapted in low confidence areas as we purposely extended the
intervals (see LABEL:{subsec:regularization}). Thus, we define an additional
criterion only for low confidence areas, called relative overestimation:

$O_{rel}=median\left(1-\frac{\Delta|d-\hat{d}|}{\overline{I}-\underline{I}}\right)$
(16)

where $\Delta|d-\hat{d}|$ is the maximal difference between the true disparity
and the predicted disparity over the low confidence area. It is therefore the
size of the optimal interval in the area. Figure 6 allows to visualize
$\Delta|d-\hat{d}|$ and $\overline{I}-\underline{I}$. $O_{rel}$ is the median
of intervals overestimation over low confidence areas.

Figure 6: A low confidence area in gray, with a representation of
$\Delta|d-\hat{d}|$ and $\overline{I}-\underline{I}$ from Eq. 16.

In Eq. 15 and Eq. 16, the median is used to evaluate the sizes of the
intervals instead of the mean in order to gain statistical robustness. It is
noteworthy that using the mean yields very similar results.

In the absence of any other method for creating disparity confidence
intervals, we compare the accuracy and relative size of our method with a
“naive” approach that serves as a baseline. This approach, referred to as
baseline in Tab. 2, consists in simply normalizing every cost curve with its
maximum and minimum values, and defining the disparity interval as the minimal
and maximal disparities for which the cost is greater that $90\%$. We compare
this to our method using possibilities without regularization and with
$\alpha$-levels taking values in $[50\%,80\%,90\%,98\%]$. We also present our
method with the regularization step and an $\alpha$-level of $90\%$, referred
to as “$90$ w/ reg”. This is the same method used in Tab. 1. This ablation
study enables us to observe the impact of different $\alpha$-levels on the
accuracy of the intervals, and proves the necessity of the regularization
step.

Section 4.3 discusses the performance of the method with regard to the
criterion from Eq. 14. We aim to validate at least $90\%$ accuracy on the
intervals over every scene. Section 4.4 evaluates criteria of Eq. 15 and Eq.
16. In high confidence areas, the relative size $S_{rel}$ needs to be as small
as possible. We consider a relative size of around $25\%$ as a satisfying
objective. In low confidence areas, a relative overestimation $O_{rel}$ of
about $30\%$ seems a feasible objective, while still providing enough
information for a later propagation into elevation intervals. The numerical
objectives are given for information purposes, as users needs may vary
depending on the application.

| Baseline | $50$ | $80$ | $90$ |  $90$ w/ reg |  $98$
---|---|---|---|---|---|---
2003  | 0.503 | 0.997 | 0.985 | 0.982 | 0.973 | 0.98
2005  | 0.592 | 0.978 | 0.958 | 0.950 | 0.963 | 0.944
2006  | 0.626 | 0.986 | 0.982 | 0.979 | 0.989 | 0.976
2014  | 0.039 | 0.543 | 0.519 | 0.494 | 0.957 | 0.474
2021  | 0.03 | 0.549 | 0.478 | 0.442 | 0.936 | 0.420
Table 2: Ablation study. Evaluation of the accuracy from Eq. 14 with different
methods using the CENSUS cost function. $50,80,90,98$ refer to different value
of $\alpha$. $90$ /w reg refers to an $\alpha$ value of $90\%$ and a
regularization step from Sec. 3.4.

###  4.2 Reference Datasets

We used $83$ scenes from Middlebury $2003,2005,2006,2014$ and $2021$ datasets
for evaluation [30, 29, 28, 31]. We use quarter-size and third-size versions
of the data for $2003,2005$ and $2006$ datasets and full resolution for $2014$
and $2021$ datasets. We use the disparity range indicated in the calibration
files. Each year contains respectively $2$, $6$, $21$, $23$ and $24$ pairs of
images with different shapes, and the size of the disparity intervals ranges
between $60$ and $1110$. We also use $120$ $1845\times 1845$ pairs of epipolar
images generated using [5] from satellite images of the region of Montpellier,
France, with a resolution of $50$cm/pixel. The disparity range for those
images is between $20$ and $50$ pixels, depending on the scene. During the
evaluation, the dataset is split into two categories: urban, for images
containing mostly buildings, and rural, for image mostly composed of forests
and fields. The ground truth disparity was retrieved using LiDAR data.

###  4.3 Accuracy Results

First, intervals accuracy can be analyzed on the specific example of Fig.
7(a). Inaccurate intervals are colored in the left image. Figures 7(b) and
7(c) present confidence interval values along a row, as well as the disparity
estimation and the true disparity. Low confidence areas are indicated by the
gray sections. Figs. 7(b) and 7(c) contain both high confidence areas with
small intervals, and low confidence areas with important disparity variations.
We observe a low confidence area between columns $1300$ and $1390$ where the
computed disparity is far from the true disparity, but the confidence
intervals remain correct.

Evaluating the accuracy statistics on each dataset yields strong results.
Scores per year for intervals computed using the CENSUS and MC-CNN cost
functions are presented in Tab. 1. CENSUS-based intervals have an accuracy
always superior to $90\%$ over each dataset. They verify the $90\%$ accuracy
objective on $80$ of the $83$ scenes from Middlebury, and on all $120$
satellite images. MC-CNN-based intervals have an accuracy superior to $95\%$
on every datasets. They validate the $90\%$ accuracy objective on all $83$
scenes from Middlebury and $120$ satellite images. Those strong performances
come nonetheless with large interval size, as detailed in section Sec. 4.4.

(a) Left epipolar image

(b) Confidence Intervals

(c) Confidence Intervals

Figure 7: Fig. 7(a) is the left image of the city of Montpellier where colored
pixels indicate wrong interval location. Figs. 7(b) and 7(c) present detailed
confidence intervals, computed disparity, and true disparity along the black
dashed line from figure Fig. 7(a). Areas with low confidence are indicated in
gray.

An ablation study is carried out in Tab. 2, highlighting the importance of the
regularization process. Without the regularization, the accuracy drops for
datasets with many low confidence areas. The value of $\alpha$ has a small
impact when compared to the regularization step. We also observe that the
naive approach of the baseline produces very inaccurate intervals in
comparison to our method.

###  4.4 Discussions on the Size of the Intervals

Although the intervals are very accurate, confidence intervals with
unnecessarily large sizes need to be avoided. In Figs. 7(b) and 7(c),
confidence intervals have a small relative size in high confidence areas, and
a larger size in low confidence areas. In those areas, the intervals are
properly adjusted to contain the predicted disparity and the true disparity
without overestimating the error. Similar observations can be made in Fig.
8(b).

Detailed statistics of intervals relative size and relative overestimation are
presented in Tab. 1 alongside accuracy results. For MC-CNN-based intervals,
the $5\%$ relative size criterion is validated for each dataset in high
confidence areas, with a relative size on all datasets of only $1.5\%$. The
global relative size is below $4\%$ for years $2001,2003,2005,2006$ and
$2014$. $S_{rel}$ is maximal for the $2021$ dataset due to the high proportion
of low confidence intervals on this dataset. Intervals in low confidence areas
are only overestimated by around $30\%$, meaning that large intervals are
unavoidable on this dataset. This can also be explained by the complexity and
high resolution of $2021$ (and $2014$) scenes, which leads to larger
confidence intervals in general. It results in poorer performances of the
disparity prediction as a majority of pixels have low confidence. Intervals
computed on satellite images have a relative size around $25\%$, which is
relatively low as most scenes have a disparity range of around $20$ pixels.
They however tend to overestimate the intervals in low confidence areas:
around $30\%$ when using the CENSUS cost function, and $70\%$ when using the
MC-CNN cost function.

(a) Left epipolar image

(b) Confidence Intervals

Figure 8: Fig. 8(a) is the left image of a rural area near Montpellier.
Colored pixels indicate wrong interval locations. Figure 8(b) details
confidence intervals, computed disparity, and true disparity along the black
dashed line from figure Fig. 8(a). Areas with low confidence are indicated in
gray.

Intervals computed using the CENSUS cost function validate the $25\%$ relative
size objective in high confidence areas. In low confidence areas, their
relative overestimation is around $13\%$, meaning that they are close to the
ideal intervals. They outperform MC-CNN based intervals on all datasets when
comparing their relative size in high confidence, and on all datasets
regarding the relative overestimation criterion. This is probably due to the
SGM regularization, which uses different weight for the CENSUS and MC-CNN cost
functions. CENSUS based weights seem to produce curves with a more
pronounced/narrow peak near the minima, resulting in smaller intervals, and
thus better relative sizes and over-estimation metrics.

##  5 Conclusion and Perspectives

To the best of our knowledge, we present the first method for creating
confidence intervals on the disparity in stereo matching problems. Our method
is designed to work with any stereo algorithm computing a 3D cost volume.
Matching cost functions are transformed into possibility distributions and
then interpreted as an expert’s opinion. We rely on the advanced theoretical
background of possibility distributions to compute robust uncertainty
estimations. Confidence intervals are deduced from the $\alpha$-cuts of
possibility distributions, and regularized in low confidence areas. Post-
processing steps handling is also taken into account to maintain consistency
between the predicted disparity map and the confidence intervals. As we have
not found existing accurate methods for comparison, we assess the intervals
based on accuracy, relative size, and overestimation. Criteria are evaluated
on the Middlebury datasets. $90\%$ accuracy objective is achieved while
maintaining a small relative size in high confidence area and without
overestimating the intervals size in low confidence areas. The accuracy of the
confidence intervals does not depend on the performance of the disparity
estimation. All of our contributions are available on our GitHub repository.
This work aims to motivate further research in detecting, locating and
quantifying the magnitude of the error in disparity maps. We demonstrate in
this paper that possibility distributions can model and process epistemic
uncertainty in a understandable and explainable way.

Future work will include propagating the disparity confidence intervals into
elevation confidence intervals for 3D reconstruction. Those intervals can then
be provided as a complementary product alongside digital surface models, often
used in many Earth Observation applications.

## References

  * Baudrit et al. [2007] Cédric Baudrit, Dominique Guyonnet, and Didier Dubois.  Joint propagation of variability and imprecision in assessing the risk of groundwater contamination.  _Journal of Contaminant Hydrology_ , 93(1-4):72–84, 2007.
  * Bárdossy et al. [1995] András Bárdossy, Axel Bronsterts, and Bruno Merz.  l-, 2- and 3-dimensional modeling of water movement in the unsaturated soil matrix using a fuzzy approach.  _Advances in Water Resources_ , 1995.
  * Chebbi et al. [2023] Mohamed Ali Chebbi, Ewelina Rupnik, Marc Pierrot-Deseilligny, and Paul Lopes.  DeepSim-Nets: Deep Similarity Networks for Stereo Image Matching.  In _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pages 2097–2105, Vancouver, BC, Canada, 2023. IEEE.
  * Chen et al. [2023] Liyan Chen, Weihan Wang, and Philippos Mordohai.  Learning the Distribution of Errors in Stereo Matching for Joint Disparity and Uncertainty Estimation, 2023.  arXiv:2304.00152 [cs].
  * Cournet et al. [2020] Myriam Cournet, Emmanuelle Sarrazin, Loïc Dumas, Julien Michel, Jonathan Guinet, David Youssefi, Véronique Defonte, and Quentin Fardet.  Ground Truth Generation and Disparity Estimation for Optical Satellite Imagery.  _The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences_ , XLIII-B2-2020:127–134, 2020.
  * Dubois and Prade [1991] Didier Dubois and Henri Prade.  Random sets and fuzzy interval analysis.  _Fuzzy Sets and Systems_ , 42(1):87–101, 1991.
  * Dubois and Prade [1992] Didier Dubois and Henri Prade.  When upper probabilities are possibility measures.  _Fuzzy Sets and Systems_ , 49(1):65–74, 1992.
  * Dumas et al. [2022] Loïc Dumas, Véronique Defonte, Yoann Steux, and Emmanuelle Sarrazin.  Improving Pairwise DSM With 3SGM: a Semantic Segmentation for SGM Using an Automatically Refined Neural Network.  _ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences_ , V-2-2022:167–175, 2022.
  * Facciolo et al. [2015] Gabriele Facciolo, Carlo de Franchis, and Enric Meinhardt.  MGM: A Significantly More Global Matching for Stereovision.  In _Procedings of the British Machine Vision Conference 2015_ , pages 90.1–90.12, Swansea, 2015. British Machine Vision Association.
  * Franchis et al. [2014] Carlo de Franchis, Enric Meinhardt, Julien Michel, Jean-Michel Morel, and Gabriele Facciolo.  An automatic and modular stereo pipeline for pushbroom images.  _ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences_ , II-3:49–56, 2014.
  * Geiger et al. [2012] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? The KITTI vision benchmark suite.  In _2012 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3354–3361, Providence, RI, 2012. IEEE.
  * Gouveia et al. [2015] Rafael Gouveia, Aristotle Spyropoulos, and Philippos Mordohai.  Confidence Estimation for Superpixel-Based Stereo Matching.  In _2015 International Conference on 3D Vision_ , pages 180–188, Lyon, France, 2015. IEEE.
  * Guo et al. [2019] Xiaoyang Guo, Kai Yang, Wukui Yang, Xiaogang Wang, and Hongsheng Li.  Group-Wise Correlation Stereo Network.  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 3268–3277, Long Beach, CA, USA, 2019. IEEE.
  * Haller et al. [2010] Istvan Haller, Cosmin D. Pantilie, Florin Oniga, and Sergiu Nedevschi.  Real-time semi-global dense stereo solution with improved sub-pixel accuracy.  In _2010 IEEE Intelligent Vehicles Symposium_ , pages 369–376, La Jolla, CA, USA, 2010. IEEE.
  * Hirschmüller [2005] Heiko Hirschmüller.  Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information.  In _2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)_ , pages 807–814, San Diego, CA, USA, 2005. IEEE.
  * Hu and Mordohai [2012] Xiaoyan Hu and Philippos Mordohai.  A Quantitative Evaluation of Confidence Measures for Stereo Vision.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 34(11):2121–2133, 2012.
  * Höllmann et al. [2020] Mark Höllmann, Max Mehltretter, and Christian Heipke.  Geometry-Based Regularization For Dense Image Matching Via Uncertainty-Driven Depth Propagation.  _ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences_ , V-2-2020:151–159, 2020.
  * Laga et al. [2022] Hamid Laga, Laurent Valentin Jospin, Farid Boussaid, and Mohammed Bennamoun.  A Survey on Deep Learning Techniques for Stereo-Based Depth Estimation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(4):1738–1764, 2022.
  * Marti et al. [2016] Renaud Marti, Simon Gascoin, Etienne Berthier, M. de Pinel, Thomas Houet, and Dominique Laffly.  Mapping snow depth in open alpine terrain from stereo satellite imagery.  _The Cryosphere_ , 10(4):1361–1380, 2016.
  * Marí et al. [2022] Roger Marí, Thibaud Ehret, and Gabriele Facciolo.  Disparity Estimation Networks for Aerial and High-Resolution Satellite Images: A Review.  _Image Processing On Line_ , 12:501–526, 2022.
  * Mehltretter and Heipke [2019] Max Mehltretter and Christian Heipke.  CNN-Based Cost Volume Analysis as Confidence Measure for Dense Matching.  In _2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)_ , pages 2070–2079, Seoul, Korea (South), 2019. IEEE.
  * Melet et al. [2020] Olivier Melet, David Youssefi, Céline L’Helguen, Julien Michel, Emmanuelle Sarrazin, Florie Languille, and Laurent Lebègue.  CO3D Mission Digital Surface Model Production Pipeline.  _The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences_ , XLIII-B2-2020, 2020.
  * Oussalah [2002] Mourad Oussalah.  On the normalization of subnormal possibility distributions: New investigations.  _International Journal of General Systems_ , 31(3):277–301, 2002.
  * Poggi and Mattoccia [2016] Matteo Poggi and Stefano Mattoccia.  Learning a General-Purpose Confidence Measure Based on O(1) Features and a Smarter Aggregation Strategy for Semi Global Matching.  In _2016 Fourth International Conference on 3D Vision (3DV)_ , pages 509–518, Stanford, CA, USA, 2016. IEEE.
  * Poggi et al. [2021] Matteo Poggi, Seungryong Kim, Fabio Tosi, Sunok Kim, Filippo Aleotti, Dongbo Min, Kwanghoon Sohn, and Stefano Mattoccia.  On the confidence of stereo matching in a deep-learning era: a quantitative evaluation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pages 1–1, 2021.
  * Rupnik et al. [2017] Ewelina Rupnik, Mehdi Daakir, and Marc Pierrot Deseilligny.  MicMac – a free, open-source solution for photogrammetry.  _Open Geospatial Data, Software and Standards_ , 2(1):14, 2017.
  * Sarrazin et al. [2021] Emmanuelle Sarrazin, Myriam Cournet, Loïc Dumas, Véronique Defonte, Quentin Fardet, Y. Steux, N. Jimenez Diaz, E. Dubois, D. Youssefi, and F. Buffe.  Ambiguity Concept In Stereo Matching Pipeline.  _The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences_ , XLIII-B2-2021:383–390, 2021.
  * Scharstein and Pal [2007] Daniel Scharstein and Chris Pal.  Learning Conditional Random Fields for Stereo.  In _2007 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1–8, Minneapolis, MN, USA, 2007. IEEE.
  * Scharstein and Szeliski [2003] Daniel Scharstein and Richard Szeliski.  High-accuracy stereo depth maps using structured light.  In _2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings._ , pages I–195–I–202, Madison, WI, USA, 2003. IEEE Comput. Soc.
  * Scharstein et al. [2001] Daniel Scharstein, Richard Szeliski, and Ramin Zabih.  A taxonomy and evaluation of dense two-frame stereo correspondence algorithms.  In _Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001)_ , pages 131–140, Kauai, HI, USA, 2001. IEEE Comput. Soc.
  * Scharstein et al. [2014] Daniel Scharstein, Heiko Hirschmüller, York Kitajima, Greg Krathwohl, Nera Nešić, Xi Wang, and Porter Westling.  High-Resolution Stereo Datasets with Subpixel-Accurate Ground Truth.  In _Pattern Recognition_ , pages 31–42. Springer International Publishing, Cham, 2014.  Series Title: Lecture Notes in Computer Science.
  * Shean et al. [2016] David E. Shean, Oleg Alexandrov, Zachary M. Moratto, Benjamin E. Smith, Ian R. Joughin, Claire Porter, and Paul Morin.  An automated, open-source pipeline for mass production of digital elevation models (DEMs) from very-high-resolution commercial stereo satellite imagery.  _ISPRS Journal of Photogrammetry and Remote Sensing_ , 116:101–117, 2016.
  * Spyropoulos et al. [2014] Aristotle Spyropoulos, Nikos Komodakis, and Philippos Mordohai.  Learning to Detect Ground Control Points for Improving the Accuracy of Stereo Matching.  In _2014 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1621–1628, Columbus, OH, USA, 2014. IEEE.
  * Tosi et al. [2019] Fabio Tosi, Matteo Poggi, and Stefano Mattoccia.  Leveraging Confident Points for Accurate Depth Refinement on Embedded Systems.  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pages 158–167, Long Beach, CA, USA, 2019. IEEE.
  * Walley [1991] Peter Walley.  _Statistical Reasoning with Imprecise Probabilities_.  Springer US, Boston, MA, 1991.
  * Youssefi et al. [2020] David Youssefi, Julien Michel, Emmanuelle Sarrazin, Fabrice Buffe, Myriam Cournet, Jean-Marc Delvit, Celine L’Helguen, Olivier Melet, Aurelie Emilien, and Julien Bosman.  CARS: A Photogrammetry Pipeline Using Dask Graphs to Construct A Global 3D Model.  In _IGARSS 2020 - 2020 IEEE International Geoscience and Remote Sensing Symposium_ , pages 453–456, Waikoloa, HI, USA, 2020. IEEE.
  * Zabih and Woodfill [1994] Ramin Zabih and John Woodfill.  Non-parametric local transforms for computing visual correspondence.  In _Computer Vision — ECCV ’94_ , pages 151–158. Springer Berlin Heidelberg, Berlin, Heidelberg, 1994.  Series Title: Lecture Notes in Computer Science.
  * Žbontar and LeCun [2016] Jure Žbontar and Yann LeCun.  Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches.  _Journal of Machine Learning Research 17_ , 2016.  arXiv: 1510.05970.


Euclidian path integral at some Euclidean time along the thermal
circle.565656Alternatively, we can slice the path integral into two pieces at
a pair of times on opposite sides of the circle to define the thermofield
double state.

We can slice this same path integral perpendicular to one of the
$\mathbb{R}^{3}$ directions to define the vacuum state of the field theory on
$\mathbb{R}^{2}\times S^{1}$ with antiperiodic boundary conditions for
fermions on the $S^{1}$.

The Casimir stress-energy tensor for this vacuum state (i.e. the stress-energy
relative to its value on Minkowski space) is directly related to the thermal
stress-energy tensor above, since both are obtained by inserting an operator
into the Euclidean path integral that defines these two states.

Specifically, if $\tau$ is the periodic coordinate and $\eta$ is the time
coordinate in the theory with a periodic direction, we have

$\displaystyle T^{Cas}_{\tau\tau}$ $\displaystyle=$ $\displaystyle-
c_{T}{1\over\beta^{4}}$ $\displaystyle T^{Cas}_{\eta\eta}$ $\displaystyle=$
$\displaystyle-{1\over 3}c_{T}{1\over\beta^{4}}$ $\displaystyle T^{Cas}_{ij}$
$\displaystyle=$ $\displaystyle\delta_{ij}{1\over 3}c_{T}{1\over\beta^{4}}$

Thus, we have a negative energy density $T^{Cas}_{\eta\eta}$ and a negative
null energy in the periodic direction of $T_{++}=T_{--}=-4/3c_{T}\beta^{-4}$.

These energy densities can be made arbitrarily negative by taking the length
$\beta$ of the periodic direction to be small.

#### Enhanced negative energies in systems with boundaries

In order to support the wormhole geometries that could give rise to realistic
cosmology, we need negative energy that extends over a spatial region that is
much larger than the scale of the negative energy itself. A naive way to
construct this would be to take this quantum field theory state from the
example above, consider the reduced state on a subsystem
$\tau\in(0,\beta-\epsilon)$ of the periodic direction, and try to purify this
in a system where the $\tau$ direction has a larger extent.

It is impossible to purify this system to a translation-invariant state of a
quantum field theory on Minkowski space, since the averaged null energy
condition (ANEC) Faulkner:2016mzt ; Hartman:2016lgu  says that integral of
$T_{++}$ over a complete null geodesic must be non-negative. So in any
purification on Minkowski, we must have regions of positive null-energy
surrounding this region of negative null energy. However, we are not aware of
any restriction on the width of a region where $T_{++}$ is bounded above by a
certain negative value. The holographic models of VanRaamsdonk:2021qgv ;
May:2021xhz , which have interfaces that create a finite spatial extent for
the CFT, provide examples to suggest that there is no such restriction. It
would be interesting to understand whether even weakly coupled quantum field
theories can exist in states with a negative energy much larger than the
geometrical scale (distance between interfaces/boundaries) for the right
choice of boundary physics.

## References

  * (1) J. M. Maldacena, _The Large N limit of superconformal field theories and supergravity_ , _Int. J. Theor. Phys._ 38 (1999) 1113–1133, [hep-th/9711200].
  * (2) S. S. Gubser, I. R. Klebanov and A. M. Polyakov, _Gauge theory correlators from noncritical string theory_ , _Phys. Lett._ B428 (1998) 105–114, [hep-th/9802109].
  * (3) E. Witten, _Anti-de Sitter space and holography_ , _Adv. Theor. Math. Phys._ 2 (1998) 253–291, [hep-th/9802150].
  * (4) O. Aharony, S. S. Gubser, J. M. Maldacena, H. Ooguri and Y. Oz, _Large N field theories, string theory and gravity_ , _Phys. Rept._ 323 (2000) 183–386, [hep-th/9905111].
  * (5) S. Perlmutter, G. Aldering, G. Goldhaber, R. Knop, P. Nugent, P. G. Castro et al., _Measurements of $\omega$ and $\lambda$ from 42 high-redshift supernovae_, _The Astrophysical Journal_ 517 (1999) 565.
  * (6) A. G. Riess, A. V. Filippenko, P. Challis, A. Clocchiatti, A. Diercks, P. M. Garnavich et al., _Observational evidence from supernovae for an accelerating universe and a cosmological constant_ , _The Astronomical Journal_ 116 (1998) 1009.
  * (7) P. J. E. Peebles and B. Ratra, _Cosmology with a Time Variable Cosmological Constant_ , _Astrophys. J. Lett._ 325 (1988) L17.
  * (8) B. Ratra and P. J. E. Peebles, _Cosmological Consequences of a Rolling Homogeneous Scalar Field_ , _Phys. Rev. D_ 37 (1988) 3406.
  * (9) R. R. Caldwell, R. Dave and P. J. Steinhardt, _Cosmological imprint of an energy component with general equation of state_ , _Phys. Rev. Lett._ 80 (1998) 1582–1585, [astro-ph/9708069].
  * (10) K. Dutta, Ruchika, A. Roy, A. A. Sen and M. M. Sheikh-Jabbari, _Beyond $\Lambda$CDM with low and high redshift data: implications for dark energy_, _Gen. Rel. Grav._ 52 (2020) 15, [1808.06623].
  * (11) L. Visinelli, S. Vagnozzi and U. Danielsson, _Revisiting a negative cosmological constant from low-redshift data_ , _Symmetry_ 11 (2019) 1035, [1907.07953].
  * (12) A. A. Sen, S. A. Adil and S. Sen, _Do cosmological observations allow a negative $\lambda$?_, 2021.
  * (13) T. Banks and W. Fischler, _An Holographic cosmology_ , hep-th/0111142.
  * (14) A. Strominger, _The dS / CFT correspondence_ , _JHEP_ 10 (2001) 034, [hep-th/0106113].
  * (15) M. Alishahiha, A. Karch, E. Silverstein and D. Tong, _The dS/dS correspondence_ , _AIP Conf. Proc._ 743 (2004) 393–409, [hep-th/0407125].
  * (16) V. Gorbenko, E. Silverstein and G. Torroba, _dS/dS and $T\overline{T}$_, _JHEP_ 03 (2019) 085, [1811.07965].
  * (17) E. Coleman, E. A. Mazenc, V. Shyam, E. Silverstein, R. M. Soni, G. Torroba et al., _de Sitter Microstates from $T\bar{T}+\Lambda_{2}$ and the Hawking-Page Transition_, 2110.14670.
  * (18) B. Freivogel, V. E. Hubeny, A. Maloney, R. C. Myers, M. Rangamani and S. Shenker, _Inflation in AdS/CFT_ , _JHEP_ 03 (2006) 007, [hep-th/0510046].
  * (19) P. McFadden and K. Skenderis, _Holography for Cosmology_ , _Phys. Rev. D_ 81 (2010) 021301, [0907.5542].
  * (20) S. Banerjee, U. Danielsson, G. Dibitetto, S. Giri and M. Schillo, _Emergent de Sitter Cosmology from Decaying Anti–de Sitter Space_ , _Phys. Rev. Lett._ 121 (2018) 261301, [1807.01570].
  * (21) L. Susskind, _Black Holes Hint Towards De Sitter-Matrix Theory_ , 2109.01322.
  * (22) G. Obied, H. Ooguri, L. Spodyneiko and C. Vafa, _De Sitter Space and the Swampland_ , 1806.08362.
  * (23) U. H. Danielsson and T. Van Riet, _What if string theory has no de Sitter vacua?_ , _Int. J. Mod. Phys. D_ 27 (2018) 1830007, [1804.01120].
  * (24) J. Hartle and S. Hawking, _Wave Function of the Universe_ , _Adv. Ser. Astrophys. Cosmol._ 3 (1987) 174–189.
  * (25) J. M. Maldacena and L. Maoz, _Wormholes in AdS_ , _JHEP_ 02 (2004) 053, [hep-th/0401024].
  * (26) B. Freivogel, V. Godet, E. Morvan, J. F. Pedraza and A. Rotundo, _Lessons on Eternal Traversable Wormholes in AdS_ , _JHEP_ 07 (2019) 122, [1903.05732].
  * (27) P. Betzios, E. Kiritsis and O. Papadoulaki, _Euclidean Wormholes and Holography_ , _JHEP_ 06 (2019) 042, [1903.05658].
  * (28) P. Saad, S. H. Shenker and D. Stanford, _JT gravity as a matrix integral_ , 1903.11115.
  * (29) D. Marolf and H. Maxfield, _Transcending the ensemble: baby universes, spacetime wormholes, and the order and disorder of black hole information_ , 2002.08950.
  * (30) A. Maloney and E. Witten, _Averaging Over Narain Moduli Space_ , 2006.04855.
  * (31) N. Afkhami-Jeddi, H. Cohn, T. Hartman and A. Tajdini, _Free partition functions and an averaged holographic duality_ , 2006.04839.
  * (32) J. Cotler and K. Jensen, _AdS 3 gravity and random CFT_, 2006.08648.
  * (33) M. Van Raamsdonk, _Comments on wormholes, ensembles, and cosmology_ , 2008.02259.
  * (34) J. Maldacena and L. Susskind, _Cool horizons for entangled black holes_ , _Fortsch. Phys._ 61 (2013) 781–811, [1306.0533].
  * (35) M. Van Raamsdonk, _Evaporating Firewalls_ , _JHEP_ 11 (2014) 038, [1307.1796].
  * (36) G. Penington, _Entanglement Wedge Reconstruction and the Information Paradox_ , 1905.08255.
  * (37) A. Almheiri, N. Engelhardt, D. Marolf and H. Maxfield, _The entropy of bulk quantum fields and the entanglement wedge of an evaporating black hole_ , 1905.08762.
  * (38) P. Simidzija and M. Van Raamsdonk, _Holo-ween_ , 6, 2020.  2006.13943.
  * (39) M. Van Raamsdonk, _Building up spacetime with quantum entanglement II: It from BC-bit_ , 1809.01197.
  * (40) S. Cooper, M. Rozali, B. Swingle, M. Van Raamsdonk, C. Waddell and D. Wakeham, _Black Hole Microstate Cosmology_ , 1810.10601.
  * (41) S. Antonini and B. Swingle, _Cosmology at the end of the world_ , _Nature Phys._ 16 (2020) 881–886, [1907.06667].
  * (42) A. Hamilton, D. N. Kabat, G. Lifschytz and D. A. Lowe, _Local bulk operators in AdS/CFT: A Boundary view of horizons and locality_ , _Phys. Rev. D_ 73 (2006) 086003, [hep-th/0506118].
  * (43) S. Antonini and B. Swingle, _Holographic boundary states and dimensionally reduced braneworld spacetimes_ , _Phys. Rev. D_ 104 (2021) 046023, [2105.02912].
  * (44) M. Van Raamsdonk, _Cosmology from confinement?_ , 2102.05057.
  * (45) Z. Wang, Z. Xu, S. Zhou and Y. Zhou, _Partial Reduction and Cosmology at Defect Brane_ , 2112.13782.
  * (46) Z.-Y. Fan, _On holographic braneworld cosmology_ , 2106.14376.
  * (47) A. Karch and L. Randall, _Localized gravity in string theory_ , _Phys. Rev. Lett._ 87 (2001) 061601, [hep-th/0105108].
  * (48) T. Takayanagi, _Holographic dual of a boundary conformal field theory_ , _Physical review letters_ 107 (2011) 101602.
  * (49) M. Fujita, T. Takayanagi and E. Tonni, _Aspects of AdS/BCFT_ , _JHEP_ 11 (2011) 043, [1108.5152].
  * (50) I. Kourkoulou and J. Maldacena, _Pure states in the SYK model and nearly- $AdS_{2}$ gravity_, 1707.02325.
  * (51) A. Almheiri, _Holographic Quantum Error Correction and the Projected Black Hole Interior_ , 1810.02055.
  * (52) G. Penington, S. H. Shenker, D. Stanford and Z. Yang, _Replica wormholes and the black hole interior_ , 1911.11977.
  * (53) X. Dong, X.-L. Qi, Z. Shangnan and Z. Yang, _Effective entropy of quantum fields coupled with gravity_ , 2007.02987.
  * (54) Y. Chen, V. Gorbenko and J. Maldacena, _Bra-ket wormholes in gravitationally prepared states_ , _JHEP_ 02 (2021) 009, [2007.16091].
  * (55) J. B. Hartle, S. W. Hawking and T. Hertog, _Accelerated expansion from negative $\lambda$_, 2012.
  * (56) L. Boyle, K. Finn and N. Turok, _C p t-symmetric universe_ , _Physical review letters_ 121 (2018) 251301.
  * (57) M. Demirtas, M. Kim, L. McAllister, J. Moritz and A. Rios-Tascon, _Exponentially Small Cosmological Constant in String Theory_ , _Phys. Rev. Lett._ 128 (2022) 011602, [2107.09065].
  * (58) M. Demirtas, M. Kim, L. McAllister, J. Moritz and A. Rios-Tascon, _Small cosmological constants in string theory_ , _JHEP_ 12 (2021) 136, [2107.09064].
  * (59) P. Betzios, E. Kiritsis and O. Papadoulaki, _Interacting systems and wormholes_ , _JHEP_ 02 (2022) 126, [2110.14655].
  * (60) E. Antonyan, J. Harvey and D. Kutasov, _Chiral symmetry breaking from intersecting D-branes_ , _Nucl. Phys. B_ 784 (2007) 1–21, [hep-th/0608177].
  * (61) A. May, P. Simidzija and M. V. Raamsdonk, _Negative energy enhancement in layered holographic conformal field theories_ , 2103.14046.
  * (62) P. Gao, D. L. Jafferis and A. Wall, _Traversable Wormholes via a Double Trace Deformation_ , _JHEP_ 12 (2017) 151, [1608.05687].
  * (63) J. Maldacena and X.-L. Qi, _Eternal traversable wormhole_ , 1804.00491.
  * (64) A. Karch and L. Randall, _Locally localized gravity_ , _JHEP_ 05 (2001) 008, [hep-th/0011156].
  * (65) A. Karch and L. Randall, _Open and closed string interpretation of SUSY CFT’s on branes with boundaries_ , _JHEP_ 06 (2001) 063, [hep-th/0105132].
  * (66) D. Gaiotto and E. Witten, _S-Duality of Boundary Conditions In N=4 Super Yang-Mills Theory_ , _Adv. Theor. Math. Phys._ 13 (2009) 721–896, [0807.3720].
  * (67) E. D’Hoker, J. Estes and M. Gutperle, _Exact half-BPS Type IIB interface solutions. I. Local solution and supersymmetric Janus_ , _JHEP_ 06 (2007) 021, [0705.0022].
  * (68) E. D’Hoker, J. Estes and M. Gutperle, _Exact half-BPS Type IIB interface solutions. II. Flux solutions and multi-Janus_ , _JHEP_ 06 (2007) 022, [0705.0024].
  * (69) O. Aharony, L. Berdichevsky, M. Berkooz and I. Shamir, _Near-horizon solutions for D3-branes ending on 5-branes_ , _Phys. Rev._ D84 (2011) 126003, [1106.1870].
  * (70) B. Assel, C. Bachas, J. Estes and J. Gomis, _Holographic Duals of D=3 N=4 Superconformal Field Theories_ , _JHEP_ 08 (2011) 087, [1106.4253].
  * (71) C. Bachas and I. Lavdas, _Massive Anti-de Sitter Gravity from String Theory_ , _JHEP_ 11 (2018) 003, [1807.00591].
  * (72) M. Van Raamsdonk and C. Waddell, _Finding AdS 5× S5 in 2+1 dimensional SCFT physics_, _JHEP_ 11 (2021) 145, [2109.04479].
  * (73) C. F. Uhlemann, _Islands and Page curves in 4d from Type IIB_ , _JHEP_ 08 (2021) 104, [2105.00008].
  * (74) M. Porrati, _Mass and gauge invariance 4. Holography for the Karch-Randall model_ , _Phys. Rev. D_ 65 (2002) 044015, [hep-th/0109017].
  * (75) T. Hartman, Y. Jiang and E. Shaghoulian, _Islands in cosmology_ , 2008.01022.
  * (76) E. Witten, _Anti-de Sitter space, thermal phase transition, and confinement in gauge theories_ , _Adv. Theor. Math. Phys._ 2 (1998) 505–532, [hep-th/9803131].
  * (77) D. Z. Freedman, S. S. Gubser, K. Pilch and N. P. Warner, _Renormalization group flows from holography–supersymmetry and a c-theorem_ , hep-th/9904017.
  * (78) J. Louis and H. Triendl, _Maximally supersymmetric ads 4 vacua in n = 4 supergravity_ , _Journal of High Energy Physics_ 2014 (Oct, 2014) .
  * (79) P. Karndumri, _Holographic RG flows and symplectic deformations of N=4 gauged supergravity_ , 2203.01766.
  * (80) P. Breitenlohner and D. Z. Freedman, _Stability in gauged extended supergravity_ , _Annals of physics_ 144 (1982) 249–281.
  * (81) D. N. Page, _Will Entropy Decrease if the Universe Recollapses?_ , _Phys. Rev. D_ 32 (1985) 2496.
  * (82) I. Affleck and M. Dine, _A New Mechanism for Baryogenesis_ , _Nucl. Phys. B_ 249 (1985) 361–380.
  * (83) S. M. Carroll, _Quintessence and the rest of the world: suppressing long-range interactions_ , _Physical Review Letters_ 81 (1998) 3067\.
  * (84) L. Amendola, _Coupled quintessence_ , _Physical Review D_ 62 (2000) 043511.
  * (85) S. Fallows and S. F. Ross, _Islands and mixed states in closed universes_ , _JHEP_ 07 (2021) 022, [2103.14364].
  * (86) R. Bousso and E. Wildenhain, _Islands in closed and open universes_ , 2022\.
  * (87) R. Bousso and J. Polchinski, _Quantization of four form fluxes and dynamical neutralization of the cosmological constant_ , _JHEP_ 06 (2000) 006, [hep-th/0004134].
  * (88) G. J. Galloway, K. Schleich, D. M. Witt and E. Woolgar, _Topological censorship and higher genus black holes_ , _Phys. Rev. D_ 60 (1999) 104039, [gr-qc/9902061].
  * (89) T. Faulkner, R. G. Leigh, O. Parrikar and H. Wang, _Modular Hamiltonians for Deformed Half-Spaces and the Averaged Null Energy Condition_ , _JHEP_ 09 (2016) 038, [1605.08072].
  * (90) T. Hartman, S. Kundu and A. Tajdini, _Averaged Null Energy Condition from Causality_ , _JHEP_ 07 (2017) 066, [1610.05308].


# Correlated frequency noise in a multimode acoustic resonator

Nuttamas Tubsrinuan<EMAIL_ADDRESS>Department of
Microtechnology and Nanoscience MC2, Chalmers University of Technology,
Göteborg, Sweden  London Centre for Nanotechnology, University College London,
Gordon Street, London, WC1H 0AH, UK  Jared H. Cole  Chemical and Quantum
Physics, School of Science, RMIT University, Melbourne, VIC 3001, Australia
Per Delsing  Department of Microtechnology and Nanoscience MC2, Chalmers
University of Technology, SE-41296 Göteborg, Sweden  Gustav Andersson
<EMAIL_ADDRESS>Department of Microtechnology and Nanoscience MC2,
Chalmers University of Technology, Göteborg, Sweden  Pritzker School of
Molecular Engineering, University of Chicago, Chicago, Illinois 60637, USA

###### Abstract

Frequency instabilities are a major source of errors in quantum devices. This
study investigates frequency fluctuations in a surface acoustic wave (SAW)
resonator, where reflection coefficients of 14 SAW modes are measured
simultaneously for more than seven hours. We report two distinct noise
characteristics. Multimode frequency noise caused by interactions with two-
level system (TLS) defects shows significant degrees of correlations that
diminish with increased detuning. This finding agrees with the current
understanding of the parasitic TLS behavior as one of the dominant noise
sources in quantum devices. In addition to the TLS-induced noise, we observe
strong anomalous frequency fluctuations with slow, anti-correlated dynamics.
These noise bursts resemble signatures of cosmic radiation observed in
superconducting quantum systems.

††preprint: APS/123-QED

Decoherence in quantum devices leads to quantum information loss and limits
their performance [1, 2]. A number of decoherence channels have been
identified, including various types of parasitic two-level systems (TLSs) [3,
4] and quasiparticles [5]. Quasiparticle generation can be induced by stray
infrared light [6] and cosmic radiation [7], while TLS defects inducing loss
and noise reside in the circuit [8, 9]. Over long timescales, these noise
sources impact the stability of device parameters [3, 10, 2], necessitating
frequent calibration or additional mitigation measures [11, 12]. In the past
decades, significant enhancement in the coherence of superconducting devices
has been achieved by improved microwave engineering [13], circuit design [14],
fabrication methods [15], and materials [16]. In spite of these developments,
decoherence remains a significant issue. A better understanding of noise
processes is important in order to mitigate decoherence mechanisms and advance
the development of solid state quantum processors.

In this work, we use a multifrequency signal (a frequency comb) to
simultaneously probe frequency fluctuations in 14 modes of a surface acoustic
wave (SAW) resonator at cryogenic temperatures. Tracking the reflection
coefficient of the SAW resonator for more than 7 hours, we observe that there
are two dominant noise processes taking place in the device. We further
investigate the noise correlations and their characteristics.

Two-level system defects residing at dielectric surfaces and interfaces of
superconducting circuits have been considered a central problem causing
frequency fluctuations and significant energy dissipation [17, 8, 4, 18].
Despite many attempts in the past decades [9], TLS-induced loss and noise
remain persistent issues in superconducting quantum processors. Although the
microscopic nature of TLSs is not well understood, the standard tunneling
model (STM) [19, 20, 9] has been successfully used as a framework for
understanding TLS interaction with quantum circuits. The STM predicts power-
dependent losses and temperature-dependent resonance frequency shifts due to
the direct coupling between quantum devices and coherent TLSs, which have
transition energies larger than $k_{B}T$. In addition to coherent TLSs there
are thermal fluctuators, two-level systems with sufficiently low transition
energies to be thermally activated at millikelvin temperatures. Direct
interaction between TLSs is a significant effect not accounted for by the STM
[8, 21, 22]. When coherent TLSs couple with surrounding thermal fluctuators,
they are subject to a continuous time-dependent energy shift, known as
spectral diffusion [23]. While undergoing spectral diffusion, they can couple
dispersively to quantum devices, resulting in frequency drift that translates
into phase noise [24, 25].

TLSs couple to both electromagnetic [17] and strain fields [26]. Their
behaviors have been studied using microwave resonators [27, 28],
superconducting qubits [17, 22], and surface acoustic wave (SAW) devices [29,
30]. A study of SAW resonators at cryogenic temperature reveals that energy
losses are caused by the coupling to parasitic TLSs [31]. Further two-tone
spectroscopy of TLSs using a SAW resonator shows that TLSs dominate the losses
[30], suggesting the potential of SAW resonators for investigating TLSs. Due
to the slow propagation, the SAW wavelength is smaller than the
electromagnetic wavelength by five orders of magnitude at the same frequency.
This characteristic allows SAW resonant structures to accommodate many
resonant frequencies in a compact design.

Here, we exploit multiple resonance modes of the SAW resonator to probe
frequency noise caused by TLSs at many different frequencies simultaneously.
According to the Jaynes-Cummings model [32], the resonance frequency of a
resonator $f$, which is dispersively coupled to an individual TLS, is given by

$f=f_{r}+\frac{g^{2}}{f_{r}-f_{TLS}}\hat{\sigma}_{z},$ (1)

where $\hat{\sigma}_{z}$ is the Pauli spin operator acting on the TLS, and $g$
represents the TLS-resonator coupling strength. The resonance frequency of the
bare resonator and the TLS are denoted by $f_{r}$ and $f_{TLS}$, respectively.
In most realistic cases, the resonator interacts with a TLS ensemble with a
frequency distribution spanning the resonator bandwidth. The effective
frequency shift is the sum of the dispersive shifts contributed by all the
individual TLSs. Due to their small wavelengths and high wavenumbers, SAW
modes are close to each other in frequency and have a large overlap in real
space. Thus, all modes interact with the same physical TLS ensemble, however,
with slightly different detuning. Therefore, the multimode frequency noise in
a SAW resonator is expected to be correlated. Additionally, the degree of
correlation should depend on the detuning, as suggested by Eq. (1).

In superconducting devices, cosmic rays and other sources of radiation can
generate high-energy phonons [33, 34, 35]. These phonons break Cooper pairs,
leading to an increased kinetic inductance of the superconducting material and
associated resonance frequency shifts [36]. A recent study on superconducting
granular aluminum resonators shows that ionizing radiation due to high-energy
cosmic rays create phonons that in turn generate quasiparticle bursts,
resulting in resonance frequency shifts and reduced quality factors [37]. This
effect has also been observed in transmon qubits with shortened relaxation and
dephasing times as a result [7, 38, 39]. Direct measurements of rare events
which were attributed to cosmic rays have also been observed in gravitational
wave experiments [40, 41, 42]. The frequency noise caused by ionizing
radiation can be suppressed by the use of a phonon absorber [43], radiation
shielding, and conducting experiments at a deep-underground facility [37].

Our SAW resonator, fabricated on a piezoelectric GaAs substrate, has two Bragg
mirrors, located 1440 $\mu$m apart. Each mirror has 800 fingers. An
interdigital transducer (IDT) is placed at the center of the cavity, serving
as both input and output port. The IDT has a split-finger design with 50
periods of 1.2 $\mu$m, as illustrated in Fig. 1(a) [30]. The device supports
14 resonance modes within the frequency range from 2.37 GHz to 2.40 GHz. The
free spectral range between each mode is approximately 2 MHz.

Figure 1: (a) Dimensions of the SAW resonator used in this study (not to
scale). The mirrors and the IDT are fabricated from aluminum patterned on a
GaAs substrate. The shorted Bragg mirrors have 800 fingers each. The IDT has a
split-finger design with 50 periods of $1.2\ \mu$m. (b) The reflection
measurement of the SAW resonator, showing the 14 resonance modes lying between
the frequency of 2.37 and 2.40 GHz. The indices represent mode numbers
arranged from the lowest to the highest frequency of the resonator stopband.
(c) The fitted magnitude and phase of resonance mode 2 highlighted by the red
box in the top figure. The quality factors are extracted using Eq. 2.

The characteristics of the SAW resonator are shown in Fig. 1(b)-(c). The
resonance linewidths are between 18 and 32 kHz. We extract the resonator
parameters based on the expression for the reflection coefficient [44]:

$S_{11}(f)=-ae^{-i(\pi-\theta)}\left[1-\frac{2(Q_{L}/Q_{C})e^{i\phi_{0}}}{1+2iQ_{L}(\frac{f-f_{r}}{f_{r}})}\right],$
(2)

where $f$ is the probe frequency and the parameter $a$ is an amplitude scaling
coefficient. A phase shift due to cable delay is denoted by $\theta$. The
loaded quality factor and the external quality factor are represented by
$Q_{L}$ and $Q_{C}$, respectively. An impedance mismatch is accounted for by
the parameter $\phi_{0}$. We drive the SAW resonator continuously so that the
estimated number of phonons is approximately 1000.

We measure resonance frequency fluctuations by tracking the amplitude and
phase response while driving 14 modes of the SAW resonator simultaneously for
$2.6\times 10^{4}$ seconds, approximately 7 hours and 15 minutes. (See
supplementary materials for the experimental setup.) The probe frequencies are
defined from the resonance frequencies extracted from the resonator profile in
Fig. 1(a) using Eq. 2. Additionally, we apply a control tone, which is off-
resonant with every SAW mode. This control tone is used as a reference for the
signal fluctuations due to other sources in the environment. (See
supplementary materials.) The long measurement duration is used in order to
capture any TLS telegraphic switching. The rates for such events have been
observed to be in the range of 75 $\mu$Hz - 2 mHz [2]. We calculate the
resonance frequency from the measured reflection coefficient $S_{11}$ using
the parameters extracted from the resonance profile in Fig. 1 and Eq. 2.

Figure 2: (a) Normalized time traces of SAW resonance frequency fluctuations
of the resonator mode 1 up to mode 14. All traces are measured simultaneously
for approximately 7 hours and 15 minutes. The traces are vertically shifted
for better visualization. Bottom panel: Time traces in specific time windows
highlighted by grey boxes in the top panel. (b) The resonance frequencies in
the “quiet” time window where fluctuations are small. Note that the traces are
shifted by a different proportion from (a). (c) - (e) The frequency noise in
these time windows show additional burst events superimposed with the TLS-
induced noise. (See discussion in the main text.)

The time traces in Fig. 2(a) show resonance frequency fluctuations of the 14
different modes within the resonator band of 2.37-2.40 GHz. We observe a
baseline of frequency fluctuations with uniform amplitude, superimposed with
occasional striking large-amplitude fluctuations. There are “quiet” time
windows where only small fluctuations are present. For instance, in the time
window highlighted in Fig. 2(b), there are no burst events. The noise
amplitude is small, 50 Hz on average, compared to the largest fluctuation of
20 kHz. There is no obvious relationship between fluctuations in different
modes. For a device having a small mode volume, we would expect to see a clear
telegraphic shift caused by an individual TLS [25]. In contrast, our SAW
resonator has a large mode volume. Thus, we can observe frequency drifts
resulting from many weakly-coupled TLSs, which makes telegraphic features
indistinct.

We also observe time windows with large fluctuation features (Fig. 2(c)-(e)).
There are anomalous frequency fluctuations with a smooth deformation dependent
on mode number. In these periods, we recognize similar features in all traces,
indicating significant correlations between them. Figure 2(c) shows a series
of substantial frequency shifts followed by a relaxation tail. The shifts are
negative in the low mode numbers. Their magnitude reduces as the mode number
increases. Above mode 10, the frequency shift changes sign to positive, while
its magnitude grows as the mode number increases.

Figure 2(d) presents frequency noise with a large amplitude. At 4:48:35 hours,
we observe a frequency increase followed by a relaxation tail. This feature
superimposes with a different frequency shift, which has a weaker amplitude
and can be either positive or negative. While the burst event shows positive
shifts in all modes, this second event has a negative sign in the first nine
modes. Above mode 10, both shifts are positive, resulting in anomalously high
amplitudes, on the order of a few kilohertz. In Fig. 2(e), we can see the
characteristic of the noise of the second kind, where the magnitude gradually
changes and switches signs. Note that the magnitude of fluctuations decreases
with increasing mode number before the frequency shift becomes positive,
similar to the features observed in Fig. 2(c). The noise bursts appear
randomly in our measurements with a rate of approximately 0.12-0.30 mHz.

To determine the relationship between frequency noise in different modes $i$
and $j$, we compute the correlation coefficients $C_{ij}$, which is given by

$C_{ij}=\frac{\langle\Delta f_{i}\Delta f_{j}\rangle}{\sqrt{\langle\Delta
f_{i}^{2}\rangle\langle\Delta f_{j}^{2}\rangle}},$ (3)

where $\langle\ \rangle$ denotes the expectation value. Figure 3(a)
illustrates the correlation matrix of the frequency noise calculated from the
entire measurement (7 hours and 15 minutes). The frequency noise correlation
reduces with increasing detuning, i.e., modes closer in frequency show
stronger correlations than far-detuned modes. As suggested by Eq. 1, at a
given frequency $f_{r}$, a TLS far from resonance contributes to the frequency
noise less than a near-resonant TLS. Furthermore, we obtain negative
correlations at a certain detuning, implying that two modes have frequency
shifts in opposite directions. This result is not surprising, given that many
large frequency shifts shown in Fig. 2(c)-(e) explicitly show opposite signs.

Figure 3: (a) The matrix showing the correlation coefficients of the frequency
noise measured for 7 hours and 15 minutes, shown in Fig. 2(a). Note that the
correlation diminishes and becomes negative when the pair of modes is far-
detuned. (b) The correlation matrix calculated from the quiet time window
shown in Fig. 2(b). For this specific period, the correlations between modes
still manifest a diminishing trend. However, we no longer observe clear
negative correlations between modes. (c) Averaged correlation coefficients as
a function of detuning calculated from the whole time trace (blue) and the
quiet window (black). For the correlations of the full time trace, the sign of
the correlation switches from positive to negative when the detuning is
approximately 18 MHz. Note that the plot is symmetric as $C_{ij}=C_{ji}$.

Based on the STM, the anti-correlated noise bursts are not likely to be
induced by TLSs. Under a continuous drive, near-resonant TLSs in the ground
state can be excited by the SAW field. If TLSs are saturated, they do not
contribute to resonance frequency shifts [30]. Thus, saturating TLSs in a
specific part of the frequency spectrum will result in an effect similar to
the absence of TLSs. The remaining unsaturated TLSs will push neighboring
modes towards the saturated region. The modes that are positively detuned from
the saturated TLSs will experience negative frequency shifts, while those that
are negatively detuned from the saturated TLSs will have positive shifts.
Therefore, we would expect anti-correlated frequency shifts for the modes on
opposite sides. Such frequency shifts caused by saturating TLSs are in the
reverse direction to the results observed in our measurements. According to
Eq. 1, the dispersive interaction between a resonator and a ground-state TLS
will shift frequencies away from each other. These frequency shifts will give
rise to negative correlation coefficients, which are similar to the effects
observed in our experiment. Anti-correlated frequency noise could arise from
the sudden appearance or elimination of TLSs at specific frequencies, and such
”scrambling” of the TLS distribution has been observed with superconducting
qubits [45]. That process, however, does not show the slow relaxation of mode
frequencies observed here and is not consistent with stronger anti-
correlations between further-detuned modes.

Considering the frequency noise only in a quiet period, we do not observe
clear negative correlations, as shown in Fig. 3(b). From the correlation
matrices, we can calculate the average correlation as a function of detuning
between modes. Figure 3(c) shows that the correlation coefficients between all
pairs of modes show a smooth variation. In the quiet time window, the noise
correlation reduces from unity to zero. For the full time trace, the
correlation switches its sign when the detuning exceeds 18 MHz. The
diminishing positive correlation characteristic agrees with our current
understanding of TLS interaction. However, this mechanism cannot explain the
anti-correlations during the burst periods.

We further perform frequency domain analysis by calculating the noise power
spectral density $S_{f}$. We consider the standard power law noise, where the
noise spectrum is expressed by [46]

$S_{f}(f)=\frac{h_{-1}}{f}+h_{0}.$ (4)

Here, $h_{-1}$ and $h_{0}$ denote the $1/f$ and the white noise level,
respectively. The power spectral density (PSD) of the frequency noise in mode
8 is plotted in Fig. 4. The PSD calculated from the quiet time window exhibits
$1/f$ characteristic, which is anticipated for the TLS-induced frequency noise
[47]. For the full trace, where the frequency bursts are present, the PSD at
low frequencies is one order of magnitude higher than that of the quiet
window, indicating that the anomalous frequency fluctuations dominate in this
frequency range. We fit a sum of $1/f$ and Lorentzian frequency noise
$PSD(f)=S_{L}(f)+S_{f}(f)$, where

$S_{L}(f)=\frac{4A^{2}\tau_{0}}{1+(2\pi f\tau_{0})^{2}}.$ (5)

Here, $A$ and $\tau_{0}$ are the Lorentzian noise amplitude and the
characteristic time scale, respectively. With this model, we obtain the
Lorentzian noise amplitude $A=53.12$ Hz, the Lorentzian characteristic time
$\tau_{0}=1.22$ s, the $1/f$ noise power $h_{-1}=145.0$ Hz2, whereas the white
noise level is negligible. The characteristic time falls in the range of the
characteristic time of relaxation tails 1.2-3.0 s, observed in the time traces
in Fig. 2. This fit does not capture the steep rolloff of the PSD at higher
frequencies, indicating a non-Lorentzian spectrum of the slow frequency
excursions. Our analysis in both time and frequency domains suggests that the
frequency noise cannot be purely induced by TLSs and the large fluctuations
are caused by an additional noise source.

Figure 4: Noise power spectral density of the frequency fluctuations in mode 8
calculated from the entire trace and from the quiet time window only
(approximately 400 second-long). The lines represent the fitted power law
noise model (Eq. 4) and the sum of the power law noise and the Lorentzian
noise model (Eq.4 + Eq. 5).

As shown in Fig. 2, frequency noise anomalies have an abrupt change followed
by a relaxation tail. This characteristic resembles the effect of ionizing
radiation in high-kinetic inductance superconducting devices caused by cosmic
rays [37]. In that case the long-time transient effects are due to long
recombination times of quasiparticles above the superconducting gap, a
mechanism absent in our mechanical resonator. The expected flux of high-energy
cosmic muons is around 10 per hour across the chip cross-section [7, 37, 48],
a rate consistent with our data, although there is no obvious interaction
mechanism. An ionization event induced by radiation could lead to the
generation of electron-hole pairs which get trapped in deep-level defects with
long relaxation times. Charge trapping and release from deep defects can give
rise to currents [49], causing noise and device instability [50]. The
correlations of the frequency noise between different modes vary smoothly with
detuning and show a uniform behaviour where the sign of frequency excursions
are consistent across different events. This suggests a geometrical effect
arising from the spatial distribution of a single or small number of defects
and the acoustic field variation with mode number.

In conclusion, we have reported two distinctive noise processes in a surface
acoustic wave resonator. There is TLS-induced noise, showing diminishing
correlations with increased detuning and a noise spectral density with $1/f$
characteristics. This noise signature provides additional evidence for TLSs as
the origin of phase noise in quantum devices. The correlated nature of phase
noise suggests that quantum information encoded in different modes of the same
resonator undergo reduced relative dephasing. In the presence of TLS noise, a
multimode quantum memory may therefore provide improved preservation of
entanglement compared to physically separated modes. Additionally, we observe
anomalous frequency fluctuations, whose origins are not clear but show some
resemblance to the previously observed impact of cosmic radiation. Our
multimode probe scheme reveals striking negative correlations in these
fluctuations, whose noise power spectral densities are approximately
represented by a Lorentzian noise model.

###### Acknowledgements.

This work was supported by the Knut and Alice Wallenberg foundation through
the Wallenberg Center for Quantum Technology (WACQT), and the Swedish Research
Council, VR. JHC is supported by the Australian Research Council Centre of
Excellence programme through Grant number CE170100026 and the Australian
National Computational Infrastructure facility.

## Data availability

The data generated and analysed in this study are available upon reasonable
request.

## Author contributions

NT performed the measurements and analysis. NT and GA developed the conceptual
idea behind the experiment. JHC provided theory support. PD and GA supervised
the project. NT wrote the manuscript with input from all coauthors.

## Competing interests

The authors declare no competing interests.

## References

  * Simmonds  _et al._ [2004] R. W. Simmonds, K. M. Lang, D. A. Hite, S. Nam, D. P. Pappas, and J. M. Martinis, Phys. Rev. Lett. 93, 077003 (2004).
  * Burnett  _et al._ [2019] J. J. Burnett, A. Bengtsson, M. Scigliuzzo, D. Niepce, M. Kudra, P. Delsing, and J. Bylander, NPJ Quantum Inf. 5, 54 (2019).
  * Müller  _et al._ [2015] C. Müller, J. Lisenfeld, A. Shnirman, and S. Poletto, Phys. Rev. B 92, 035442 (2015).
  * Martinis  _et al._ [2015] J. Martinis, K. Cooper, R. McDermott, M. Steffen, M. Ansmann, K. Osborn, K. Cicak, S. Oh, D. Pappas, R. Simmonds, and C. Yu, Phys. Rev. Lett. 95, 210503 (2015).
  * Wilen  _et al._ [2021] C. D. Wilen, S. Abdullah, N. A. Kurinsky, C. Stanford, L. Cardani, G. D’Imperio, C. Tomei, L. Faoro, L. B. Ioffe, C. H. Liu, A. Opremcak, B. G. Christensen, J. L. DuBois, and R. McDermott, Nature 594, 369 (2021).
  * Barends  _et al._ [2011] R. Barends, J. Wenner, M. Lenander, Y. Chen, R. C. Bialczak, J. Kelly, E. Lucero, P. O’Malley, M. Mariantoni, D. Sank, H. Wang, T. C. White, Y. Yin, J. Zhao, A. N. Cleland, J. M. Martinis, and J. J. A. Baselmans, Appl. Phys. Lett. 99, 113507 (2011).
  * Vepsäläinen  _et al._ [2020] A. P. Vepsäläinen, A. H. Karamlou, J. L. Orrell, A. S. Dogra, B. Loer, F. Vasconcelos, D. K. Kim, A. J. Melville, B. M. Niedzielski, J. L. Yoder, S. Gustavsson, J. A. Formaggio, B. A. VanDevender, and W. D. Oliver, Nature 584, 551 (2020).
  * Burnett  _et al._ [2014] J. Burnett, L. Faoro, V. G. I. Wisby, A. Chernykh, G. Mikhailov, V. Tulin, R. Shaikhaidarov, V. Antonov, P. Meeson, A. Tzalenchuk, and T. Linström, Nat. Commun. 5, 4119 (2014).
  * Müller  _et al._ [2019] C. Müller, J. Cole, and J. Lisenfeld, Rep. Prog. Phys. 82, 124501 (2019).
  * Klimov  _et al._ [2018] P. V. Klimov, J. Kelly, Z. Chen, M. Neeley, A. Megrant, B. Burkett, R. Barends, K. Arya, B. Chiaro, Y. Chen, A. Dunsworth, A. Fowler, B. Foxen, C. Gidney, M. Giustina, R. Graff, T. Huang, E. Jeffrey, E. Lucero, J. Y. Mutus, O. Naaman, C. Neill, C. Quintana, P. Roushan, D. Sank, A. Vainsencher, J. Wenner, T. C. White, S. Boixo, R. Babbush, V. N. Smelyanskiy, H. Neven, and J. M. Martinis, Phys. Rev. Lett. 121, 090502 (2018).
  * Matityahu  _et al._ [2021] S. Matityahu, A. Shnirman, and M. Schechter, Phys. Rev. Applied 16, 044036 (2021).
  * You  _et al._ [2022] X. You, Z. Huang, U. Alyanak, A. Romanenko, A. Grassellino, and S. Zhu, Stabilizing and improving qubit coherence by engineering noise spectrum of two-level systems (2022), arXiv:2206.10669v1 [quant-ph] .
  * Chiaro  _et al._ [2016] B. Chiaro, A. Megrant, A. Dunsworth, Z. Chen, R. Barends, B. Campbell, Y. Chen, A. Fowler, I. C. Hoi, E. Jeffrey, J. Kelly, J. Mutus, C. Neill, P. J. J. O’Malley, C. Quintana, P. Roushan, D. Sank, A. Vainsencher, J. Wenner, T. C. White, and J. M. Martinis, Supercond. Sci. Technol. 29, 104006 (2016).
  * Koch  _et al._ [2007] J. Koch, T. M. Yu, J. Gambetta, A. A. Houck, D. I. Schuster, J. Majer, A. Blais, M. H. Devoret, S. M. Girvin, and R. J. Schoelkopf, Phys. Rev. A 76, 042319 (2007).
  * Cicak  _et al._ [2010] K. Cicak, D. Li, J. A. Strong, M. S. Allman, F. Altomare, A. J. Sirois, J. D. Whittaker, J. D. Teufel, and R. W. Simmonds, Appl. Phys. Lett. 96, 093502 (2010).
  * Chang  _et al._ [2013] J. B. Chang, M. R. Vissers, A. D. Córcoles, M. Sandberg, J. Gao, D. W. Abraham, J. M. Chow, J. M. Gambetta, M. Beth Rothwell, G. A. Keefe, M. Steffen, and D. P. Pappas, Appl. Phys. Lett. 103, 012602 (2013).
  * Martinis  _et al._ [2005] J. M. Martinis, K. B. Cooper, R. McDermott, M. Steffen, M. Ansmann, K. D. Osborn, K. Cicak, S. Oh, D. P. Pappas, R. W. Simmonds, and C. C. Yu, Phys. Rev. Lett. 95, 210503 (2005).
  * Lisenfeld  _et al._ [2019] J. Lisenfeld, A. Bilmes, A. Megrant, R. Barends, J. Kelly, P. Klimov, G. Weiss, J. M. Martinis, and A. V. Ustinov, NPJ Quantum Inf. 5, 105 (2019).
  * Phillips [1972] W. Phillips, J. Low Temp. Phys. 7, 351 (1972).
  * Anderson  _et al._ [1972] P. W. Anderson, B. I. Halperin, and C. M. Varma, Philos. Mag. 25, 1 (1972).
  * Faoro and Ioffe [2015] L. Faoro and L. B. Ioffe, Phys. Rev. B 91, 014201 (2015).
  * Lisenfeld  _et al._ [2015] J. Lisenfeld, G. J. Grabovskij, C. Müller, J. H. Cole, G. Weiss, and A. V. Ustinov, Nat. Commun. 6, 1 (2015).
  * Black and Halperin [1977] J. Black and B. Halperin, Phys. Rev. B 69, 2879 (1977).
  * Gao  _et al._ [2007] J. Gao, J. Zmuidzinas, B. A. Mazin, H. G. LeDuc, and P. K. Day, Appl. Phys. Lett. 90, 102507 (2007).
  * Niepce  _et al._ [2021] D. Niepce, J. J. Burnett, M. Kudra, J. H. Cole, and J. Bylander, Science Advances 7, eabh0462 (2021).
  * Grabovskij  _et al._ [2012] G. J. Grabovskij, T. Peichl, J. Lisenfeld, G. Weiss, and A. V. Ustinov, Science 338, 232 (2012).
  * Gao  _et al._ [2008] J. Gao, M. Daal, A. Vayonakis., S. Kumar, J. Zmuidzinas, B. Sadoulet, B. Mazin, P. Day, and H. Leduc, Appl. Phys. Lett. 92, 152505 (2008).
  * Pappas  _et al._ [2011] D. P. Pappas, M. R. Vissers, D. S. Wisbey, J. S. Kline, and J. Gao, IEEE Trans. Appl. Supercond. 21, 871 (2011).
  * Anghel  _et al._ [2007] D. V. Anghel, T. Kühn, Y. M. Galperin, and M. Manninen, Phys. Rev. B 75, 064202 (2007).
  * Andersson  _et al._ [2021] G. Andersson, A. L. Bilobran, M. Scigliuzzo, M. M. de Lima, J. H. Cole, and P. Delsing, NPJ Quantum Inf. 7, 15 (2021).
  * Manenti  _et al._ [2016] R. Manenti, M. J. Peterer, A. Nersisyan, E. B. Magnusson, A. Patterson, and P. J. Leek, Phys. Rev. B 93, 041411 (2016).
  * Jaynes and Cummings [1963] E. Jaynes and F. Cummings, Proc. IEEE 51, 89 (1963).
  * Grünhaupt  _et al._ [2018] L. Grünhaupt, N. Maleeva, S. T. Skacel, M. Calvo, F. Levy-Bertrand, A. V. Ustinov, H. Rotzinger, A. Monfardini, G. Catelani, and I. M. Pop, Phys. Rev. Lett. 121, 117001 (2018).
  * Swenson  _et al._ [2010] L. J. Swenson, A. Cruciani, A. Benoit, M. Roesch, C. S. Yung, A. Bideaud, and A. Monfardini, Appl. Phys. Lett. 96, 263511 (2010).
  * Moore  _et al._ [2012] D. C. Moore, S. R. Golwala, B. Bumble, B. Cornell, P. K. Day, H. G. LeDuc, and J. Zmuidzinas, Appl. Phys. Lett. 100, 232601 (2012).
  * Catelani  _et al._ [2011] G. Catelani, R. J. Schoelkopf, M. H. Devoret, and L. I. Glazman, Phys. Rev. B 84, 064517 (2011).
  * Cardani  _et al._ [2021] L. Cardani, F. Valenti, N. Casali, G. Catelani, T. Charpentier, M. Clemenza, I. Colantoni, A. Cruciani, G. D’Imperio, L. Gironi, L. Grünhaupt, D. Gusenkova, F. Henriques, M. Lagoin, M. Martinez, G. Pettinari, C. Rusconi, O. Sander, C. Tomei, A. V. Ustinov, M. Weber, W. Wernsdorfer, M. Vignati, S. Pirro, and I. M. Pop, Nat. Commun. 12, 2733 (2021).
  * de Graaf  _et al._ [2020] S. E. de Graaf, L. Faoro, L. B. Ioffe, S. Mahashabde, J. J. Burnett, T. Lindström, S. E. Kubatkin, A. V. Danilov, and A. Y. Tzalenchuk, Sci. Adv. 6, 5055 (2020).
  * McEwen  _et al._ [2021] M. McEwen, L. Faoro, K. Arya, A. Dunsworth, T. Huang, S. Kim, B. Burkett, A. Fowler, F. Arute, J. C. Bardin, A. Bengtsson, A. Bilmes, B. B. Buckley, N. Bushnell, Z. Chen, R. Collins, S. Demura, A. R. Derk, C. Erickson, M. Giustina, S. D. Harrington, S. Hong, E. Jeffrey, J. Kelly, P. V. Klimov, F. Kostritsa, P. Laptev, A. Locharla, X. Mi, K. C. Miao, S. Montazeri, J. Mutus, O. Naaman, M. Neeley, C. Neill, A. Opremcak, C. Quintana, N. Redd, P. Roushan, D. Sank, K. J. Satzinger, V. Shvarts, T. White, Z. J. Yao, P. Yeh, J. Yoo, Y. Chen, S. Vadim, J. M. Martinis, H. Neven, A. Megrant, L. Ioffe, and R. Barends, Nat. Phys. 18, 107 (2021).
  * Astone  _et al._ [2001] P. Astone, M. Bassan, P. Bonifazi, P. Carelli, E. Coccia, S. D’Antonio, V. Fafone, G. Federici, A. Marini, G. Mazzitelli, Y. Minenkov, I. Modena, G. Modestino, A. Moleti, G. Pallottino, V. Pampaloni, G. Pizzella, L. Quintieri, F. Ronga, R. Terenzi, M. Visco, and L. Votano, Phys. Lett. B 499, 16 (2001).
  * Astone  _et al._ [2008] P. Astone, D. Babusci, M. Bassan, P. Bonifazi, G. Cavallari, E. Coccia, S. D’Antonio, V. Fafone, G. Giordano, C. Ligi, A. Marini, G. Mazzitelli, Y. Minenkov, I. Modena, G. Modestino, A. Moleti, G. Pallottino, G. Pizzella, L. Quintieri, A. Rocchi, F. Ronga, R. Terenzi, and M. Visco, Astropart. Phys. 30, 200 (2008).
  * Goryachev  _et al._ [2021] M. Goryachev, W. M. Campbell, I. S. Heng, S. Galliou, E. N. Ivanov, and M. E. Tobar, Phys. Rev. Lett. 127, 071102 (2021).
  * Karatsu  _et al._ [2019] K. Karatsu, A. Endo, J. Bueno, P. J. de Visser, R. Barends, D. J. Thoen, V. Murugesan, N. Tomita, and J. J. A. Baselmans, Appl. Phys. Lett. 114, 032601 (2019).
  * Probst  _et al._ [2015] S. Probst, S. Song, P. Bushev, A. Ustinov, and M. Weides, Rev. Sci. Instrum. 86, 024706 (2015).
  * Thorbeck  _et al._ [2022] T. Thorbeck, A. Eddins, I. Lauer, D. T. McClure, and M. Carroll, Tls dynamics in a superconducting qubit due to background ionizing radiation (2022), arXiv:2210.04780 [quant-ph] .
  * Rubiola [2008] E. Rubiola, Phase noise and feedback theory, in  _Phase Noise and Frequency Stability in Oscillators_, The Cambridge RF and Microwave Engineering Series (Cambridge University Press, 2008) p. 88–124.
  * Burnett  _et al._ [2013] J. Burnett, T. Lindström, M. Oxborrow, Y. Harada, Y. Sekine, P. Meeson, and A. Y. Tzalenchuk, Phys. Rev. B 87, 140501 (2013).
  * Tsuji  _et al._ [1998] S. Tsuji, T. Katayama, K. Okei, T. Wada, I. Yamamoto, and Y. Yamashita, J. Phys. G: Nucl. Part. Phys. 24, 1805 (1998).
  * Zuleeg [1985] R. Zuleeg, Chapter 11 - Radiation Effects in GaAs Integrated Circuits (Elsevier, 1985) pp. 391–436.
  * Peaker  _et al._ [2018] A. R. Peaker, V. P. Markevich, and J. Coutinho, J. Appl. Phys. 123, 161559 (2018).