# The Intrinsic Dimension of Images
and Its Impact on Learning

Phillip Pope1, Chen Zhu1, Ahmed Abdelkader2, Micah Goldblum1, Tom Goldstein1
1Department of Computer Science, University of Maryland, College Park
2Oden Institute for Computational Engineering and Sciences, University of
Texas at Austin
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

It is widely believed that natural image data exhibits low-dimensional
structure despite the high dimensionality of conventional pixel
representations. This idea underlies a common intuition for the remarkable
success of deep learning in computer vision. In this work, we apply dimension
estimation tools to popular datasets and investigate the role of low-
dimensional structure in deep learning. We find that common natural image
datasets indeed have very low intrinsic dimension relative to the high number
of pixels in the images. Additionally, we find that low dimensional datasets
are easier for neural networks to learn, and models solving these tasks
generalize better from training to test data. Along the way, we develop a
technique for validating our dimension estimation tools on synthetic data
generated by GANs allowing us to actively manipulate the intrinsic dimension
by controlling the image generation process. Code for our experiments may be
found here.

##  1 Introduction

The idea that real-world data distributions can be described by very few
variables underpins machine learning research from manifold learning to
dimension reduction (Besold & Spokoiny, 2019; Fodor, 2002). The number of
variables needed to describe a data distribution is known as its _intrinsic
dimension_ (ID). In applications, such as crystallography, computer graphics,
and ecology, practitioners depend on data having low intrinsic dimension
(Valle & Oganov, 2010; Desbrun et al., 2002; Laughlin, 2014). The utility of
representations which are low-dimensional has motivated a variety of deep
learning techniques including autoencoders and regularization methods (Hinton
& Salakhutdinov, 2006; Vincent et al., 2010; Gonzalez & Balajewicz, 2018; Zhu
et al., 2018).

It is also known that dimensionality plays a strong role in learning function
approximations and non-linear class boundaries. The exponential cost of
learning in high dimensions is easily captured by the trivial case of sampling
a function on a cube; in $d$ dimensions, sampling only the cube vertices would
require $2^{d}$ measurements. Similar behaviors emerge in learning theory. It
is known that learning a manifold requires a number of samples that grows
exponentially with the manifold’s intrinsic dimension (Narayanan & Mitter,
2010). Similarly, the number of samples needed to learn a well-conditioned
decision boundary between two classes is an exponential function of the
intrinsic dimension of the manifold on which the classes lie (Narayanan &
Niyogi, 2009). Furthermore, these learning bounds have no dependence on the
ambient dimension in which manifold-structured datasets live.

In light of the exponentially large sample complexity of learning high-
dimensional functions, the ability of neural networks to learn from image data
is remarkable. Networks learn complex decision boundaries from small amounts
of image data (often just a few hundred or thousand samples per class). At the
same time, generative adversarial networks (GANs) are able to learn image
“manifolds” from merely a few thousand samples. The seemingly low number of
samples needed to learn these manifolds strongly suggests that image datasets
have extremely low-dimensional structure.

Figure 1:  Estimates of the intrinsic dimension of commonly used datasets
obtained using the MLE method with $k=3,5,10,20$ nearest neighbors (left to
right). The trends are consistent using different $k$’s.

Despite the established role of low dimensional data in deep learning, little
is known about the intrinsic dimension of popular datasets and the impact of
dimensionality on the performance of neural networks. Computational methods
for estimating intrinsic dimension enable these measurements. We adopt tools
from the dimension estimation literature to shed light on dimensionality in
settings of interest to the deep learning community. Our contributions can be
summarized as follows:

  * •

We verify the reliability of intrinsic dimension estimation on high-
dimensional data using generative adversarial networks (GANs), a setting in
which we can a priori upper-bound the intrinsic dimension of generated data by
the dimension of the latent noise vector.

  * •

We measure the dimensionality of popular datasets such as MNIST, CIFAR-10, and
ImageNet. In our experiments, we find that natural image datasets whose images
contain thousands of pixels can, in fact, be described by orders of magnitude
fewer variables. For example, we estimate that ImageNet, despite containing
$224\times 224\times 3=150528$ pixels per image, only has intrinsic dimension
between $26$ and $43$; see Figure 1.

  * •

We train classifiers on data, synthetic and real, of various intrinsic
dimension and find that this variable correlates closely with the number of
samples needed for learning. On the other hand, we find that extrinsic
dimension, the dimension of the ambient space in which data is embedded, has
little impact on generalization.

Together, these results put experimental weight behind the hypothesis that the
unintuitively low dimensionality of natural images is being exploited by deep
networks, and suggest that a characterization of this structure is an
essential building block for a successful theory of deep learning.

##  2 Related Work

While the hypothesis that natural images lie on or near a low-dimensional
manifold is controversial, Goodfellow et al. (2016) argue that the low-
dimensional manifold assumption is at least approximately correct for images,
supported by two observations. First, natural images are locally connected,
with each image surrounded by other highly similar images reachable through
image transformations (e.g., contrast, brightness). Second, natural images
seem to lie on a low-dimensional structure, as the probability distribution of
images is highly concentrated; uniformly sampled pixels can hardly assemble a
meaningful image. It is widely believed that the combination of natural scenes
and sensor properties yields very sparse and concentrated image distributions,
as has been supported by several empirical studies on image patches (Lee et
al., 2003; Donoho & Grimes, 2005; Carlsson et al., 2008). This observation
motivated work on efficient coding (Olshausen & Field, 1996) and served as a
prior in computer vision (Peyré, 2009). Further, rigorous experiments have
been conducted clearly supporting the low-dimensional manifold hypothesis for
many image datasets (Ruderman, 1994; Schölkopf et al., 1998; Roweis & Saul,
2000; Tenenbaum et al., 2000; Brand, 2003); see also (Fefferman et al., 2016)
for principled algorithms on verifying the manifold hypothesis.

The generalization literature seeks to understand why some models generalize
better from training data to test data than others. One line of work suggests
that the loss landscape geometry explains why neural networks generalize well
(Huang et al., 2019). Other generalization work predicts that data with low
dimension, along with other properties which do not include extrinsic
dimension, characterize the generalization difficulty of classification
problems (Narayanan & Niyogi, 2009). In the context of deep learning, Gong et
al. (2019) found that neural network features are low-dimensional. Ansuini et
al. (2019) further found that the intrinsic dimension of features decreases in
late layers of neural networks and observed interesting trends in the
dimension of features in early layers. In contrast to Gong et al. (2019) and
Ansuini et al. (2019), who find that the intrinsic dimension of internal
representations is inversely correlated with high performance, we study the
dimensionality of _data_ and its impact on performance, and we make a similar
finding. Zhu et al. (2018) proposed a regularizer derived from the intrinsic
dimension of images augmented with their corresponding feature vectors.
Another line of work in deep learning has found that neural networks rely
heavily on textures which are low-dimensional (Geirhos et al., 2018; Brendel &
Bethge, 2019). Similarly, some have suggested that natural images can be
represented as mixtures of textures which lie on a low-dimensional manifold
(Vacher & Coen-Cagli, 2019; Vacher et al., 2020).

##  3 Intrinsic Dimension Estimation

Given a set of sample points $\mathcal{P}\subset\mathbb{R}^{N}$, it is common
to assume that $\mathcal{P}$ lies on or near a low-dimensional manifold
$\mathcal{M}\subseteq\mathbb{R}^{N}$ of intrinsic dimension
$\dim(\mathcal{M})=d\ll N$. As a measure of the degrees of freedom in a
dataset, as well as the information content, there is great interest in
estimating the intrinsic dimension $d$. In the remainder of this section, we
briefly describe the dimension estimation method we use in this paper; for
further information, see (Kim et al., 2019) and references therein.

One of the main approaches to intrinsic dimension estimation is to examine a
neighborhood around each point in the dataset, and compute the Euclidean
distance to the $k^{th}$ nearest neighbor. Assuming that density is constant
within small neighborhoods, the _Maximum Likelihood Estimation (MLE)_ of
Levina & Bickel (2005) uses a Poisson process to model the number of points
found by random sampling within a given radius around each sample point. By
relating the rate of this process to the surface area of the sphere, the
likelihood equations yield an estimate of the ID at a given point $x$ as:

$\hat{m}_{k}(x)=\left[\frac{1}{k-1}\sum_{j=1}^{k-1}\log\frac{T_{k}(x)}{T_{j}(x)}\right]^{-1},$
(1)

where $T_{j}(x)$ is the Euclidean ($\ell_{2}$) distance from $x$ to its
$j^{th}$ nearest neighbor. Levina & Bickel (2005) propose to average the local
estimates at each point to obtain a global estimate
$\bar{m}_{k}=\frac{1}{n}\sum_{i=1}^{n}\hat{m}_{k}(x_{i})$. MacKay & Ghahramani
(2005) suggestion a correction based on averaging of inverses

$\bar{m}_{k}=\left[\frac{1}{n}\sum_{i=1}^{n}\hat{m}_{k}(x_{i})^{-1}\right]^{-1}=\left[\frac{1}{n(k-1)}\sum_{i=1}^{n}\sum_{j=1}^{k-1}\log\frac{T_{k}(x_{i})}{T_{j}(x_{i})}\right]^{-1},$
(2)

where $n$ is the number of samples. We use Equation (2) as our MLE estimator
throughout this paper.

Since the geometry of natural images is complex and unknown, we face two
challenges when verifying the accuracy of MLE on natural image datasets.
First, we need to choose a proper value of $k$. As shown by MacKay &
Ghahramani (2005), the positive bias of the corrected estimator Equation (2)
increases as $k$ increases, but the variance decreases. In order to navigate
this bias-variance tradeoff, we try various values of $k$ in Section 4.
Second, in addition to the aforementioned local uniformity assumption, MLE
assumes that data arises as a sequence of i.i.d. random variables which can be
written as a continuous and sufficiently smooth function of a random variable
with smooth density, which may or may not be true for natural image datasets.
While the truth of these assumptions is unknown on natural images, we verify
the accuracy of our MLE estimates in a controlled setting in the following
section.

We briefly discuss other notable techniques for dimensionality estimation.
GeoMLE (Gomtsyan et al., 2019) attempts to account for non-uniformity of
density and nonlinearity of manifold using a polynomial regression of standard
MLE based on distances to nearest neighbors in different sized neighborhoods.
However, GeoMLE chooses to approximate averages of $\hat{m}_{k}(x_{i})$,
instead of averaging its reciprocal like Equation (2), resulting in a
potentially wrong maximum likelihood estimator. As a result, we find its
estimation deviates significantly from expected dimensionalities. TwoNN (Facco
et al., 2017) is based on the ratio of the distances to the first and second
nearest neighbors. Finally, the approach of (Granata & Carnevale, 2016)
considers the distribution of geodesic distances over the data manifold,
approximated by distances through kNN graphs, compared to the distribution of
distances over hyperspheres of varying dimension. Unlike MLE, our preliminary
experiments suggest that these techniques do not provide reasonable estimates
for some natural and synthetic images which are key to this work; see Appendix
A.5 for further discussion.

##  4 Validating Dimension Estimation with Synthetic Data

Dimensionality estimates are often applied on “simple” manifolds or toy
datasets where the dimensionality is known, and so the accuracy of the methods
can be validated. Image manifolds, by contrast, are highly complex, may
contain many symmetries and modes, and are of unknown dimension. In principle,
there is no reason why MLE-based dimensionality estimates cannot be applied to
image datasets. However, because we lack knowledge of the exact dimensionality
of image datasets, we cannot directly verify that MLE-based dimensionality
estimates scale up to the complexity of image structures.

There is an inherent uncertainty in estimating the ID of a given dataset.
First, we cannot be sure if the dataset actually resembles a sampling of
points on or near a manifold. Second, there are typically no guarantees that
the sampling satisfies the conditions assumed by the ID estimators we are
using.

Towards a principled application of ID estimates in contexts of practical
relevance to deep learning, we begin by validating that MLE methods can
generate accurate dimensionality estimates for complex image data. We do this
by generating synthetic image datasets using generative models for which the
intrinsic dimensionality can be upper-bounded a priori. We believe such
validations are essential to put recent findings in perspective (Gong et al.,
2019; Ansuini et al., 2019).

#### GAN Images

We use the BigGAN variant with $128$ latent entries and outputs of size
$128\times 128\times 3$ trained on the ImageNet dataset (Deng et al., 2009).
Using this GAN, we generate datasets with a varying number of images, where we
fix most entries of the latent vectors to zero leaving only $\bar{d}$ free
entries to be chosen at random. As we increase the number of free entries, we
expect the intrinsic dimension to increase with $\bar{d}$ as an upper bound;
see Section A.1 for further discussion.

Figure 2:  Visualization of basenji GAN samples of varying intrinsic
dimension.

In particular, we create several synthetic datasets of varying intrinsic
dimensionality using the ImageNet class, basenji, and check if the estimates
match our expectation. As seen in Figure 2, we observe increasing diversity
with increasing intrinsic dimension. In Figure 3, we show convergence of the
MLE estimate on basenji data with dimension bounded above by $\bar{d}=10$. We
observe that the estimates can be sensitive to the choice of $k$ as discussed
in prior work; see Appendix A.2 for additional GAN classes.

Figure 3:  Validation of MLE estimate on synthetic basenji data with
$\bar{d}=10$ free entries. We observe the estimates to converge around the
expected dimensionality of 10. Standard errors plotted with $N=5$ replicates
over random samples of the data.

#### Scaling to large datasets.

We develop a practical approach for estimating the ID of large datasets such
as ImageNet. In this approach, we randomly select a fraction $\alpha$ of the
dataset as anchors. Then, we evaluate the MLE estimate using only the anchor
points, where nearest-neighbors are computed over the entire dataset. Note
that, when anchors are chosen randomly, this acceleration has no impact on the
expected value of the result. See Appendix A.3 for an evaluation of this
approach.

##  5 The Intrinsic Dimension of Popular Datasets

In this section, we measure the intrinsic dimensions of a number of popular
datasets including MNIST (Deng, 2012), SVHN (Netzer et al., 2011), CIFAR-10
and CIFAR-100 (Krizhevsky et al., 2009), ImageNet (Deng et al., 2009), MS-COCO
(Lin et al., 2014), and CelebA (Liu et al., 2015). Using three different
parameter settings for the MLE ID estimator, we find that the ID is indeed
much smaller than the number of pixels; see Table 1. Notice that the rank
order of datasets by dimension does not depend on the choice of $k$. A
comparison of state-of-the-art (SOTA) classification accuracy on each
respective dataset111Values from https://paperswithcode.com/task/image-
classification. with the dimension estimates suggests a negative correlation
between the intrinsic dimension and test accuracy. In the next section, we
take a closer look at this phenomenon through a series of dedicated
experiments.

Dataset | MNIST | SVHN | CIFAR-100 | CelebA | CIFAR-10 | MS-COCO | ImageNet
---|---|---|---|---|---|---|---
MLE ($k$=3) | 7 | 9 | 11 | 9 | 13 | 22 | 26
MLE ($k$=5) | 11 | 14 | 18 | 17 | 21 | 33 | 38
MLE ($k$=10) | 12 | 18 | 22 | 24 | 25 | 37 | 43
MLE ($k$=20) | 13 | 19 | 23 | 26 | 26 | 36 | 43
SOTA Accuracy | 99.84 | 99.01 | 93.51 | - | 99.37 | - | 88.55
Table 1: The MLE estimates for practical image datasets, and the state-of-the-
art test-set image classification accuracy (for classification problems only)
for these datasets.

##  6 Intrinsic Dimension and Generalization

Learning theory work has established that learning a manifold requires a
number of samples that grows exponentially with the manifold’s intrinsic
dimension (Narayanan & Mitter, 2010), but the required number of samples is
independent of the extrinsic dimension. Specifically, the number of samples
needed to learn a well-conditioned decision boundary between two classes is
exponential in the intrinsic dimension of the manifold on which the classes
lie (Narayanan & Niyogi, 2009).

We leverage dimension estimation tools to empirically verify these theoretical
findings using a family of binary classification problems defined over both
synthetic and real datasets of varying intrinsic dimension. In these
experiments, we observe a connection between the intrinsic dimension of data
and generalization. Specifically, we find that classification problems on data
of lower intrinsic dimensionality are easier to solve.

###  6.1 Synthetic GAN data: Sample complexity depends on intrinsic (not
extrinsic) dimensionality

The synthetic GAN data generation technique described in Section 4 provides a
unique opportunity to test the relationship between generalization and the
intrinsic/extrinsic dimensionality of images. By creating datasets with
controlled intrinsic dimensionality, we may compare their sample complexity,
that is the number of samples required to obtain a given level of test error.
Specifically we test the following two hypotheses (1) data of lower intrinsic
dimensionality has lower sample complexity than that of higher intrinsic
dimensionality and (2) extrinsic dimensionality is irrelevant for sample
complexity.

To investigate hypothesis (1), we create four synthetic datasets of varying
intrinsic dimensionality: $16,32,64,128$, fixed extrinsic dimensionality:
$3\times 32\times 32$, and two classes: basenji and beagle. For each dataset
we fix a test set of size $N=1700$. For all experiments, we use the ResNet-18
($\text{width}=64$) architecture (He et al., 2016). We then train models until
they fit their entire training set with increasing amounts of training samples
and measure the test error. We show these results in Figure 4. Observing the
varying rates of growth, we see that data of higher intrinsic dimension
requires more samples to achieve a given test error.

For hypothesis (2), we carry out the same experiment with the roles of
intrinsic and extrinsic dimension switched. We create four synthetic datasets
of varying extrinsic dimensionality by resizing the images with nearest-
neighbor interpolation. Specifically we create 6 datasets of square, 3-channel
images of sizes $16,32,64,128,256$, fixed intrinsic dimensionality of size
$128$, and all other experimental details the same. We show these results in
Figure 5. Observing the lack of variable growth rates, we see that extrinsic
dimension has little to no effect on sample complexity.

To the best of our knowledge, this is the first experimental demonstration
that intrinsic but not extrinsic dimensionality matters for the generalization
of deep networks.

Figure 4:  Sample complexity of synthetic datasets of varying intrinsic
dimensionality. Figure 5:  Sample complexity of synthetic datasets of varying
extrinsic dimensionality.

###  6.2 Real Data: Intrinsic dimensionality matters for generalization

Next, we examine the sample complexity of binary classification tasks from
four common image datasets: MNIST, SVHN, CIFAR-10, and ImageNet. This case
differs from the synthetic case in that we have no control over each dataset’s
intrinsic dimension. Instead, we estimate it via the MLE method discussed in
Section 3. To account for variable difficulty of classes, we randomly sample
$5$ class pairs from each dataset and run the previously described sample
complexity experiment. Note that these subsets differ from those used in Table
1, where the estimates are taken from the entire dataset and across all
classes.

On these sampled subsets, we find the MLE estimates as shown in Table 2. Note
that these estimates are consistent with expectation, e.g. MNIST is
qualitatively simpler then SVHN or CIFAR-10.

We conduct the same sample complexity experiment as the previous section on
the datasets. Because these datasets are ordinarily of varying extrinsic
dimensionality, we resize all to size $32\times 32\times 3$ (before applying
MLE). We report results in Figure 6, where we overall observe trends ordered
by intrinsic dimensionality estimate. These results are consistent with
expectation of the relative hardness of each dataset. However, there are some
notable differences from the synthetic case. Several unexpected cross-over
points exist in the low-sample regime, and the gap between SVHN and CIFAR-10
is smaller than one may expect based on their estimated intrinsic dimension.

From these observations we conclude that intrinsic dimensionality is indeed
relevant to generalization on real data, but it is not the only feature of
data that influences sample complexity.

| MNIST | SVHN | CIFAR-10 | ImageNet
---|---|---|---|---
$k=3$ | 7.5 (0.2) | 8.5 (0.1) | 11.4 (0.2) | 15.4 (0.8)
$k=4$ | 9.8 (0.3) | 11.6 (0.1) | 15.9 (0.2) | 19.8 (0.9)
$k=5$ | 10.9 (0.4) | 13.2 (0.1) | 18.3 (0.3) | 21.6 (1.0)
Table 2: Mean and standard error of estimated intrinsic dimensions for
practical datasets under the same resolution $32\times 32\times 3$ using
different $k$’s. These results are consistent with the test accuracies on
these datasets under the same resolution. Figure 6:  Sample complexity of real
datasets. Standard errors are shown $N=5$ class pairs.

###  6.3 Real data: Adding noise changes dimensionality to affect
generalization

In this section, we examine an alternative technique for changing the
intrinsic dimension of a real dataset: adding noise to images. Here we
leverage the fact that uniformly sampled noise in $[0,1]^{\underline{d}}$ has
dimension $\underline{d}$. We thus add independent noise, drawn uniformly from
a fixed randomly oriented $\underline{d}$-dimensional unit hypercube embedded
in pixel space, to each sample in a dataset. This procedure ensures that the
dataset has dimension at least $\underline{d}$. Since the natural data we use
has low dimension, and the hypercubes have high dimension, this procedure
specifically increases dimensionality. We note that estimation error may occur
when there is an insufficient number of samples to achieve a proper estimate.
Since the variation in images in a dataset may still be dominated by non-noise
directions, we expect to underestimate the new increased dimensions of these
noised datasets.

Starting with CIFAR-10 data, we add noise of varying dimensions, where we
replace pixels at random in the image. We only add noise to an image once to
keep the augmented dataset the same size as the original. We use the following
noise dimensionalities: $256,512,1024,2048,2560$. The estimated dimensions of
the noised datasets are listed in Table 3. We see that intrinsic dimension
increases with increasing noise dimensionality, but dimensionality does not
saturate to the maximum true dimension, likely due to a poverty of samples.

On these noisy CIFAR-10 datasets, we again carry out the sample complexity
experiment of the previous sections. We show results in Figure 7. We observe
sample complexity largely in the same order as intrinsic dimension.

| $\underline{d}=256$ | $\underline{d}=512$ | $\underline{d}=1024$ | $\underline{d}=1536$ | $\underline{d}=2048$ | $\underline{d}=2560$
---|---|---|---|---|---|---
$k=3$ | 19.7 | 30.9 | 57.1 | 77.8 | 110.0 | 136.1
$k=4$ | 25.2 | 39.1 | 72.8 | 101.3 | 142.1 | 177.7
$k=5$ | 27.6 | 42.5 | 78.3 | 110.2 | 153.4 | 196.6
Table 3: Estimated intrinsic dimension of the CIFAR-10 dataset after adding
different dimensions ($d$) of uniformly sampled noise using different $k$’s.
The estimated dimension consistently increases with $d$ under different $k$’s.
Figure 7:  Sample complexity of noisy datasets. Standard errors are shown
$N=5$ random subsets of the data.

###  6.4 Manipulating the intrinsic dimensionality of FONTS

In this section, we describe a final technique for studying the effect of
intrinsic dimensionality on sample complexity on the recently proposed FONTS
dataset (Stutz et al., 2019). Beginning with a collection of characters and
font types, termed a prototype set by the authors, FONTS datasets are
constructed using a fixed set of data augmentations: scaling, translation,
rotation, and sheering. In principle, these augmentations each increase the
intrinsic dimension of the prototype set allowing us to synthetically alter
the intrinsic dimension by varying the number of augmentations used.

We construct 5 FONTS datasets in this way, FONTS-$\\{0,1,2,3,4\\}$, where the
suffix denotes the number of transformations used in the data generation
process. The MLE estimates on each of the datasets are given in Table 4.

Consistent with expectation, we observe that MLE methods consistently resolve
the increased dimesionality of transformed datasets. Carrying out the sample
complexity experiment of the previous section, we report results in Figure 8.
We observe again that, on the whole, sample complexity is ordered by intrinsic
dimension.

| FONTS-0 | FONTS-1 | FONTS-2 | FONTS-3 | FONTS-4
---|---|---|---|---|---
$k=3$ | 1.8 (0.1) | 3.8 (0.1) | 5.1 (0.2) | 5.8 (0.2) | 6.1 (0.3)
$k=4$ | 2.7 (0.1) | 5.2 (0.1) | 6.9 (0.2) | 7.8 (0.3) | 8.3 (0.4)
$k=5$ | 3.2 (0.1) | 5.9 (0.1) | 7.8 (0.3) | 8.8 (0.4) | 9.4 (0.4)
Table 4: Mean and standard errror of MLE estimates on the FONTS dataset under
different number of transforms using different $k$’s. Again, the ranks of the
estimated IDs are consistent, and the estimated IDs increase with the number
of transforms. Figure 8:  Sample complexity of FONTS datasets.

##  7 Discussion

In this work, we measure the intrinsic dimension of popular image datasets and
show that the intrinsic dimension of data matters for deep learning. While
there may be many factors, such as class separation and the number of classes,
which determine generalization, we build the case that intrinsic dimension is
one of these important factors. Along the way, we introduce a technique for
using GANs to synthesize data while manipulating dimensionality. This
technique is useful not only for validating dimension estimation methods but
also for examining the learning behavior of neural networks under a dimension-
controlled environment. In addition to synthetic data, we verify that
dimension plays a large role in learning on natural data. Our results support
the commonly held belief that low dimensional structure underpins the success
of deep learning on high-resolution data.

These findings raise a number of salient directions for future work. Methods
for enhancing neural network learning on high-dimensional data could improve
generalization on hard vision problems. To this end, a deeper understanding of
the mechanisms by which dimensionality plays a role in learning may enable
such methods. Additionally, our work indicates that different dimensionality
estimation tools are better suited for different settings. Future work on
computing tighter and more reliable estimates specific to image data would
allow the community to more precisely study the relationship between the
dimensionality of image datasets and learning.

## Acknowledgements

This work was supported by the DARPA GARD and DARPA QED programs. Further
support was provided by the AFOSR MURI program, and the National Science
Foundation’s DMS division. Computation resources were funded by the Sloan
Foundation.

## References

  * Ansuini et al. (2019) Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan.  Intrinsic dimension of data representations in deep neural networks.  In _Advances in Neural Information Processing Systems_ , pp. 6111–6122, 2019.
  * Besold & Spokoiny (2019) Franz Besold and Vladimir Spokoiny.  Adaptive manifold clustering.  _arXiv preprint arXiv:1912.04869_ , 2019.
  * Brand (2003) Matthew Brand.  Charting a manifold.  In _Advances in neural information processing systems_ , pp. 985–992, 2003.
  * Brendel & Bethge (2019) Wieland Brendel and Matthias Bethge.  Approximating cnns with bag-of-local-features models works surprisingly well on imagenet.  _arXiv preprint arXiv:1904.00760_ , 2019.
  * Brock et al. (2018) Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale GAN training for high fidelity natural image synthesis.  _arXiv preprint arXiv:1809.11096_ , 2018.
  * Carlsson et al. (2008) Gunnar Carlsson, Tigran Ishkhanov, Vin de Silva, and Afra Zomorodian.  On the local behavior of spaces of natural images.  _International Journal of Computer Vision_ , 76(1):1–12, 2008.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  ImageNet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pp. 248–255. Ieee, 2009.
  * Deng (2012) Li Deng.  The MNIST database of handwritten digit images for machine learning research [best of the web].  _IEEE Signal Processing Magazine_ , 29(6):141–142, 2012.
  * Desbrun et al. (2002) Mathieu Desbrun, Mark Meyer, and Pierre Alliez.  Intrinsic Parameterizations of Surface Meshes.  _Computer Graphics Forum_ , 2002.
  * Donoho & Grimes (2005) David L. Donoho and Carrie Grimes.  Image manifolds which are isometric to Euclidean space.  _Journal of Mathematical Imaging and Vision_ , 23(1):5–24, 2005.
  * Facco et al. (2017) Elena Facco, Maria d’Errico, Alex Rodriguez, and Alessandro Laio.  Estimating the intrinsic dimension of datasets by a minimal neighborhood information.  _Scientific Reports_ , 7(1):12140, 2017.
  * Fefferman et al. (2016) Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan.  Testing the manifold hypothesis.  _Journal of the American Mathematical Society_ , 29(4):983–1049, 2016.
  * Fodor (2002) Imola K Fodor.  A survey of dimension reduction techniques.  Technical report, Lawrence Livermore National Lab., CA (US), 2002.
  * Geirhos et al. (2018) Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel.  Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness.  _arXiv preprint arXiv:1811.12231_ , 2018.
  * Gomtsyan et al. (2019) Marina Gomtsyan, Nikita Mokrov, Maxim Panov, and Yury Yanovich.  Geometry-aware maximum likelihood estimation of intrinsic dimension.  In _Asian Conference on Machine Learning_ , pp. 1126–1141, 2019\.
  * Gong et al. (2019) Sixue Gong, Vishnu Naresh Boddeti, and Anil K Jain.  On the intrinsic dimensionality of image representations.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 3987–3996, 2019.
  * Gonzalez & Balajewicz (2018) Francisco J Gonzalez and Maciej Balajewicz.  Deep convolutional recurrent autoencoders for learning low-dimensional feature dynamics of fluid systems.  _arXiv preprint arXiv:1808.01346_ , 2018.
  * Goodfellow et al. (2016) Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio.  _Deep learning_ , volume 1.  MIT press Cambridge, 2016.
  * Granata & Carnevale (2016) Daniele Granata and Vincenzo Carnevale.  Accurate estimation of the intrinsic dimension using graph distances: Unraveling the geometric complexity of datasets.  _Scientific reports_ , 6:31377, 2016.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * Hinton & Salakhutdinov (2006) Geoffrey E Hinton and Ruslan R Salakhutdinov.  Reducing the dimensionality of data with neural networks.  _science_ , 313(5786):504–507, 2006.
  * Huang et al. (2019) W. Ronny Huang, Zeyad Emam, Micah Goldblum, Liam Fowl, Justin K. Terry, Furong Huang, and Tom Goldstein.  Understanding generalization through visualizations, 2019.
  * Kim et al. (2019) Jisu Kim, Alessandro Rinaldo, and Larry Wasserman.  Minimax Rates for Estimating the Dimension of a Manifold.  _Journal of Computational Geometry_ , 10(1), 2019.
  * Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  University of Toronto, 2009.  Master’s thesis.
  * Laughlin (2014) Daniel C. Laughlin.  The intrinsic dimensionality of plant traits and its relevance to community assembly.  _Journal of Ecology_ , 102(1):186–193, 2014.
  * Lee et al. (2003) Ann B. Lee, Kim S. Pedersen, and David Mumford.  The nonlinear statistics of high-contrast patches in natural images.  _International Journal of Computer Vision_ , 54(1):83–103, 2003.
  * Levina & Bickel (2005) Elizaveta Levina and Peter J Bickel.  Maximum likelihood estimation of intrinsic dimension.  In _Advances in neural information processing systems_ , pp. 777–784, 2005.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft COCO: Common objects in context.  In _European conference on computer vision_ , pp. 740–755. Springer, 2014.
  * Liu et al. (2015) Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 3730–3738, 2015.
  * MacKay & Ghahramani (2005) David J.C. MacKay and Zoubin Ghahramani.  Comments on ‘Maximum Likelihood Estimation of Intrinsic Dimension’ by E. Levina and P. Bickel (2004), 2005.  URL http://www.inference.org.uk/mackay/dimension/.
  * Narayanan & Mitter (2010) Hariharan Narayanan and Sanjoy Mitter.  Sample complexity of testing the manifold hypothesis.  In _Advances in neural information processing systems_ , pp. 1786–1794, 2010.
  * Narayanan & Niyogi (2009) Hariharan Narayanan and Partha Niyogi.  On the sample complexity of learning smooth cuts on a manifold.  In _COLT_ , 2009.
  * Netzer et al. (2011) Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng.  Reading digits in natural images with unsupervised feature learning.  In _NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011_ , 2011.  URL http://ufldl.stanford.edu/housenumbers/nips2011_housenumbers.pdf.
  * Olshausen & Field (1996) Bruno A Olshausen and David J Field.  Natural image statistics and efficient coding.  _Network: Computation in Neural Systems_ , 7(2):333–339, 1996.
  * Peyré (2009) Gabriel Peyré.  Manifold models for signals and images.  _Computer Vision and Image Understanding_ , 113(2):249 – 260, 2009.
  * Roweis & Saul (2000) Sam T Roweis and Lawrence K Saul.  Nonlinear dimensionality reduction by locally linear embedding.  _science_ , 290(5500):2323–2326, 2000.
  * Ruderman (1994) Daniel L Ruderman.  The statistics of natural images.  _Network: Computation in Neural Systems_ , 5(4):517–548, 1994.
  * Sard (1942) Arthur Sard.  The measure of the critical values of differentiable maps.  _Bulletin of the American Mathematical Society_ , 48(12):883–890, 1942.
  * Schölkopf et al. (1998) Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller.  Nonlinear component analysis as a kernel eigenvalue problem.  _Neural computation_ , 10(5):1299–1319, 1998\.
  * Stutz et al. (2019) David Stutz, Matthias Hein, and Bernt Schiele.  Disentangling adversarial robustness and generalization.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE Computer Society, 2019.
  * Tenenbaum et al. (2000) Joshua B Tenenbaum, Vin De Silva, and John C Langford.  A global geometric framework for nonlinear dimensionality reduction.  _science_ , 290(5500):2319–2323, 2000.
  * Vacher & Coen-Cagli (2019) Jonathan Vacher and Ruben Coen-Cagli.  Combining mixture models with linear mixing updates: multilayer image segmentation and synthesis.  _arXiv preprint arXiv:1905.10629_ , 2019.
  * Vacher et al. (2020) Jonathan Vacher, Aida Davila, Adam Kohn, and Ruben Coen-Cagli.  Texture interpolation for probing visual perception.  _arXiv preprint arXiv:2006.03698_ , 2020.
  * Valle & Oganov (2010) Mario Valle and Artem R. Oganov.  Crystal fingerprint space – a novel paradigm for studying crystal-structure sets.  _Acta Crystallographica Section A_ , 66(5):507–517, 2010.
  * Vincent et al. (2010) Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou.  Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.  _Journal of machine learning research_ , 11(12), 2010.
  * Zhu et al. (2018) Wei Zhu, Qiang Qiu, Jiaji Huang, Robert Calderbank, Guillermo Sapiro, and Ingrid Daubechies.  LDMNet: Low dimensional manifold regularized neural networks.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2018.

##  Appendix A Validation of ID Estimates

In this section, we present additional discussion results and discussion
relevant to the ID estimation and related validation experiments in Section 4.

###  A.1 GAN properties

We devise a method for validating ID measurements in a controlled setting
using images generated by GANs. To justify this method, we first note that the
image of $\mathbb{R}^{d}$ under a locally Lipschitz function can be a manifold
with dimension at most $d$. Then, consider that the BigGAN generator, a
convolutional neural network with ReLU activations, is a function with this
property (Brock et al., 2018).

Specifically, BigGAN can be written as a composition of linear functions,
translations, and ReLU activation functions. Individually, these operations do
not increase dimension, and by a composition property, their composition
cannot increase dimensionality either. The more general fact that the image of
$\mathbb{R}^{d}$ under a locally Lipschitz function can be a manifold with
dimension at most $d$ follows from Sard’s theorem (Sard, 1942).

###  A.2 Convergence for More GAN Classes

We include additional results on the estimation of ID for synthetic GAN images
from various ImageNet classes with $\bar{d}=10$ free entries out of the
128-dimensional latent vector input to the GAN; see Figures 10 and 11 below.
As observed earlier in Section 4, the MLE estimates are sensitive to the
choice of $k$, where we expect the ID to be close to $10$ given the way we
sample the latent vectors to use for the GAN. We note that for a number of
classes, all choices of $k$ we considered seem to underestimate the ID.

Figure 9:  Validation of MLE estimates on synthetic daisy data with
$\bar{d}=10$. Figure 10:  Validation of MLE estimate on synthetic soap-bubbles
data with $\bar{d}=10$. Figure 11:  Validation of MLE estimate on synthetic
coffee data with $10$ free entries. Note that the estimates do not converge
around the upper bound of $\bar{d}=10$, which suggests that data generated
from this class is not of full dimension.

###  A.3 Subsampling for Large Datasets

In Figure 12 we validate the anchor approximation on basenji data of dimension
10 for varying anchor ratio $\alpha$. Then, in Figure 13 we validate the
anchor approximation on tree-frog data with $\bar{d}=32$ for varying $k$ while
fixing the anchor ratio at $\alpha=0.001$.

Figure 12:  Validation of anchor approximation on basenji with $\bar{d}=10$.
Figure 13:  Validation of anchor approximation on tree-frog with $\bar{d}=32$
and $\alpha=0.001$.

###  A.4 Relationship between $k$ in the MLE method and dataset dimension

The choice of $k$ may affect the dimensionality estimates. To better
understand this relationship, we conducted additional studies for the MLE
method using synthetic GAN data generated as described in Section 4. Table 5
shows the estimation results for various $k$ and basenji data with
$\bar{d}\in\\{2,4,8,16,32,64,128\\}$. We use a fixed number of samples
($n=10000$). We observe that the estimated intrinsic dimension increases with
$k$, and large values can yield overestimates for ID. These results agree with
the work of Levina & Bickel (2005) who suggest that low values of $k$ may
produce an estimator with higher variance while higher values of $k$ produce
an estimator with higher positive bias. Given that we have access to large
amounts of synthetic data, we opt for lower values of $k$ in our work. We do
not choose a particular value of $k$, and we report all experiments with
multiple values.

$k$ | $\bar{d}$
---|---
| 2 | 4 | 8 | 16 | 32 | 64 | 128
3 | 1.1 | 2.6 | 6.1 | 10.5 | 16.0 | 20.0 | 20.0
4 | 1.5 | 3.6 | 8.2 | 14.0 | 21.0 | 26.0 | 26.0
5 | 1.7 | 4.1 | 9.3 | 15.7 | 23.5 | 28.7 | 28.5
6 | 1.8 | 4.4 | 9.9 | 16.6 | 24.9 | 30.3 | 29.9
7 | 1.9 | 4.6 | 10.4 | 17.2 | 25.8 | 31.2 | 30.6
8 | 1.9 | 4.7 | 10.7 | 17.6 | 26.4 | 31.7 | 31.1
9 | 2.0 | 4.9 | 10.9 | 18.0 | 26.8 | 31.9 | 31.5
10 | 2.0 | 5.0 | 11.1 | 18.2 | 27.1 | 32.1 | 31.7
15 | 2.1 | 5.3 | 11.6 | 18.8 | 27.8 | 32.3 | 31.7
20 | 2.2 | 5.5 | 11.8 | 19.0 | 27.9 | 31.9 | 31.3
25 | 2.2 | 5.7 | 12.0 | 19.2 | 27.9 | 31.5 | 30.8
Table 5:  Additional results on the relationship between $k$ and MLE
dimensionality estimate for synthetic basenji images with varying $\bar{d}$.

###  A.5 Comparing MLE to Other Estimators in a Controlled Setting

To validate MLE in comparison to other estimators, we evaluate three other
dimensionality estimation methods: GeoMLE (Gomtsyan et al., 2019), TwoNN
(Facco et al., 2017) and kNN graph distances (Granata & Carnevale, 2016). For
GeoMLE, we sample a total of 20 bootstrap subsets ($M=20$) and use
$k_{1}=20,k_{2}=55$ as recommended by Gomtsyan et al. (2019). To extend the
kNN graph distance method to large datasets, we randomly sample a subset of
samples (fixed to 10,000 for datasets with more than 10,000 samples) and use
shortest graph distances to their $k$ nearest neighbors to estimate the IDs.
Other settings are as default in the implementations of Granata & Carnevale
(2016).

First, we validate each method on datasets of uniformly sampled from
$d$-dimensional hypercubes. We report these results in Figure 14. Each method
works reasonable on low-dimensional cubes. We observed the Shortest Path
method to give erratic estimates on cubes of higher dimension, and have
omitted these. TwoNN has poor sample efficiency for higher dimensional cubes.
Interestingly, GeoMLE estimates these high dimensional cubes well.

Next, we report estimation results on basenji_10 for TwoNN, kNN graph
distance, and GeoMLE methods in Figure 15. Comparing against the MLE results
in Figure 3, we observe that each other method does not achieve an accurate
estimate in this sample regime, thus motivating our focus on MLE. Notably,
GeoMLE and TwoNN severely overestimate dimension, while the kNN graph distance
method severely underestimates dimension.

On MNIST, CIFAR-10, CIFAR-100, and SVHN, the results of these other estimation
methods also deviate from expectation (Table 6). For example, TwoNN assigns a
significantly higher dimension estimate to MNIST than to CIFAR-100, which
contradicts both intuition and the results of other estimators. We set $k=4$
and number of bins to 1000, and use default settings for all other parameters
including $r_{\text{MAX}}$.

Figure 14:  The TwoNN, Shortest-Path, and GeoMLE methods on $d$-dimensional Hypercube data. Each method estimates low-dimensional cubes well, validating their implementation. Figure 15:  The TwoNN, Shortest-Path, and GeoMLE methods on basenji_10 data. The estimates do not converge around the expected value of $\bar{d}=10$ in this sample regime. Dataset | MNIST | CIFAR-10 | CIFAR-100 | SVHN
---|---|---|---|---
MLE ($k=5$) | 11 | 21 | 18 | 14
GeoMLE ($k_{1}=20$,$k_{2}=55$) | 25 | 96 | 93 | 21
TwoNN | 15 | 11 | 9 | 7
kNN Graph Distance | 7 | 7 | 8 | 6
Table 6: Additional ID estimators on popular datasets.


# Optical microcavities as platforms for entangled photon spectroscopy

Ravyn Malatesta  School of Chemistry and Biochemistry and School of Physics,
Georgia Institute of Technology, Atlanta, Georgia 30332, USA.  Lorenzo Uboldi
Department of Physics and Center for Functional Materials, Wake Forest
University, 2090 Eure Drive, Winston Salem, North Carolina 27109, USA.
Dipartimento di Fisica, Politecnico di Milano, Piazza Leonardo da Vinci 32,
Milano, Italy.  Evan J. Kumar  Department of Physics and Center for Functional
Materials, Wake Forest University, 2090 Eure Drive, Winston Salem, North
Carolina 27109, USA.  Esteban Rojas-Gatjens  School of Chemistry and
Biochemistry and School of Physics, Georgia Institute of Technology, Atlanta,
Georgia 30332, USA.  Department of Physics and Center for Functional
Materials, Wake Forest University, 2090 Eure Drive, Winston Salem, North
Carolina 27109, USA.  Luca Moretti  Dipartimento di Fisica, Politecnico di
Milano, Piazza Leonardo da Vinci 32, Milano, Italy.  Andy Cruz  Department of
Physics, City College of New York, City University of New York, NY 10031, USA.
Vinod Menon  Department of Physics, City College of New York, City University
of New York, NY 10031, USA.  Giulio Cerullo  Dipartimento di Fisica,
Politecnico di Milano, Piazza Leonardo da Vinci 32, Milano, Italy.  Ajay Ram
Srimath Kandada<EMAIL_ADDRESS>Department of Physics and Center for
Functional Materials, Wake Forest University, 2090 Eure Drive, Winston Salem,
North Carolina 27109, USA.

###### Abstract

Optical microcavities are often proposed as platforms for spectroscopy in the
single- and few-photon regime due to strong light-matter coupling. For
classical-light spectroscopies, an empty microcavity simply acts as an optical
filter. However, we find that in the single- or few-photon regime treating the
empty microcavity as an optical filter does not capture the full effect on the
quantum state of the transmitted photons. Focusing on the case of entangled
photon-pair spectroscopy, we consider how the propagation of one photon
through an optical microcavity changes the joint spectrum of a frequency-
entangled photon pair. Using the input-output treatment of a Dicke model, we
find that propagation through a strongly coupled microcavity above a certain
coupling threshold enhances the entanglement entropy between the signal and
idler photons. These results show that optical microcavities are not neutral
platforms for quantum-light spectroscopies and their effects must be carefully
considered when using change in entanglement entropy as an observable.

††preprint: APS/123-QED

##  I Introduction

Due to spectacular advances within the field of quantum optics,
experimentalists can now control non-classical states of light with high
levels of precision in optics laboratories. Combined with advances in single-
photon detection, these innovations lay the groundwork for the growing field
of quantum-light spectroscopy[1]. There are many advantages to using quantum-
light for spectroscopy, including access to information otherwise inaccessible
using classical spectroscopies[2][3][4], and importantly superior signal-to-
noise ratio that can enable spectroscopy at extremely low excitation fluence.

Quantum light refers to any state of light that cannot be described
classically, such as single photons, squeezed light, or entangled photon
pairs. Entangled photon pairs exhibit non-classical correlations that provide
an advantage for both linear and nonlinear spectroscopies[5][6][7][8][9]. In
the single- or few-photon regime, classical spectroscopic signals are swamped
with noise but entanglement-enhanced spectroscopies can surpass the shot-noise
limit by taking advantage of quantum correlations[10][11]. Similarly,
entangled light can enhance signal-to-noise ratios of nonlinear
spectroscopies, resulting in sharper spectroscopic features and greater
simultaneous time-frequency resolution[12]. Furthermore, entangled-photon
pairs provide direct access to nonlinear processes even at low-level
excitation, facilitating the study of nonlinear processes in photo-sensitive
systems that might bleach or otherwise be destroyed at excitation powers[13].
Hao Li et al. describe theoretically how the entanglement entropy of biphoton
states (photon pair states) can be used as a probe of many-body correlations
that are often elusive or obscured in classical nonlinear spectroscopic
measurements[14].

In the single- or few-photon regime, a challenge arises for spectroscopists
because of the low proability of light-matter interactions. One popular method
to address this problem is to use an optical microcavity to couple to optical
excitations in materials and thus enhance the processes of
interest[15][16][17][18]. Optical microcavities are extremely controllable
platforms for light-matter interaction; they are used to manipulate molecular
states, enhance spontaneous emission, and drive chemical reactions[19]. For
all of their uses, microcavities are an extremely versatile platform for
quantum spectroscopy, but they are not neutral platforms and cannot be treated
as such.

To demonstrate this, we consider how the joint spectrum and entanglement
entropy of a frequency-entangled biphoton state changes after one photon (the
idler) propagates through an optical microcavity. We first briefly describe
the modeling of the biphoton joint spectrum and its transformation using
input-output theory. We then consider an empty microcavity. Although for
classical light an empty microcavity behaves as a simple optical filter, we
find experimentally that treating the microcavity as an optical filter does
not capture the full effect on the transmitted biphoton state. With simple
input-output theory, we can model the filtering effect of the empty
microcavity but cannot explain the full joint spectral transformation. We next
move on to a simple model of a microcavity coupled to $N$ two-level systems
and consider how strong-coupling transforms the joint spectrum and
entanglement entropy of the biphoton state after the idler passes through an
active microcavity. We find that above a certain coupling strength, passing
through the microcavity system alone, regardless of detuning, enhances the
entanglement entropy even without including many-body interactions in the
model. These results confirm that optical microcavities are not neutral
platforms for quantum-light spectroscopies and their effects must be carefully
considered when using change in entanglement entropy as a spectroscopic
observable.

##  II Biphoton state transformation

Figure 1: Transmission function, joint spectral intensity, and applied phase
shift of a (a) one-sided and (b) two-sided empty microcavity following
Gardiner and Collett.

###  II.1 Joint spectrum and entanglement

Sources of entangled photons that are based on spontaneous parametric
downconversion (SPDC) generate two daughter photons, historically called the
signal and idler, from a single pump photon according to energy- and momentum-
conservation. Following the development of Zielnicki et al.[20], a generic
biphoton state of a signal and idler pair can be written as

$\displaystyle\ket{\psi_{s,i}}=\int\int
d\omega_{s}d\omega_{i}\mathcal{F}\left(\omega_{s},\omega_{i}\right)a_{1}^{{\dagger}}(\omega_{s})a_{2}^{{\dagger}}(\omega_{i})\ket{0},$
(1)

where the creation operators $a_{1}^{{\dagger}}(\omega_{s})$ and
$a_{2}^{{\dagger}}(\omega_{i})$ operate on the vacuum state to create photons
at frequency $\omega_{s}$ and $\omega_{i}$, respectively. The joint spectral
amplitude, $\mathcal{F}\left(\omega_{s},\omega_{i}\right)$, describes the
frequency-correlations between the signal and idler photons.

Experimentally, we typically measure the joint spectral intensity (JSI),

$\displaystyle|\mathcal{F}\left(\omega_{s},\omega_{i}\right)|^{2}=|A\left(\omega_{s},\omega_{i}\right)|^{2}|\Phi\left(\omega_{s},\omega_{i}\right)|^{2},$
(2)

where $A\left(\omega_{s},\omega_{i}\right)$ is based on the spectral amplitude
of the pump beam and $\Phi\left(\omega_{s},\omega_{i}\right)$ is determined by
the phase-matching conditions and spatial profile of the pump.

To quantify the entanglement between the signal and idler photons, we compute
the von Neumann entanglement entropy, $S$. We first normalize the joint
spectral amplitude, and then use singular value decomposition to find the
Schmidt coefficients $\lambda_{j}$ which satisfy the normalization condition
$\sum_{j}\lambda_{j}^{2}=1$. We then calculate the entanglement entropy as

$\displaystyle S=-\sum_{i}\lambda_{i}^{2}\ln\left(\lambda_{i}^{2}\right).$ (3)

###  II.2 Application of input-output theory

In their seminal work in 1984 [21], Collett and Gardiner develop a general
input-ouput theory that relates output operators to input operators via
internal dynamics of a cavity system governed by quantum Langevin equations.
For a single photon mode, if we express the input-output transformation as a
frequency-dependent function $C(\omega_{i})$, we simply write the output
creation operator in terms of the input as

$\displaystyle\tilde{a}^{{\dagger}}(\omega_{i})=C\left(\omega_{i}\right)a^{{\dagger}}(\omega_{i}).$
(4)

Now, considering the case of a biphoton state where we allow only the idler
photon to propagate through a microcavity system, we replace the original
idler creation operator
$a_{2}^{{\dagger}}(\omega_{i})\rightarrow\tilde{a}_{2}^{{\dagger}}(\omega_{i})$
to get the transformed biphoton state

$\displaystyle\ket{\Psi}=\int\int
d\omega_{s}d\omega_{i}\mathcal{F}\left(\omega_{s},\omega_{i}\right)C\left(\omega_{i}\right)a_{1}^{{\dagger}}(\omega_{s})a_{2}^{{\dagger}}(\omega_{i})\ket{0}.$
(5)

In this simple approach, the transformed JSA is the product
$\mathcal{F}\left(\omega_{s},\omega_{i}\right)\cdot C\left(\omega_{i}\right)$.
The transformed state bears similarity to the expression developed by
Kalashnikov et al. before they act with a beamsplitter to see how interaction
with a resonant medium changes the quantum interference pattern in Hong-Ou-
Mandel interferometry[22].

##  III Propagation through an empty microcavity

###  III.1 Theory

Following the input-output formalism of Collett and Gardiner[21], we consider
an empty cavity confining a single optical mode. Starting with a one-sided
empty microcavity, i.e. a microcavity with substantial loss through a single
mirror, the transformation of the output photon creation operator in terms of
the input is

$\displaystyle\tilde{a}^{\dagger}\left(\omega\right)=\frac{\frac{1}{2}\gamma-i\left(\omega-\omega_{0}\right)}{\frac{1}{2}\gamma+i\left(\omega-\omega_{0}\right)}a^{\dagger}\left(\omega\right),$
(6)

where $\gamma$ is the coupling strength of the cavity photons to input(output)
photons and $\omega_{0}$ is the frequency of the cavity mode. The coupling
strength $\gamma$ is directly related to the cavity photon lifetime,
$\tau=1/\gamma$. For all our simulations, we choose $\gamma$ such that the
cavity photon lifetime $\tau$ is 150 fs.

As noted by Collett and Gardiner, the one-sided cavity imposes a frequency-
dependent relative phase shift, but does not change the JSI. Therefore the JSI
shown in Fig. 1(a) is identical to that of the input biphoton state. For all
simulations shown here, we use the same input state, assuming a Gaussian pump
with a central down-converted wavelength of 685 nm for both signal and idler
photons. To replicate experimental conditions, we apply detection filters to
both the signal and idler. For the filter shape, we choose a Gaussian squared,
centered at 685 nm with an 8 nm bandwidth. Until we consider the effect of the
pump bandwidth on entanglement entropy, the pump bandwidth is set at 6 nm.

Next, we move on to a two-sided empty microcavity, i.e. a cavity with leaky
mirrors on both sides, and so with two input and two output modes. We assume
the coupling to be same for both mirrors, $\gamma_{1}=\gamma_{2}=\gamma$, and
a single input mode. Thus in transmission, the output photon creation operator
is

$\displaystyle\tilde{a}^{\dagger}\left(\omega\right)=\frac{\gamma}{\gamma+i\left(\omega-\omega_{0}\right)}a^{\dagger}\left(\omega\right).$
(7)

Now we see the filtering effect of the empty microcavity acting on the idler
photon, as shown in Fig. 1(b). The centering and bandwidth of the transmission
function are determined by the cavity mode frequency and the cavity photon
lifetime, respectively. The filtering effect of the microcavity slightly
affects the entanglement entropy bringing the base entanglement entropy from
$S=0.395$ to $S=0.359$.

Figure 2: Joint spectral intensity before and after propagating through an
empty microcavity from (a, b) experimental measurement and (c, d) input-output
theory.

###  III.2 Experiment

To further test the formalism developed in the previous section, we
experimentally measure the JSI of a biphoton state with one of the photons
transmitted through an empty microcavity. The spectrally entangled state is
generated in a Type-I $\beta$-Barium Borate (BBO) crystal phase-matched for
SPDC close to the degeneracy at the pump wavelength of 343 nm. The pump beam
here is the third harmonic of a femtosecond laser oscillator (Pharos, Light
Conversion) output operating at 1030 nm and 75 MHz. The photons are spatially
separated and transmitted through a translating-wedge-based identical pulses
encoding system (GEMINI, Nireos srl) and a co-incidence detection system
(Hydraharp, Picoquant), which enable measurement of spectral correlations
between the photons. More details of the measurement system can be found in
Ref. 23. The JSI spectrum of the as-prepared biphoton state is shown in Fig.
2(a), in which the spectral correlation between the signal and idler photons
is evident through the diagonal feature.

We transmit the idler photon of this state through a planar optical
microcavity, which is built on distributed Bragg reflectors (DBR) and has an
optical resonance at 691 nm with a full width at half maximum of 8 nm at
normal incidence. The JSI spectrum of the transmitted biphoton state is shown
in Fig. 2(b). We observe clear spectral filtering of the biphoton state with
the peak of the JSI map at the peak resonance of the cavity. On closer
inspection, we observe a reduction in the degree of spectral correlation in
the transmitted state with the previously extended diagonal feature flattening
along the idler-axis, close to the cavity resonance. To reproduce this
behavior we consider a biphoton state whose JSI spectrum follows Eq. 2, and
shown in Fig. 2(c). Based on the formalism developed in the previous section,
we estimate the JSI spectrum of the biphoton state whose idler photon is
transmitted through a microcavity. By setting the cavity resonance to 690 nm
in our simulation, we can approximate the experimentally measured transmission
function of the empty microcavity. While the filtering effect is reproduced,
we miss the effects of the microcavity that depend on the joint spectrum -
that is we miss the effects of the microcavity that depend simultaneously on
the signal and idler frequency even though only the idler propagated through
the microcavity.

Figure 3: Changes to the biphoton state due to idler propagation through a
strongly-coupled microcavity. (a) Transmission function, joint spectral
intensity, and applied phase shift of a strongly-coupled ($\lambda=\gamma$)
microcavity with a 150 fs lifetime with zero detuning, (b) dependence of the
microcavity induced phase shift on coupling strength, and (c) change in
entanglement entropy with coupling strength for variable cavity detuning for a
molecular resonance at 685 nm, compared to input state entropy (dotted gray).

##  IV Propagation through a strongly-coupled microcavity

Having already established that an empty microcavity has a non-trivial effect
on the entanglement of frequency-entangled photon pairs, we now consider a
simple model of a strongly-coupled microcavity system. Taking inspiration from
Li et al.[2], we use a Dicke model of $N$ identical 2-level emitters coupled
to an optical cavity described by the following Hamiltonian:

$\displaystyle\hat{H}$ $\displaystyle=$
$\displaystyle\sum_{j}\frac{\hbar\omega_{e}}{2}\hat{\sigma}_{z,j}+\sum_{k}\hbar(\omega_{k}-i\gamma)\hat{\psi}^{\dagger}_{k}\hat{\psi}_{k}$
(8)
$\displaystyle+\sum_{k,j}\frac{\hbar\lambda_{kj}}{\sqrt{N}}(\hat{\psi}^{\dagger}_{k}+\hat{\psi}_{k})(\hat{\sigma}^{+}_{j}+\hat{\sigma}^{-}_{j}),$

where $\omega_{e}$ is the frequency of the emitter,
$\\{\hat{\sigma}_{z,j},\hat{\sigma}_{j}^{\pm}\\}$ are the corresponding
spin-1/2 operators for site $j$, $\hat{\psi}^{\dagger}_{k}$ is the cavity
photon creation operator, and $\lambda_{kj}$ is the coupling between a cavity
photon and a molecular excitation at site $j$. As before, $\gamma$ is the
coupling of the cavity photon mode to an external photon mode. For simplicity,
we consider only the normal cavity mode, $k=0$, with frequency $\omega_{0}$.
We also constrain ourselves to the strong coupling regime, $\lambda>\gamma/2$,
but stay well below the critical point $\lambda_{c}$ where the system
undergoes a quantum phase transition.

Using an input-output treatment of this model, Li et al. develop an analytical
expression for the response function of the strongly-coupled system [2] which
we use to define the transformation function $C(\omega_{i})$ for an idler
photon propagating through the strongly-coupled microcavity. The
transformation function depends on many parameters: the frequency of the
molecular emitter $\omega_{e}$, the frequency of the cavity mode $\omega_{0}$,
the cavity lifetime $1/\gamma$, and the strength of the coupling $\lambda$.

Applying this transformation to the same input biphoton state as before (Fig.
2(c)), we immediately find much different behavior than for transmission
through an empty microcavity. We analyze transmission through a strongly-
coupled microcavity with both the molecular and cavity resonance at 685 nm, a
150 fs cavity photon lifetime, and equal coupling to the molecular excitation
and external photons ($\gamma=\lambda$). The resulting transmission spectrum
and the JSI map shown in Fig. 3(a) are composed of two peaks associated with
the lower and upper polariton states. While this is an expected result, two
intriguing details emerge on deeper analysis. Firstly, we see sharp
discontinuity in the applied phase shift corresponding to the molecular
resonance. As the strength of the light-matter coupling increases with respect
to the coupling to external photons, the applied phase shift begins to
resemble a step function, as shown in Fig. 3(b).

Secondly, the entanglement entropy of the transformed state is $S=0.437$,
which is a higher value than the entanglement entropy of the input state
$S=0.395$. But the increase in the entropy is curiously not monotonically
related to the strength of light-matter coupling. The transformation of the
biphoton state due to propagation of the idler photon through the strongly-
coupled microcavity system increases only above a certain coupling-strength
threshold, see Fig. 3(c). Below this threshold, the entropy substantially
reduces, possibly due to the spectral filtering of the idler photons by the
dominant molecular transition. Of course, the exact coupling strength at which
the enhanced entanglement entropy surpasses that of the input depends on the
specific state and microcavity system parameters including the molecular
resonance, cavity photon lifetime, and cavity detuning. In general, across
several cavity detunings, shown in Fig. 3(c), at the lower end of the strong-
coupling limit when coupling to external photons out-competes coupling to the
molecular excitation ($\gamma>\lambda>\gamma/2$), propagation through the
coupled microcavity system suppresses the entanglement entropy even below
propagation through an empty microcavity. As the coupling strength increases,
we reach a regime where the microcavity system improves the entanglement
entropy past that of the input state for a wide range of cavity-detuning
values, until the dependence of the entanglement entropy on cavity-detuning
plateaus.

Figure 4: Entanglement entropy by pump bandwidth for increasing coupling
strength compared to an empty microcavity (dashed black).

We find a similar ebb and flow of entropy improvement when considering how the
entanglement entropy changes with the bandwidth of the Gaussian pump
generating the input biphoton state, seen in Fig. 4. For very narrow pump
bandwidths, the frequencies of the signal and idler are strongly anti-
correlated and the input state thus has a relatively high entanglement
entropy. Within this limit, propagation of the idler through a sufficiently
strongly coupled microcavity system still improves the entanglement entropy.
Beyond $\lambda=1.35\gamma$, the benefit weakens but the entanglement entropy
remains firmly above that for an empty microcavity.

##  V Conclusion

In summary, we show with experiment and simple input-output theory that
propagation through empty optical microcavities exerts a non-trivial effect on
the state of frequency-entangled biphotons. We also theoretically consider the
case of strongly coupled microcavities and identify peculiar transformations
of the spectral correlations of the output biphoton state. From our
experimental measurements of an empty microcavity we expect there to be
further correlated effects in these systems that our simple input-output
approach does not capture, but understanding the interplay of the modeled and
unmodeled changes requires further theoretical development and
experimentation. Nevertheless, we show even with a simple theoretical
treatment that the microcavity platform has notable effects on entanglement
entropy of biphoton states.

Notably, the experimental configuration and the model we consider here simply
correspond to the linear response of microcavities. Previous works propose to
use the entanglement entropy of the biphoton state transmitted through the
cavity as a probe of many-body processes, including polariton-polariton
interactions. While these treatments show the biphoton entanglement entropy is
sensitive to such many-body interactions, we have to also consider the non-
trivial entropy changes identified in this work that can manifest even in the
absence of any correlating mechanisms. While optical microcavities can indeed
be excellent platforms that enable spectroscopy with entangled photons, care
must be taken to design systems with light-matter coupling strengths that
minimize the linear-response induced variations in the JSI, so that the
transformation of the biphoton state can be directly correlated with many-body
dynamics.

###### Acknowledgements.

A.R.S.K. acknowledges the start-up funds provided by Wake Forest University
and funding from the Center for Functional Materials and the Office of
Research and Sponsored Programs at WFU. The authors thank Prof Carlos Silva,
Prof Eric Bittner and Dr Andrei Piryatinski for insightful discussions. This
material is based upon work supported by the National Science Foundation
Graduate Research Fellowship under Grant No. DGE-2039655. Any opinion,
findings, and conclusions or recommendations expressed in this material are
those of the authorss and do not necessarily reflect the views of the National
Science Foundation.

## References

  * Mukamel  _et al._ [2020] S. Mukamel, M. Freyberger, W. Schleich, M. Bellini, A. Zavatta, G. Leuchs, C. Silberhorn, R. W. Boyd, L. L. Sánchez-Soto, A. Stefanov, _et al._ , Journal of physics B: Atomic, molecular and optical physics 53, 072002 (2020).
  * Li  _et al._ [2017] H. Li, A. Piryatinski, J. Jerke, A. R. S. Kandada, C. Silva, and E. R. Bittner, Quantum Science and Technology 3, 015003 (2017).
  * Cuevas  _et al._ [2018] Á. Cuevas, J. C. López Carreño, B. Silva, M. De Giorgi, D. G. Suárez-Forero, C. Sánchez Muñoz, A. Fieramosca, F. Cardano, L. Marrucci, V. Tasco, _et al._ , Science Advances 4, eaao6814 (2018).
  * Asban and Mukamel [2021] S. Asban and S. Mukamel, Science advances 7, eabj4566 (2021).
  * Gea-Banacloche [1989] J. Gea-Banacloche, Physical review letters 62, 1603 (1989).
  * Oka [2010] H. Oka, Physical Review A 81, 063819 (2010).
  * Szoke  _et al._ [2020] S. Szoke, H. Liu, B. P. Hickam, M. He, and S. K. Cushing, Journal of Materials Chemistry C 8, 10732 (2020).
  * Schlawin and Mukamel [2013] F. Schlawin and S. Mukamel, The Journal of chemical physics 139 (2013).
  * Dorfman  _et al._ [2016] K. E. Dorfman, F. Schlawin, and S. Mukamel, Reviews of Modern Physics 88, 045008 (2016).
  * Dorfman  _et al._ [2021] K. Dorfman, S. Liu, Y. Lou, T. Wei, J. Jing, F. Schlawin, and S. Mukamel, Proceedings of the National Academy of Sciences 118, e2105601118 (2021).
  * Srimath Kandada and Cerullo [2021] A. R. Srimath Kandada and G. Cerullo, Proceedings of the National Academy of Sciences 118, e2112897118 (2021).
  * Raymer  _et al._ [2013] M. G. Raymer, A. H. Marcus, J. R. Widom, and D. L. Vitullo, The Journal of Physical Chemistry B 117, 15559 (2013).
  * Kojima and Nguyen [2004] J. Kojima and Q.-V. Nguyen, Chemical physics letters 396, 323 (2004).
  * Li  _et al._ [2019] H. Li, A. Piryatinski, A. R. Srimath Kandada, C. Silva, and E. R. Bittner, The Journal of chemical physics 150 (2019).
  * Carnio  _et al._ [2021] E. G. Carnio, A. Buchleitner, and F. Schlawin, The Journal of Chemical Physics 154 (2021).
  * Levinsen  _et al._ [2019] J. Levinsen, F. M. Marchetti, J. Keeling, and M. M. Parish, Physical Review Letters 123, 266401 (2019).
  * Osaka  _et al._ [2014] Y. Osaka, N. Yokoshi, M. Nakatani, and H. Ishihara, Physical Review Letters 112, 133601 (2014).
  * Gu and Mukamel [2020] B. Gu and S. Mukamel, The Journal of Physical Chemistry Letters 11, 8177 (2020).
  * Flick  _et al._ [2017] J. Flick, M. Ruggenthaler, H. Appel, and A. Rubio, Proceedings of the National Academy of Sciences 114, 3026 (2017).
  * Zielnicki  _et al._ [2018] K. Zielnicki, K. Garay-Palmett, D. Cruz-Delgado, H. Cruz-Ramirez, M. F. O’Boyle, B. Fang, V. O. Lorenz, A. B. U’Ren, and P. G. Kwiat, Journal of Modern Optics 65, 1141 (2018).
  * Collett and Gardiner [1984] M. Collett and C. Gardiner, Physical Review A 30, 1386 (1984).
  * Kalashnikov  _et al._ [2017] D. A. Kalashnikov, E. V. Melik-Gaykazyan, A. A. Kalachev, Y. F. Yu, A. I. Kuznetsov, and L. A. Krivitsky, Scientific reports 7, 11444 (2017).
  * Moretti  _et al._ [2023] L. Moretti, E. Rojas-Gatjens, L. Uboldi, D. O. Tiede, E. J. Kumar, C. Trovatello, F. Preda, A. Perri, C. Manzoni, G. Cerullo, _et al._ , The Journal of Chemical Physics 159 (2023).