11institutetext: University of Cambridge, Department of Computer Science and
Technology, Cambridge CB3 0FD, United Kingdom
11email: {ct538, jz426<EMAIL_ADDRESS>

# Arbitrary Scale Super-Resolution for Brain MRI Images††thanks: Supported by
University of Cambridge.

Chuan Tan  Jin Zhu  Pietro Lio

###### Abstract

Recent attempts at Super-Resolution for medical images used deep learning
techniques such as Generative Adversarial Networks (GANs) to achieve
perceptually realistic single image Super-Resolution. Yet, they are
constrained by their inability to generalise to different scale factors. This
involves high storage and energy costs as every integer scale factor involves
a separate neural network. A recent paper has proposed a novel meta-learning
technique that uses a Weight Prediction Network to enable Super-Resolution on
arbitrary scale factors using only a single neural network. In this paper, we
propose a new network that combines that technique with SRGAN, a state-of-the-
art GAN-based architecture, to achieve arbitrary scale, high fidelity Super-
Resolution for medical images. By using this network to perform arbitrary
scale magnifications on images from the Multimodal Brain Tumor Segmentation
Challenge (BraTS) dataset, we demonstrate that it is able to outperform
traditional interpolation methods by up to 20$\%$ on SSIM scores whilst
retaining generalisability on brain MRI images. We show that performance
across scales is not compromised, and that it is able to achieve competitive
results with other state-of-the-art methods such as EDSR whilst being fifty
times smaller than them. Combining efficiency, performance, and
generalisability, this can hopefully become a new foundation for tackling
Super-Resolution on medical images.

###### Keywords:

Super-Resolution Medical Image Analysis Meta-Learning Image Processing

##  1 Introduction

In this paper, we seek to apply elements of Meta-Learning to Generative
Adversarial Networks (GANs) to tackle Super-Resolution in medical images,
specifically brain MRI images. We are the first to apply such a scale-free
Super-Resolution technique on these images. Super-Resolution is the task of
increasing the resolution of images. It helps radiological centres located in
rural areas which do not have high-fidelity instruments achieve comparable
diagnostic results as their advanced counterparts in the city. The importance
of Super-Resolution is growing as cross modality analysis requires combining
different types of information (such as PET and NMR scans) of varying
resolution.

Traditionally, Super-Resolution has been done using interpolation, such as
bicubic interpolation, but recent attempts have involved the use of deep
learning methods to extract high level information from data, which can be
used to supply additional information to increase the resolution of the image.
Current deep learning techniques for medical images rely heavily on Generative
Adversarial Networks (GANs) for they are able to generate realistic and
sharper images by using a different loss function that yields high perceptual
quality [5]. For example, mDCSRN [5], Lesion-focussed GAN [15], ESRGAN [4],
are all GAN-based solutions tackling Super-Resolution for medical images. Most
of them are based off SRGAN - a GAN-based network tackling Super-Resolution
developed by Ledig et al. [9]. We therefore use SRGAN as a foundation to which
we apply our modification.

Despite their better performance, almost all the networks relied on the sub-
pixel convolution layer introduced by Shi et al. [13], which tied a particular
upscaling factor to a particular network architecture. This incurs high
storage and energy costs for medical professionals who may wish to conduct
Super-Resolution on different scaling factors. In this paper, we combine the
Meta-Upscale Module introduced by Hu et al. [7] with SRGAN to create a novel
network lovingly termed Meta-SRGAN. This breaks the constraint imposed by Shi
et al’s layer and is capable of tackling Super-Resolution on any scale, even
non-integer ones, and hence can reduce storage, energy costs and lay the
foundations for real-time Super-Resolution. We first show that Meta-SRGAN
outperforms the baseline of bicubic interpolation on the BraTS dataset [11, 2,
3], and also show that Meta-SRGAN is capable of performing similarly to SRGAN,
but yet is able to super-resolve images of arbitrary scales. We also compare
the memory footprint and show that Meta-SRGAN is $\approx$ 98$\%$ smaller than
EDSR, a state-of-the-art Super-Resolution technique, but yet is able to
achieve similar performance.

The rest of the paper is organised as follows. Background introduces Super-
Resolution formally and illustrates how Hu et al. reframes the problem. We
then introduce the architecture of Meta-SRGAN in Methods. Experiments
elaborates on dataset preparation and training details. Finally, we show the
outcomes of the experiments in Results.

###  1.1 Background

####  1.1.1 Super-Resolution

Deep learning-based Super-Resolution techniques are generally supervised
learning algorithms. They analyse relationships between the low-resolution
(LR) image and its corresponding high-resolution (HR) image, and use this
relationship to obtain the super-resolved (SR) image. This SR image is then
evaluated against the HR image to see how well the algorithm is performing.
Any algorithm that is capable of extracting these relationships can be used to
do Super-Resolution. Dong et al [6] showed that any such algorithms can be
thought of as convolutional neural networks (CNNs). Figure 1 shows a typical
high level recipe for a Super-Resolution task.

Figure 1: How a Super-Resolution task is often carried out. A low-resolution
input image undergoes some convolutions to generate feature maps. Feature maps
are some sort of representation that captures the features of the low-
resolution image. Depending on the upscaling factor (it must be a multiple of
$2^{n}$), there would be some number $n$ of upscaling modules applied to the
representation. This upscaling unfolds the image into the high-resolution
output.

####  1.1.2 Meta-Upscale Module

The Upsampling module in the network is often the efficient sub-pixel
convolution layer proposed by Shi et al. [13]. Instead of explicitly enlarging
feature maps, it expands the channels of the output features to store the
extra points to increase resolution. It then rearranges these points to obtain
the super-resolved output image. Almost every network tackling Super-
Resolution uses this Upsampling module. An example is EDSR [10], a
convolutional neural network tackling Super-Resolution. Hu et al. then
proposed replacing the Upsampling module with a Meta-Upscale Module [7]. This
Meta-Upscale Module consists of a Weight Prediction Network and is an example
of Meta-Learning as it generalises a network to tackle more than one upscaling
factor. Essentially, given a few training examples (small amount of images
from some scales), we train a network to tackle arbitrary scale factors. The
Weight Prediction Network is able to predict weights for different upscaling
factors. This extra layer of abstraction enables the underlying neural network
to generalise across tasks. The framework proposed by Hu et al. for enabling
arbitrary scale Super-Resolution is illustrated in Figure 2.

Figure 2: The framework for tackling Super-Resolution tasks, as proposed by Hu
et al. There is no constraint on the upscaling factor. Note that the
Upsampling module has been replaced with a matrix multiplication and a Weight
Prediction Network. The matrix multiplication and Weight Prediction Network
are collectively called the Meta-Upscale Module.

Hu et al. reframes Super-Resolution as a matrix multiplication problem:

$Y=Wx+b$ (1)

where $Y$ is the super-resolved image, $W$ is some matrix of weights we learn,
$x$ is the input low-resolution image, and $b$ is some constant. This may seem
like an oversimplification and in many ways it is (for example it doesn’t take
account how pixels close to each other tend to exert more influence on each
other), but it gives us a rather intuitive understanding for why a separate
weight prediction network works for arbitrary scale Super-Resolution. Consider
$x$ as a $4\times 1$ vector, and $W$ as a $8\times 4$ matrix. The output $Y$
is therefore a $8\times 1$ vector. In some sense we have doubled the size of
$x$. We can think of W as providing an upscaling of $2$. The idea proposed by
Hu et al. is to predict W, for every upscaling factor. This is done by passing
in the input dimensions to an external function to create the shape of W,
before using a weight prediction network to predict the values of $W$, as in
the equation below:

$W=f_{\theta}(x)$ (2)

where $W$ is the matrix of weights, $f_{\theta}$ is the weight prediction
network with parameters $\theta$, and $x$ is the input low-resolution image.

The architecture of the Meta-Upscale Module is shown in Figure 3, and includes
a weight prediction network that outputs weights which are then matrix
multiplied to the input features to obtain the super-resolved image.

Figure 3: Architecture of Meta-Upscale Module. $H$ and $W$ refers to the
height and width of the high-resolution image, and $out\\_channels$ refers to
the number of channels in the original image (typically $3$ for colour, $1$
for grayscale). $64$ refers to the number of channels in the input features
fed into the Meta-Upscale Module. $3$ refers to the kernel size used in the
last convolution layer in the network producing the input features fed into
the Meta-Upscale Module.

The Weight Prediction Network consists of three layers, and is trained
alongside the main network. Of significant importance is the shape of the
input into the Weight Prediction Network. It must contain information relating
to the shape of the high-resolution image, as that determines the size of the
weight matrix that is multiplied with the input features fed into the Module.
Additional details can be found in [7].

##  2 Methods

###  2.1 Baselines

We used Bicubic Interpolation as a baseline, and implemented EDSR as a state-
of-the-art technique that we compare Meta-SRGAN to. We also wanted to
investigate how well using the Meta-Upscale Module on a GAN will affect its
performance, so we implemented SRGAN too. EDSR and SRGAN were trained on x2
upsampling tasks, and were trained on DIV2K for 160k updates before
transferring their learning onto the BraTS dataset for an additional 160k
updates. EDSR was trained with 256 number of features, 32 residual blocks, and
a residual scaling of 0.1 (see [10] for more information). EDSR was trained
using L1 Loss, whilst SRGAN was trained using the same combination of losses
used to train Meta-SRGAN.

###  2.2 Meta-SRGAN

To reap the benefits of both GANs and meta-learning, we combine Ledig et al.’s
SRGAN with Hu et al.’s Meta-Upscale Module to obtain a new architecture called
Meta-SRGAN. The nature of a GAN enables the generator to generate realistic
images. The ability to predict weights for different upscaling factors enables
the network to tackle arbitrary scales. Combined, they result in a network
that is both hugely powerful and highly generalisable.

There are two networks being trained in a Generative Adversarial Network.
There is the generator which generates an image (in this case it super-
resolves an image) which is then fed into the discriminator which
discriminates between a real and a generated image (in this case it determines
whether the image presented is the ground truth high-resolution, or not). This
feedback from the discriminator is then used to further train the generator.
Both generator and discriminator are playing a game to outbid each other. The
generator tries to improve its ability to generate images that can fool the
discriminator whilst the discriminator tries to improve its ability to discern
real images from generated images. This adversarial learning enables us to
generate more realistic and detailed images. The architecture of the generator
and discriminator are shown in figures 4 and 5 respectively.

####  2.2.1 Generator

The generator is used to generate super-resolved images, which the
discriminator will take in and output a classification that says whether it is
a generated image or a real image.

(a)

(b)

Figure 4: (Top) Architecture of Generator. Generator consists of Convolutions,
Residual Blocks, Batch Normalisation, and the Meta-Upscale Module. PReLU
refers to the Parametric Leaky ReLU activation function. (Bottom) Architecture
of Residual Block used in Generator. ReLU refers to the ReLU activation layer.
The number 64 refers to the number of kernels, and can be modified like any
hyperparameter.

Residual Blocks are used in the generator to stabilise training by
incorporating a feedback loop back onto the input. Parametric ReLU was used as
it was empirically found to stabilise training [9].

####  2.2.2 Discriminator

The discriminator consists of several Leaky ReLU layers and an average pooling
layer. These was empirically found to stabilise training [9]. A sigmoid layer
was used to facilitate binary classification.

Figure 5: Architecture of Discriminator. Leaky ReLU refers to the Leaky ReLU
activation function. AdaptiveAveragePool2d refers to an average pooling layer,
whilst Sigmoid refers to the sigmoid activation function that outputs a binary
value.

####  2.2.3 L1 Loss

L1 Loss is the typical loss function used by other networks that calculates
pixel-wise errors. We incorporate it to train the generator in Meta-SRGAN. We
denote an input image by $x$, and a neural network and its parameters by
$f_{\theta}$. Letting $\hat{y}=f_{\theta}(x)$, $i$ be just an index, $x_{i}$
be the low-resolution input image and $y_{i}$ be the high-resolution target
image, $C\times H\times W$ be the dimensions of an image, and $||\dots||$ be
the Frobenius norm, L1 Loss is defined as

$\ell_{l1}(\hat{y},y)=\frac{1}{CHW}\sum_{i}\begin{minipage}[t]{433.62pt}\rm\ignorespaces
\trivlist\par\trivlist@item@[]\hbox{\hbox{\hbox to0.0pt{\indent\hskip
0.0pt\hss}}\hskip
0.0pt\hbox{\hbox{}\hbox{\ignorespaces$\ignorespaces||$}}}\par\endtrivlist\ignorespaces\end{minipage}\hat{y}_{i}-y_{i}\begin{minipage}[t]{433.62pt}\rm\ignorespaces
\trivlist\par\trivlist@item@[]\hbox{\hbox{\hbox to0.0pt{\indent\hskip
0.0pt\hss}}\hskip
0.0pt\hbox{\hbox{}\hbox{\ignorespaces$\ignorespaces||$}}}\par\endtrivlist\ignorespaces\end{minipage}$
(3)

L1 Loss minimises the mean absolute error between pixels and helps Meta-SRGAN
match its output as closely as possible to the high-resolution target.

####  2.2.4 Adversarial Loss

The generator tries to minimise the following function whilst the
discriminator tries to maximise it. $D(x)$ is the discriminator’s output that
the high-resolution image is high-resolution, $G(z)$ is the super-resolved
image produced by the generator, and $D(G(z))$ is the discriminator’s output
that the super-resolved image is high-resolution.

$E_{x}[log(D(x))]+E_{z}[log(1-D(G(z)))]$ (4)

The above function is termed Adversarial Loss, and it allows Meta-SRGAN to
achieve realistic super-resolved images.

####  2.2.5 Perceptual Loss

Meta-SRGAN also incorporates Perceptual Loss, a type of loss introduced by
Johnson et al. to improve performance on Super-Resolution tasks [8]. The idea
was to encourage the generated image to have similar feature representations
as computed by a separate network. We use a 19-layer VGG network [14]
pretrained on the ImageNet dataset[12], and we extract the features before the
last layer of the network. Letting $\hat{y}$ and $y$ be the super-resolved and
high-resolution images respectively, $\phi$ be the network that extracts their
feature representations, and assuming that the feature representations are of
shape $C\times H\times W$, we define the Perceptual Loss as the Euclidean
distance between the feature representations:

$\ell_{p}(\hat{y},y)=\frac{1}{CHW}||\phi(\hat{y})-\phi(y)||^{2}$ (5)

In the following experiments we combined the loss functions as

$\ell_{total}(\hat{y},y)=\ell_{l1}(\hat{y},y)+0.001\ell_{adversarial}(\hat{y},y)+0.006\ell_{p}(\hat{y},y)$
(6)

and use it to train Meta-SRGAN.

##  3 Experiments

We performed experiments on the Multimodal Brain Tumor Segmentation (BraTS)
dataset [11, 2, 3]. We first trained Meta-SRGAN on the DIV2K dataset [1] \- a
dataset curated for Super-Resolution tasks, and then transferred that learning
onto the BraTS dataset.

###  3.1 Datasets and Data Preparation

To demonstrate how well Meta-SRGAN performs on medical images, we trained and
tested Meta-SRGAN on the Multimodal Brain Tumor Segmentation (BraTS) dataset
[11, 2, 3]. This dataset contains several versions, including t1, t1ce, and t2
brain MRI scans, and serves as a good proxy for medical images. We picked the
2D t1ce version as it was two dimensional and was simple to work with. There
were $15140$ training images and $3784$ validation images. Training images
were normalised using a mean and standard deviation of $0.370$ and $0.117$
respectively before they were passed to the network to be trained on. These
values were calculated using pixel values from the training images, after the
maximal informational crop was applied.

Figure 6: Workflow for handling the BraTS dataset, for a x2 upsampling task.

The BraTS dataset is tricky to deal with because every image has sparse
information. Only about $50\%$ of each image contained useful information
(i.e. the brain), the rest was black. To tackle this problem, we cropped the
image with the most pixel information. This corresponded to crops of the
brain. Then, from this crop, we then randomly sampled $16$ $96$ by $96$
patches to provide sufficient coverage per image. For an upscaling factor of
2, the low-resolution input image (LR) was of size $48$ by $48$ whilst the
high-resolution target image (HR) and super-resolved output image (SR) were of
size $96$ by $96$. This workflow is summarised in Figure 6.

###  3.2 Training Parameters and Experiment Settings

We first trained Meta-SRGAN on the DIV2K dataset for 360k updates, with a
mini-batch size of 8. We then did transfer learning and trained Meta-SRGAN on
the BraTS dataset for an additional 360k updates. We trained Meta-SRGAN on a
range of scales from 1.1 to 4.0 (with 0.1 increments). Higher upscaling
factors were not chosen as that would have made the input image too small to
be useful. Each minibatch of images was associated with a scale, and each
scale was chosen uniformly at random from 1.1 to 4.0. We optimised the network
using a combination of L1 Loss, Adversarial Loss, and Perceptual Loss as
described in the previous section. We used an Adam optimiser with a learning
rate of $1*10^{-4}$ on each of the generator and discriminator, and decreased
the learning rate by $20\%$ every 60k updates. Pixel values were clamped to
between 0 and 255 to give the generator an edge. The network was trained on a
NVIDIA TITAN X GPU and took three days. We calculated the PSNR and SSIM scores
only on the luminance channel of the maximal information crops.

##  4 Results

The results of our experiments are tabulated in Table 1. We see that Meta-
SRGAN clearly outperformed the baseline of Bicubic Interpolation, which
suggests that the inclusion of the Meta-Upscale Module did not hinder the
network’s performance. This is further reinforced by the fact that Meta-SRGAN
was able to achieve similar performance on the x2 upsampling task as SRGAN,
the architecture it was based on. It also had the added benefit of a smaller
number of parameters compared to SRGAN. Visual quality is indicated by a proxy
metric, Structural Similarity Index Metric (SSIM), which measures the
perceived image similarity between two images. The high SSIM scores achieved
by the GANs bodes well for brain MRI imaging which demands accurate images. We
also see that Meta-SRGAN has the lowest number of parameters, which means its
memory footprint is really small ($\approx$ 98$\%$ smaller than EDSR), which
suggests that it can be deployed easily.

Table 1: A comparison of PSNR and SSIM scores between Bicubic Interpolation,
SRGAN and Meta-SRGAN on the BraTS dataset. Best results are bolded.

x2 Upsampling Task, BraTS Testing Set
---
| Bicubic Interpolation | EDSR | SRGAN | Meta-SRGAN
| PSNR (dB) | SSIM | PSNR (dB) | SSIM | PSNR (dB) | SSIM | PSNR (dB) | SSIM
mean | 17.98 | 0.6091 | 18.93 | 0.6885 | 18.42 | 0.7295 | 18.72 | 0.7279
std | 5.674 | 0.1708 | 4.2909 | 0.1814 | 5.4975 | 0.1308 | 5.338 | 0.1422
#parameters | - | 40.7M | 0.565M | 0.561M

Sampled images are shown in Figure 7. The higher SSIM scores of SRGAN and
Meta-SRGAN than the baseline of Bicubic Interpolation highlights the fact that
they generate perceptually better images than the baseline. The very fact that
Meta-SRGAN is even able to have comparable scores to EDSR despite being
trained on a whole range of scales, and with considerably less parameters,
attests to not just the power of adversarial training, but also the robustness
of the Meta-Upscale Module.

Figure 7: Sample images from Bicubic Interpolation, EDSR, SRGAN and Meta-
SRGAN. This is from a x2 upsampling task.

Meta-SRGAN was also able to produce a sequence of images corresponding to
different upscaling factors. This is shown in Figure 8. The relatively
consistent PSNR and SSIM scores indicate that performance across different
scales are not compromised. All these results suggests that we have a memory
efficient network capable of generating super-resolved brain MRI images of
high visual quality, and can be used to generate images of any scale.

With better visual quality than the baseline, lower memory footprint, and
ability to tackle any upscaling factor with negligible loss in performance,
Meta-SRGAN clearly can deliver comparable performance to state-of-the-art
Super-Resolution techniques.

Figure 8: (Top) Image going through various upsampling factors using Bicubic
Interpolation. (Bottom) Image going through various upsampling factors using
Meta-SRGAN. PSNR and SSIM scores are calculated against a target image
downscaled by a factor of 4.

##  5 Conclusions

The implications of the results on the BraTS dataset are three-fold. The low
memory footprint of Meta-SRGAN enables it to be wrapped as a helper tool to
aid medical tasks such as MRI and endoscope videography. The ability to super-
resolve images on any arbitrary scale allows one to easily extend this to an
application of real-time zooming and enhancing of an image, which will be
useful in medical screening and surgical monitoring. The fact that its
performance is not hindered across multiple upscaling factors affords it the
flexibility to be used in other works such segmentation, de-noising, and
registration. Despite only testing it on brain MRI images, Meta-SRGAN can
easily be extended to datasets of other modalities such as Ultrasound and CT
scans. Its low memory footprint, high visual quality, and generalisability
suggests that Meta-SRGAN can become a new foundation on which other
architectures can be built to enhance the performance for medical images.

We have built a network that combines the ability to generate images of high
visual quality with the ability to tackle arbitrary scales, and are the first
to show that this does not compromise performance or memory footprint for
brain MRI images. This means that unlike other state-of-the-art methods, our
method works on arbitrary scales which means that only a single network is
required to perform Super-Resolution on any upscaling factors. In future work
we hope to enhance the performance of Meta-SRGAN and apply it to other
modalities.

##  6 Acknowledgements

Jin Zhu’s PhD research is funded by China Scholarship Council
(grant No.201708060173), whilst Pietro Lio’ is supported by the GO-DS21 EU
grant proposal.

## References

  * [1] Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: Dataset and study. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (July 2017)
  * [2] Bakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., Kirby, J., et al.: Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features. Scientific Data 4 (09 2017). https://doi.org/10.1038/sdata.2017.117
  * [3] Bakas, S., Reyes, M., Jakab, A., Bauer, S., Rempfler, M., Crimi, A., et al.: Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge. CoRR abs/1811.02629 (2018), http://arxiv.org/abs/1811.02629
  * [4] Bing, X., Zhang, W., Zheng, L., Zhang, Y.: Medical image super resolution using improved generative adversarial networks. IEEE Access 7, 145030–145038 (2019). https://doi.org/10.1109/ACCESS.2019.2944862
  * [5] Chen, Y., Shi, F., Christodoulou, A.G., Zhou, Z., Xie, Y., Li, D.: Efficient and accurate MRI super-resolution using a generative adversarial network and 3d multi-level densely connected network. CoRR abs/1803.01417 (2018), http://arxiv.org/abs/1803.01417
  * [6] Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. CoRR abs/1501.00092 (2015), http://arxiv.org/abs/1501.00092
  * [7] Hu, X., Mu, H., Zhang, X., Wang, Z., Tan, T., Sun, J.: Meta-sr: A magnification-arbitrary network for super-resolution. CoRR abs/1903.00875 (2019), http://arxiv.org/abs/1903.00875
  * [8] Johnson, J., Alahi, A., Li, F.: Perceptual losses for real-time style transfer and super-resolution. CoRR abs/1603.08155 (2016), http://arxiv.org/abs/1603.08155
  * [9] Ledig, C., Theis, L., Huszar, F., Caballero, J., Aitken, A.P., Tejani, A., Totz, J., Wang, Z., Shi, W.: Photo-realistic single image super-resolution using a generative adversarial network. CoRR abs/1609.04802 (2016), http://arxiv.org/abs/1609.04802
  * [10] Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. CoRR abs/1707.02921 (2017), http://arxiv.org/abs/1707.02921
  * [11] Menze, B.H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., et al.: The multimodal brain tumor image segmentation benchmark (brats). IEEE Transactions on Medical Imaging 34(10), 1993–2024 (Oct 2015). https://doi.org/10.1109/TMI.2014.2377694
  * [12] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M.S., Berg, A.C., Li, F.: Imagenet large scale visual recognition challenge. CoRR abs/1409.0575 (2014), http://arxiv.org/abs/1409.0575
  * [13] Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. CoRR abs/1609.05158 (2016), http://arxiv.org/abs/1609.05158
  * [14] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2014)
  * [15] Zhu, J., Yang, G., Lio, P.: How can we make gan perform better in single medical image super-resolution? a lesion focused multi-scale approach. 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019) (Apr 2019). https://doi.org/10.1109/isbi.2019.8759517, http://dx.doi.org/10.1109/ISBI.2019.8759517


11institutetext: Universidad de Granada, Avenida de la Fuente Nueva S/N, 18071
Granada, Spain 22institutetext: Instituto de Astrofísica de Andalucía (IAA-
CSIC), Glorieta de la Astronomía s/n, 18008 Granada, Spain 33institutetext:
Max-Planck Institute for Astronomy, Königstuhl 17, 69117 Heidelberg, Germany

# Detecting hot stars in the Galactic centre with combined near- and mid-
infrared photometry

M. Cano-González 11 R. Schödel 22 F. Nogueras-Lara 33

###### Abstract

Context. The Galactic centre (GC) is a unique astrophysical laboratory to
study the stellar population of galactic nuclei because it is the only
galactic nucleus whose stars can be resolved down to milliparsec scales.
However, the extreme and spatially highly variable interstellar extinction
towards the GC poses a serious obstacle to photometric stellar classification.

Aims. Our goal is to identify hot, massive stars in the nuclear stellar disc
(NSD) region through combining near-infrared (NIR) and mid-infrared (MIR)
photometry, and thus to demonstrate the feasibility of this technique, which
may gain great importance with the arrival of the James Webb Space Telescope
(JWST).

Methods. We combined the GALACTICNUCLEUS NIR survey with the IRAC/Spitzer MIR
survey of the GC. We applied the so-called Rayleigh-Jeans colour excess (RJCE)
de-reddening method to our combined NIR-MIR data to identify potential hot
stars in colour-magnitude diagrams (CMDs).

Results. Despite the very low angular resolution of IRAC we find 12 clear
candidates for young massive stars among the $1\,065$ sources that meet our
selection criteria. Seven out of these 12 stars are previously known hot,
massive stars belonging to the Arches and Quintuplet clusters, as well as
sources detected by the Hubble Space Telescope/NICMOS Paschen-$\alpha$ survey.
Five of our massive star candidates have not been previously reported in the
literature.

Conclusions. We show that the RJCE method is a valuable tool to identify hot
stars in the GC using photometry alone. Upcoming instruments with high angular
resolution MIR imaging capabilities such as the JWST could surely make more
substantial use of this de-reddening method and help establish a far more
complete census of hot, young stars in the GC area than what is possible at
the moment.

###### Key Words.:

Galaxy: centre – Interstellar medium: dust, extinction – stars: early-type,
massive.

##  1 Introduction

The Galactic centre (GC) is a fundamental astrophysical environment to study
the astrophysics of the nuclear environment of large spiral galaxies because
the nearness of the GC provides us unrivalled insight because it enables us to
study stars individually.

The GC region presents the most extreme conditions in the Galaxy. Stellar
crowding reaches $10^{7}$ pc-3 in the inner $0\aas@@fstack{\prime\prime}1$ of
the nuclear star cluster (NSC), the massive ($2.5\pm 0.4\times 10^{7}\
M_{\sun}$) compact ($R_{hl}=4.2\pm 0.4$ pc) cluster harbouring the $\approx
4\times 10^{6}\ M_{\sun}$ massive black hole Sagittarius A* (Sgr A*) located
at the dynamical centre of the Galaxy (Ghez et al. 2008; Genzel et al. 2010;
Schödel et al. 2014, 2018). Crowding is less extreme ($\sim 10^{5}$ pc-3) in
the nuclear stellar disc (NSD), a compact, disc-like structure of radius $\sim
160$–$230$ pc and height of $\sim 45$ pc that is often also referred to as the
nuclear bulge (Launhardt et al. 2002; Nishiyama & Schödel 2013; Gallego-Cano
et al. 2020).

One of the many reasons to study the GC is that it has been a prolific,
massive star forming region in the past few tens of millions of years, as is
witnessed by the presence of three of the most massive clusters of the Galaxy
(the central, Arches and Quintuplet star clusters) and numerous apparently
isolated massive hot stars distributed throughout the nuclear bulge, and by
its star formation history (e.g. Figer et al. 2004; Mauerhan et al. 2010; Dong
et al. 2011; Nogueras-Lara et al. 2020a). The detection of classical Cepheids
shows that about 25 Myr ago, lasting until recently, the GC underwent an
increase in star formation rate relative to the previous $40$ Myr, forming
around $10^{6}\ M_{\odot}$ of stars (Matsunaga et al. 2011). However, less
than $10\%$ of this mass can be attributed to the three known young clusters.
This may be a result of massive clusters dissolving into the crowded
background within a few million years (Portegies Zwart et al.
2002).Alternatively, massive stars may form in isolation at the GC, contrary
to what we find in other parts of the Milky Way. Therefore, more massive stars
or clusters are expected to reside in the GC, but have not been identified
yet. The conditions in the GC are a proxy to what we can find in starburst
galaxies and there is also good reason to believe that the initial mass
function is top-heavy (or bottom light) at the GC (see, e.g., Lu et al. 2013;
Bartko et al. 2010; Hosek et al. 2019). Hence, there is great interest to
obtain a complete census of massive, hot, young stars in the GC.

Observations of the GC suffer from ($A_{V}\sim 30$ mag) extinction (see
Nishiyama et al. 2009; Fritz et al. 2011; Nogueras-Lara et al. 2020b, and
references therein), making it only observable at infrared (IR) wavelengths.
An additional complication is that extinction varies significantly on
arcsecond scales (e.g., Scoville et al. 2003; Schödel et al. 2010; Hosek et
al. 2019; Nogueras-Lara et al. 2018).The spectroscopic identification of stars
is hampered by the extreme source crowding and by massive observational
overheads. Other ways to identify massive stars are, for example, narrow band
photometry (Dong et al. 2011) or proper motion studies (Shahzamanian et al.
2019). Being able to use sensitive broad-band photometry from large surveys
could provide significant progress.

Most of observations of the GC stellar population are carried out in the near-
infrared (NIR) regime, in particular in the $J$ ($\sim 1.25\ \mu\mathrm{m}$),
$H$ ($\sim 1.65\ \mu\mathrm{m}$), and $K_{S}$ ($\sim 2.15\ \mu\mathrm{m}$)
bands. Unfortunately, it is very difficult to unambiguously identify hot stars
in colour-magnitude diagrams (CMDs) or colour-colour diagrams (CCDs) using NIR
photometry alone because of the degeneracy between reddening and stellar type
(see Figs. 1 and 4 in Majewski et al. 2011). Mid-infrared (MIR) photometry can
be used to break this degeneracy. Certain NIR-MIR colour combinations present
little variation for a wide range of stellar types (from A0 to M, see Majewski
et al. 2011).

If the intrinsic colour of a star is known, then its colour excess (the
difference between the observed colour and the intrinsic colour) can be
computed once an extinction law has been assumed. In particular, we can use
the following expressions:

$A_{K_{S}}=\frac{E([\mathrm{NIR}]-[\mathrm{MIR}])}{\left(\frac{\lambda_{\mathrm{NIR}}}{\lambda_{K_{S}}}\right)^{-\alpha_{\mathrm{NIR}}}-\left(\frac{A_{\mathrm{MIR}}}{A_{K_{S}}}\right)}$
(1)

and

$\mathrm{mag}(\lambda)_{0}=\mathrm{mag}(\lambda)_{\mathrm{obs}}-\left(\frac{A_{\lambda}}{A_{K_{S}}}\right)A_{K_{S}}$
(2)

to calculate extinction in the $K_{S}$ band for combined NIR-MIR photometry
and thus retrieve magnitudes and colours. In these equations,
$E([\mathrm{NIR}]-[\mathrm{MIR}])$ denotes the colour excess,
$\alpha_{\mathrm{NIR}}$ represents the extinction index in the NIR (assuming
an extinction law of the form: $A_{\lambda}\propto\lambda^{-\alpha}$, see
Nishiyama et al. 2009; Fritz et al. 2011; Nogueras-Lara et al. 2019b), and
$A_{\lambda}$ denotes extinction at a certain wavelength, $\lambda$. An
extensive description of the so-called Rayleigh-Jeans colour excess (RJCE) de-
reddening method can be found in Majewski et al. (2011). Briefly stated, this
method relies on the fact that intrinsic stellar colours of a broad range of
stars are small and almost constant for suitable combinations of NIR and MIR
filters. In particular, Majewski et al. (2011) demonstrated the usefulness of
the filter combinations of $(H-[4.5\mu])$ and $(K_{S}-[3.6\mu])$.

In this work we explore the application of this method to the stellar
population of the GC. This paper is organised as follows: Section 2 offers a
brief description of the NIR and MIR data used. In Sect. 3 we provide details
on source selection and other methodological aspects of our work. In Sect. 4
we present the main results and in Sect. 5 the conclusions of our study.

##  2 Data

Our NIR data correspond to the central field of the high-resolution
($0\aas@@fstack{\prime\prime}2$) $J$, $H$, and $K_{S}$ GALACTICNUCLEUS survey
(Nogueras-Lara et al. 2019a). Its high resolution is achieved after applying
the holographic imaging technique described in Schödel et al. (2013). Details
regarding data reduction and survey performance can be found in Nogueras-Lara
et al. (2018). Statistical uncertainties are below $0.05$ mag for $J\sim
21\,\mathrm{mag}$, $H\sim 19$ mag and $K_{S}\sim 18$ mag and the zero-point
uncertainty is $\lesssim 0.04$ mag for all bands. The GALACTICNUCLEUS survey
suffers from non-linearity and saturation effects in sources brighter than
$K_{s}\lesssim 11.5$ mag (Nogueras-Lara et al. 2019a).

For MIR photometry we used the IRAC/Spitzer Galactic Center survey (Stolovy et
al. 2006). This survey has a considerably lower resolution than
GALACTICNUCLEUS with mean full width at half maximum (FWHM) of
1$\aas@@fstack{\prime\prime}$66, 1$\aas@@fstack{\prime\prime}$72,
1$\aas@@fstack{\prime\prime}$88, and 1$\aas@@fstack{\prime\prime}$98 for
channels 1, 2, 3, and 4, respectively, which correspond to broadband MIR
filters approximately centred on $3.6\ \mu\mathrm{m}$, $4.5\ \mu\mathrm{m}$,
$5.8\ \mu\mathrm{m}$, and $8.0\ \mu\mathrm{m}$. We used the IRAC point source
catalogue provided by Ramírez et al. (2008). Its average confusion limits are
12.4, 12.1, 11.7, and 11.2 mag for channels 1 to 4.

Data from the Two Micron All Sky Survey (2MASS) at $J$, $H$, $K_{S}$ are also
included in the Ramírez et al. (2008) catalogue. The 2MASS survey provides
point sources with $\gtrsim 2\aas@@fstack{\prime\prime}5$ mean FWHM for
sources in the brightness range $(4\ \mathrm{mag}\lesssim
K_{S}^{\mathrm{2MASS}}\lesssim 14\ \mathrm{mag})$ (Skrutskie et al. 2006).

In addition, we also used multi-epoch data from the SIRIUS/IRSF GC survey
(Nagayama et al. 2003; Nishiyama et al. 2006; Hatano et al. 2013) to detect
and deselect variable stars (see Sect. 3.2). The SIRIUS GC survey is seeing-
limited (FWHM: $\gtrsim 1\arcsec$) with $10\sigma$ limiting magnitude averages
of $J^{\mathrm{SIRIUS}}=17.1$ mag, $H^{\mathrm{SIRIUS}}=16.6\,\mathrm{mag}$,
and $K_{S}^{\mathrm{SIRIUS}}=15.6\,\mathrm{mag}$ (Nishiyama et al. 2006).
Saturation and non-linearity effects are prominent for magnitudes brighter
than $9.5$, $9.5$, and $9.0$ mag for $J^{\mathrm{SIRIUS}}$,
$H^{\mathrm{SIRIUS}}$, and $K_{S}^{\mathrm{SIRIUS}}$ bands, respectively
(Matsunaga et al. 2009).

##  3 Methodology

In the following subsections we describe how we combined the catalogues, how
we deselected variable stars and outliers, and how we identified the hot star
candidates.

###  3.1 Catalogue merging

Since there is about an order of magnitude difference in the angular
resolution between the IRAC/Spitzer, 2MASS, SIRIUS data, and GALACTICNUCLEUS
data (see Fig. 1), we applied the following merging criteria:

1) We imposed a preliminary matching radius of
$0\aas@@fstack{\prime\prime}265$. 2) We determined the mean coordinate offsets
between the data and corrected them. Subsequently, we repeated cross-
identification with a smaller matching radius of
$0\aas@@fstack{\prime\prime}133$. 3) Some apparently single IRAC/2MASS/SIRIUS
sources have several NIR counterparts in GALACTICNUCLEUS because of the higher
resolution of the latter survey. Therefore, we deselected sources with close
neighbours that may contaminate the IRAC photometry. We excluded stars that
had another GALACTICNUCLEUS detection within the IRAC channel 2 mean half
width at half maximum (HWHM) ($0\aas@@fstack{\prime\prime}86$; Fazio et al.
2004).

With these criteria we obtained a total of $5\,585$ matching NIR-MIR
detections.

Figure 1: Left: Detail of IRAC $[3.6\mu]$ image. Right: Same area as seen by
GALACTICNUCLEUS $K_{S}$ band. This image illustrates the considerable
difference in angular resolution between the main surveys used in this work.
The dark rings present in GALACTICNUCLEUS brightest stars (also seen in Figs.
5 and 6) are a consequence of a combined effect of the holographic data
reduction technique and the image scaling. These stars are not marked or
outlined in any way.

###  3.2 Variability

All data used in this work were taken at different epochs. The 2MASS survey
collected data between 1997 and 2001, Spitzer performed its GC survey with
IRAC in 2005, and GALACTICNUCLEUS started in 2015. These differences in
observing epochs result in unphysical observed stellar colours for many
variable stars. To account for variability, we used two SIRIUS/IRSF NIR data
sets (Nishiyama et al. 2006; Hatano et al. 2013) as well as GALACTICNUCLEUS
and 2MASS data from Ramírez et al. (2008) for bright, saturated sources in
GALACTICNUCLEUS. We took the following steps: 1) We used the SIRIUS survey as
a photometric reference and adjusted the median magnitude offsets of
GALACTICNUCLEUS, 2MASS, and the other SIRIUS data (polarimetry study by Hatano
et al. 2013) to the SIRIUS photometry. Only bright ($K_{S}<13.5$ mag), non-
saturated detections were used for this purpose. 2) For each NIR band and each
catalogue combination (SIRIUS-SIRIUSpol, SIRIUS-GALACTICNUCLEUS, SIRIUSpol-
GALACTICNUCLEUS) we computed

$\chi_{\mathrm{var}}=\frac{|\mathrm{mag_{1}}-\mathrm{mag_{2}}|}{\sqrt{\mathrm{emag_{1}}^{2}+\mathrm{emag_{2}}^{2}}}$
(3)

for every common detection. If this quantity was larger than three for any
band and any combination of two measurements, that particular star was flagged
as variable.

With this procedure, $2\,628$ stars appeared to be non-variable and $1\,907$
stars were flagged as variable for at least one catalogue combination.
Therefore, $\approx 42\%$ of the stars in our sample are flagged as
potentially variable. Finally, $1\,050$ stars were not detected in some of the
above-mentioned catalogues, making it thus impossible to infer their
variability with this method. We excluded both these unclassified stars and
the variable stars from our selection.

###  3.3 De-reddening

We now proceeded to de-redden the stars in our combined NIR-MIR GC catalogue
of non-variable stars with the RJCE method. We computed extinction in the
$K_{S}$ band and intrinsic magnitudes and colours with Eqs. (1) and (2). We
used the filter combinations with the narrowest distribution of intrinsic
colours according to Majewski et al. (2011): $(H-[4.5\mu])$ and
$(K_{S}-[3.6\mu])$. We assumed intrinsic colour values of
$(H-[4.5\mu])_{0}=0.08\,\mathrm{mag}$ and
$(K_{S}-[3.6\mu])_{0}=0.08\,\mathrm{mag}$ (based on Fig. 3 in Majewski et al.
2011, and on our own estimates from stellar atmosphere models).

The resulting distributions of $A_{K_{S}}$ extinction and its corresponding
uncertainties (based on the photometric uncertainties) for the two colours are
shown in Fig. 2. We found that $(H-[4.5\mu])$ provided more reliable results
in terms of de-reddened colours and uncertainties than $(K_{S}-[3.6\mu])$.
This could be explained by the larger wavelength range covered by the former,
hence reducing systematics related to filter width, and the deeper dust
penetration capabilities of the latter. For this reason we only use
$(H-[4.5\mu])$ henceforth. We only used stars with with uncertainties $<0.1$
mag in both GALACTICNUCLEUS and IRAC photometry. We excluded foreground stars
belonging to the Galactic disc and bulge using a $(H-K_{S})<1.3\,\mathrm{mag}$
colour cut (Nogueras-Lara et al. 2020a, 2021). For the extinction curve, we
adopted values of $\alpha_{\mathrm{NIR}}=2.11\pm 0.06$ and
$A_{[4.5\mu]}/A_{K_{S}}=0.396\pm 0.082$, $A_{[3.6\mu]}/A_{K_{S}}=0.547\pm
0.052$ from Fritz et al. (2011).

Figure 2: Left: $A(K_{S})$ histograms for the combined NIR-MIR sources that
met the photometric quality criteria and the foreground cut: $(H-[4.5\mu])$
(blue) and $(K_{S}-[3.6\mu])$ (red). Right: Corresponding distributions of the
uncertainties for the same stars and colours. The considerably higher
uncertainties of $(K_{S}-[3.6\mu])$ are noted.

##  4 Results and discussion

The $(J-K_{S},K_{S})$ CMD of our final sample is shown in Fig. 3. We chose
this colour combination because the $H$ band was already used in the de-
reddening process, and because the intrinsic colour of stars is expected to be
large for this filter combination

Figure 3: Final $K_{S}$ vs $J-K_{S}$ CMD. Both observed (black) and de-
reddened (blue) data are plotted. We fitted the de-reddened RGB cloud to a 9
Gyr, $Z=0.039$ isochrone (red line). We separate $12$ hot stars (blue circles)
from the RGB.

All stars plotted in Fig. 3 are detected in all three GALACTICNUCLEUS bands
and in the IRAC channel 2 ($4.5\ \mu$m), meet the non-variability criteria,
have photometric uncertainties below $0.1$ mag, and are not foreground
sources. Additionally, we excluded faint $(K_{S}>13.5\ \mathrm{mag})$ sources
because their de-reddened counterparts presented extremely blue
($(J-K_{S})_{0}\approx-2,-3$ mag) colours. We suppose that this over-
correction was related to systematic errors of the IRAC photometry of such
faint sources in this extremely crowded field.

We fitted by eye a 9 Gyr, $Z=0.039$, solar-scaled BaSTI isochrone model to the
main cloud of points in the CMD (Pietrinferni et al. 2013; Nogueras-Lara et
al. 2020a). For this purpose we used $\alpha_{\mathrm{NIR}}=2.11$ (Fritz et
al. 2011) and the ratio $A[4.5\mu]/A_{K_{S}}=0.32$. The latter ratio is lower
than – but within the uncertainties provided for this ratio by Fritz et al.
(2011). Small changes in the extinction index and/or ratios translate to small
variations in the linear dependence of extinction with respect to colour
excess, that is $A(K_{S})=\varphi\ E(H-[4.5\mu])$. In other words, small
changes ($A_{[4.5\mu]}/A_{K_{S}}=0.32$ instead of
$A_{[4.5\mu]}/A_{K_{S}}=0.396$) in the $\varphi$ coefficient yield changes for
all detections, modulated by their colour excess. This effect is therefore of
no concern when separating hot stars in a de-reddened CMD using photometry
alone, which is our main purpose rather than providing an accurate measurement
of de-reddened magnitudes and colours.

We note that reasonably varying either age $(7-10\ \mathrm{Gyr})$ and/or
metallicity $(0.02<Z<0.04)$ resulted in little change in the position of the
isochrone in the CMD. In comparison to the isochrone, the red giant branch
(RGB) appears somewhat tilted towards the blue with increasing brightness. We
were not able to identify the reason for this behaviour, which may be caused
by a complex relation between effects such as non-linearity of photometry in
some of the data, extinction correction, and effective wavelength in broad-
band filters, or the precise composition of the stellar population; the
effective wavelength results from a complex function of filter, stellar
spectral energy distribution (SED), extinction curve, and absolute extinction.
The important observation is, however, that the RGB stars cluster along a
tight sequence in the de-reddened RGB.

We separated the de-reddened RGB cloud into approximately 0.5 magnitude bins
($(K_{S})_{0}<7.5\,\mathrm{mag}$,
$7.5\,\mathrm{mag}<(K_{S})_{0}<8\,\mathrm{mag}$,
$8\,\mathrm{mag}<(K_{S})_{0}<8.5\,\mathrm{mag}$,
$8.5\,\mathrm{mag}<(K_{S})_{0}<9\,\mathrm{mag}$,
$9\,\mathrm{mag}<(K_{S})_{0}<9.5\,\mathrm{mag}$,
$9.5\,\mathrm{mag}<(K_{S})_{0}<10\,\mathrm{mag}$ and
$10\,\mathrm{mag}<(K_{S})_{0}$) and computed the median and standard deviation
of $(J-K_{S})_{0}$ and $(K_{S})_{0}$ for each bin. We chose this bin width so
at least 100 stars per bin are used. Each bin contains $\sim 150$ stars on
average.

We identified as hot, and presumably young, those stars bluer than $2.5\sigma$
from the median $(J-K_{S})_{0}$ in each bin. As shown in Fig. 3, the de-
reddened cloud widens for magnitudes brighter than $(K_{S})_{0}\sim
10\,\mathrm{mag}$. Therefore, to separate real hot detections from the blue
tail of the RGB quasi-Gaussian distribution, we interpolated with a spline
degree two (blue, dashed line) using as nodes those values located $2.5\sigma$
bluer than the median $(J-K_{S})_{0}$ of each bin. The position of the nodes
in the magnitude axis is given by the median $(K_{S})_{0}$ of each bin. Thus,
$12$ out of the $1\,065$ stars plotted are identified as hot (blue circles).
We interpret one star, which is bluer than $7\sigma$ from the median
$(J-K_{S})_{0}$, as an outlier (red circle). We proceeded analogously with the
red tail of the de-reddened RGB. Only five sources are redder than $2.5\sigma$
from the median $(J-K_{S})_{0}$, suggesting that our hot candidates are hot
stars and not residual errors from the de-reddened distribution.

Since IRAC photometry is less affected by interstellar extinction than
$K_{S}$-band photometry, in principle, some highly reddened sources could be
detected at $[3.6\mu]$ or $[4.5\mu]$ while being fainter than the $K_{S}$
detection limit. This scenario could result in possible contamination due to
chance alignments of MIR sources that are not detected in the NIR and
therefore not deselected in the NIR-only neighbour exclusion criteria
explained in Sect. 3.1. To test whether this concerns our case we used average
intrinsic (non-reddened) $(K_{S}-[4.5\mu])_{0}$ colours of several stellar
types and reddened them with typical GC extinction values using the extinction
coefficients from Fritz et al. (2011). For RGB stars, we averaged the values
given in the IRAC K-III primary calibrators (Reach et al. 2005, Table 1) and
the intrinsic colours for M-type giants provided in Jian et al. (2017, Table
11). Both K-type and M-type giants showed similar intrinsic NIR-MIR colours:
$(K_{S}-[4.5\mu])_{0}\sim 0\,\mathrm{mag}$. Therefore, given that our faintest
$[4.5\mu]$ detections are $\sim 12\,\mathrm{mag}$, we estimated how much
fainter such detections would appear in the $K_{S}$ band and checked if they
were above the GALACTICNUCLEUS survey $K_{S}$ detection limit ($K_{S}\sim
18\,\mathrm{mag}$). In an average extinction scenario ($A(K_{S})\sim
2\,\mathrm{mag}$ and $A([4.5\mu])\sim 1\,\mathrm{mag}$) such stars would be
clearly detectable by GALACTICNUCLEUS since they would be around $K_{S}\sim
15\,\mathrm{mag}$:
$(K_{S}-[4.5\mu])=(K_{S}-[4.5\mu])_{0}+A(K_{S})+A([4.5\mu])$. Even in a high
extinction scenario ($A(K_{S})\sim 3.5\,\mathrm{mag}$ and $A([4.5\mu])\sim
1.5\,\mathrm{mag}$), such highly reddened RGB stars would present $K_{S}\sim
17\,\mathrm{mag}$, brighter than GALACTICNUCLEUS detection limit. We proceeded
analogously with asymptotic giant branch (AGB) stars, whose intrinsic colours
we derived from Reiter et al. (2015, Tables 1 and 2). We used the five closest
stars from each sub-category (O-rich, C-rich and S-rich) to minimise
extinction, and averaged the values. We obtained the following results:
$(K_{S}-[4.5\mu])_{0}\approx 0.2\pm 0.3\,\mathrm{mag},0.5\pm
0.2\,\mathrm{mag}$ and $0.3\pm 0.3\,\mathrm{mag}$ for O-rich, C-rich, and
S-rich AGB stars respectively. We note that using the multi-epoch $[4.5\mu]$
values given in Reiter et al. (2015, Table 2) did not change our average
results. Thus, even if we assume the redder colours from C-rich AGB stars and
a high extinction scenario, the $K_{S}$-band counterpart of such stars would
be around $K_{S}\sim 17.5\,\mathrm{mag}$ and therefore detectable by
GALACTICNUCLEUS. Regarding the possibility of contamination due to objects
with intrinsically redder SEDs such as young stellar objects (YSOs) we rely on
the results from Nandakumar et al. (2018). This spectroscopic study showed
that many of the objects previously identified as YSOs with CMD were cool,
evolved AGB stars, indicating how similar NIR and MIR colours are for AGB
stars and YSOs. Furthermore, the total amount of YSOs detected in the central
molecular zone (CMZ) is relatively small (a few hundreds, according to Yusef-
Zadeh et al. 2009); therefore, the chance alignment of such an object within
$\sim 1\arcsec$ of one of our target stars is very small.

We used the NICMOS HST Paschen-$\alpha$ survey (Wang et al. 2010; Dong et al.
2011) to test whether we could find any previously detected massive stars. The
majority of hot stars reported by Dong et al. (2011) are not present in our
final data set because of our selection criteria. However, we find seven hot
massive stars from their primary Paschen-$\alpha$ emitting candidates (Dong et
al. 2011, Table 3). We find none of their secondary emitting Paschen-$\alpha$
sources. Our hot stars candidates, and their photometric properties are listed
in Table 1 and their positions on the sky are represented in Fig. 4.

Table 1: Hot star candidates. J2000 right ascension (R.A.) and declination ($\delta$) taken from GALACTICNUCLEUS. All NIR measurements are those of GALACTICNUCLEUS, except for stars brighter than $K_{S}=11.5$ mag whose $K_{S}$ photometry is taken from 2MASS. The IRAC MIR photometry is that of Ramírez et al. (2008) catalogue. The last column shows the ID of previously known hot stars: $a$: Dong et al. (2011, Table 3); $b$: Clark et al. (2018a); and $c$: Clark et al. (2018b). The label NaN means no detection in a particular band. R.A. (deg) |  $\delta$ (deg) | $J$ | e$J$ | $H$ | e$H$ | $K_{S}$ | e$K_{S}$ | $[3.6\mu]$ | e$[3.6\mu]$ | $[4.5\mu]$ | e$[4.5\mu]$ | ID
---|---|---|---|---|---|---|---|---|---|---|---|---
$266.38721$ | $-28.91581$ | $18.57$ | $0.10$ | $14.57$ | $0.08$ | $12.66$ | $0.09$ | NaN | NaN | $10.16$ | $0.01$ |
$266.16985$ | $-29.19812$ | $15.85$ | $0.09$ | $11.95$ | $0.09$ | $9.77$ | $0.03$ | $7.89$ | $0.01$ | $7.69$ | $0.01$ |
$266.31870$ | $-28.90582$ | $20.20$ | $0.10$ | $14.64$ | $0.09$ | $11.42$ | $0.05$ | $9.01$ | $0.01$ | $8.57$ | $0.01$ |
$266.30472$ | $-28.91789$ | $17.41$ | $0.09$ | $13.12$ | $0.09$ | $10.78$ | $0.02$ | $8.64$ | $0.01$ | $7.77$ | $0.01$ |
$266.45230$ | $-28.83499$ | $17.29$ | $0.09$ | $13.27$ | $0.09$ | $11.02$ | $0.05$ | $9.10$ | $0.01$ | $8.48$ | $0.01$ | $96^{a}$
$266.55600$ | $-28.81649$ | $14.58$ | $0.10$ | $11.58$ | $0.10$ | $9.69$ | $0.03$ | $8.45$ | $0.01$ | $8.03$ | $0.01$ | $4^{a},31^{c}$
$266.54169$ | $-28.92569$ | $15.03$ | $0.08$ | $12.36$ | $0.10$ | $10.75$ | $0.04$ | $9.72$ | $0.01$ | $9.38$ | $0.01$ | $23^{a}$
$266.41382$ | $-28.88921$ | $14.57$ | $0.09$ | $11.79$ | $0.09$ | $10.19$ | $0.05$ | $8.94$ | $0.01$ | $8.52$ | $0.01$ | $106^{a}$
$266.40787$ | $-29.10816$ | $15.74$ | $0.09$ | $13.03$ | $0.09$ | $11.53$ | $0.02$ | $10.54$ | $0.01$ | $10.09$ | $0.01$ | $40^{a}$
$266.46439$ | $-28.82381$ | $16.78$ | $0.08$ | $13.19$ | $0.08$ | $11.20$ | $0.07$ | $9.72$ | $0.01$ | $8.99$ | $0.01$ | $78^{a},1^{b}$
$266.44547$ | $-28.77219$ | $17.81$ | $0.09$ | $14.37$ | $0.08$ | $12.63$ | $0.09$ | $10.86$ | $0.02$ | $10.56$ | $0.02$ |
$266.56671$ | $-28.82275$ | $15.18$ | $0.09$ | $12.24$ | $0.09$ | $10.47$ | $0.05$ | $9.37$ | $0.05$ | $9.03$ | $0.06$ | $72^{a},21^{c}$
Figure 4: Hot star candidates shown over the IRAC $[4.5\mu]$ image. The image
is approximately centred on the Milky Way NSC. The HST NICMOS Paschen-$\alpha$
survey area is indicated by the white dashed lines. The dark blue circles show
previously known massive stars belonging to our candidate list. The light blue
circles show the positions of our hot candidates that have not been reported
in previous literature.

According to recent census of massive young stars in the Arches and Quintuplet
clusters (Clark et al. 2018a, b), we detect one star belonging to the former
cluster (top circled detection of Fig 5) and two stars belonging to the latter
cluster (both circled stars in Fig. 6).

Figure 5: Hot stars (blue circles) identified in the outskirts of the Arches
cluster; only the top-most hot stars belongs to the cluster. From left to
right: IRAC channel 2 ($[4.5\mu\mathrm{m}]$), HST NICMOS Paschen-$\alpha$
image and GALACTICNUCLEUS $K_{S}$ band image. Figure 6: Same as in Fig. 5 but
for the Quintuplet cluster.

Presumably, we do not detect more known massive stars belonging to either
cluster because of our strict close neighbour exclusion, and because the IRAC
high confusion limits in such crowded fields.

##  5 Conclusions

Studies of star formation at the GC are limited by interstellar extinction
that complicates the application of broad-band photometry for even a rough
stellar classification. The RJCE method combines NIR and MIR data and uses the
fact that most stars have a very narrow distribution of intrinsic colours in
given combinations of NIR and MIR filter and is of great interest for highly
extinguished regions of the Galaxy. Unfortunately there is no way to image
large fields from the ground in the MIR, which can only be done from space.
The upcoming JWST, with its high angular resolution ($\lesssim
0\aas@@fstack{\prime\prime}2$ at wavelengths ($\lesssim 5\,\mu$m)) can
therefore provide a breakthrough in our understanding of star formation at the
GC.

Currently, we have only low angular resolution MIR photometry from
IRAC/Spitzer available. We used these data and combined them with the high
angular resolution NIR GALACTICNUCLEUS survey in a pilot study to demonstrate
the application of the RJCE method to the GC. Even though we are only left
with about 1 000 stars in our sample after applying the necessary data
selection criteria, we can identify candidates for hot stars in the de-
reddened CMD. The latter are placed in a separate sequence that is clearly
placed to the blue of the RGB. About half of our candidates are known from
previous studies and some of these are known members of the Arches and
Quintuplet clusters. We also report on five hot star candidates that have not
been previously reported in the literature.

###### Acknowledgements.

R. S. acknowledges financial support from the State Agency for Research of the
Spanish MCIU through the “Center of Excellence Severo Ochoa” award for the
Instituto de Astrofísica de Andalucía (SEV-2017-0709). R. S. acknowledges
financial support from national project PGC2018-095049-B-C21 (MCIU/AEI/FEDER,
UE). F.N.-L. gratefully acknowledges support by the Deutsche
Forschungsgemeinschaft (DFG, German Research Foundation) – Project-ID
138713538 – SFB 881 (”The Milky Way System”, subproject B8).

## References

  * Bartko et al. (2010) Bartko, H., Martins, F., Trippe, S., et al. 2010, ApJ, 708, 834
  * Clark et al. (2018a) Clark, J. S., Lohr, M. E., Najarro, F., Dong, H., & Martins, F. 2018a, A&A, 617, A65
  * Clark et al. (2018b) Clark, J. S., Lohr, M. E., Patrick, L. R., et al. 2018b, A&A, 618, A2
  * Dong et al. (2011) Dong, H., Wang, Q. D., Cotera, A., et al. 2011, MNRAS, 417, 114
  * Fazio et al. (2004) Fazio, G. G., Hora, J. L., Allen, L. E., et al. 2004, ApJS, 154, 10
  * Figer et al. (2004) Figer, D. F., Rich, R. M., Kim, S. S., Morris, M., & Serabyn, E. 2004, ApJ, 601, 319
  * Fritz et al. (2011) Fritz, T. K., Gillessen, S., Dodds-Eden, K., et al. 2011, ApJ, 737, 73
  * Gallego-Cano et al. (2020) Gallego-Cano, E., Schödel, R., Nogueras-Lara, F., et al. 2020, A&A, 634, A71
  * Genzel et al. (2010) Genzel, R., Eisenhauer, F., & Gillessen, S. 2010, Reviews of Modern Physics, 82, 3121
  * Ghez et al. (2008) Ghez, A. M., Salim, S., Weinberg, N. N., et al. 2008, ApJ, 689, 1044
  * Hatano et al. (2013) Hatano, H., Nishiyama, S., Kurita, M., et al. 2013, AJ, 145, 105
  * Hosek et al. (2019) Hosek, Matthew W., J., Lu, J. R., Anderson, J., et al. 2019, ApJ, 870, 44
  * Jian et al. (2017) Jian, M., Gao, S., Zhao, H., & Jiang, B. 2017, AJ, 153, 5
  * Launhardt et al. (2002) Launhardt, R., Zylka, R., & Mezger, P. G. 2002, A&A, 384, 112
  * Lu et al. (2013) Lu, J. R., Do, T., Ghez, A. M., et al. 2013, ApJ, 764, 155
  * Majewski et al. (2011) Majewski, S. R., Zasowski, G., & Nidever, D. L. 2011, ApJ, 739, 25
  * Matsunaga et al. (2009) Matsunaga, N., Kawadu, T., Nishiyama, S., et al. 2009, MNRAS, 399, 1709
  * Matsunaga et al. (2011) Matsunaga, N., Kawadu, T., Nishiyama, S., et al. 2011, Nature, 477, 188
  * Mauerhan et al. (2010) Mauerhan, J. C., Muno, M. P., Morris, M. R., Stolovy, S. R., & Cotera, A. 2010, ApJ, 710, 706
  * Nagayama et al. (2003) Nagayama, T., Nagashima, C., Nakajima, Y., et al. 2003, in Instrument Design and Performance for Optical/Infrared Ground-based Telescopes, ed. M. Iye & A. F. M. Moorwood, Vol. 4841, International Society for Optics and Photonics (SPIE), 459 – 464
  * Nandakumar et al. (2018) Nandakumar, G., Schultheis, M., Feldmeier-Krause, A., et al. 2018, A&A, 609, A109
  * Nishiyama et al. (2006) Nishiyama, S., Nagata, T., Kusakabe, N., et al. 2006, ApJ, 638, 839
  * Nishiyama & Schödel (2013) Nishiyama, S. & Schödel, R. 2013, A&A, 549, A57
  * Nishiyama et al. (2009) Nishiyama, S., Tamura, M., Hatano, H., et al. 2009, ApJ, 696, 1407
  * Nogueras-Lara et al. (2018) Nogueras-Lara, F., Gallego-Calvente, A. T., Dong, H., et al. 2018, A&A, 610, A83
  * Nogueras-Lara et al. (2019a) Nogueras-Lara, F., Schödel, R., Gallego-Calvente, A. T., et al. 2019a, A&A, 631, A20
  * Nogueras-Lara et al. (2020a) Nogueras-Lara, F., Schödel, R., Gallego-Calvente, A. T., et al. 2020a, Nature Astronomy, 4, 377
  * Nogueras-Lara et al. (2019b) Nogueras-Lara, F., Schödel, R., Najarro, F., et al. 2019b, A&A, 630, L3
  * Nogueras-Lara et al. (2021) Nogueras-Lara, F., Schödel, R., & Neumayer, N. 2021, arXiv e-prints, arXiv:2106.04529
  * Nogueras-Lara et al. (2020b) Nogueras-Lara, F., Schödel, R., Neumayer, N., et al. 2020b, A&A, 641, A141
  * Pietrinferni et al. (2013) Pietrinferni, A., Cassisi, S., Salaris, M., & Hidalgo, S. 2013, ApJ, 558, A46
  * Portegies Zwart et al. (2002) Portegies Zwart, S. F., Makino, J., McMillan, S. L. W., & Hut, P. 2002, ApJ, 565, 265
  * Ramírez et al. (2008) Ramírez, S. V., Arendt, R. G., Sellgren, K., et al. 2008, ApJS, 175, 147
  * Reach et al. (2005) Reach, W. T., Megeath, S. T., Cohen, M., et al. 2005, PASP, 117, 978
  * Reiter et al. (2015) Reiter, M., Marengo, M., Hora, J. L., & Fazio, G. G. 2015, MNRAS, 447, 3909
  * Schödel et al. (2014) Schödel, R., Feldmeier, A., Kunneriath, D., et al. 2014, A&A, 566, A47
  * Schödel et al. (2018) Schödel, R., Gallego-Cano, E., Dong, H., et al. 2018, A&A, 609, A27
  * Schödel et al. (2010) Schödel, R., Najarro, F., Muzic, K., & Eckart, A. 2010, A&A, 511, A18
  * Schödel et al. (2013) Schödel, R., Yelda, S., Ghez, A., et al. 2013, MNRAS, 429, 1367
  * Scoville et al. (2003) Scoville, N., Stolovy, S. R., & Christopher, M. 2003, Astronomische Nachrichten Supplement, 324, 591
  * Shahzamanian et al. (2019) Shahzamanian, B., Schödel, R., Nogueras-Lara, F., et al. 2019, A&A, 632, A116
  * Skrutskie et al. (2006) Skrutskie, M. F., Cutri, R. M., Stiening, R., et al. 2006, ApJ, 131, 1163
  * Stolovy et al. (2006) Stolovy, S., Ramirez, S., Arendt, R. G., et al. 2006, in Journal of Physics Conference Series, Vol. 54, Journal of Physics Conference Series, 176–182
  * Wang et al. (2010) Wang, Q. D., Dong, H., Cotera, A., et al. 2010, MNRAS, 402, 895
  * Yusef-Zadeh et al. (2009) Yusef-Zadeh, F., Hewitt, J. W., Arendt, R. G., et al. 2009, ApJ, 702, 178


*

# Rejoinder: Gaussian Differential Privacy

Jinshuo Dong Graduate Group in Applied Mathematics and Computational Science,
University of Pennsylvania. Email<EMAIL_ADDRESS>Aaron Roth
Department of Computer and Information Sciences, University of Pennsylvania.
Email<EMAIL_ADDRESS>Weijie J. Su Department of Statistics, the
Wharton School, University of Pennsylvania. Email<EMAIL_ADDRESS>
(April 2021)

We warmly thank Editor Paul Smith for selecting our paper for discussion and
are extremely grateful to all the discussants for taking their valuable time
to provide engaging and stimulating feedback on our work. These insights
situate our work in context and provide promising directions for future
research. We are excited to see that thoughts about theoretical complements
and new applications are already emerging.

A general view, shared by all discussants, is that privacy is a first-order
concern in many data science problems. We are very pleased to learn that our
statistics community welcomes new foundational development and methodological
contributions that allow for privacy protections in statistical data analysis.

In this rejoinder, we aim to address two broad issues that cover most comments
made in the discussion. First, we discuss some theoretical aspects of our work
and comment on how this work might impact the theoretical foundation of
privacy-preserving data analysis. Taking a practical viewpoint, we next
discuss how $f$-differential privacy ($f$-DP) and Gaussian differential
privacy (GDP) can make a difference in a range of applications.

##  1 Theoretical Aspects of $f$-DP

As echoed by many discussants, the formalization of $f$-DP in our paper (Dong
et al., 2021a) starts from a decision-theoretic interpretation of a
“differential” privacy attack, which originates from the work of Wasserman and
Zhou (2010). The binary nature of the decision-theoretic problems renders the
classical theory of hypothesis testing a basic tool. Specifically, we use the
trade-off between type I and type II errors of the hypothesis testing problem
as our privacy measure. In response to a question raised by Dr. Jorge Mateu,
we remark that this trade-off is concerned with a thought experiment in which
someone is trying to determine if an individual’s data is in the dataset or
not, rather than a hypothesis test that is actually conducted. It can
therefore always be reasoned about analytically/formally, without needing
simulation, even if the algorithms themselves are complex or simulation based.

This treatment of privacy cost in $f$-DP comes with several technical
properties that can facilitate the development of better differentially
private algorithms. As highlighted by Dr. Borja Balle, $f$-DP gives tight and
analytically tractable formulas for composition. This appealing feature arises
from applying the central limit theorem to the privacy loss random variables,
thereby making GDP a canonical single-parameter family of privacy definitions
within the $f$-DP class. While we did not attempt to push hard on weakening
assumptions for the privacy central limit theorems, there are several possible
extensions. For example, one may be able to identify a necessary and
sufficient condition for the privacy central limit theorem to hold, just like
the Lindeberg–Feller condition for the usual central limit theorems. Another
possibility is to sharpen the central limit theorem by leveraging a refined
analysis of the privacy loss random variables (Zheng et al., 2020). More
specifically, Dr. Sebastian Dietz suggested a very interesting direction for
improving the composition formulas by making use of a third-moment-free
central limit theorem (see, for example, DasGupta (2008)). A successful
investigation in this direction might extend the applicability of the
composition formulas to $(\varepsilon,\delta)$-DP and others. More broadly, it
would also be interesting to explore central limit theorem phenomena of
privacy beyond composition. For example, Dong et al. (2021b) recently showed
that a related central limit theorem occurs in high-dimensional query
answering and yet privacy cost is best described in the framework of $f$-DP.
We see all these as interesting future directions for broadening the scope of
the hypothesis testing viewpoint on differential privacy.

In addition to composition, subsampling is another important primitive that is
involved in many algorithm designs. As pointed out by Dr. Borja Balle, while
divergence-based privacy definitions face technical difficulties in describing
privacy amplification by subsampling, $f$-DP gives a relatively concise and
coherent expression for understanding how privacy is amplified using this
primitive. This also gives a sharper privacy analysis of subsampling than can
be obtained by directly using $(\varepsilon,\delta)$-DP. An interesting
observation made by Dr. Borja Balle is that the significant gap between the
two frameworks seems surprising, and warrants further investigation. It is
also worth developing similar privacy analyses for the various flavors of
subsampling schemes used in training deep learning models (not all of which
involve _independent_ sampling across rounds).

##  2 Applications of $f$-DP

Our main hope for $f$-DP is to see as many applications as possible to
improving the privacy analyses of the diverse algorithms used in a variety of
data science problems. Encouragingly, we found many such possibilities in the
discussants’ contributions that either tackle important problems or show great
promise.

Understanding the tradeoff between privacy and utility for various statistical
and computational tasks is _the_ central object of study in the differentially
privacy literature. The main point of $f$-DP and GDP is to make it possible to
capture this fundamental tradeoff more precisely. As a result, the $f$-DP
framework allows us to obtain better tradeoffs between privacy guarantees and
data usability. This tradeoff is different in different applications, and
requires analyses on a case by case basis. We emphasize, as pointed out by
several of the discussants, that privacy protection does inevitably come with
utility loss. Indeed, this is a consequence of the “fundamental law of
information recovery”, which applies not just to differential privacy but to
any method of releasing data. So while it is true that differential privacy
can harm utility (especially for small datasets), this is not an artifact of
differential privacy, but an actual, fundamental tradeoff that we have to
grapple with as a society. We can choose to get exact statistics about our
data, but we should understand that this means giving up on privacy.
Differential privacy takes no stand on how we should mediate this fundamental
tradeoff: rather it provides a precise language in which to talk about it.

###  2.1 $f$-DP for Stochastic Optimization

To appreciate how sharply this tradeoff can be characterized using a given
privacy definition, perhaps the best benchmark is stochastic gradient descent
(SGD), the basic foundation for many machine learning algorithms. Owing to its
effectiveness in handling composition and subsampling, $f$-DP gives a tighter
privacy analysis of SGD than the moments accountant technique (Abadi et al.,
2016), which further feeds back into improved test accuracy of trained deep
learning models at fixed privacy guarantees (Bu et al., 2020). We are
delighted that Dr. Borja Balle wrote “I would encourage practitioners to take
note of this and start using Gaussian DP accounting in their DP-SGD
implementations.”

Moving forward, Dr. Marco Avella-Medina raised several interesting and
important questions regarding private SGD with $f$-DP guarantees. Although
gradient clipping is a necessary step in private SGD that ensures bounded
sensitivity to any single data point, this step can lead to inconsistency for
some problems. To go around this difficulty, Avella-Medina suggested using a
consistent bounded influence $M$-estimator from robust statistics, which we
believe is a promising approach worthy of future research effort. Moreover, we
are glad to see that Avella-Medina et al. (2021) introduced a kind of noisy
gradient descent and analyzed its Gaussian differential privacy properties.
This opens an exciting research avenue to understand when noisy gradient
descent outperforms SGD.

###  2.2 Other Applications

Differential privacy has applications beyond machine learning. A promising
application area — due to strict privacy regulation — is in the analysis and
sharing of medical data. A challenge in medical applications is that the size
of the relevant datasets is often relatively small. The improved tradeoff
between privacy and utility is especially important in the challenging small
data regime. As noted by Drs. Peter Krusche and Frank Bretz, there are
obstacles to combining data across hospitals — for which we think differential
privacy might be able to help. Noisy access to a large dataset might be better
— even from the perspective of utility — than exact access to only a small
local dataset. When applying privacy protections to small data, it is
especially important not be as tight as possible in accounting for privacy
loss, which is one of the main benefits of the $f$-DP framework.

Dr. Jorge Mateu brought up privacy issues that arise when analyzing trajectory
data. In principle, $f$-DP and GDP can be applied to any kind of data, such as
trajectory data. It would make sense, for example, to think about the question
of releasing statistics about trajectory data or a synthetic dataset
consisting of trajectories that maintain consistency with the real data with
respect to various statistics of interest, so long as those statistics have
low sensitivity and vary only mildly with the data of individuals. These types
of problems deserve further study. Of course, providing useful analyses of a
single _individual_ ’s trajectory is by design prevented by technologies that
aim to preserve individual privacy. A related question, asked by Dr. Christine
Chai, was whether the $f$-DP framework can be applied to COVID-19 contact
tracing data. Differential privacy ($(\varepsilon,\delta)$-DP, $f$-DP, or any
related variant) is not directly applicable to what is most commonly known as
contact tracing — letting contacts know that someone with COVID-19 has been
near them — since by design, this is highly sensitive to a single data point.
However, GDP (as well as other differential privacy variants) can be used to
improve population level statistics related to contact tracing, such as how
crowded grocery stores are by time and mobility data, or even what fraction of
visitors to a grocery store in a given day have had potential COVID-19
exposure. More generally, we believe that $f$-DP has many more connections to
various aspects of data science.

Finally, we remark that there are many heuristic approaches to privacy that do
not come with the guarantees of differential privacy. There is a vast
literature of pros and cons among these approaches, which is beyond the scope
of this paper — but in general, “syntactic” approaches do not stand up to
attack by a determined adversary. In particular, synthetic data is known to be
neither necessary nor sufficient for privacy — but also not incompatible with
differential privacy. For example, there is a large literature on generating
differentially private synthetic data (see, e.g., Blum et al. (2013); Gaboardi
et al. (2014); Vietri et al. (2020); Aydore et al. (2021); Jordon et al.
(2018); Beaulieu-Jones et al. (2019)), most of which we believe can be
improved by $f$-DP style analyses.

## References

  * Abadi et al. [2016] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang.  Deep learning with differential privacy.  In _Proceedings of the 2016 ACM SIGSAC conference on computer and communications security_ , pages 308–318, 2016.
  * Avella-Medina et al. [2021] M. Avella-Medina, C. Bradshaw, and P.-L. Loh.  Differentially private inference via noisy optimization.  _arXiv preprint arXiv:2103.11003_ , 2021.
  * Aydore et al. [2021] S. Aydore, W. Brown, M. Kearns, K. Kenthapadi, L. Melis, A. Roth, and A. Siva.  Differentially private query release through adaptive projection.  _arXiv preprint arXiv:2103.06641_ , 2021.
  * Beaulieu-Jones et al. [2019] B. K. Beaulieu-Jones, Z. S. Wu, C. Williams, R. Lee, S. P. Bhavnani, J. B. Byrd, and C. S. Greene.  Privacy-preserving generative deep neural networks support clinical data sharing.  _Circulation: Cardiovascular Quality and Outcomes_ , 12(7):e005122, 2019.
  * Blum et al. [2013] A. Blum, K. Ligett, and A. Roth.  A learning theory approach to noninteractive database privacy.  _Journal of the ACM (JACM)_ , 60(2):1–25, 2013\.
  * Bu et al. [2020] Z. Bu, J. Dong, Q. Long, and W. J. Su.  Deep learning with Gaussian differential privacy.  _Harvard Data Science Review_ , 2020(23), 2020.
  * DasGupta [2008] A. DasGupta.  _Asymptotic theory of statistics and probability_.  Springer Science & Business Media, 2008.
  * Dong et al. [2021a] J. Dong, A. Roth, and W. J. Su.  Gaussian differential privacy.  _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , 2021a.
  * Dong et al. [2021b] J. Dong, W. J. Su, and L. Zhang.  A central limit theorem for differentially private query answering.  _arXiv preprint arXiv:2103.08721_ , 2021b.
  * Gaboardi et al. [2014] M. Gaboardi, E. J. G. Arias, J. Hsu, A. Roth, and Z. S. Wu.  Dual query: Practical private query release for high dimensional data.  In _International Conference on Machine Learning_ , pages 1170–1178. PMLR, 2014.
  * Jordon et al. [2018] J. Jordon, J. Yoon, and M. Van Der Schaar.  Pate-gan: Generating synthetic data with differential privacy guarantees.  In _International Conference on Learning Representations_ , 2018.
  * Vietri et al. [2020] G. Vietri, G. Tian, M. Bun, T. Steinke, and S. Wu.  New oracle-efficient algorithms for private synthetic data release.  In _International Conference on Machine Learning_ , pages 9765–9774. PMLR, 2020.
  * Wasserman and Zhou [2010] L. Wasserman and S. Zhou.  A statistical framework for differential privacy.  _Journal of the American Statistical Association_ , 105(489):375–389, 2010.
  * Zheng et al. [2020] Q. Zheng, J. Dong, Q. Long, and W. J. Su.  Sharp composition bounds for Gaussian differential privacy via Edgeworth expansion.  In _International Conference on Machine Learning_ , pages 11420–11435, 2020.