11institutetext:  University of Tübingen 22institutetext: Max Planck Institute
for Intelligent Systems

# Uncertainty-Guided Progressive GANs for Medical Image Translation

Uddeshya Upadhyay 11 Yanbei Chen 11 Tobias Hepp 1122
Sergios Gatidis 1122 Zeynep Akata 1122

###### Abstract

Image-to-image translation plays a vital role in tackling various medical
imaging tasks such as attenuation correction, motion correction, undersampled
reconstruction, and denoising. Generative adversarial networks have been shown
to achieve the state-of-the-art in generating high fidelity images for these
tasks. However, the state-of-the-art GAN-based frameworks do not estimate the
uncertainty in the predictions made by the network that is essential for
making informed medical decisions and subsequent revision by medical experts
and has recently been shown to improve the performance and interpretability of
the model. In this work, we propose an uncertainty-guided progressive learning
scheme for image-to-image translation. By incorporating aleatoric uncertainty
as attention maps for GANs trained in a progressive manner, we generate images
of increasing fidelity progressively. We demonstrate the efficacy of our model
on three challenging medical image translation tasks, including PET to CT
translation, undersampled MRI reconstruction, and MRI motion artefact
correction. Our model generalizes well in three different tasks and improves
performance over state of the art under full-supervision and weak-supervision
with limited data. Code is released here:
https://github.com/ExplainableML/UncerGuidedI2I

###### Keywords:

Image-to-image translation Uncertainty estimation Progressive GANs PET CT MRI
Artefact correction

##  1 Introduction

In the medical domain, each imaging modality reflects particular physical
properties of the tissue under examination. This results in images with
different dimensionality, spatial resolution, and contrast. Various imaging
modalities provide a complimentary stream of information for clinical
diagnostics or technical pre and post-processing steps. Moreover, acquiring
medical images is susceptible to various kinds of noise and modality-specific
artefacts. To remedy these issues, translating images between different
domains is of great importance.

Inter-modal image-to-image translation can potentially replace additional
acquisition procedures, reducing examination costs and time. Besides, intra-
modality image-to-image translation enables complex artefact and noise
correction. For example, attenuation correction of positron emission
tomography (PET) data is challenging in situations where no density
distribution is available from computed tomography (CT) data, as in the case
for stand-alone PET scanners or combined PET/magnetic resonance imaging (MRI).
In these situations, the generation of pseudo-CTs from PET data can be
helpful. Further examples are related to image reconstruction and/or
correction in MRI: Reconstruction of undisturbed artifact-free images is hard
to achieve with traditional methods; deep-learning-based image-to-image
translation can solve this challenge. In particular, generative adversarial
networks (GAN) based on convolutional neural networks (CNN) have proven to
provide a high visual quality of the generated synthetic images. However,
predictions of GANs can be unreliable, and particularly in medical
applications, the quantification of uncertainty is of high importance for the
interpretation of the results. In this work, we propose a generic end-to-end
model that introduces high-capacity conditional progressive GANs to synthesize
high-quality images, using aleatoric uncertainty estimates as the guide to
focus on improving image quality in regions where the network is highly
uncertain about the prediction. We perform experiments on three challenging
and vital medical imaging tasks: PET to CT translation, undersampled MRI
reconstruction, and motion correction in MRI. Moreover, we empirically
demonstrate the efficacy of our model under weak supervision with limited
data.

##  2 Related Works

Traditional machine learning techniques for medical image translation rely on
explicit feature representations [6, 35, 12, 17]. More recently, convolutional
neural networks have been proposed for various image translation tasks [13,
19, 4, 5, 8, 3] and state-of-the-art performance is achieved by generative
adversarial networks [15, 29, 30, 32, 7, 31, 9, 2, 1, 21, 22]. The existing
methods propose conditional GAN architectures with deterministic outputs that
typically uses $\mathcal{L}_{1}/\mathcal{L}_{2}$-based fidelity loss for the
generator assumes a pixel-wise homoscedasticity and also assumes the pixel-
wise error (i.e., residual) to be independent and identically distributed
(i.i.d) following a Laplace or Gaussian distribution. This is a limiting
assumption as explained in [10, 23, 25]. While these methods can provide
synthetic images of high visual quality, the image content may still deviate
significantly from the corresponding ground-truth. This results in
overconfidence or misinterpretation with negative consequences, particularly
in the medical domain. There have been recent works on quantifying aleatoric
and epistemic uncertainty in task-specific medical imaging algorithms like
classification, segmentation, super-resolution etc [14, 25, 26, 27, 20]
quantifying it for general image-to-image translation problem largely remains
unexplored. Thus, the central motivation of our work is to provide measures of
uncertainty for image-to-image translation tasks that can contribute to safe
applications of results.

Moreover, recent work has shown that high-capacity generators that are
progressive in nature lead to high-quality results as described in [2, 1, 9].
However, the progressive generation of high-quality images remains unguided
without specifically attending to poorly translated regions. Prior works
indicate a correlation between estimated uncertainty and prediction error [33,
20, 23]. We exploit this relationship for the progressive enhancement of
synthetic images, which has not been investigated by prior work before.

##  3 Uncertainty-Guided Progressive GAN (UP-GAN)

Let $A$ and $B$ be two image domains with a set of images
$S_{A}:=\\{a_{1},a_{2}...a_{n}\\}$ and $S_{B}:=\\{b_{1},b_{2}...b_{m}\\}$
where $a_{i}$ and $b_{i}$ represent the $i^{th}$ image from domain $A$ and $B$
respectively. Let each image drawn from an underlying unknown probability
distribution $\mathcal{P}_{AB}$, i.e.,
$(a_{i},b_{i})\sim\mathcal{P}_{AB}\forall i$ have $K$ pixels, and $u_{ik}$
represent the $k^{th}$ pixel of a particular image $u_{i}$. Our goal is to
learn a mapping from domain $A$ to $B$ ($A\rightarrow B$) in a paired manner,
i.e., learning the underlying conditional distribution $\mathcal{P}_{B|A}$
from the set of given samples $\\{(a_{i},b_{i})\\}$, following the
distribution $\mathcal{P}_{AB}$. For a given image $a_{i}$ in domain $A$, the
estimated image in domain $B$ is called $\hat{b}_{i}$. The pixel wise error is
defined as $\epsilon_{ij}=\hat{b}_{ij}-b_{ij}$. While the existing framework
models the residual as the i.i.d as described above, we relax that assumption
by modelling the residual as non i.i.d variables and learning the optimal
distribution from the dataset, as described in the following.

Figure 1:  Uncertainty-guided Progressive GANs (UP-GAN): The primary GAN takes
the input image from domain $A$, while subsequent GANs absorb outputs from the
preceding GAN (see Eq. 3 and 4). Explicitly guided by the attention maps, the
uncertainty maps are estimated from the preceding GAN.

Figure 1 shows our model that consists of cascaded GANs, where each generator
is capable of estimating the aleatoric uncertainty, along with generating
images. Our solution alleviates the aforementioned limitations of recent
methods by modelling the underlying per-pixel residual distribution as
independent but non-identically distributed zero-mean generalized Gaussian
distribution (GGD) as in [23], where the network learns to predict the optimal
scale ($\alpha$) and shape ($\beta$) of the GGD for every pixel, Therefore,
$\hat{b}_{ij}=b_{ij}+\epsilon_{ij}$ with, $\epsilon_{ij}\sim
GGD(\epsilon;0,\alpha_{ij},\beta_{ij})\equiv\beta_{ij}(2\alpha_{ij}\Gamma(\beta^{-1}_{ij}))^{-1}\exp{\left(-\alpha_{ij}^{-1}|\epsilon|^{\beta_{ij}}\right)}$.
We generate images in multiple phases, with each phase generating output
images along with the aleatoric uncertainty estimates. The outputs from one
phase serve as the input to the subsequent GAN in the next phase, explicitly
guided by the attention map derived from uncertainty estimates. Importantly,
this uncertainty-based guidance enforces the model to focus on refining the
uncertain regions that are likely to be poorly synthesized, resulting in
progressively improving quality.

Our framework is composed of a sequence of $M$ GANs, where the $m^{th}$ GAN is
represented by a pair of networks, generator and discriminator, given by,
$(\mathcal{G}_{m}(\cdot;\theta_{m}),\mathcal{D}_{m}(\cdot;\phi_{m}))$. Both
the generator and discriminator can have arbitrary network architecture as
long as generator can estimate aleatoric uncertainty as described in [23]. We
choose all the discriminators to be the patch discriminators from [7] and
generators to be modified U-Net [16], where the head is split into three to
estimate the parameters of the GGD as shown in Figure 1 and in [23].

Primary GAN. We train the first GAN ($\mathcal{G}_{0}$) using the dataset
$S_{A}$ and $S_{B}$. The predictions of the generator are given by
$(\hat{\alpha}_{[0]i},\hat{\beta}_{[0]i},\hat{b}_{[0]i})$. The network is
trained with an adaptive fidelity loss function
$\mathcal{L}^{G}_{\alpha\beta}$ [23] and an adversarial loss
$\mathcal{L}^{G}_{\text{adv}}$ [36], combined as
$\mathcal{L}^{G}_{\text{tot}}$ for the generator
($\mathcal{G}_{0}(\cdot;\theta_{0}):A\rightarrow B$):

$\displaystyle\mathcal{L}^{G}_{\alpha\beta}(\hat{b}_{[0]i},\hat{\alpha}_{[0]i},\hat{\beta}_{[0]i},b_{i})=\frac{1}{K}\bm{\sum}_{j}\left(\frac{|\hat{b}_{[0]ij}-b_{ij}|}{\hat{\alpha}_{[0]ij}}\right)^{\hat{\beta}_{[0]ij}}-\log\frac{\hat{\beta}_{[0]ij}}{\hat{\alpha}_{[0]ij}}+\log\Gamma(\hat{\beta}_{[0]ij}^{-1})$
(1)
$\displaystyle\mathcal{L}_{\text{adv}}^{G}=\mathcal{L}_{2}(\mathcal{D}_{1}(\hat{b}_{[0]i}),1)\text{
and
}\mathcal{L}^{G}_{\text{tot}}=\lambda_{1}\mathcal{L}^{G}_{\alpha\beta}+\lambda_{2}\mathcal{L}_{\text{adv}}^{G}.$
(2)

The patch discriminator ($\mathcal{D}_{1}$) is trained using the adversarial
loss from [36] given by
$\mathcal{L}^{D}_{\text{adv}}=\mathcal{L}_{2}(\mathcal{D}^{A}(b_{i}),1)+\mathcal{L}_{2}(\mathcal{D}^{A}(\hat{b}_{[0]i}),0)$.

Subsequent GANs. The $m^{th}$ GAN (where $m>0$) takes the output produced by
the $(m-1)^{th}$ GAN, i.e.
$(\hat{\alpha}_{[m-1]i},\hat{\beta}_{[m-1]i},\hat{b}_{[m-1]i})$, along with
the original sample $a_{i}$ from domain $A$ as its input and generates a
refined output. The image estimated by the $(m-1)^{th}$ GAN along with its
uncertainty map learns to create the input feature $f_{[m]i}$ for the $m^{th}$
GAN, where the uncertainty map serves as an attention mechanism to highlight
the uncertain regions in the image. The input $a_{[m]i}$ for the $m^{th}$
generator is given by concatenating $a_{i}$ and $f_{[m]i}$, i.e.,

$\displaystyle\hat{\sigma}_{[m-1]i}=\hat{\alpha}_{[m-1]i}\sqrt{\frac{\Gamma(3/\hat{\beta}_{[m-1]i})}{\Gamma(1/\hat{\beta}_{[m-1]i})}}\text{,
and
}f_{[m]i}=\hat{b}_{[m-1]i}\odot\frac{\hat{\sigma}_{[m-1]i}}{\bm{\sum}_{j}\hat{\sigma}_{[m-1]ij}}$
(3) $\displaystyle a_{[m]i}=\mathtt{concat}(f_{[m]i},a_{i})$ (4)

The input $a_{[m]i}$ for the $m^{th}$ GAN encourages the generator to further
refine the highly uncertain regions in the image given the original input
context. The generator and the discriminator are trained using
$\mathcal{L}^{G}_{\text{tot}}$ and $\mathcal{L}^{D}_{\text{adv}}$,
respectively.

Progressive training scheme. We initialize the parameters $\theta\cup\phi$
sequentially. First, we initialize $\theta_{1}\cup\phi_{1}$ using the training
set $(S_{A},S_{B})$ to minimize the loss function given by
$\mathcal{L}^{G}_{\text{tot}}$ and $\mathcal{L}^{D}_{\text{adv}}$. Then, for
the subsequent GANs, we initialize the $\theta_{m}\cup\phi_{m}$ ($m>1$) by
fixing the weights of all the previous generators and training the $m^{th}$
GAN alone (see Eq. 3 and 4 with losses $\mathcal{L}^{G}_{\text{tot}}$ and
$\mathcal{L}^{D}_{\text{adv}}$). Once all the parameters have been initialized
(i.e., $\theta_{m}\cup\phi_{m}\forall m$), we do further fine tuning by
training all the networks end-to-end by combining the loss functions of all
the intermediate phases and a significantly smaller learning-rate.

##  4 Experiments

In this section, we first detail the experimental setup and comparative
methods in Section 4.1, and present the corresponding results in Section 4.2.

###  4.1 Experimental Setup

Tasks and datasets. We evaluate our method on the following three tasks.

(i) PET to CT translation: We synthesize CT images from PET scans to be used
for the attenuation correction, e.g. for PET-only scanners or PET/MRI. We use
paired data sets of non-attenuation-corrected PET and the corresponding CT of
the head region of 49 patients acquired on a state-of-the-art PET/CT scanner
(Siemens Biograph mCT), approved by ethics committee of the Medical Faculty of
the University of Tübingen. Data is split into 29/5/15 for training/val/test
sets. Figure 2 shows exemplary slices for co-registered PET and CT.

(ii) Undersampled MRI reconstruction: We translate undersampled MRI images to
fully-sampled MRI images. We use MRI scans from the open-sourced IXI 111from
https://brain-development.org/ixi-dataset/ dataset that consists of
T1-weighted (T1w) MRI scans. We use a cohort of 500 patients split into
200/100/200 for training/val/test, and retrospectively create the undersampled
MRI with an acceleration factor of $12.5\times$, i.e., we preserve only $8\%$
of the fully-sampled k-space measurement (from the central region) to obtain
the undersampled image.

(iii) MRI Motion correction: We generate sharp images from motion corrupted
images. We retrospectively create the motion artefacts in the T1w MRI from IXI
following the transformations in the k-space as described in [18]. Figure
3-(ii) shows the input MRI scan with artefacts and ground-truth.

Training details and evaluation metrics. All GANs are first initialized using
the aforementioned progressive learning scheme with
$(\lambda_{1},\lambda_{2})$ in Eq. 2 set to $(1,0.001)$. We use Adam [11],
with the hyper-parameters $\beta_{1}:=0.9$, $\beta_{2}:=0.999$, an initial
learning rate of 0.002 for initialization and 0.0005 post-initialization that
decays based on cosine annealing over 1000 epochs, using a batch size of 8. We
use three widely adopted metrics to evaluate image generation quality: PSNR
measures $20\log{\text{MAX}_{I}/\sqrt{\text{MSE}}}$, where $\text{MAX}_{I}$ is
the highest possible intensity value in the image and MSE is the mean-squared-
error between two images. SSIM computes the structural similarity between two
images [28]. MAE computes the mean absolute error between two images. Higher
PSNR, SSIM, and lower MAE indicate a higher quality of the generated images
(wrt ground-truth).

Compared methods. We compare our model to representative state-of-the-art
methods for medical image translation, including Pix2pix [7], a baseline
conditional adversarial networks for image-to-image translation tasks using
GANs, PAN [24], and MedGAN [2], a GAN-based method that relies on external-
pre-trained feature extractors, with a generator that refines the generated
images progressively. MedGAN is shown to perform superior to methods like,
Fila-sGAN [34], ID-cGAN [32], and achieve state-of-the-art performance for
several medical image-to-image translation problems.

Figure 2: Outputs from different phases of UP-GAN (with M=3). (Top) The input
(uncorrected PET), the corresponding ground-truth CT, mean residual values
over different phases, mean uncertainty values over different phases. (Bottom)
Each row shows the predicted output, the residual between the prediction and
the ground-truth, the predicted scale ($\alpha$) map, the predicted shape
($\beta$) map, the uncertainty map, and the uncertainty in high residual
regions.

Figure 3: Qualitative results. (Top) PET to CT translation. (Bottom)
Undersampled MRI reconstruction (left), and MRI motion correction (right).

Methods | PET to CT | Undersampled MRI Recon. | MRI Motion Correction
---|---|---|---
SSIM | PSNR | MAE | SSIM | PSNR | MAE | SSIM | PSNR | MAE
pix2pix [7] | 0.89$\pm$0.04 | 26.0$\pm$2.0 | 38.5$\pm$10.7 | 0.92$\pm$0.03 | 28.5$\pm$0.9 | 27.6$\pm$9.3 | 0.94$\pm$0.06 | 29.6$\pm$1.4 | 26.3$\pm$8.2
PAN [24] | 0.90$\pm$0.08 | 26.5$\pm$4.5 | 37.2$\pm$15.6 | 0.93$\pm$0.05 | 28.8$\pm$0.7 | 26.2$\pm$10.4 | 0.95$\pm$0.10 | 30.1$\pm$2.8 | 24.9$\pm$9.7
MedGAN [2] | 0.90$\pm$0.04 | 27.1$\pm$2.5 | 35.4$\pm$11.8 | 0.94$\pm$0.02 | 29.7$\pm$1.9 | 24.2$\pm$8.7 | 0.95$\pm$0.04 | 30.8$\pm$1.8 | 23.6$\pm$9.1
UP-GAN | 0.95$\pm$0.05 | 28.9$\pm$0.4 | 24.7$\pm$12.9 | 0.97$\pm$0.07 | 29.4$\pm$2.1 | 24.1$\pm$7.5 | 0.96$\pm$0.03 | 32.1$\pm$0.3 | 22.8$\pm$11.1

Table 1:  Evaluation of various methods on three medical image translation
tasks.

###  4.2 Results and Analysis

Qualitative results. Figure 2 visualizes the (intermediate) outputs of the
generators at different phases of the framework. The visual quality of the
generated image content increasingly improves along the network phases (as
shown in the first column, second row onward). At the same time, prediction
error and uncertainty decrease continuously (second column and fifth column,
second row onward, respectively). High uncertainty values are found in
anatomical regions with fine osseous structures, such as the nasal cavity and
the inner ear in the petrous portion of the temporal bone. Particularly in
such regions of high uncertainty, we achieve a progressive improvement in the
level of detail.

Figure 3-(Top) visualizes the generated CT images from the PET for all the
compared methods along with our methods. We observe that more high-frequency
features are present in our prediction compared to the previous state-of-the-
art model (MedGAN). We also observe that the overall residual is significantly
lower for our method compared to the other baselines. MedGAN performs better
than pix2pix in synthesizing high-frequency features and sharper images.
Figure 3-(Bottom) shows similar results for the undersampled MRI
reconstruction task and MRI motion correction task. In both cases, our model
yields superior images, as can be seen via relatively neutral residual maps.

Quantitative results. Table 4 shows the quantitative performance of all the
methods on the three tasks; for all the tasks, our method outperforms the
recent models. In particular, for the most challenging task, PET to CT
translation, our method with uncertainty-based guide outperforms the previous
state-of-the-art method, MedGAN (that relies on task-specific external feature
extractor), without using any external feature extractor. Therefore, the
uncertainty guidance reduces the burden of having an externally trained task-
specific feature extractor to achieve high fidelity images. The same trend
holds for undersampled MRI reconstruction and motion correction in MRI. The
statistical tests on SSIM values of MedGAN and our UP-GAN gives us a p-value
of 0.016 for PET-to-CT translation, 0.021 for undersampled MRI reconstruction,
and 0.036 for MRI motion correction. As all the p-values are $<0.05$, results
are statistically significant.

Ablation study. We study the model that does not utilize the estimated
uncertainty maps as attention maps and observe that the model without the
uncertainty as the guide performs inferior to the UP-GAN with a performance
(SSIM/PSNR/MAE) of (0.87/25.4/40.7), (0.93/27.3/38.7), and (0.92/26.2/35.1)
for PET to CT translation, undersampled MRI reconstruction, and MRI motion
correction, respectively. UP-GAN model leverages the uncertainty map to refine
the predictions where the model is uncertain, which is also correlated to the
regions where the translation is poor. The model without uncertainty-based
guidance does not focus on the regions mentioned above in the prediction and
is unable to perform as well as UP-GAN.

Figure 4:  Quantitative results in the presence of limited labeled training
data.

Evaluating models with weak supervision. We evaluate all the models for PET to
CT synthesis by limiting the number of paired image samples used for training.
We define five supervision levels corresponding to different amounts of cross-
domain pairwise training sample slices. For this experiment, we train the
recent state-of-the-art models with a varying number of patients in the
training stage, i.e., we use 5, 10, 15, 20, and 29 patients, respectively.
Figure 4 shows the performance of all the models at varying supervision
levels. We observe that our model with uncertainty guidance outperforms all
the baselines at full supervision (with 29 patients). Moreover, our model
sharply outperforms the baselines with limited training data (with $<$ 29
patients). UP-GAN produces intermediate uncertainty maps that have higher
values under weak supervision (compared to the full supervision case), but
this still allows UP-GAN to focus on highly uncertain regions, that the
current state-of-the-art models do not have access to, hence are not able to
leverage that to refine the predicted images.

##  5 Conclusion

In this work, we propose a new generic model for medical image translation
using uncertainty-guided progressive GANs. We demonstrate how uncertainty can
serve as an attention map in progressive learning schemes. We demonstrate the
efficacy of our method on three challenging medical image translation tasks,
including PET to CT translation, undersampled MRI reconstruction, and motion
correction in MRI. Our method achieves state-of-the-art in various tasks.
Moreover, it allows the quantification of uncertainty and shows better
generalizability with smaller sample sizes than recent approaches.

Acknowledgements. This work has been partially funded by the ERC (853489 -
DEXIM) and by the DFG (2064/1 – Project number 390727645). The authors thank
the International Max Planck Research School for Intelligent Systems (IMPRS-
IS) for support.

## References

  * [1] Armanious, K., Jiang, C., Abdulatif, S., Küstner, T., Gatidis, S., Yang, B.: Unsupervised medical image translation using cycle-medgan. In: European Signal Processing Conference (EUSIPCO) (2019)
  * [2] Armanious, K., Jiang, C., Fischer, M., Küstner, T., Hepp, T., Nikolaou, K., Gatidis, S., Yang, B.: Medgan: Medical image translation using gans. Computerized Medical Imaging and Graphics (2020)
  * [3] Chen, H., Zhang, Y., Kalra, M.K., Lin, F., Chen, Y., Liao, P., Zhou, J., Wang, G.: Low-dose ct with a residual encoder-decoder convolutional neural network. IEEE TMI (2017)
  * [4] Dou, Q., Chen, H., Yu, L., Qin, J., Heng, P.A.: Multilevel contextual 3-d cnns for false positive reduction in pulmonary nodule detection. IEEE Transactions on Biomedical Engineering (2016)
  * [5] Havaei, M., Davy, A., Warde-Farley, D., Biard, A., Courville, A., Bengio, Y., Pal, C., Jodoin, P.M., Larochelle, H.: Brain tumor segmentation with deep neural networks. Medical Image Analysis (2017)
  * [6] Huynh, T., Gao, Y., Kang, J., Wang, L., Zhang, P., Lian, J., Shen, D.: Estimating ct image from mri data using structured random forest and auto-context model. IEEE TMI (2015)
  * [7] Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: IEEE CVPR (2017)
  * [8] Kamnitsas, K., Ledig, C., Newcombe, V.F., Simpson, J.P., Kane, A.D., Menon, D.K., Rueckert, D., Glocker, B.: Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation. Medical Image Analysis (2017)
  * [9] Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of gans for improved quality, stability, and variation. In: ICLR (2018)
  * [10] Kendall, A., Gal, Y.: What uncertainties do we need in bayesian deep learning for computer vision? In: NIPS (2017)
  * [11] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. preprint arXiv:1412.6980 (2014)
  * [12] Küstner, T., Schwartz, M., Martirosian, P., Gatidis, S., Seith, F., Gilliam, C., Blu, T., Fayad, H., Visvikis, D., Schick, F., et al.: Mr-based respiratory and cardiac motion correction for pet imaging. Medical Image Analysis (2017)
  * [13] Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian, M., Van Der Laak, J.A., Van Ginneken, B., Sánchez, C.I.: A survey on deep learning in medical image analysis. Medical Image Analysis (2017)
  * [14] Nair, T., Precup, D., Arnold, D.L., Arbel, T.: Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation. Medical Image Analysis (2020)
  * [15] Nie, D., Trullo, R., Lian, J., Wang, L., Petitjean, C., Ruan, S., Wang, Q., Shen, D.: Medical image synthesis with deep convolutional adversarial networks. IEEE Transactions on Biomedical Engineering (2018)
  * [16] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI (2015)
  * [17] Rueda, A., Malpica, N., Romero, E.: Single-image super-resolution of brain mr images using overcomplete dictionaries. Medical Image Analysis (2013)
  * [18] Shaw, R., Sudre, C., Ourselin, S., Cardoso, M.J.: Mri k-space motion artefact augmentation: model robustness and task-specific uncertainty. In: MIDL (2019)
  * [19] Shin, H.C., Roth, H.R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D., Summers, R.M.: Deep convolutional neural networks for computer-aided detection: Cnn architectures, dataset characteristics and transfer learning. IEEE TMI (2016)
  * [20] Tanno, R., Worrall, D.E., Ghosh, A., Kaden, E., Sotiropoulos, S.N., Criminisi, A., Alexander, D.C.: Bayesian image quality transfer with cnns: exploring uncertainty in dmri super-resolution. In: MICCAI (2017)
  * [21] Upadhyay, U., Awate, S.P.: A mixed-supervision multilevel gan framework for image quality enhancement. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 556–564. Springer (2019)
  * [22] Upadhyay, U., Awate, S.P.: Robust super-resolution gan, with manifold-based and perception loss. In: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019). pp. 1372–1376. IEEE (2019)
  * [23] Upadhyay, U., Chen, Y., Akata, Z.: Uncertainty-aware generalized adaptive cyclegan. preprint arXiv:2102.11747 (2021)
  * [24] Wang, C., Xu, C., Wang, C., Tao, D.: Perceptual adversarial networks for image-to-image transformation. IEEE TIP (2018)
  * [25] Wang, G., Li, W., Aertsen, M., Deprest, J., Ourselin, S., Vercauteren, T.: Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks. Neurocomputing (2019)
  * [26] Wang, G., Li, W., Ourselin, S., Vercauteren, T.: Automatic brain tumor segmentation using convolutional neural networks with test-time augmentation. In: International MICCAI Brainlesion Workshop (2018)
  * [27] Wang, G., Li, W., Ourselin, S., Vercauteren, T.: Automatic brain tumor segmentation based on cascaded convolutional neural networks with uncertainty estimation. Frontiers in Computational Neuroscience (2019)
  * [28] Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE TIP (2004)
  * [29] Wolterink, J.M., Dinkla, A.M., Savenije, M.H., Seevinck, P.R., van den Berg, C.A., Išgum, I.: Deep mr to ct synthesis using unpaired data. In: International workshop on Simulation and Synthesis in Medical Imaging (2017)
  * [30] Yang, Q., Yan, P., Zhang, Y., Yu, H., Shi, Y., Mou, X., Kalra, M.K., Zhang, Y., Sun, L., Wang, G.: Low-dose ct image denoising using a generative adversarial network with wasserstein distance and perceptual loss. IEEE TMI (2018)
  * [31] Yi, X., Walia, E., Babyn, P.: Generative adversarial network in medical imaging: A review. Medical Image Analysis (2019)
  * [32] Zhang, H., Sindagi, V., Patel, V.M.: Image de-raining using a conditional generative adversarial network. IEEE Transactions on Circuits and Systems for Video Technology (2019)
  * [33] Zhang, Z., Romero, A., Muckley, M.J., Vincent, P., Yang, L., Drozdzal, M.: Reducing uncertainty in undersampled mri reconstruction with active acquisition. In: IEEE CVPR (2019)
  * [34] Zhao, H., Li, H., Cheng, L.: Synthesizing filamentary structured images with gans. preprint arXiv:1706.02185 (2017)
  * [35] Zhong, L., Lin, L., Lu, Z., Wu, Y., Lu, Z., Huang, M., Yang, W., Feng, Q.: Predict ct image from mri data using knn-regression with learned local descriptors. In: IEEE ISBI (2016)
  * [36] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: IEEE ICCV (2017)

## Appendix

The following presents the numerical results for the weak supervision
evaluation.

Methods | PET to CT
---|---
SSIM | PSNR | MAE | log(MAE)
pix2pix [7] | 0.02$\pm$0.003 | 2.91$\pm$0.29 | 1281$\pm$46.94 | 7.15$\pm$0.035
PAN [24] | 0.01$\pm$0.006 | 8.3$\pm$0.93 | 557$\pm$68.6 | 6.31$\pm$0.12
MedGAN [2] | 0.05$\pm$0.008 | 4.6$\pm$0.25 | 1090$\pm$35.7 | 6.99$\pm$0.03
UP-GAN | 0.29$\pm$0.08 | 13.2$\pm$1.39 | 158.1$\pm$43.41 | 5.02$\pm$0.32

Table 2:  Evaluation of different models trained with data from 5 patients.

Methods | PET to CT
---|---
SSIM | PSNR | MAE | log(MAE)
pix2pix [7] | 0.03$\pm$0.005 | 3.55$\pm$0.36 | 1214$\pm$53.98 | 7.10$\pm$0.044
PAN [24] | 0.14$\pm$0.04 | 11.68$\pm$1.36 | 383$\pm$78.4 | 5.94$\pm$0.20
MedGAN [2] | 0.26$\pm$0.08 | 8.86$\pm$0.85 | 512$\pm$88.8 | 6.22$\pm$0.18
UP-GAN | 0.78$\pm$0.09 | 14.09$\pm$1.41 | 98.9$\pm$42.64 | 4.48$\pm$0.55

Table 3:  Evaluation of different models trained with data from 15 patients.

Methods | PET to CT
---|---
SSIM | PSNR | MAE | log(MAE)
pix2pix [7] | 0.36$\pm$0.11 | 13.56$\pm$1.53 | 269$\pm$72.54 | 5.56$\pm$0.27
PAN [24] | 0.54$\pm$0.13 | 14.38$\pm$2.70 | 221$\pm$90.4 | 5.30$\pm$0.44
MedGAN [2] | 0.83$\pm$0.07 | 23.65$\pm$2.65 | 57.48$\pm$20.72 | 3.99$\pm$0.33
UP-GAN | 0.90$\pm$0.07 | 26.40$\pm$0.74 | 38.9$\pm$20.69 | 3.49$\pm$0.62

Table 4:  Evaluation of different models trained with data from 25 patients.


For any partition $\mu \in \msc{P}(r)$ of $r$, we denote by $M_\mu \subset \GL_r$ the standard Levi subgroup. The Weyl group of $M_\mu$ is denoted by $W_\mu \subset W$. We write

$$\lambda:=(n^a b) \in \msc{P}(r).$$

Since $\mca{N}^{\rm max}_{\rm tr}(\Theta(\pi^\dag, \nu)) =\set{\lambda}$, then <cit.> and <cit.> together with the periodicity of theta representations (see <cit.>) give the first equality in

$$c_\mca{O}=\dim \Wh_\psi(\Theta(\wt{M}_\lambda^{(n)})) =\angb{\varepsilon_{W_\lambda}}{\sigma^\msc{X}}_{W_\lambda} = \angb{{\rm Ind}_{W_\lambda}^W (\varepsilon_{W_\lambda})}{\sigma^\msc{X}}_W  ,$$

where the theta representation $\wt{M}_\lambda^{(n)}$ is associated with $\pi^\dag$ and $\nu$, and the second equality follows from <cit.>.
On the other hand, the Weyl subgroup $W_\nu$ is a parabolic Weyl subgroup associated with

$$\lambda^\top =((a+1)^b a^{n-b}).$$

\angb{j_{W_\nu}^W(\varepsilon_{W_\nu})}{ \varepsilon_W \otimes \sigma^\msc{X} }_W & = \angb{j_{W_{\lambda^\top}}^W(\varepsilon_{W_{\lambda^\top}})}{ \varepsilon_W \otimes \sigma^\msc{X} }_W \\
& = \angb{\varepsilon_W \otimes j_{W_{\lambda^\top}}^W(\varepsilon_{W_{\lambda^\top}})}{  \sigma^\msc{X} }_W \\
& = \angb{ j_{W_\lambda}^W(\varepsilon_{W_\lambda}) }{ \sigma^\msc{X} }_W,
\end{aligned} $$

where the last equality follows from <cit.>.

We have

$${\rm Ind}_{W_\lambda}^W (\varepsilon_{W_\lambda}) = j_{W_\lambda}^W(\varepsilon_{W_\lambda}) + \sum_{\substack{\lambda \lest \mu \\ \mu \ne \lambda}}  j_{W_\mu}^W(\varepsilon_{W_\mu}),$$

see <cit.>. Also, for any $\mu > \lambda$,

$$\angb{ j_{W_\mu}^W(\varepsilon_{W_\mu}) }{ \sigma^\msc{X} }_W \lest \angb{ \Ind_{W_\mu}^W(\varepsilon_{W_\mu}) }{ \sigma^\msc{X} }_W = \angb{ \varepsilon_{W_\mu} }{ \sigma^\msc{X} }_{W_\mu}= \dim \Wh_\psi(\Theta(\wt{M}_\mu)),$$

where the last equality follows from <cit.>. Since $\mu > \lambda$, there is a component in the partition of $\mu$ strictly greater than $n$ and thus one has

$$\dim \Wh_\psi(\Theta(\wt{M}_\mu))=0.$$

All the above together give that $c_\mca{O} = \angb{j_{W_\nu}^W(\varepsilon_{W_\nu})}{ \varepsilon_W \otimes \sigma^\msc{X} }_W$.  This completes the proof.

Note that the restriction to Kazhdan–Patterson cover of $\GL_r$ is only for convenience. Results in Theorem <ref> and other parts of paper hold for general Brylinski–Deligne covers of $\GL_r$, where the only essential alternation in the statement is to replace $n$ by $n_\alpha$.

author=Achar, Pramod N.,
title=An order-reversing duality map for conjugacy classes in Lusztig's canonical quotient,
journal=Transform. Groups,

author=Adams, Jeffrey,
author=Barbasch, Dan,
author=Paul, Annegret,
author=Trapa, Peter E.,
author= Vogan, David A., Jr.,
title=Unitary Shimura correspondences for split real groups,
journal=J. Amer. Math. Soc.,

author=Bai, Zhanqiang,
author=Xie, Xun,
title=Gelfand-Kirillov dimensions of highest weight Harish-Chandra modules for $SU(p,q)$,
journal=Int. Math. Res. Not. IMRN,

author=Bai, Zhanqiang,
author=Xiao, Wei,
author=Xie, Xun,
title=Gelfand–Kirillov Dimensions and Associated Varieties of Highest Weight Modules,
journal=Int. Math. Res. Not. IMRN,

author=Bai, Zhanqiang,
author=Ma, Jiajun,
author=Wang, Yutong,
title=A combinatorial characterization of the annihilator varieties of highest weight modules for classical Lie algebras,
status=preprint, available at https://arxiv.org/abs/2304.03475v1,

author=Ban, Dubravka,
author=Jantzen, Chris,
title=The Langlands quotient theorem for finite central extensions of $p$-adic groups,
journal=Glas. Mat. Ser. III,

author=Barbasch, Dan,
author=Vogan, David A., Jr.,
title=The local structure of characters,
journal=J. Functional Analysis,

author=Barbasch, Dan,
author=Vogan, David,
title=Primitive ideals and orbital integrals in complex classical groups,
journal=Math. Ann.,

author=Barbasch, Dan,
author=Vogan, David,
title=Primitive ideals and orbital integrals in complex exceptional groups,
journal=J. Algebra,

author=Borel, Armand,
author=Tits, Jacques,
title=Homomorphismes “abstraits” de groupes algébriques simples,
journal=Ann. of Math. (2),

author=Bourbaki, Nicolas,
title=Lie groups and Lie algebras. Chapters 4–6,
series=Elements of Mathematics (Berlin),
note=Translated from the 1968 French original by Andrew Pressley,
publisher=Springer-Verlag, Berlin,

author=Brylinski, Jean-Luc,
author=Deligne, Pierre,
title=Central extensions of reductive groups by $\bold K_2$,
journal=Publ. Math. Inst. Hautes Études Sci.,

author=Bump, Daniel,
author=Friedberg, Solomon,
author=Ginzburg, David,
title=Small representations for odd orthogonal groups,
journal=Int. Math. Res. Not.,

author=Bump, Daniel,
author=Friedberg, Solomon,
author=Ginzburg, David,
title=Lifting automorphic representations on the double covers of orthogonal groups,
journal=Duke Math. J.,

author=Bump, Daniel,
author=Furusawa, Masaaki,
author=Ginzburg, David,
title=Non-unique models in the Rankin-Selberg method,
journal=J. Reine Angew. Math.,

author=Cai, Yuanqing,
title=Fourier coefficients for theta representations on covers of general linear groups,
journal=Trans. Amer. Math. Soc.,

author=Carter, Roger W.,
title=Finite groups of Lie type,
series=Wiley Classics Library,
note=Conjugacy classes and complex characters; Reprint of the 1985 original; A Wiley-Interscience Publication,
publisher=John Wiley & Sons, Ltd., Chichester,

author=Ciubotaru, Dan,
author=Mason-Brown, Lucas,
author=Okada, Emile T.,
title=The wavefront sets of Iwahori-spherical representations of reductive $p$-adic groups,
status=preprint, available at https://arxiv.org/abs/2112.14354v4,

author=Collingwood, David H.,
author=McGovern, William M.,
title=Nilpotent orbits in semisimple Lie algebras,
series=Van Nostrand Reinhold Mathematics Series,
publisher=Van Nostrand Reinhold Co., New York,

author=Duflo, Michel,
title=Construction de représentations unitaires d'un groupe de Lie,
language=French, with English summary,
conference= title=Harmonic analysis and group representations, ,
book= publisher=Liguori, Naples, ,

author=Friedberg, Solomon,
author=Ginzburg, David,
title=Metaplectic theta functions and global integrals,
journal=J. Number Theory,

author=Friedberg, Solomon,
author=Ginzburg, David,
title=Theta functions on covers of symplectic groups,
journal=Bull. Iranian Math. Soc.,

author=Friedberg, Solomon,
author=Ginzburg, David,
title=Descent and theta functions for metaplectic groups,
journal=J. Eur. Math. Soc. (JEMS),

author=Friedberg, Solomon,
author=Ginzburg, David,
title=Classical theta lifts for higher metaplectic covering groups,
journal=Geom. Funct. Anal.,

author=Gan, Wee Teck,
author=Gao, Fan,
title=The Langlands-Weissman program for Brylinski-Deligne extensions,
language=English, with English and French summaries,
note=L-groups and the Langlands program for covering groups,

author=Gao, Fan,
title=Distinguished theta representations for certain covering groups,
journal=Pacific J. Math.,

author=Gao, Fan,
title=Kazhdan–Lusztig representations and Whittaker space of some genuine representations,
journal=Math. Ann.,

author=Gao, Fan,
author=Shahidi, Freydoon,
author=Szpruch, Dani,
title=On the local coefficients matrix for coverings of $\rm SL_2$,
conference= title=Geometry, algebra, number theory, and their information technology applications, ,
book= series=Springer Proc. Math. Stat., volume=251, publisher=Springer, Cham, ,

author=Gao, Fan,
author=Tsai, Wan-Yu,
title=On the wavefront sets associated with theta representations,
journal=Math. Z.,

author=Geck, Meinolf,
author=Pfeiffer, Götz,
title=Characters of finite Coxeter groups and Iwahori-Hecke algebras,
series=London Mathematical Society Monographs. New Series,
publisher=The Clarendon Press, Oxford University Press, New York,

author=Ginzburg, David,
title=Certain conjectures relating unipotent orbits to automorphic representations,
journal=Israel J. Math.,

author=Ginzburg, David,
author=Rallis, Stephen,
author=Soudry, David,
title=The descent map from automorphic representations of ${\rm GL}(n)$ to classical groups,
publisher=World Scientific Publishing Co. Pte. Ltd., Hackensack, NJ,

author=Gomez, Raul,
author=Gourevitch, Dmitry,
author=Sahi, Siddhartha,
title=Generalized and degenerate Whittaker models,
journal=Compos. Math.,

author=Gomez, Raul,
author=Gourevitch, Dmitry,
author=Sahi, Siddhartha,
title=Whittaker supports for representations of reductive groups,
language=English, with English and French summaries,
journal=Ann. Inst. Fourier (Grenoble),

author=Gourevitch, Dmitry,
author=Sahi, Siddhartha,
title=Degenerate Whittaker functionals for real reductive groups,
journal=Amer. J. Math.,

author=Gurevich, Maxim,
title=A triangular system for local character expansions of Iwahori-spherical representations of general linear groups,
language=English, with English and French summaries,
journal=C. R. Math. Acad. Sci. Paris,

title=Admissible invariant distributions on reductive $p$-adic groups,
series=University Lecture Series,
note=With a preface and notes by Stephen DeBacker and Paul J. Sally, Jr.,
publisher=American Mathematical Society, Providence, RI,

author=Howe, Roger,
title=The Fourier transform and germs of characters (case of ${\rm Gl}_{n}$ over a $p$-adic field),
journal=Math. Ann.,

author=Jackson, Steven Glenn,
author=Noël, Alfred G.,
title=Prehomogeneous spaces associated with complex nilpotent orbits,
journal=J. Algebra,

author=Jiang, Dihua,
title=Automorphic integral transforms for classical groups I: Endoscopy correspondences,
conference= title=Automorphic forms and related geometry: assessing the legacy of I. I. Piatetski-Shapiro, ,
book= series=Contemp. Math., volume=614, publisher=Amer. Math. Soc., Providence, RI, ,

author=Jiang, Dihua,
author=Liu, Baiying,
title=Fourier coefficients for automorphic forms on quasisplit classical groups,
conference= title=Advances in the theory of automorphic forms and their $L$-functions, ,
book= series=Contemp. Math., volume=664, publisher=Amer. Math. Soc., Providence, RI, ,

author=Jiang, Dihua,
author=Liu, Baiying,
title=On wavefront sets of global Arthur packets of classical groups: upper bound,
journal=J. European Math. Society.,

author=Jiang, Dihua,
author=Liu, Baiying,
author=Savin, Gordan,
title=Raising nilpotent orbits in wave-front sets,
journal=Represent. Theory,

author=Jiang, Dihua,
author=Liu, Dongwen,
author=Zhang, Lei,
title=Arithmetic wavefront sets and generic $L$-packets,
status=preprint, available at https://arxiv.org/abs/2207.04700,

author=Jiang, Dihua,
author=Zhang, Lei,
title=Automorphic integral transforms for classical groups II: Twisted descents,
conference= title=Representation theory, number theory, and invariant theory, ,
book= series=Progr. Math., volume=323, publisher=Birkhäuser/Springer, Cham, ,

author=Jiang, Dihua,
author=Zhang, Lei,
title=Arthur parameters and cuspidal automorphic modules of classical groups,
journal=Ann. of Math. (2),

author=Kaplan, Eyal,
title=The double cover of odd general spin groups, small representations, and applications,
journal=J. Inst. Math. Jussieu,

author=Li, Wen-Wei,
title=La formule des traces pour les revêtements de groupes réductifs connexes. II. Analyse harmonique locale,
language=French, with English and French summaries,
journal=Ann. Sci. Éc. Norm. Supér. (4),

author=Liu, Baiying,
author=Shahidi, Freydoon,
title=Jiang's conjecture on local Arthur packets,
journal=Kudla Proceedings,

author=Liu, Baiying,
author=Shahidi, Freydoon,
title=Jiang's conjecture on the wave front sets of local Arthur packets,

author=Liu, Baiying,
author=Xu, Bin,
title=On top Fourier coefficients of certain automorphic representations of $\mathrm {GL}_n$,
journal=Manuscripta Math.,
fjournal=Manuscripta Mathematica,
mrclass=11F70 (11F30 22E50 22E55),
mrreviewer=o Ngọc Diep,

author=McNinch, George J.,
author=Sommers, Eric,
title=Component groups of unipotent centralizers in good characteristic,
note=Special issue celebrating the 80th birthday of Robert Steinberg,
journal=J. Algebra,

author=Mœglin, C.,
title=Front d'onde des représentations des groupes classiques $p$-adiques,
language=French, with French summary,
journal=Amer. J. Math.,

author=Mœglin, C.,
author=Waldspurger, J.-L.,
title=Modèles de Whittaker dégénérés pour des groupes $p$-adiques,
journal=Math. Z.,

author=Nevins, Monica,
title=Admissible nilpotent coadjoint orbits of $p$-adic reductive Lie groups,
journal=Represent. Theory,

author=Nevins, Monica,
title=Admissible nilpotent orbits of real and $p$-adic split exceptional groups,
journal=Represent. Theory,

author=Okada, Emile T.,
title=The wavefront set of spherical Arthur representations,
status=preprint, available at https://arxiv.org/abs/2107.10591v1,

author=Prakash Patel, Shiv,
title=A theorem of Mœglin and Waldspurger for covering groups,
journal=Pacific J. Math.,

author=Savin, Gordan,
title=Dual pair $G_{\scr J}\times {\rm PGL}_2$ [where] $G_{\scr J}$ is the automorphism group of the Jordan algebra ${\scr J}$,
journal=Invent. Math.,

author=Savin, Gordan,
title=A nice central extension of ${\rm GL}_r$,

author=Serre, Jean-Pierre,
title=Galois cohomology,
series=Springer Monographs in Mathematics,
edition=Corrected reprint of the 1997 English edition,
note=Translated from the French by Patrick Ion and revised by the author,
publisher=Springer-Verlag, Berlin,

author=Sommers, Eric,
title=A generalization of the Bala-Carter theorem for nilpotent orbits,
journal=Internat. Math. Res. Notices,

author=Sommers, Eric,
title=Lusztig's canonical quotient and generalized duality,
journal=J. Algebra,

author=Torasso, Pierre,
title=Méthode des orbites de Kirillov-Duflo et représentations minimales des groupes simples sur un corps local de caractéristique nulle,
journal=Duke Math. J.,

author=Tsai, Cheng-Chiang,
title=Geometric wave-front set may not be a singleton,

author=Tsai, Wan-Yu,
title=Some genuine small representations of a nonlinear double cover,
journal=Trans. Amer. Math. Soc.,

author=Varma, Sandeep,
title=On a result of Moeglin and Waldspurger in residual characteristic 2,
journal=Math. Z.,

author=Weissman, Martin H.,
title=L-groups and parameters for covering groups,
language=English, with English and French summaries,
note=L-groups and the Langlands program for covering groups,