# Joint Semi-supervised 3D Super-Resolution and Segmentation with Mixed
Adversarial Gaussian Domain Adaptation

Nicolò Savioli  Antonio de Marvao  Wenjia Bai  Shuo Wang  Stuart A. Cook
Calvin W. L. Chin  Daniel Rueckert  Declan P. O'Regan
<EMAIL_ADDRESS>MRC London Institute of Medical Sciences,
Imperial College London, London, UK  Department of Computing, Imperial College
London, London, UK  Department of Brain Sciences, Imperial College London,
London, UK  National Heart Research Institute Singapore, National Heart Center
Singapore, Singapore  Institute for Artificial Intelligence and Informatics,
Klinikum rechts der Isar, Technical University of Munich, Germany

###### Abstract

Optimising the analysis of cardiac structure and function requires accurate 3D
representations of shape and motion. However, techniques such as cardiac
magnetic resonance imaging are conventionally limited to acquiring contiguous
cross-sectional slices with low through-plane resolution and potential inter-
slice spatial misalignment. Super-resolution in medical imaging aims to
increase the resolution of images but is conventionally trained on features
from low resolution datasets and does not super-resolve corresponding
segmentations. Here we propose a semi-supervised multi-task generative
adversarial network (Gemini-GAN) that performs joint super-resolution of the
images and their labels using a ground truth of high resolution 3D cines and
segmentations, while an unsupervised variational adversarial mixture
autoencoder (V-AMA) is used for continuous domain adaptation. Our proposed
approach is extensively evaluated on two transnational multi-ethnic
populations of $1,331$ and $205$ adults respectively, delivering an
improvement on state-of-the-art methods in terms of Dice index, peak
signal‐to‐noise ratio, and structural similarity index measure. This framework
also exceeds the performance of state-of-the-art generative domain adaptation
models on external validation (Dice index $0.81$ vs $0.74$ for the left
ventricle). This demonstrates how joint super-resolution and segmentation,
trained on 3D ground-truth data with cross-domain generalization, enables
robust precision phenotyping in diverse populations.

###### keywords:

Deep Learning , Super-Resolution (SR), Variational Inference , MRI , Joint
Segmentation

††journal: ArXiv

##  1 Introduction

Image super-resolution (SR) is a class of techniques in computer vision that
is used to reconstruct a high resolution (HR) image from observed low
resolution (LR) images. Improving the details of an image is valuable across a
range of natural image applications including data compression [5] and feature
extraction [28]. It is particularly relevant in medical imaging where there
are physical constraints on the acquired spatial resolution and high-
resolution structural detail may hold discriminative clinical information [8].
SR representations are important for cardiac imaging where detailed variations
in structure and motion are informative for genetic association studies and
prognostic stratification [3, 36]. While acquiring three-dimensional cardiac
magnetic resonance (CMR) cine imaging is feasible [24], routine clinical
practice and population biobanks are still dominated by multi-slice 2D cine
imaging, which limits accurate assessment of geometry due to poor through-
plane resolution.

Dictionary-based approaches, such as deformable patch-match methods, aim to
recover HR patches from LR images via priors between the target image and
training data [46]. However, such methods are computationally demanding as the
candidate patches have to be searched to find the most suitable match. In
contrast, SR convolutional neural networks learn an end-to-end mapping between
LR and HR images, and have significantly improved the quality and efficiency
of reconstructions [16, 33, 6, 7, 11]. These methods use a convolutional
neural network (CNN) that codifies non-linear transformations (i.e., patch
extraction, nonlinear mapping, and reconstruction) with a combination of
ResNet blocks and generative adversarial networks (GANs) [23]. Making use of
multiple images acquired from different slice directions can further improve
and constrain the HR image reconstruction [25, 9]. Lately, optical flow
interpolation has been proposed for intensity cardiac image SR [42]. This
method is based on recent advances in the field of video interpolation [2],
where intermediate sections are synthesized through an optical flow
transformation of the cine data. However, the proposed method cannot eliminate
slice misalignment and inter-slice motion artifacts (i.e., where differing
breath-hold positions during scanning generate imperfect alignment between
slices).

These strategies perform segmentation independently of processing the
intensity image and do not share the same feature encoder for the related
tasks of super-resolving the greyscale image and its semantic label map.
Multi-task deep learning approaches with atlas propagation for shape-refined
segmentation can produce anatomically smooth models that are robust to the
presence of artifacts in the input CMR volumes [13]. A limitation is the
computational cost needed to perform atlas propagation and the output is only
HR label maps without an end-to-end super-resolved image. Joint SR and
segmentation using GANs (segSR-GAN) have been proposed for brain imaging [10]
as an extension of SR-GAN [23]. The segSR-GAN model initially performs both
interpolation and patch image subdivisions of the LR input and maps it to an
output within two separate up-convolutional layers. However, the initial pre-
interpolation step may amplify artifacts such as the motion between successive
slices or intensity bias within a given slice. Also, both pre-interpolation
and/or patch image subdivisions have long computational times and may not be
suitable for clinical applications or large datasets.

In common with other classification tasks, if the training and test images for
SR are drawn from different distributions, domain adaptation is needed to
reduce sample bias to improve generalization performance across populations
[39]. There are three main techniques for domain adaptation: divergence-based
domain adaptation, adversarial-based domain adaptation, and reconstruction-
based domain adaptation [37, 18, 43]. The divergence-based domain adaptation
method works by minimizing a divergence criterion between source and target
distributions for achieving domain invariant feature representation. In
adversarial-based domain adaptation, a synthetic target data domain is
generated from the source data by applying GANs. A domain confusion loss is
alternately applied to balance the distributions of the source and target
domain in one shot within a confusion metric in the final regression layer
[38]. Finally, in the reconstruction-based domain adaptation approach, a
shared representation of the domain simultaneously solves classification and
reconstruction maintaining information in the target domain [15]. However, the
adversarial-based domain adaptation is the most used method in the medical
field [26, 12], but is still prone to mode collapse [14] that leads the domain
adaption model to collapse in the wrong distribution. The variational
adversarial autoencoder model proposed here has two main advantages: it solves
the mode collapse problem by decoupling the distribution transformation within
two variational networks and, at the same time, simplifies the mapping
operation between source and target distributions.

In this work, we propose: i) a generative adversarial network called Gemini-
GAN that jointly performs SR of greyscale cine images and their label maps and
(ii) a generative domain adaptation approach (Variational Adversarial Mixture
Autoencoder, V-AMA) which together with joint SR, generalizes across different
populations. This approach enforces cross-domain consistency and benefits from
complementary learning of super-resolving images and their segmentations. In
contrast to previous work, we use a natively HR ground truth for training and
validate our approach on two independent multi-ethnic cardiac MR datasets.

##  2 Datasets

We used paired natively acquired LR/HR images and their respective
segmentations for model training, taking advantage of the near-isotropic HR 3D
cine datasets available in the UK Digital Heart Project (UKDHP). We then
generalized the model to an external population with equivalent paired LR/HR
data available (National Heart Center Singapore Biobank), and then to a
further population with only LR data available (UK Biobank) which is an
intended use case. Table 1 summarizes the phenotypic characteristics of the
datasets used. Images were stored on an open-source database (MRIdb, Imperial
College London, UK) [41]. Ground truth labels for both LR and HR datasets were
derived as previously described [1]. In each case ethical approval and written
informed consent was obtained.

###  2.1 UK Digital Heart Project (UKDHP)

A dataset of $1331$ healthy adults was used from the UK Digital Heart Project
at Imperial College London. High-spatial resolution 3D balanced steady-state
free precession cine sequences were used that assessed the left and right
ventricles in their entirety in a single breath-hold (60 sections,
reconstructed voxel size 1.2 × 1.2 × 2 mm, 20 cardiac phases, typical breath-
hold 20s). Conventional single slice multi-breath-hold images were also
acquired in the same geometry (10 sections, reconstructed voxel size 1.8 × 1.8
× 8 mm). Imaging was performed on a 1.5-T Philips Achieva system (Best, the
Netherlands).

###  2.2 National Heart Center Singapore Biobank (SG)

External validation was performed on a dataset of $205$ healthy adults
recruited to the National Heart Center Singapore Biobank using the same LR/HR
image acquisition parameters as the UKDHP but obtained on a 1.5T Aera (Siemens
Healthcare, Erlangen, Germany).

###  2.3 UK Biobank (UKBB)

Generalisation to a second external dataset with only LR cine images was
performed in $1331$ adults prospectively recruited to UK Biobank [4].
Conventional LR conventional single slice multi-breath-hold images were
acquired (10 sections, reconstructed voxel size 1.8 × 1.8 × 8 mm) performed to
a standard protocol on a 1.5T Aera (Siemens Healthcare, Erlangen, Germany)
[27]. Data was processed under access number 40616.

Characteristic | UKDHP | SG | UKBB∗
---|---|---|---
| (n=1,331) | (n=205) | (n=1,331)
Age (years) | 37.7 $\pm$ 12.51 | 50.2 $\pm$ 14.69 | 56.5 $\pm$ 8.1
Sex (male) | 578.0 (43.4$\%$) | 106 (51.7$\%$) | 86.7 (54.9$\%$)
Ethnicity |  |  |
Caucasian | 922.0 (69.3$\%$) | - | 150.4 (95.2$\%$)
Chinese | - | 191 (93.17$\%$) | -
South Asian | 180 (13.5$\%$) | 11 (5.3$\%$) | -
African | 145 (10.9$\%$) | - | -
Other | 84 (6.3$\%$) | 3 (1.4$\%$) | -
BSA ($m^{2}$) | 1.8$\pm$0.2 | 1.7$\pm$0.2 | 1.9$\pm$0.22
SBP ($mmHg$) | 118.3$\pm$14.3 | 134.4$\pm$18 | 139.7$\pm$19.6
DBP ($mmHg$) | 77.9$\pm$9.5 | 80.1$\pm$11.6 | 82.2$\pm$10.7

Table 1: Population characteristics. Values are n ($\%$) or mean $\pm$
standard deviation. ∗ Characteristics shown for whole population. SG,
Singapore biobank; UKDHP, UK Digital Heart Project; UKBB, UK Biobank; BSA,
body surface area; SBP, systolic blood pressure; DBP, diastolic blood
pressure.

##  3 Methods

Here we describe the proposed Gemini-GAN for joint super-resolution of
greyscale images and segmentation labels from routine 2D cine sequences. We
then describe how V-AMA uses a novel domain transfer technique to map a source
distribution to the target distribution where Gemini-GAN was previously
trained. The whole framework combines both methods (V-AMA and Gemini-GAN) and
enables joint SR to be generalised across domains.

Fig. 1: The figure shows the main components of the proposed $G_{Gemini}$-GAN
model. It is constituted by a (blue) Convolution Neural Network decoder that
takes as input the LR volume that is sent into the latent space $Z$ (red),
which contains both SR and 3D segmentation latent information. Subsequently,
the $Z$ information is sent in two encoder networks constituted by up-
convolutions. The first decoder (red) reconstructs the 3D segmentation
$I^{SR}_{seg}$, while the other (green) $I^{SR}_{grey}$, the volumetric SR
images. The dashed lines represent the skip path connections that concatenate
the features of each decoder layer with the features of the corresponding
upsampling layer. Then these ``fake''-generated volumes $I^{SR}_{seg}$ and
$I^{SR}_{grey}$ are both concatenated and sent to the SR discriminator to fool
it into believing that they are similar to the concatenation of samples from
real dataset.

###  3.1 Gemini-GAN model

The proposed Gemini-GAN (Fig. 1) is based on the UNet style model [30], and
consists of one encoder path and two decoder paths (i.e., one for SR and one
for segmentation reconstruction). The encoder path is composed of repeated
$3\times 3$ convolutions, followed by a Rectified Linear Unit (ReLU) and a
Batch Normalisation (BN) layer. The twin's decoder branches are made by up-
convolutions concatenated with the corresponding cropping feature maps from
the decoder path. The networks take as input a 2D stack of LR slices $d\times
w\times h$, where $d$ is the number of slices along the z-axis and $w\times h$
is the size of each cardiac slice. The input channels $d$ of the network are
equal to the size of the $z$ axis's input volume. The feature maps in the
latent space are split into two decoders: one for reconstructing the SR
greyscale image and one for the 3D segmentation. Each feature map of each
decoder layer is concatenated to the upsampling layer for both SR and 3D
segmentation through skip path connections. We refer to this network as
Generator $G_{Gemini}$. The energy function for the segmentation encoder
branch of $G_{Gemini}$ is a cross-entropy criterion that combines log-softmax
and log-likelihood.

The log softmax [Eq. 1] takes as input a 3D dimensional input.

$p_{f}(x)=\frac{e^{z_{f}(x)}}{\sum_{i=1}^{K}e^{z_{i}(x)}}$ (1)

Here $z_{d}(x)$ in the numerator denotes the activation map at the feature
channel $f$ and pixel position $x\in\mathbb{R}^{2}$ for each cardiac slice.
While the sum of the denominator is a normalization across $K$ segmentation
class and ensures that the sum of probability components in the output vector
$p_{f}(x)$ are equal. A Cross Entropy ($CE_{Loss}$) loss is then defined as
the log-likelihood of probability of log-softmax distribution $p_{f}(x)$. The
weight map $w_{f}(x)$ assigning importance to each segmentation $K$ class
pixel as follows:

$CE_{Loss}=w_{f}(x)\cdot log(p_{f}(x))$ (2)

Instead in the SR reconstruction branch a Mean Square Error (MSE) is used:

$MSE_{Loss}=\frac{1}{n}\sum_{i=1,j=1}^{n}(o_{i,j}-y_{i,j})^{2}$ (3)

The $o_{i,j}$ is the output from the SR decoder and $y_{i,j}$ the
corresponding SR ground truth, $n$ is the number of image pixels $[i,j]$.

The L2 penalty loss is then defined as the square root of $MSE$:

$\begin{split}L2_{Loss}=\beta\cdot{\sqrt{(o_{i,j}-y_{i,j})^{2}}}=\beta\cdot\sqrt{nMSE_{Loss}}\end{split}$
(4)

To increase the performance of the network an SR discriminator $D_{SR}$ is
also coupled which corresponds to a network formed by eight $3\times 3$
reiterated convolutions followed by BN and LeakyReLU with a threshold value
set at $0.2$. The generator network $G_{Gemini}$ maps the LR volume as a
``fake'' greyscale SR and 3D segmentation volumes. The concatenation operation
($+$) is applied to both ``fake''-generated volumes sampling by
$I^{SR}_{fake}\sim(I^{SR}_{seg}+I^{SR}_{grey})$ where $I^{SR}_{seg}$ and
$I^{SR}_{grey}$ are outputs of $G$ net. Then, the concatenation
$I^{SR}_{fake}$ is sent to the SR discriminator $D_{SR}$ [Fig. 1] trained to
distinguish it from the corresponding ``real''-dataset concatenation
$I^{SR}_{real}\sim(I^{SR}_{seg}+I^{SR}_{grey})$ of greyscale SR and 3D
segmentation sampling from training dataset distribution. The final GAN loss
is formulated as follows:

$\begin{split}GAN_{Loss}=\min_{G}\max_{D_{SR}}L_{GAN_{Gemini}}(D_{SR},G)=\\\
=\mathbb{E}_{I^{SR}_{real}\sim
UKDHP_{({I^{SR}_{seg}+I^{SR}_{grey}})}}[log(D_{SR}(I^{SR}_{real})]+\\\
\mathbb{E}_{I^{SR}_{fake}}\sim
G_{({I^{SR}_{seg}+I^{SR}_{grey}})}[log(1-D_{SR}(I^{SR}_{fake})]\end{split}$
(5)

The final joint SR and 3D segmentation loss is a weighted combination of
cross-entropy, MSE, GAN, and L2 losses mediate by $\lambda$, $\omega$, and
$\beta$ constants:

$\begin{split}Loss_{G_{Gemini}}=CE+\lambda\cdot MSE+\omega\cdot GAN+\beta\cdot
L2\end{split}$ (6)

Fig. 2: The figure shows the novel Variational Adversarial Mixture Autoencoder
(V-AMA) used for the domain adaption. Particularly, the aim of the network is
to adapt the LR information, from an unknown source dataset (i.e., SG, UKBB),
to a target one (i.e., UKDHP dataset). The V-AMA model is composed of three
distinct blocks: LR rigid alignment block, variational target block, and
variational source mixture block. (a) The LR rigid alignment block allows the
transformation process through an adaption network we calculated a rigid
transformation among them within a set of six landmark points extracted from
the apex and base of their respective ventricles. This process is fully
automated as we used the respective LR segmentation both for source and target
domain volumes. (b) The Variational target block takes input the concatenation
of low-resolution target input and low resolution source input (i.e.,
$I_{i}=I_{t}+I_{s}$) and with network encoder (blue in figure) a posterior
target distribution $log(q_{\phi}(z_{t}|I_{i})_{t}$ (i.e., eq 33) is
estimated. The image output $I_{t}^{out}$ is reconstructing from
($\mu_{t}$,$\sigma_{t}$), where the input target image $I_{t}$ is added to
target output $I^{*}_{t}$. Notably, to enforce the match between the generate
target image $I_{t}^{out}$ and the target volume $I_{t}$, a discriminator
$D_{t}$ is used to minimize a GAN loss. c) Variational source mixture block
uses the previous target posterior information ($\mu_{t}$,$\sigma_{t}$) to
move the ($\mu_{s}$,$\sigma_{s}$) closer the final target distribution.

###  3.2 Variational Adversarial Mixture Autoencoder (V-AMA)

Here we describe our domain adaption solution. A source domain is defined as
$S_{d}\in\mathbb{R}^{3}$ and a target domain as $T_{d}\in\mathbb{R}^{3}$,
where $S_{d}$ is a LR volume from a specific unknown LR dataset, while $T_{d}$
is the LR training set volume used by Gemini-GAN. We then train a novel
network called Variational Adversarial Mixture Autoencoder (V-AMA) (Fig. 2).
The V-AMA network maps an unknown source LR input volume $I_{s}\in S_{d}$ to
the training target $I_{t}\in T_{d}$ LR volume. The V-AMA is composed of three
distinct blocks: LR rigid alignment block, variational target network, and
variational source mixture network.

####  3.2.1 Low Resolution Rigid Alignment Block

The LR rigid alignment block allows the alignment between LR source volume of
$S_{d}$ and a target volume of $T_{d}$ through rigid transformation between
them within a set of six landmark points extracted between the apex and base
of the ventricles. This simplifies the adaption task; as it only needs to
adapt the intensity distribution of the source domain within its target one.
The rigid registration process is based on a Singular Value Decomposition
(SVD) of the point locations matrix. This method determines the least-squares
fit for the distance of the corresponding domain landmarks. Future
developments of this block include the automatic prediction of landmarks
through a regression neural network and the use of a deep learning algorithm
to perform the final rigid transformation. This block is only essential in the
training phase to align the target and source volumes and have a correct
mapping between them.

####  3.2.2 Variational Target Network Block

The variational target network block is a deep latent variable generative
model for estimating the log-likelihood and the posterior $S_{t}$ target
distribution. The model is composed of an UNet style [30] encoder-decoder
variational target generator network $G_{t}$, followed by a discriminator
network $D_{t}$. Equivalently, for Gemini-GAN, the decoder path is constituted
by $3\times 3$ reiterated convolutions followed by a ReLU and BN. The outputs
encoder is made of up-convolutions concatenated with cropped feature maps from
the input decoder path. During the training, the variational target generator
network $G_{t}$ (Fig. 2), takes the concatenation of both LR source $I_{s}\in
T_{s}$ and target volume $I_{t}\in T_{d}$ as a batch of 2D slices (i.e.,
subscript $s$ denotes the source domain and $t$ denotes the target domain).

The input to the target network (i.e., blue encoder in the variational target
block 2) is indicated as a concatenation operation (+) between the target
volume $I_{t}$ and source volume $I_{s}$ to better learn the consistency
between both (i.e., $I_{i}=I_{t}+I_{s}$). After the encoding process $I_{i}$
is mapped to a Gaussian posterior distribution in the latent space
$q_{\phi}(z_{t}|I_{i})_{t}=\mathcal{N}(z_{t};\mu_{t},diag(\sigma_{t}))$. In
particular, $\phi$ are the variational parameters, $z_{t}$ the latent variable
of the target $t$ encoder, and $I_{t}\in T_{d}$ the LR target volume. The
following equations explain the variational target encoding (i.e., please
refer to theorem A for more precise mathematical explanation):

$\begin{split}I_{i}=I_{t}+I_{s}\\\
(\mu_{t},log(\sigma_{t}))=Encoding_{{G_{t}}_{\phi}}(I_{i})\\\
q_{\phi}(z_{t}|I_{i})_{t}=\mathcal{N}(z_{t};\mu_{t},diag(\sigma_{t}))\\\
log(q_{\phi}(z_{t}|I_{i})_{t})=\sum_{k}log(\mathcal{N}(z_{{t}_{k}},0,1))-log(\sigma_{{t}_{k}})\\\
=\sum_{k}\frac{1}{2}(log(\sigma_{t_{k}})-\mu^{2}_{t_{k}}-\sigma^{2}_{t_{k}}+1)\end{split}$
(7)

The reconstructed target output $I^{*}_{t}$ (i.e., Fig. 2) is concatenated
with $I_{t}$ in order to constrain the match between the $I_{i}$ and $I_{t}$.
The reconstruction target $I^{out}_{t}$ (in output from the variational target
network) is given by:

$I^{out}_{t}=I_{t}+\lambda_{t}\dot{I}^{*}_{t}$ (8)

where $\lambda_{t}$ constant mediates the concatenation between $I_{t}$ and
$I^{*}_{t}$. Also, to better enforce the match between the generated target
volume $I^{out}_{t}$ and the input target volume $I_{t}$ (i.e., sample from
the UKDHP dataset), a discriminator $D_{t}$ is also used to minimize the
following GAN loss for the variational target generator network
$G_{{t}_{\phi}}$ (i.e., please also see theorem B):

$\begin{split}GAN^{t}_{Loss}=\min_{G_{{t}_{\phi}}}\max_{D_{t}}L_{GAN_{t}}(D_{t},G_{{t}_{\phi}})=\\\
=\mathbb{E}_{I^{real}_{t}\sim{UKDHP(I_{t})}}[log(D_{t}(I^{t}_{real}))]+\\\
\mathbb{E}_{I^{fake}_{t}\sim
G_{{t}_{\phi}}(I_{i})}[log(1-D_{t}(I^{t}_{fake})]\end{split}$ (9)

Then, the total target loss [Eq. 11], for the variational target network
block, is composed by the sum of ELBO loss (i.e., Eq 28 in the appendix), GAN
loss [Eq. 9] and L1 loss. If we rearranging [Eq. 26, 27, and 28] we have:

$\begin{split}VAELoss_{\theta,\phi}(I_{i})=log(p_{\theta}(I_{i}))-log(q_{\phi}(z_{t}|I_{i})+p_{\theta}(z_{t}|I_{i})\\\
=log(p_{\theta}(I_{i})\cdot
p_{\theta}(z_{t}|I_{i}))-log(q_{\phi}(z_{t}|I_{i})\\\
=p_{\theta}(I_{i},z_{t})-log(q_{\phi}(z_{t}|I_{i}))\\\ \end{split}$ (10)

The $p_{\theta}(I_{i},z_{t})$ distribution is approximated by a MSE loss
between reconstruction target $I^{out}_{t}$ (i.e., Eq. 8) and variational
network input (i.e., $I_{i}=I_{t}+I_{s}$). While the posterior distribution
$q_{\phi}(z_{t}|I_{i})$ is given by the [Eq. 34] in the appendix.

$\begin{split}VAELoss_{\theta,\phi}(I_{i})_{t}=MSE(I_{i},I^{out}_{t})+\\\
0.5\cdot\sum_{i}\frac{1}{2}(\sigma^{2}_{t_{i}}+\mu^{2}_{t_{i}}-log(\sigma_{t_{i}})-1)\end{split}$
(11)

Final target loss (i.e., variational target block in Fig. 2) is:

$\begin{split}Target_{Loss}=VAELoss_{\theta,\phi}(I_{i})+\\\
L1_{Loss}(I_{i},I^{out}_{t})+0.05*GAN^{t}_{Loss}\end{split}$ (12)

####  3.2.3 Variational Source Mixture Network Block

After training the variational target block, both encoding features (i.e., at
each variational encoding layers) and inference $(\mu_{t}$ and $\sigma_{t})$
[Eq. 33] are transferred to the source encoder (i.e., indicated with
$G_{{s}_{\phi}}$) to equally inject the same amount of target distribution
(Fig.2). The posterior distribution $q_{\phi}(z_{s}|I_{s})_{s}$ (i.e., $s$ in
the following equations represent the source domain $T_{s}$) is derived by:

$\begin{split}(\mu_{s},log(\sigma_{s}))=Encoding_{{G_{s}}_{\phi}}(I_{s})\\\
q_{\phi}(z_{s}|I_{s})_{s}=\mathcal{N}(z_{s};\mu_{s},diag(\sigma_{s}))\\\
log(q_{\phi}(z_{s}|I_{s})_{s})=\sum_{k}log(\mathcal{N}(z_{{s}_{k}},0,1))-log(\sigma_{{s}_{k}})\\\
=\sum_{k}\frac{1}{2}(log(\sigma_{s_{k}})-\mu^{2}_{s_{k}}-\sigma^{2}_{s_{k}}+1)\end{split}$
(13)

where the previous posterior inference of target network
$q_{{\phi}(z|I_{s})_{t}}$ and the new posterior inference of source network
$q_{{\phi}(z_{s}|I_{s})_{s}}$ are added together in the latent space of
$G_{{s}_{\phi}}$ to constitute the final mixture KL posterior
$q_{{\phi}(z_{s}|I_{s})_{m}}$. Indeed, the source network $G_{{s}_{\phi}}$
loss is:

$\begin{split}VAELoss_{\theta,\phi}(I_{s})=MSE(I_{s},I^{out}_{s})+\\\
0.5\cdot\sum_{k}\frac{1}{2}(log(\sigma_{{m}_{k}})-\mu^{2}_{m_{k}}-\sigma^{2}_{m_{k}}+1)\end{split}$
(14)

Given that $(\sigma_{t},\mu_{t})$ and $(\sigma_{s},\mu_{s})$ are normally
distributed independent random variables, then their sum is also normally
distributed. Hence, we obtain $\sigma_{{m}^{2}_{k}}$, $log(\sigma_{{m}_{k})}$,
$\mu_{{m}^{2}_{k}}$, and $z_{{m}_{k}}$ as:

$\begin{split}\sigma_{{m}^{2}_{k}}=\sigma_{t_{k}}^{2}+\sigma_{s_{k}}^{2}\\\
log(\sigma_{{m}_{k}})=log(\sigma_{t_{k}})+log(\sigma_{s_{k}})\\\
\mu_{{m}^{2}_{k}}=(\mu_{t_{k}}+\mu_{s_{k}})^{2}\\\
k_{{m}_{k}}\sim\mathcal{N}(0,I)\\\
z_{{m}_{k}}=(\mu_{t_{k}}+\mu_{s_{k}})+(\sigma_{s_{k}}+\sigma_{t_{k}})\odot
k_{m_{k}}\\\ \end{split}$ (15)

The reconstruction target $I^{out}_{s}$ in output from $G_{{s}_{\phi}}$ is
given by:

$I^{out}_{s}=I_{s}+\lambda_{s}\dot{I}^{*}_{s}$ (16)

Where $\lambda_{s}$ constant mediates the interaction between the source
network input $I_{s}$ and the output $I^{*}_{s}$. As for $G_{{t}_{\phi}}$, to
better enforce the match between the generated source volume $I^{out}_{s}$ and
the target volume distribution $I_{t}$ (i.e., sample from the UKDHP dataset),
we used a discriminator $D_{s}$ to minimize the following GAN loss for the
$G_{{s}_{\phi}}$ network:

$\begin{split}GAN^{s}_{Loss}=min_{G_{{t}_{\phi}}}max_{D_{s}}L_{GAN_{s}}(D_{s},G_{{s}_{\phi}})=\\\
=\mathbb{E}_{I^{real}_{t}\sim{UKDHP(I_{t})}}[log(D_{s}(I^{t}_{real}))]+\\\
\mathbb{E}_{{I^{fake}_{s}}\sim
G_{{s}_{\phi}}(I_{s})}[log(1-D_{s}(I^{s}_{fake})]\end{split}$ (17)

The total source loss, for the variational source network block, is composed
by the sum of posterior mixture KL loss (i.e., eq 14 and 15), GAN loss [Eq 17]
and L1 loss.

$\begin{split}Soruce_{Loss}=VAELoss_{\theta,\phi}(I_{s})+\\\
L1_{Loss}(I_{s},I^{out}_{s})+0.05*GAN^{s}_{Loss}\end{split}$ (18)

The variational source mixture network uses the target posterior information
to fully incorporate the target distribution in source one so that the source
reconstruction $I^{out}_{s}$ will look like the target image UKDHP style. The
obvious benefit is linked to a better inference task in the domain adaption
process where the variational target network is only involved to find the
posterior distribution that captures the UKDHP style. However, in order to
optimize the inference speed, the LR rigid alignment block is used only during
the training stage where during test inference both variational target and
variational source mixture networks take in input the source $I_{s}$.

###  3.3 Training details

The number of UKDHP patients utilized for training is 1131 with $100$ epochs
iterations until convergence and finally valid and testing in 100 UKDHP's
patients. Data argumentation was also applied through the extraction of random
cropping, within vertical and horizontal flip; where the final inference is
performed for all networks comparison on the entire volume at LR. The learning
rate (lr) used is $1e-4$ with a weight decay (i.e., L2 penalty) of $\beta$ set
to $1e-6$ [Eq 4]. Also the $\lambda=1$, for giving equal importance to SR than
3D segmentation during the training; while the GAN constant is set to
$\omega=1e-3$. The LR images, and their corresponding segmentation and SR
images, are used to train the network in a supervised way with ADAM
optimization with random horizontal and vertical flipping and random cropping
[20]. While for V-AMA an ADAM optimization is also used with an lr of $0.0002$
and a momentum (i.e., $beta_{1}$) of 0.5 where $\lambda_{t}=1$ and
$\lambda_{s}=1$. All networks were consistently compared in inference using
the full cardiac LR-volume (i.e., no initial cropping and/or interpolation)
inputs where the training hyperparameters were kept constant for all training
phases. The GPU used is an NVIDIA V100 with a smaller batch size for
minimizing the overhead between the GPU and the Hard Disk Drive (HDD) and
maximizing the use of the overall deep-learning system.

##  4 Results

First we describe the metrics used to quantify algorithm performance. We then
compare the performance of Gemini-GAN (i.e., super-resolution performances) to
several benchmark algorithms used for either sequential or joint SR and
segmentation tasks, and then evaluate V-AMA (i.e., domain-adaption
performances) in an external dataset against other domain adaptation methods.

###  4.1 Segmentation and image quality metrics

####  4.1.1 Dice Index (DI)

Segmentation performance is evaluated for the left ventricular (LV) cavity, LV
myocardium and right ventricular (RV) cavity both in end-systole (ED) and end-
diastole (ED) with the Dice Index (DI). Given two sets of binary mask X and Y,
the DI is defined by the following equation:

$Dice=\frac{2|X\displaystyle\cap Y|}{|A|+|B|}$ (19)

Here $|X|$ and $|Y|$ represent the number of elements for X and Y.

####  4.1.2 Peak Signal-to-Noise Ratio (PSNR)

The PSNR represents the ratio between the maximum possible power of a signal
and the power of corrupting noise that affects the fidelity of its
representation. The mathematical definition of PSNR is expressed
logarithmically. Given two images $I$ and $K$ of size $m\times n$ we define
PSNR (in decibel scale) as:

$\begin{split}PSNR=10\cdot log_{10}(\frac{I^{2}_{Max}}{MSE_{I,K}})=\\\ 20\cdot
log_{10}(\frac{I_{Max}}{\sqrt{MSE_{I,K}}})=20\cdot log_{10}(I_{Max})\\\
-10\cdot log_{10}(MSE_{I,K})\end{split}$ (20)

Here $I_{Max}$ is the maximum possible value of the image $I$, while
$MSE_{I,K}$ is the Mean Squared Error between the image $I$ and $K$.

$MSE_{I,K}=\frac{1}{m\cdot
n}\sum^{m-1}_{i=0}\sum^{n-1}_{j=0}[I(i,j)-K(i,j)]^{2}$ (21)

####  4.1.3 Structural Similarity Index Measure (SSIM)

The structural similarity index measure (SSIM) is a metric for the assessment
of the visual quality of images. Structural information measures the pixel
inter-dependencies that emphasize visual structure and take into consideration
contrast and luminance [40]. Given two images x and y of size $m\times n$ the
SSIM index is defined as:

$SSIM(x,y)=\frac{(2\mu_{x}\mu_{y}+c_{1})\cdot(2\sigma_{xy}+c_{2})}{(\mu^{2}_{x}+\mu^{2}_{y}+c_{1})\cdot(\sigma^{2}_{x}+\sigma^{2}_{y}+c_{2})}$
(22)

Here $\mu_{x}$ is the average of image x, $\mu_{y}$ is the average of image
y,$\sigma^{2}_{x}$ is the variance of image x, $\sigma^{2}_{y}$ is the
variance of image y, and $\sigma_{xy}$ the covariance of image x and y. The
constants $c_{1}$ and $c_{2}$ stabilize the division in the case of a weak
denominator.

###  4.2 Deep-learning methods comparison

We used three state-of-the-art methods for both SR and segmentation
comparison: UNet, SR-GAN, and SegSR-GAN. The UNet model is a advanced
segmentation model [30] based on an encoding path, that captures the general
context of the input image, and a decoding path that produces the final
segmentation. While the SR-GAN is based on two networks a ResNet [17]
generator with a GAN discriminator (i.e., similar to Gemini-GAN). Lastly, the
SegSR-GAN [10] model, a leading-edge joint segmentation method, is further
compared. The SegSR-GAN is essentially similar to SR-GAN in architecture with
the exception of having an upsampling block that gives both segmentation and
SR outputs. However, to be fully consistent with other model's comparisons
(i.e., UNet, SR-GAN, and Gemini-GAN), no initial interpolation and/or image
patch extraction was performed to the input (i.e., as in the original paper
the SegSR-GAN). We instead prefer to respect the initial architecture proposed
by SR-GAN (i.e., adding a skip connection from the first convolution block to
the last before the upsampling layer).

###  4.3 Performance for Super-resolution

We evaluate the super-resolution performances through two main methods: (i)
classic interpolation-based methods, (ii) deep learning approaches in the
UKDHP dataset which contains paired LR and HR ground truth (Table 2). An
example of the output from Gemini-GAN is shown in Figs. 3 and 4. The first
three classic interpolation-based methods models evaluated were nearest
neighbor (NN), linear, and B-spline which are all sub-optimal both in terms of
3D Segmentation and SR (and they cannot be used for an initial pre-
interpolation as in SegSR-GAN paper). The performance of UNet [30] for SR and
segmentation shows good performance in either task individually. We also show
that the SegSR-GAN joint algorithm under-performs the proposed Gemini-GAN
model which generally has the highest segmentation accuracy across all
comparisons while achieving comparable super-resolved image quality to
disjoint methods. As segSR-GAN is a direct derivation of SR-GAN we hypothesize
that the decrease in performance for both networks is linked to two main
causes: firstly the number of parameters increases with respect to the size of
our dataset especially when the ResNet [17] backbone is used. Secondly, the
lack of skip-path connections in the upsampling layers causes a loss of high-
level information responsible for shape and object detection. The number of
ResNet layers utilized does not decisively influence the network performance
(i.e., since an data argumentation mechanism, allows to prevent overfitting
due to extra network parameters).

Fig. 3: Joint 3D super-resolution and segmentation of the left and right
ventricles cavity in long axis cross-section using the Gemini-GAN network
showing the consistency between super-resolved greyscale images and labels
prediction. Fig. 4: Three dimensional renderings of the surface and long-axis
of the heart (yellow: left ventricular myocardium; green: left ventricular
cavity; blue: right ventricular cavity) with corresponding greyscale images
and segmentation contours in a long axis reconstruction. A comparison is shown
between the acquired low resolution (LR) images and Gemini-GAN's joint 3D
segmentation and super-resolution.

Metric Dice PSNR SSIM Anatomy LV cavity ED LV cavity ES LV myocardium ED LV
myocardium ES RV cavity ED RV cavity ES ED ES ED ES Linear 0.02(0.01)
0.01(0.01) 0.003(0.002) 0.004(0.006) 0.001(0.002) 0.0008(0.004) 12.69(0.50)
12.57(0.46) 0.46(0.03) 0.44(0.03) Nearest neighbour 0.02(0.01) 0.02(0.009)
0.004(0.002) 0.005(0.006) 0.001(0.002) 0.0009(0.004) 12.68(0.50) 12.55(0.46)
0.46(0.03) 0.44(0.03) B-Spline 0.001(0.00) 0.001(0.00) 3.24e-05(0.00)
9.93e-06(0.00) 3.84e-05(0.00) 3.52e-05(0.00) 13.39(0.52) 13.41(0.48)
0.46(0.03) 0.44(0.03) UNet [30] (Only Seg) 0.84(0.05) 0.72(0.04) 0.72(0.04)
0.72(0.06) 0.71(0.05) 0.60(0.06) - - - - UNet [30] (Only Img) - - - - - -
27.86(0.65) 28.12(0.71) 0.90(0.01) 0.91(0.01) SR-GAN [23] (Only Seg)
0.85(0.05) 0.50(0.05) 0.70(0.04) 0.58(0.05) 0.71(0.05) 0.59(0.07) - - - - SR-
GAN [23] (Only Img) - - - - - - 23.66(0.37) 23.53(0.38) 0.77(0.01) 0.77(0.01)
SegSR-GAN [10] (Joint) 0.82(0.05) 0.70(0.05) 0.70(0.04) 0.71(0.04) 0.72(0.06)
0.58(0.08) 19.90(0.36) 20.10(0.29) 0.53(0.02) 0.54(0.01) Gemini-GAN [Our]
(Joint) 0.87(0.05) 0.69(0.06) 0.75(0.03) 0.75(0.04) 0.74(0.05) 0.64(0.05)
26.93(0.59) 27.12(0.60) 0.88(0.01) 0.88(0.01)

Table 2: A comparison of each super-resolution model showing segmentation
accuracy for each cardiac structure (Dice) and greyscale image quality metrics
(PSNR and SSIM) with mean and standard deviation (Left ventricle, LV; right
ventricle, RV; end diastole, ED; end systole; ES).

Metric Dice PSNR SSIM Anatomy LV cavity ED LV cavity ES LV myocardium ED LV
myocardium ES RV cavity ED RV cavity ES ED ES ED ES No Adaptation 0.01(0.01)
0.01(0.01) 0.01(0.01) 0.01(0.01) 0.01(0.03) 0.01(0.02) 10.68(0.71) 10.55(0.71)
0.14(0.09) 0.13(0.10) CycleGAN [44] 0.72(0.06) 0.31(0.06) 0.67(0.04)
0.45(0.05) 0.63(0.06) 0.33(0.10) 23.35(0.46) 23.50(0.43) 0.49(0.02) 0.48(0.02)
BicycleGAN [45] 0.48(0.06) 0.09(0.03) 0.54(0.04) 0.16(0.04) 0.50(0.08)
0.14(0.07) 22.21(1.13) 21.67(1.26) 0.49(0.02) 0.48(0.02) MUNIT [19] 0.74(0.06)
0.26(0.06) 0.66(0.04) 0.41(0.05) 0.62(0.06) 0.32(0.10) 23.12(0.44) 22.87(0.42)
0.49(0.02) 0.48(0.02) V-AMA [Our] 0.81(0.05) 0.40(0.06) 0.71(0.04) 0.53(0.06)
0.64(0.04) 0.41(0.09) 25.05(0.52) 24.80(0.54) 0.49(0.02) 0.48(0.02)

Table 3: A comparison in the external validation set (SG) between the
presented V-AMA model and state of art networks (i.e., CycleGAN [44],
BicycleGAN [45], and MUNIT [19]) used for image domain adaption relative to
``No Adaptation''. Segmentation accuracy for each cardiac structure (Dice) and
greyscale image quality metrics (PSNR and SSIM) with mean and standard
deviation are shown. (Left ventricle, LV; right ventricle, RV; end diastole,
ED; end systole; ES). Fig. 5: Three dimensional renderings of the heart
comparing Gemini-GAN with the current state-of-the-art for joint segmentation
and SR (segSR-GAN [10]) which does not accurately reconstruct fine anatomic
details in the greyscale image. Fig. 6: Three dimensional renderings of the
heart comparing 3D segmentations produced by Gemini-GAN after adaptation with
different state-of-the-art methods and our proposed V-AMA method. Our proposed
framework maintains a complete and smooth geometry of the heart while other
methods show discontinuities in the segmentation of the mid-ventricular and
apical myocardium (arrowheads). Fig. 7: Three dimensional renderings of the
heart comparing demonstrating that our Gemini-GAN + V-AMA framework
generalises well to a further external dataset where only LR images are
available (UKBB) demonstrating its potential value for routine clinical
imaging and biobank samples. a) The full volume reconstruction from the 3D
segmentation prediction, b) the volume in cross-section, and c) the greyscale
SR with the 3D segmentation contours.

###  4.4 Performance for domain adaptation

Here we present the performance of the V-AMA network in the external
validation (SG) dataset. Three state-of-the-art networks (CycleGAN [44],
BicycleGAN [45], and MUNIT [19]) used for image domain adaption are compared
to V-AMA (Table 3 and Fig.6).

The main goal of CycleGAN is to learn the mapping with a generator from a
source distribution to a target one by an adversarial and a cycle consistent
loss that boosts the mapping invertibility [44]. To solve this problem, we use
BicycleGAN - where two networks are trained with cycle loss and random noise
injection. The first network is a conditional variational auto-encoder, that
encodes the target image in the latent target space and uses it to map the
target image (i.e., from the source image to latent target space to target
image). The second network is a conditional latent regressor network and uses
the source image and random noise to produce the latent target image [45]. The
injection of random noise in the second BicycleGAN network could potentially
lead to instability and is reflected by a performance decrease for the Dice
index.

In the MUNIT model, the latent space source volume input is decomposed into a
style space and content space. This is performed by content encoder and style
encoder networks. While the style encoder network is composed of several
stride convolutions, followed by global pooling and MPL layer to produce a set
of parameters. The content encoder network processes the source volume to give
as input to the decoding network with adjusted parameters by the style
encoder. However, the style of the content encoder network is limited by the
previous global pooling operation in the style encoder that is insufficient to
extract detailed information due to the intrinsic limitation of pooling
operation [31] (i.e., pooling takes the neuron with high peak information but
not the one relevant for the specific encoding task to performed).

To address this problem, V-AMA is proposed to decouple the source latent space
through two adversarial variational networks without any pooling in the style
content-encoding where the variational mechanism is instead used to share
style information. In the external validation set, this provided the highest
segmentation accuracy for all cardiac structures as well as the best image
quality metrics.

We further tested V-AMA in the UKBB dataset (which is representative of
routine imaging and population data), but since there is no specific SR
ground-truth both in 3D segmentation and greyscale in UKBB, we limit our
analysis to a qualitative evaluation; confirming the system's ability to
generalize well to a further external dataset (Fig 7).

##  5 Discussion and Conclusion

The proposed deep-learning system creates a hybrid between supervised and
unsupervised learning algorithms for training the Gemini-GAN and V-AMA
algorithms respectively. We have shown the improved performance of Gemini-GAN
compared to best in class networks for jointly determining 3D Segmentation and
SR from a natively LR input volume while maintaining high PSNR and SSIM.

We compared our network with SR-GAN which showed relatively poor performance
for segmentation accuracy and image quality. This was primarily due to a lack
of a large dataset (i.e, 350,000 ImageNet images were originally used to train
SR-GAN) to satisfy the depth of the ResNet networks. Furthermore, the lack of
skip path connections in both SR-GAN and Seg-SRGAN reduces the segmentation
performance, as the local information in the high-level features in the first
convolutions layers is lost. A further limitation of Seg-SRGAN, is related to
the initial input B-spline interpolation which can introduce a prediction
bias. Furthermore, interpolating the volume with a classical method, as in the
original Seg-SRGAN paper, may be computationally ineffective. This can be
unfavorable if we additionally want to determine the spatial information from
a full volumetric LR input, where erasing of the skip path, at each up-
sampling level, reduces the possibility of reaching the correct 3D spatial
information between the individual channels at each encoding layer. However,
Gemini-GAN exhibits high performance for both 3D segmentation and SR directly
from LR greyscale volumes even with limited dataset size.

Our unsupervised domain adaptation algorithm, V-AMA, is also proposed here.
After spatial alignment of the source and target volume within a rigid
alignment algorithm, paired distinct adversarial VAE networks are utilized.
The first network, called variation target block, extracts the variational
source volume parameters in terms of mean and std, transferred to a second
network, a variational source mixture block, which mixes both source and
target VAE's parameters. The V-AMA model has several advantages, in the first
place it solves the mode collapse problem [14], where the optimization of
multidimensional non-convex space can lead to instability and collapse the
model to a single distribution (i.e., only source but not target). Secondly,
it simplifies the mapping operation between source to target distributions by
disassociating the inference process within two distinct deep networks. To
quantify the ability of V-AMA to adapt its domain in a small dataset we used
an external dataset and three modern domain adaption networks demonstrating
its accurate and robust performance.

Mode collapse is one of the main problems for other algorithms, which could be
partially solved by injecting input noise into the network as in BicycleGAN
[45]. The direct injection of random noise in the network could lead to
instability especially when the source distribution differs from the target
one. Notably, any additional random noise could amplify this behavior.
Instead, the MUNIT model decomposes the latent space into style space and
content space within two networks decreasing the mode collapse problem.
Notably, one of the main limitations of MUNIT is related to global pooling in
style encoding, where the global pooling layer loses relevant information and
neglecting the spatial relation between the objects in the volume (i.e., LV vs
RV cavity position in the volume input) [32]. Instead, our V-AMA model avoids
the pooling by mixing the source and target style directly in variational
inference fashion.

In summary, we have experimentally demonstrated how our deep learning system
can achieve high performance in joint cardiac SR and 3D segmentation tasks
with domain adaption to new distributions. Our method will enable future
applications in the field of cardiac shape and motion analysis where precision
phenotypes are needed for prediction and classification tasks across diverse
datasets.

## Data availability

The code [34] and segmentations [35] used in this study are open access.

## Acknowledgments

The study was supported by Bayer AG; Medical Research Council (MC-A658-5QEB0);
National Institute for Health Research (NIHR) Imperial College Biomedical
Research Centre; British Heart Foundation (NH/17/1/32725, RG/19/6/34387,
RE/18/4/34215); Academy of Medical Sciences (SGL015/1006); Mason Medical
Research Trust grant; and the Engineering and Physical Sciences Research
Council (EP/P001009/1).

## References

  * Bai et al. [2015] Bai, W., Shi, W., de Marvao, A., Dawes, T.J., O’Regan, D.P., Cook, S.A., Rueckert, D., 2015.  A bi-ventricular cardiac atlas built from 1000+ high resolution MR images of healthy subjects and an analysis of shape and motion.  Med Image Anal 26, 133–145.
  * Bao et al. [2019] Bao, W., Lai, W.S., Ma, C., Zhang, X., Gao, Z., Yang, M.H., 2019.  Depth-aware video frame interpolation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3703–3712.
  * Bello et al. [2019] Bello, G.A., Dawes, T.J., Duan, J., Biffi, C., de Marvao, A., Howard, L.S., Gibbs, J.S.R., Wilkins, M.R., Cook, S.A., Rueckert, D., et al., 2019\.  Deep-learning cardiac motion analysis for human survival prediction.  Nat Mach Intell 1, 95–104.
  * Bycroft et al. [2018] Bycroft, C., Freeman, C., Petkova, D., Band, G., Elliott, L.T., Sharp, K., Motyer, A., Vukcevic, D., Delaneau, O., O’Connell, J., et al., 2018\.  The UK biobank resource with deep phenotyping and genomic data.  Nature 562, 203–209.
  * Cao et al. [2020] Cao, S., Wu, C.Y., Krähenbühl, P., 2020.  Lossless image compression through super-resolution.  arXiv preprint arXiv:2004.02872 .
  * Chaudhari et al. [2018] Chaudhari, A.S., Fang, Z., Kogan, F., Wood, J., Stevens, K.J., Gibbons, E.K., Lee, J.H., Gold, G.E., Hargreaves, B.A., 2018.  Super-resolution musculoskeletal MRI using deep learning.  Magn Reson Med 80, 2139–2154.
  * Chen et al. [2018] Chen, Y., Shi, F., Christodoulou, A.G., Xie, Y., Zhou, Z., Li, D., 2018.  Efficient and accurate MRI super-resolution using a generative adversarial network and 3D multi-level densely connected network, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer. pp. 91–99.
  * Cui et al. [2014] Cui, Z., Chang, H., Shan, S., Zhong, B., Chen, X., 2014\.  Deep network cascade for image super-resolution, in: European Conference on Computer Vision, Springer. pp. 49–64.
  * Davatzikos et al. [2003] Davatzikos, C., Tao, X., Shen, D., 2003.  Hierarchical active shape models, using the wavelet transform.  IEEE Trans Med Imaging 22, 414–423.
  * Delannoy et al. [2020] Delannoy, Q., Pham, C.H., Cazorla, C., Tor-Díez, C., Dollé, G., Meunier, H., Bednarek, N., Fablet, R., Passat, N., Rousseau, F., 2020\.  SegSRGAN: Super-resolution and segmentation using generative adversarial networks—application to neonatal brain MRI.  Comput Biol Med 120, 103755\.
  * Dong et al. [2014] Dong, C., Loy, C.C., He, K., Tang, X., 2014.  Learning a deep convolutional network for image super-resolution, in: European conference on computer vision, Springer. pp. 184–199.
  * Dou et al. [2018] Dou, Q., Ouyang, C., Chen, C., Chen, H., Heng, P.A., 2018\.  Unsupervised cross-modality domain adaptation of convnets for biomedical image segmentations with adversarial loss.  arXiv preprint arXiv:1804.10916 .
  * Duan et al. [2019] Duan, J., Bello, G., Schlemper, J., Bai, W., Dawes, T.J.W., Biffi, C., de Marvao, A., Doumoud, G., O’Regan, D.P., Rueckert, D., 2019\.  Automatic 3D bi-ventricular segmentation of cardiac images by a shape-refined multi- task deep learning approach.  IEEE Trans Med Imaging 38, 2151–2164.  doi:10.1109/TMI.2019.2894322.
  * Durall et al. [2020] Durall, R., Chatzimichailidis, A., Labus, P., Keuper, J., 2020\.  Combating mode collapse in GAN training: An empirical analysis using Hessian Eigenvalues.  arXiv preprint arXiv:2012.09673 .
  * Ghifary et al. [2016] Ghifary, M., Kleijn, W.B., Zhang, M., Balduzzi, D., Li, W., 2016.  Deep reconstruction-classification networks for unsupervised domain adaptation, in: European Conference on Computer Vision, Springer. pp. 597–613.
  * Gholipour et al. [2010] Gholipour, A., Estroff, J.A., Warfield, S.K., 2010.  Robust super-resolution volume reconstruction from slice acquisitions: application to fetal brain MRI.  IEEE Trans Med Imaging 29, 1739–1758.
  * He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J., 2015.  Deep residual learning for image recognition.  arXiv preprint arXiv:1512.03385 .
  * Hoffman et al. [2018] Hoffman, J., Tzeng, E., Park, T., Zhu, J.Y., Isola, P., Saenko, K., Efros, A., Darrell, T., 2018\.  Cycada: Cycle-consistent adversarial domain adaptation, in: International conference on machine learning, PMLR. pp. 1989–1998.
  * Huang et al. [2018] Huang, X., Liu, M.Y., Belongie, S., Kautz, J., 2018\.  Multimodal unsupervised image-to-image translation, in: Proceedings of the European conference on computer vision (ECCV), pp. 172–189.
  * Kingma and Ba [2014] Kingma, D.P., Ba, J., 2014.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980 .
  * Kingma and Welling [2013] Kingma, D.P., Welling, M., 2013\.  Auto-encoding variational Bayes.  arXiv preprint arXiv:1312.6114 .
  * Kingma and Welling [2019] Kingma, D.P., Welling, M., 2019\.  An introduction to variational autoencoders.  arXiv preprint arXiv:1906.02691 .
  * Ledig et al. [2017] Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al., 2017\.  Photo-realistic single image super-resolution using a generative adversarial network, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4681–4690.
  * Moghari et al. [2018] Moghari, M.H., Barthur, A., Amaral, M.E., Geva, T., Powell, A.J., 2018.  Free-breathing whole-heart 3D cine magnetic resonance imaging with prospective respiratory motion compensation.  Magn Reson Med 80, 181–189.
  * Oktay et al. [2017] Oktay, O., Ferrante, E., Kamnitsas, K., Heinrich, M., Bai, W., Caballero, J., Cook, S.A., De Marvao, A., Dawes, T., O‘Regan, D.P., et al., 2017\.  Anatomically constrained neural networks (ACNNs): application to cardiac image enhancement and segmentation.  IEEE Trans Med Imaging 37, 384–395.
  * Perone et al. [2019] Perone, C.S., Ballester, P., Barros, R.C., Cohen-Adad, J., 2019\.  Unsupervised domain adaptation for medical imaging segmentation with self-ensembling.  NeuroImage 194, 1–11.
  * Petersen et al. [2015] Petersen, S.E., Matthews, P.M., Francis, J.M., Robson, M.D., Zemrak, F., Boubertakh, R., Young, A.A., Hudson, S., Weale, P., Garratt, S., et al., 2015\.  UK Biobank’s cardiovascular magnetic resonance protocol.  J Cardiovasc Magn Reson 18, 8\.
  * Rasti et al. [2016] Rasti, P., Uiboupin, T., Escalera, S., Anbarjafari, G., 2016\.  Convolutional neural network super resolution for face recognition in surveillance monitoring, in: International conference on articulated motion and deformable objects, Springer. pp. 175–184.
  * Rezende et al. [2014] Rezende, D.J., Mohamed, S., Wierstra, D., 2014.  Stochastic backpropagation and approximate inference in deep generative models (2014).  arXiv preprint arXiv:1401.4082 .
  * Ronneberger et al. [2015] Ronneberger, O., Fischer, P., Brox, T., 2015.  U-net: Convolutional networks for biomedical image segmentation, in: International Conference on Medical image computing and computer-assisted intervention, Springer. pp. 234–241.
  * Sabour et al. [2017a] Sabour, S., Frosst, N., Hinton, G., 2017a.  Dynamic routing between capsules.
  * Sabour et al. [2017b] Sabour, S., Frosst, N., Hinton, G.E., 2017b.  Dynamic routing between capsules.  arXiv preprint arXiv:1710.09829 .
  * Sánchez and Vilaplana [2018] Sánchez, I., Vilaplana, V., 2018\.  Brain MRI super-resolution using 3D generative adversarial networks.  arXiv preprint arXiv:1812.11440 .
  * Savioli et al. [2021a] Savioli, N., de Marvao, A., Bai, W., Wang, S., Cook, S.A., Chin, C.W.L., Rueckert, D., O’Regan, D.P., 2021a.  ImperialCollegeLondon/Gemini-GAN.  Zenodo doi:10.5281/zenodo.5005942.
  * Savioli et al. [2021b] Savioli, N., de Marvao, A., O’Regan, D.P., 2021b.  Cardiac super-resolution label maps.  Mendeley Data V1.  doi:10.17632/pw87p286yx.1.
  * Schafer et al. [2017] Schafer, S., De Marvao, A., Adami, E., Fiedler, L.R., Ng, B., Khin, E., Rackham, O.J., Van Heesch, S., Pua, C.J., Kui, M., et al., 2017\.  Titin-truncating variants affect heart function in disease cohorts and the general population.  Nat Genet 49, 46–53.
  * Shui et al. [2020] Shui, C., Chen, Q., Wen, J., Zhou, F., Gagné, C., Wang, B., 2020.  Beyond H-divergence: Domain adaptation theory with Jensen-Shannon divergence.  arXiv preprint arXiv:2007.15567 .
  * Tzeng et al. [2014] Tzeng, E., Hoffman, J., Zhang, N., Saenko, K., Darrell, T., 2014.  Deep domain confusion: Maximizing for domain invariance.  arXiv preprint arXiv:1412.3474 .
  * Wang and Deng [2018] Wang, M., Deng, W., 2018.  Deep visual domain adaptation: A survey.  Neurocomputing 312, 135–153.
  * Wang et al. [2004] Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P., 2004\.  Image quality assessment: from error visibility to structural similarity.  IEEE Trans Image Process 13, 600–612.
  * Woodbridge et al. [2013] Woodbridge, M., Fagiolo, G., O’Regan, D.P., 2013.  MRIdb: medical image management for biobank research.  J Digit Imaging 26, 886–890.
  * Xia et al. [2021] Xia, Y., Ravikumar, N., Greenwood, J.P., Neubauer, S., Petersen, S.E., Frangi, A.F., 2021\.  Super-resolution of cardiac MR cine imaging using conditional GANs and unsupervised transfer learning.  Med Image Anal 71, 102037\.
  * Yang et al. [2020] Yang, J., An, W., Wang, S., Zhu, X., Yan, C., Huang, J., 2020.  Label-driven reconstruction for domain adaptation in semantic segmentation, in: European Conference on Computer Vision, Springer. pp. 480–498.
  * Zhu et al. [2017a] Zhu, J.Y., Park, T., Isola, P., Efros, A.A., 2017a.  Unpaired image-to-image translation using cycle-consistent adversarial networks, in: Proceedings of the IEEE international conference on computer vision, pp. 2223–2232.
  * Zhu et al. [2017b] Zhu, J.Y., Zhang, R., Pathak, D., Darrell, T., Efros, A.A., Wang, O., Shechtman, E., 2017b.  Toward multimodal image-to-image translation.  arXiv preprint arXiv:1711.11586 .
  * Zhu et al. [2014] Zhu, Y., Zhang, Y., Yuille, A.L., 2014.  Single image super-resolution using deformable patches, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2917–2924.

##  Appendix A Theorem 1

In this appendix section, we are going to demonstrate how the VAE loss is
given by the combination of Evidence Lower Bound (ELBO) and Kullback-Leibler
(KL) divergence and how derivative a gradient forms from it. Indeed, the
latent variable $z$ is part of the variational target generator model but we
can't directly observe it. In this specific case the direct graphical model is
represented by a joint distribution $p_{\theta}(x;z)$ over both LR input data
target volume $x$ and latent variable $z$. Where, the marginal distribution
$p_{\theta}(x)$, over the observed variable $x$, is given by:

$\begin{split}p_{\theta}(x)=\int p_{\theta}(x,z)dz\end{split}$ (23)

The $\theta$ are the parameters that minimize the integral. However, this
marginal distribution $p_{\theta}(x)$ can be complicated and need to be
approximate through a factoring process:

$p_{\theta}(x,z)=p_{\theta}(x)p_{\theta}(x|z)$ (24)

The distribution $p(z)$ is also called prior distribution over latent $z$
variables. The challenge is due to the intractability of $p_{\theta}(x,z)$,
where no analytic solution is given. To solve this intractability, a
parametric inference model $q_{\phi}(z|x)$ is then introduce trough an encoder
network (i.e., $G_{\phi}$) that optimize the variational parameters $\phi$ as
$q_{\phi}(z|x)\approx p_{\theta}(z|x)$. In the variational inference, this is
called lower bound objective (ELBO) optimisation problem [Kingma and Welling,
2019]. Then, for any type of parametric inference model $q_{\phi}(z|x)$, is
given:

$\begin{split}log(p_{\theta}(x))=\mathbb{E}_{q_{\phi}(z|x)}[logp_{\theta}(x)]=\\\
=\mathbb{E}_{q_{\phi}(z|x)}[log(\frac{p_{\theta}(x,z)}{p_{\theta}(z|x)})]\\\
=\mathbb{E}_{q_{\phi}(z|x)}[log(\frac{p_{\theta}(x,z)}{q_{\phi}(z|x)}\frac{q_{\phi}(z|x)}{p_{\phi}(z|x)})]\\\
=\mathbb{E}_{q_{\phi}(z|x)}[log(\frac{p_{\theta}(x,z)}{q_{\phi}(z|x)})]+\mathbb{E}_{q_{\phi}(z|x)}[log(\frac{q_{\phi}(z|x)}{p_{\theta}(z|x)}))]\end{split}$
(25)

The first term, of the last equation line, is the ELBO (i.e., the final
variational loss $Loss_{\theta,\phi}(x)$) and is always lower bound for
$log(p_{\theta}(x))$ (i.e., due to the positive value of the second term). The
ELBO is obtained by substitute the [Eq. 24] on [Eq. 25] (i.e., first term of
last equation line).

$Loss_{\theta,\phi}(x)=\mathbb{E}_{q_{\phi}(z|x)}[log(\frac{p_{\theta}(x)p_{\theta}(x|z)}{q_{\phi}(z|x)})]$
(26)

$\begin{split}Loss_{\theta,\phi}(x)=logp_{\theta}(x)-D_{KL}(q_{\phi}(z|x)||p_{\theta}(z|x))\leq
logp_{\theta}(x)\end{split}$ (27)

The second term is the Kullback-Leibler (KL) divergence among the parametric
inference model $q_{\phi}(z|x)$ and the marginal distribution
$p_{\theta}(z|x)$ (i.e., also positive):

$D_{KL}(q_{\phi}(z|x)||p_{\theta}(z|x))=\mathbb{E}_{q_{\phi}(z|x)}[log(\frac{q_{\phi}(z|x)}{p_{\theta}(z|x)})]\geq
0$ (28)

Finally, we estimate the gradient for both parameters $\phi$ and $\theta$. In
particular for $\theta$ we have:

$\begin{split}\nabla_{\theta}Loss_{\theta,\phi}(x)=\nabla_{\theta}\mathbb{E}_{q_{\phi(z|x)}}[log(p_{\theta}(x,z))-log(q_{\phi}(z|x))]\\\
=\mathbb{E}_{q_{\phi}(z|x)}[\nabla_{\theta}(log(p_{\theta}(x,z))-log(q_{\phi}(z|x))]\\\
\simeq\nabla_{\theta}(log(p_{\theta}(x,z))-log(q_{\phi}(z|x)))=\nabla_{\theta}(logp_{\theta}(x,z))\end{split}$
(29)

Where the last term of this equation represents the Monte Carlo estimator for
$z$ latent variable, randomly sampled from the parametric inference model
$q_{\phi}(z|x)$ [Kingma and Welling, 2013]. However, for $\phi$ parameters
there is no valid computational operation between expectation and gradient.
Nonetheless, the $G_{{1}_{\phi}}$ encoder is fully differentiable and the
latent variable $z$ used continued. Therefore, is calculate the gradient for
$\phi$ variable with a reparameterization step [Rezende et al., 2014]. The
reparameterization expresses the random variable $z$ in terms of another
variable $k$ within a differentiable invertible transformation function
$f(\cdot)$. The random noise of the reparametrization variable $k$ is sampled
from a univariate Gaussian distribution $\mathcal{N}(0,I)$.

$z=f(k,\phi,x)$ (30)

The random distribution variable $k$ is independent both of the input variable
$x$ and the parameter $\phi$. In this case the commutation operation between
expectation and gradient (i.e., both for the parameters $\theta$ and $\phi$)
is guaranteed. Then, the new Monte Carlo estimator with $z=f(k,\phi,x)$ and
random noise sample from $k\sim p(k)$, is given by the following equation:

$\begin{split}\mathbb{E}_{q_{\phi(z|x)}}[Loss_{\theta,\phi}(z)]=\mathbb{E}_{p(k)}[Loss_{\theta,\phi}(z)]\\\
=\nabla_{\phi}\mathbb{E}_{q_{\phi(z|x)}}[Loss_{\theta,\phi}(z)]\\\
=\mathbb{E}_{p(k)}[\nabla_{\phi}Loss_{\theta,\phi}(z)]\\\
\simeq\nabla_{\phi}Loss_{\theta,\phi}(z)\end{split}$ (31)

After the reparameterization trick:

$\begin{split}k\sim\mathcal{N}(0,I)\\\
(\mu_{t},log\sigma_{t})=Encoding_{{G_{t}}_{\phi}}(x)\\\
z=\mu_{t}+\sigma_{t}\odot k\end{split}$ (32)

Given the $q_{\phi}(z|x)$ is Gaussian and is approximate by
$Encoding_{{G_{x}}_{\phi}}$, the final inference is given by:

$\begin{split}(\mu_{x},log(\sigma_{x}))=Encoding_{{G_{x}}_{\phi}}(I_{x})\\\
q_{\phi}(z_{x}|I_{i})_{x}=\mathcal{N}(z_{x};\mu_{x},diag(\sigma_{x}))\\\
\end{split}$ (33)

Where $I_{x}$ indicated a general input image, $Encoding_{{G_{x}}_{\phi}}$ a
generic encoding network, $z_{x}$ a generic latent variables of
$Encoding_{{G_{x}}_{\phi}}$, and $\mu_{x}$, $\sigma_{x}$ the generic mean and
standard deviation (i.e., predicted by $Encoding_{{G_{x}}_{\phi}}$) of the
Gaussian distribution $\mathcal{N}(z_{x};\mu_{x},diag(\sigma_{x}))$, $k$ is
the index of the sum operation. We can then conclude

$\begin{split}log(q_{\phi}(z_{x}|I_{x})_{x})=\sum_{k}log(\mathcal{N}(z_{{x}_{k}},0,1))-log(\sigma_{{x}_{k}})\\\
=\sum_{k}\frac{1}{2}(log(\sigma_{x_{k}})-\mu^{2}_{x_{k}}-\sigma^{2}_{x_{k}}+1)\end{split}$
(34)

##  Appendix B Theorem 2

In this appendix section, we are going to demonstrate that if the variational
generator $G_{{t}_{\phi}}$ and its discriminator $D_{t}$ are enough
numerically stable, at each step iteration, the discriminator $D_{t}$ reach
its optimum where the target output probability $p^{*}_{t}(x)$ of $I^{*}_{t}$
converges to the target data $I_{t}$ UKDHP distribution probability
$p_{t}(x)$. Then, if we set $K(p^{*}_{t}(x),D_{t})$ a function of the ``fake''
target probability distribution generate by $G_{{t}_{\phi}}$, the function
$K(p^{*}_{t}(x),D_{t})$ shows a convexity in this ``fake'' probability
distribution $p^{*}_{t}(x)$.

To prove this, we let:

$\begin{split}K(p^{*}_{t}(x),D_{t})=sup_{\alpha}K_{\alpha}(p^{*}_{t}(x),D_{t})\end{split}$
(35)

Where the function $K_{\alpha}(p^{*}_{t}(x),D_{t})$ are convex on some convex
domain $S$. We suppose that at particular point $p^{*}_{t}(x)$ in S,

$\begin{split}\beta=argsup_{\alpha}K_{\alpha}(p^{*}_{t}(x),D_{t})\end{split}$
(36)

Then,

$\begin{split}K(p^{*}_{t}(x),D_{t})=K_{\beta}(p^{*}_{t}(x),D_{t})\end{split}$
(37)

Let m be any subgradient of $K_{\beta}(p^{*}_{t}(x),D_{t})$, for definition of
subgradient,

$\begin{split}K_{\beta}(p^{*}_{t}(y),D_{t})\geq
K_{\beta}(p^{*}_{t}(x),D_{t})+m^{T}(y-x)\end{split}$ (38)

Since $K(p^{*}_{t}(y),D_{t})\geq K_{\beta}(p^{*}_{t}(x),D_{t})$ for $y\in S$,

$\begin{split}K(p^{*}_{t}(y),D_{3})\geq
K(p^{*}_{t}(x),D_{3})+m^{T}(y-x)\end{split}$ (39)

Thus, we have $m\in\partial K(p^{*}_{t}(x),D_{t})$, but also for any
subgradient $\partial K_{\beta}(p^{*}_{t}(x),D_{t})\in\partial
K(p^{*}_{t}(x),D_{t})$. This is equivalent to calculate any update of
Stochastic Gradient Descent (SDG) for $p^{*}_{t}(x)$ at the optimal
discriminator value $D^{optim}_{t}(x)$, given the variational generator
$G_{t\phi}$, to observed that any small update of generative ``fake''
distribution $p^{*}_{t}(x)$ and UKDHP data ``real'' distribution $p_{t}(x)$
converges to the data ``real'' distribution $p^{*}_{t}(x)$; as
$sup_{D^{optim}_{t}(x)}K(p^{*}_{t}(x),D^{optim}_{t}(x))$ is convex in
$p^{*}_{t}(x)$, $p^{*}_{t}\xrightarrow{}p_{t}(x)$, concluding the proof.


# Getting Topology and Point Cloud Generation to Mesh

Austin Dill, Chun-Liang Li⋆, Songwei Ge, Eunsu Kang
Carnegie Mellon University
Pittsburgh, PA, United States
{abdill, chunlial, songweig<EMAIL_ADDRESS>

equal contribution

###### Abstract

In this work, we explore the idea that effective generative models for point
clouds under the autoencoding framework must acknowledge the relationship
between a continuous surface, a discretized mesh, and a set of points sampled
from the surface. This view motivates a generative model that works by
progressively deforming a uniform sphere until it approximates the goal point
cloud. We review the underlying concepts leading to this conclusion from
computer graphics and topology in differential geometry, and model the
generation process as deformation via deep neural network parameterization.
Finally, we show that this view of the problem produces a model that can
generate quality meshes efficiently.

##  1 Introduction

The proliferation of 3D point cloud data from LiDAR sensors for self-driving
cars and commercial sensors like Microsoft’s Kinect has led to an increase in
the interest for machine learning techniques to interpret 3D data. Just as
generative models for text and images gained traction following the success of
classification models for those domains, the success of neural networks on
point clouds (Qi et al., 2017a, b; Zaheer et al., 2017) has led to a growth in
research into point cloud generative models (Yang et al., 2018; Li et al.,
2018; Groueix et al., 2018a; Yang et al., 2019; Groueix et al., 2018b; Li et
al., 2019).

In this paper, we study point cloud autoencoders, which are building
components of a full point cloud generative model (Li et al., 2018; Yang et
al., 2019). The decoder itself is a generative model which samples from a
corresponding 3-dimensional distribution of the input point cloud. If we
naively train a neural network to generate point clouds, we will end up with
an extremely under-constrained problem. This will result in low reconstruction
loss but a non-uniform distribution of points on the surface of the object and
it will require an additional meshing process.

In order to remedy this, we draw inspiration from computer graphics and
topology by learning a generative model by way of deformation. We begin with a
goal object we hope to reconstruct and an initial point cloud. At each step,
we will learn a slight modification of the initial point cloud so the output
will be close to the goal point cloud. As the underlying object that generated
the point cloud sample is a solid object, we will constrain our model to learn
deformations that obey the rules that govern valid topological and mesh
deformations. As we will see, this will limit our network to invertible
deformations. Incidentally, by focusing on this type of deformation, we can
frequently use a mesh from the initial point cloud as a template for our final
object, simplifying the meshing pipeline.

##  2 Background

Traditional computer graphics approaches such as ball-pivoting, marching
cubes, and Poisson reconstruction work by algorithmically building a mesh to
fit a fixed set of points.

Ezuz et al. (2019) connect this generation of meshes to the problem of
matching points on corresponding meshes. Their problem then becomes finding a
plausible, smooth, and accurate map between triangular meshes. As investigated
in their work, this requires a harmonic and reversible mapping between meshes
that can be found by optimizing a loss function for individual pairs of meshes
with a true underlying correspondence. This approach has the benefit of
allowing the transference of texture and other mesh properties from the source
mesh to the target mesh.

The closest work to ours in the deep learning community is that of FoldingNet
(Yang et al., 2018). Their model learns a series of deformations (or folds)
from an initial fixed 2D grid of points to a final object. It seems intuitive
that starting from a 3D surface could lead to an easier learning problem, just
as it is simpler to mold clay than to fold origami. This idea is generalized
with AtlasNet where multiple 2D grids are used with multiple generators. In
addition they explore using a sphere to sample points, but neglect the
theoretical advantages connecting this approach to topology and the concept of
deformations (Groueix et al., 2018a).

While our work focuses on the generation of point clouds, a similar vein of
work has been explored in the mesh reconstruction field by works like Wang et
al. (2018) and Kanazawa et al. (2018). These methods both proceed by deforming
an initial mesh (given a priori or learned respectively) into a final shape.
These methods employ a graph-based method that does not allow for sampling an
arbitrary number of points.

##  3 Theoretical Justification

We must examine the complexity of transforming various initial distributions
to the type of surfaces encoded with point clouds to justify our claim that
our model is a more natural generative model for point clouds. To simplify
matters, we will limit our discussion to maps from a $3$-ball and a
3-dimensional isotropic Gaussian to a $2$-sphere.

Point clouds are typically sampled from the surfaces of real-world objects or
realistic meshes. Intuitively, this means that the majority of points are
located on the boundary of objects and not on the interior. If we hope to
perfectly capture the surface of objects in point clouds with a continuous,
invertible map as has become common practice in many generative models, we
must consider the topology of our initial shape (Rezende and Mohamed, 2015;
Grathwohl et al., 2018; Behrmann et al., 2018).

######  Theorem 1.

There is no continuous invertible map between the 3-ball and the 2-sphere that
respects the boundary.

###### Proof.

This follows from Brouwer’s fixed point theorem. ∎

######  Theorem 2.

There is no continuous invertible map between $\mathbb{R}^{3}$ and the
2-sphere that respects the boundary.

###### Proof.

This follows from the relationship between Hausdorf spaces and compact
subspaces. ∎

These results show that if we wish to learn a transformation that is
continuous, invertible, and achieves no error on the boundary, we must choose
an initial point cloud that is topologically close to our goal point cloud.
Otherwise, our efforts will be thwarted by the underlying topology. For these
reasons we choose to start from a hollow sphere of points with radius 1 as our
initial shape. We believe that this is the most topologically similar
structure that is simple to sample from. As we will see in section 4.2, this
decision gives us additional advantages.

##  4 Method

###  4.1 Architecture

The architecture for our network is built on the idea of repeated deformations
to an initial point cloud based on the encoding generated by a Deep Set model
(Zaheer et al., 2017). This model takes inspiration from FoldingNet with its
series of "folds" replaced by deformations and its graph-based encoder
exchanged for a set-based encoder (Yang et al., 2018).

On top of this basic framework, we introduce a forward deformation network
(going from a random sphere to the goal point cloud) and a backward pass (from
the goal point cloud to a sphere). Training both of these networks
simultaneously is meant to regularize the transformation, as inspired by the
computer graphics community’s requirement for an invertible function without
limiting ourselves to models with an analytic inverse. The forward
architecture is depicted in Figure 1 and the backward architecture is
identical.

Figure 1: A residual structure allows for gradual deformation from a random
sphere to the final shape.

###  4.2 Loss Function

Our loss function encodes our desire to minimize distortion. While the
majority of point cloud generative models are trained using Chamfer distance
for autoencoder models or maximum likelihood for normalizing flow models, our
loss function takes inspiration from the computer graphics community. Note
that each $p$ is a point and $\phi(\cdot)$ is our learned function.

$\mathcal{L}_{deform}=\sum_{i}\sum_{j\in
N(i)}(p_{i}-p_{j})^{2}-(\phi(p_{i})-\phi(p_{j}))^{2}$ (1)

Equation 1 can be seen as an approximation to the Laplacian loss frequently
used in mesh generation tasks (Wang et al., 2018). While the true Laplacian
would require knowledge of the neighborhood of each point, we can use
properties of the sphere to approximate it and then enforce that the
neighborhood persists in the output point cloud. For each point on the sphere,
its neighborhood may be simply approximated as the $k$-nearest neighbors in
Euclidean distance. This allows us to define the neighborhood function
$N(\cdot)$ required in equation 1. Our final loss function is a weighted
combination of the Chamfer loss in both directions and the deformation loss.

##  5 Experimental Results

###  5.1 Dataset

In order to train and test our model, we sample points uniformly from the
surface of meshes provided in the ShapeNet dataset (Chang et al. (2015)). The
51,300 meshes cover 55 distinct categories including airplanes, cars, lamps,
and doors. All of the results in the following sections are trained on a
portion of each category.

###  5.2 Metrics and Results

Taking our cue from previous works, we will use Distance to Face (D2F) and
Coverage as measures for the quality of our reconstruction (Li et al., 2018).
These metrics are used by Lucic et al. (2018) to evaluate GANs with the
different names called precision and recall.

$D2F\left(\left\\{x_{i}\right\\}_{i=1}^{n},\left\\{F_{j}\right\\}_{j=1}^{m}\right)=\frac{1}{n}\sum_{i=1}^{n}\min_{j}\mathcal{D}\left(x_{i},F_{j}\right)$
(2)

$\text{Cov}(\left(\left\\{x_{i}\right\\}_{i=1}^{n},\left\\{F_{j}\right\\}_{j=1}^{m}\right)=\frac{|\\{F:\exists
x_{i},F=\operatorname*{arg\,min}_{F}D(x_{i},F)\\}|}{m}$ (3)

The results in Table 1 show that our topologically motivated approach is
competitive with traditional methods for point cloud generation. In other
words, we do not pay a substantial cost for incorporating topological
similarity into our loss function.

While these metrics give us an insight into the quality of our reconstruction,
their failure to capture structure leads to its poor performance as a measure
of accuracy for the underlying surface the point cloud describes. As can be
seen in Figure 2, our model is able to produce plausible meshes without a
secondary meshing procedure. This is accomplished simply by feeding the
vertices of a sphere mesh into our pretrained network. While omitted here for
brevity, our ablation experiments show that omitting our deformation loss
leads to under-constrained transformations that cause intersecting faces.

Category | D2F ($\times 10^{-2}$) | Coverage ($\times 10^{-2}$)
---|---|---
PF | FN | PCGAN | Ours | PF | FN | PCGAN | Ours
Airplanes | 2.35 | 1.59 | 1.44 | 1.59 | 2.00 | 5.44 | 7.40 | 6.71
Benches | 7.96 | 2.23 | 1.92 | 2.34 | 11.8 | 11.5 | 13.8 | 12.66
Cars | 3.67 | 3.98 | 1.46 | 1.24 | 1.62 | 1.01 | 1.44 | 1.31
Chairs | 8.80 | 7.35 | 7.55 | 7.73 | 17.2 | 13.2 | 14.0 | 14.57
Cups | 7.11 | 1.97 | 2.11 | 2.21 | 19.1 | 12.5 | 15.0 | 14.73
Guitars | 5.02 | 5.76 | 0.995 | 1.29 | 3.80 | 2.29 | 3.43 | 3.56
Lamps | 12.9 | 10.7 | 2.84 | 3.17 | 8.94 | 6.85 | 12.7 | 12.29
Laptops | 8.55 | 4.86 | 1.20 | 1.30 | 9.47 | 8.80 | 12.3 | 11.66
Sofas | 6.31 | 6.15 | 1.87 | 1.66 | 11.6 | 8.11 | 11.1 | 9.95
Tables | 10.9 | 9.88 | 9.86 | 10.1 | 13.1 | 12.0 | 12.3 | 13.5
Table 1: Our method is comparable with other methods for point cloud
reconstruction.

(a) Airplane

(b) Car

(c) Chair

Figure 2: Meshes automatically generated by our approach.

Because our model is limited to deformations between topologically similar
objects, the standard reconstruction metrics show a decrease in performance
when compared to objects trained for reconstruction loss alone. These
shortcomings may be due to the deformation loss incentivizing the deletion of
smaller details.

##  6 Conclusions

Traditionally, generative models for point clouds have been based entirely on
the properties of sets: permutation invariance and conditional independence of
the points given the underlying shape. Although these properties are crucial
for efficiently modeling point cloud distributions, they ignore the
relationship between point cloud and mesh, making mesh generation less
effective. Our preliminary results show that methods that incorporate this
knowledge can be trained solely on point sets and yet produce a generative
process for meshes. Our hope is that this progress motivates further research
into how set models can benefit from external structure, either as
regularization or as a means for improving downstream tasks.

## References

  * Qi et al. [2017a] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 652–660, 2017a.
  * Qi et al. [2017b] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  In _Advances in neural information processing systems_ , pages 5099–5108, 2017b.
  * Zaheer et al. [2017] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov, and Alexander J Smola.  Deep sets.  In _Advances in neural information processing systems_ , pages 3391–3401, 2017.
  * Yang et al. [2018] Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian.  Foldingnet: Point cloud auto-encoder via deep grid deformation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 206–215, 2018.
  * Li et al. [2018] Chun-Liang Li, Manzil Zaheer, Yang Zhang, Barnabas Poczos, and Ruslan Salakhutdinov.  Point cloud gan, 2018.
  * Groueix et al. [2018a] Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry.  Atlasnet: A papier-mâché approach to learning 3d surface generation.  _arXiv preprint arXiv:1802.05384_ , 2018a.
  * Yang et al. [2019] Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan.  Pointflow: 3d point cloud generation with continuous normalizing flows.  _arXiv preprint arXiv:1906.12320_ , 2019.
  * Groueix et al. [2018b] Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry.  3d-coded: 3d correspondences by deep deformation.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 230–246, 2018b.
  * Li et al. [2019] Chun-Liang Li, Tomas Simon, Jason Saragih, Barnabás Póczos, and Yaser Sheikh.  Lbs autoencoder: Self-supervised fitting of articulated meshes to point clouds.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 11967–11976, 2019.
  * Ezuz et al. [2019] Danielle Ezuz, Justin Solomon, and Mirela Ben-Chen.  Reversible harmonic maps between discrete surfaces.  _ACM Transactions on Graphics_ , 38(2):1–12, Mar 2019.  ISSN 0730-0301.  doi: 10.1145/3202660.  URL http://dx.doi.org/10.1145/3202660.
  * Wang et al. [2018] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang.  Pixel2mesh: Generating 3d mesh models from single rgb images.  _Lecture Notes in Computer Science_ , page 55–71, 2018.  ISSN 1611-3349.  doi: 10.1007/978-3-030-01252-6_4.  URL http://dx.doi.org/10.1007/978-3-030-01252-6_4.
  * Kanazawa et al. [2018] Angjoo Kanazawa, Shubham Tulsiani, Alexei A. Efros, and Jitendra Malik.  Learning category-specific mesh reconstruction from image collections.  _Lecture Notes in Computer Science_ , page 386–402, 2018.  ISSN 1611-3349.  doi: 10.1007/978-3-030-01267-0_23.  URL http://dx.doi.org/10.1007/978-3-030-01267-0_23.
  * Rezende and Mohamed [2015] Danilo Jimenez Rezende and Shakir Mohamed.  Variational inference with normalizing flows.  _arXiv preprint arXiv:1505.05770_ , 2015.
  * Grathwohl et al. [2018] Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud.  Ffjord: Free-form continuous dynamics for scalable reversible generative models, 2018.
  * Behrmann et al. [2018] Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Jörn-Henrik Jacobsen.  Invertible residual networks, 2018.
  * Chang et al. [2015] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  ShapeNet: An Information-Rich 3D Model Repository.  Technical Report arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago, 2015.
  * Lucic et al. [2018] Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet.  Are gans created equal? a large-scale study.  In _Advances in neural information processing systems_ , pages 700–709, 2018.