# 3D Volumetric Super-Resolution in Radiology Using 3D RRDB-GAN

##  1 Abstract

This study introduces the 3D Residual-in-Residual Dense Block GAN (3D RRDB-
GAN) for 3D super-resolution for radiology imagery. A key aspect of 3D RRDB-
GAN is the integration of a 2.5D perceptual loss function, which contributes
to improved volumetric image quality and realism. The effectiveness of our
model was evaluated through 4x super-resolution experiments across diverse
datasets, including Mice Brain MRH, OASIS, HCP1200, and MSD-Task-6. These
evaluations, encompassing both quantitative metrics like LPIPS and FID and
qualitative assessments through sample visualizations, demonstrate the model’s
effectiveness in detailed image analysis. The 3D RRDB-GAN offers a significant
contribution to medical imaging, particularly by enriching the depth, clarity,
and volumetric detail of medical images. Its application shows promise in
enhancing the interpretation and analysis of complex medical imagery from a
comprehensive 3D perspective.

Index Terms—  3D super resolution, T1/ T2 MRI, CT, MRH, GAN, perception loss

##  2 Introduction

Super-resolution (SR) is a crucial image processing task aimed at enhancing
the resolution of images, making it an essential step for numerous
applications. In the realm of medical imaging, the significance of SR is
magnified as it directly impacts the diagnostic capabilities, thereby
affecting clinical decisions. Achieving high-resolution (HR) imaging is
especially critical when the target is to delineate fine anatomical structures
and pathologies in three dimensions, facilitating a comprehensive
understanding of the medical conditions under study.

The traditional methods of SR predominantly operate in two dimensions, which
may not capture the full essence of volumetric data present in medical
imaging. This necessitates the development of 3D SR techniques that can
efficiently handle the inherent 3D nature of medical images, preserving and
enhancing the information across the axial, sagittal, and coronal views.

In this work, we propose a 3D Residual-in-Residual Dense Block GAN (3D RRDB-
GAN) aiming to generate realistic high-resolution volumetric medical images in
the scaling factor of 4. Furthermore, we compared our methods to other
3D-based SR methods to test its capabilities over others existing methods. Our
approach diverges from standard SR methods by introducing a 2.5D perceptual
loss function designed to enhance the visual realism in all dimensions.

##  3 Related Methods

###  3.1 Deep Learning Approaches for Super Resolution

Deep learning has transformed the field of super-resolution, offering models
that can learn complex, hierarchical representations from the data.
Convolutional Neural Networks (CNNs), for instance, have demonstrated superior
performance in SR tasks over traditional methods [1]. RRDBNet [2], RCAN [3],
SRResNet [4], and DCSRN [5] are landmark models that harness the power of CNNs
for image upscaling. The introduction of Generative Adversarial Networks
(GANs) further boosted the SR capabilities by enabling the generation of high-
resolution images that are perceptually closer to real images [4].

###  3.2 2D vs. 3D Models in Medical Imagery

While 2D Super-Resolution (SR) models have demonstrated efficacy in domains
such as natural image processing, and have even found utility in enhancing
medical images [6], their application in 3D medical imaging scenarios like CT
and MRI remains suboptimal. Predominantly, 2D SR models have been employed for
image enhancement in the axial view, overlooking the depth dimension intrinsic
to 3D medical imagery and limiting the resolution enhancement only in one
2-dimensional view. This oversight restricts their applicability in 3D
radiology, given the volumetric nature of the data at hand. The advent of 3D
Convolutional Neural Networks (3D CNNs) has marked a pivotal shift by
incorporating the depth dimension, thereby providing a richer contextual
framework and preserving spatial relationships within medical images [7].
Unlike their 2D counterparts, 3D models endeavor to preserve and augment
information across not only the axial but also the sagittal and coronal views,
presenting a more holistic approach to medical image super-resolution.

###  3.3 Perception Loss

Perceptual loss functions [8, 9] have been introduced to enhance the visual
realism of synthetically generated images, including super-resolved images. In
addition to pixel-wise accuracy, this loss function aims to generate
perceptually meaningful results by focusing on maintaining structural and
textural information in the SR images. Perceptual loss is typically deriveed
from pre-trained networks such as VGG models. Due to lack of pre-trained
models in 3D medical imaging, we utilized 2.5D perception loss by aggregation
of 2D perception loss across 3 different directions (axial, sagittal, and
coronal) using pre-trained 2D VGG model.

##  4 Methods

###  4.1 Model Development

For the development of our super-resolution model, we utilized a 3D RRDB-GAN
(Figure LABEL:fig:model_arch). The RRDB network has shown the state-of-the-art
performance in 2D super resolution in natural images [10]. By extending this
network to operate in 3D, we are able to fully leverage the volumetric nature
of medical imaging data. In addition, we introduce 3D UNet discriminator and
2.5D perceptual loss to ensure that the super-resolved images are perceptually
similar to the high-resolution target images in 3D space.

###  4.2 Loss functions

As we utilize GAN framework, our losses (eq.1) includes generator’s loss (
$L_{gen}$) and discriminator’s loss ($L_{disc}$).

$\displaystyle L=$ $\displaystyle L_{gen}+L_{disc}$ (1) $\displaystyle
L_{gen}$
$\displaystyle=\lambda_{1}L_{pixel}+\lambda_{2}L_{perc}+\lambda_{3}L_{adv}$
(2) $\displaystyle L_{perc}$
$\displaystyle=\sum_{view}^{a,c,s}|VGG(I_{sr})-VGG(I_{hr})|$ (3)

$L_{gen}$ (eq.2) contains 3 loss functions such as pixel-loss ($L_{pixel}$),
2.5D perceptual loss ($L_{perc}$), and adversarial loss ($L_{adv}$) with
tunable weights. For our experiments, we used $\lambda_{1}=1$,
$\lambda_{2}=1$, $\lambda_{3}=0.01$). We used mean-absolute-error (MAE) for
$L_{pixel}$. For $L_{disc}$ and $L_{adv}$, we followed traditional binary
classification loss functions, but in voxel-wise level using 3D UNet
discriminator [2] (depicted as discriminator in Figure LABEL:fig:model_arch).
Lastly, $L_{perc}$ (eq.3) is aggregated 2D VGG19 perceptual loss across every
axial, coronal, and sagittal view, thus 2.5D. This 2.5D aggregation loss
allows volumetric image quality enhancement.

###  4.3 k-space Image Degradation

In the domain of super-resolution, the practice of image degradation to create
paired low-resolution and high-resolution images is required. We adopted
k-space image degradation method, aligning with the intrinsic nature of
medical image acquisition via k-space. The low-resolution image through
k-space degradation is attained through a series of steps: (1) applying 3D
Fast Fourier Transform (FFT) to the 3D image, (2) center cropping the
resultant 3D frequency matrix by the upscaling factor, (3) zero-padding the
cropped 3D frequency matrix to match the original image dimensions, (4)
reverting the frequency matrix into image space using 3D Inverse Fast Fourier
Transform (IFFT), and (5) executing nearest-neighbor interpolation to
downsample the image into a low-resolution version.

###  4.4 Evaluation setting

The evaluation of the SR models was done by comparing the generated SR images
against the original HR images. For a comprehensive evaluation, we employed
four key metrics: Structural Similarity Index (SSIM [11]), Peak Signal-to-
Noise Ratio (PSNR), Learned Perceptual Image Patch Similarity (LPIPS [12]),
and Frechet Inception Distance (FID [13]). Traditional evaluation metrics such
as SSIM and PSNR are used to evaluate structural similarities and fidelity
between synthetic and real images. On the other hand, modern evaluation
approaches such as LPIPS and FID captures the perceptual similarities between
synthetic and real images by comparing high-level features. The main
difference between traditional metrics (SSIM/PSNR) and modern approaches
(LPIPS/FID) is that traditional metrics focus on pixel-level accuracy, where
modern approaches focus on high-level feature representation, producing more
human-like assessment of image quality [12]. Furthermore, we adopted 5-fold
cross-validation to ensure the robustness and generalizability of our
evaluation.

Dataset | Method |  SSIM $\uparrow$ |  PSNR $\uparrow$ |  LPIPS $\downarrow$ |  FID $\downarrow$
---|---|---|---|---|---
Mice Brain MRH | Trilinear | 0.79 | 29.38 | 0.32 | 6.69
3D SRResNet [4] | 0.85 | 33.30 | 0.18 | 4.07
3D DCSRN [5] | 0.82 | 32.34 | 0.22 | 4.29
3D RCAN [3] | 0.84 | 32.73 | 0.12 | 1.91
Ours | 0.82 | 32.21 | 0.06 | 0.35
OASIS | Trilinear | 0.79 | 26.55 | 0.25 | 6.89
3D SRResNet | 0.88 | 31.95 | 0.12 | 4.17
3D DCSRN | 0.82 | 29.96 | 0.17 | 5.41
3D RCAN | 0.89 | 32.27 | 0.11 | 3.69
Ours | 0.82 | 29.50 | 0.04 | 0.18
HCP1200 | Trilinear | 0.77 | 26.94 | 0.29 | 4.67
3D SRResNet | 0.92 | 33.74 | 0.10 | 1.85
3D DCSRN | 0.86 | 31.06 | 0.17 | 3.07
3D RCAN | 0.93 | 34.09 | 0.10 | 1.60
Ours | 0.88 | 31.61 | 0.04 | 0.09
MSD Task 6 | Trilinear | 0.80 | 28.82 | 0.27 | 3.94
3D SRResNet | 0.93 | 37.89 | 0.08 | 1.70
3D DCSRN | 0.88 | 33.88 | 0.14 | 2.72
3D RCAN | 0.92 | 37.79 | 0.07 | 1.38
Ours | 0.89 | 36.01 | 0.03 | 0.18
Table 1: 3D super-resolution (4x upscaling) results from 4 datasets with 4
evaluation metrics. Trilinear is a baseline method with simple interpolation.
3D SRRestNet, 3D DCSRN, and 3D RCAN are comparison methods, and our proposed
method (ours) is based on 3D RRDB-GAN. Best performed scores are marked with
bold font.

##  5 Results

###  5.1 Datasets and Implementation

In our study, four diverse datasets were employed: Mice Brain MRH [14, 15,
16], OASIS [17], HCP1200 [18], and MSD-Task-6 [19]. The Mice Brain MRH dataset
comprises 11 high-resolution MRH images of mice brains, each with a voxel size
of $25\mu m^{3}$ and resolution of $512\times 720\times 304$. The OASIS
dataset includes 1,668 human T1-weighted brain MRIs, each with a voxel size of
$1mm^{3}$ and a median resolution of $176\times 256\times 256$. From the
HCP1200 dataset, which contains 1,068 human brain scans in multiple
modalities, we utilized T2-weighted MRIs with a voxel size of $0.7mm^{3}$ and
resolution of $260\times 312\times 260$. The MSD-Task-6 dataset features 60
human abdominal CT scans, with voxel sizes ranging from $0.6$ to $1mm$ for the
axial and coronal axes, and $0.6$ to $2.5mm$ for the sagittal axis. To
standardize the voxel size, images were spatially rescaled to a median voxel
unit of $0.785\times 0.785\times 1.25$, resulting in a median resolution of
$512\times 512\times 242$.

For our experiments, patch-wise training and inference were conducted using
patches of size $96\times 96\times 96$, optimizing computing memory resources.
This approach facilitates our model to perform super-resolution tasks scaling
from $24\times 24\times 24$ to $96\times 96\times 96$. Weighted sampling was
employed during training to consistently include regions of interest in the
samples. Inference was executed using a sliding window approach with the Monai
Python package [20], maintaining the same patch size used in training. Each
experiment utilized a single GPU from cluster nodes (NVIDIA A100 GPUs and AMD
EPYC 7742 CPUs).

###  5.2 Experiment results

In this study, we conducted 4x upscaling task to assess the capabilities of
our super resolution model, the 3D RRDB-GAN. Our evaluation utilized four
distinct datasets, encompassing two species (mice and humans), two anatomical
regions (brain and abdomen), and four imaging modalities (MRH, T1/T2 MRI, and
CT). The outcomes of these evaluations are systematically presented in Table
1, where we compare various super-resolution methods.

When considering traditional metrics such as SSIM and PSNR, both 3D SRResNet
and 3D RCAN demonstrated strong performance. However, our 3D RRDB-GAN model
exhibited notable superiority in modern evaluation metrics, including LPIPS
and FID, across all experimental datasets. Beyond these quantitative results,
we also provide visualizations of sample cases from each dataset in Figure
LABEL:fig:sample_figure. These visualizations offer a more intuitive
comparison from a human perception standpoint, particularly evident in the
zoomed-in regions where detailed imaging characteristics are highlighted.

##  6 Conclusion

Our study demonstrates the efficacy of the 3D Residual-in-Residual Dense Block
GAN (3D RRDB-GAN) in enhancing the quality of 3D medical images through super-
resolution techniques. Our approach, distinguished by the integration of a
2.5D perceptual loss function, has shown its capability in significantly
improving the volumetric detail and realism of images across various datasets.
While our approach showed more pronounced results in visualizations and modern
evaluation metrics like LPIPS and FID, indicating a strong perceptual quality,
it encountered challenges in traditional metrics such as SSIM and PSNR. These
outcomes highlight the specialized capability of the 3D RRDB-GAN in enhancing
perceptual aspects of medical imaging, suggesting its potential utility in
scenarios where detailed visual interpretation is crucial.

The breadth of datasets used in our study, covering different species, body
regions, and modalities, underscores the versatility and adaptability of the
3D RRDB-GAN. While our model represents a significant step forward in medical
imaging technology, future work will aim to explore its potential in broader
clinical applications and its integration into diagnostic workflows. By
continuously refining and adapting our model, we aspire to contribute further
to the advancements in medical imaging, striving towards more accurate and
efficient analysis and interpretation of complex medical data.

##  7 Acknowledgments

This work was supported in part by Indiana University Internal Research Seed
Fund, and in part by the NIH R01 NS125020, Indiana Center for Diabetes and
Metabolic Diseases Pilot and Feasibility Grant, Roberts Drug Discovery Fund &
TREAT-AD Center Grant.

## References

  * [1] J. Kim, J. Lee, and K. Lee,  “Deeply-recursive convolutional network for image super-resolution,”  in IEEE Conference on Computer Vision and Pattern Recognition, 2016.
  * [2] X. Wang, K. Yu, S. Wu, et al.,  “Esrgan: Enhanced super-resolution generative adversarial networks,”  in The European Conference on Computer Vision Workshops (ECCVW), 2018.
  * [3] B. Lee, S. Son, H. Kim, et al.,  “Enhanced deep residual networks for single image super-resolution,”  in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2017.
  * [4] C. Ledig, L. Theis, F. Huszar, et al.,  “Photo-realistic single image super-resolution using a generative adversarial network,”  in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [5] Y. Chen, Y. Xie, et al.,  “Brain mri super resolution using 3d deep densely connected neural networks,”  in 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), 2018.
  * [6] H. Yang, Z. Wang, X. Liu, et al.,  “Deep learning in medical image super resolution: a review,”  Applied Intelligence, 2023.
  * [7] ”Ö. Çiçek, A. Abdulkadir, S. Lienkamp, and others”,  “3d u-net: Learning dense volumetric segmentation from sparse annotation,”  in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2016, 2016.
  * [8] J. Johnson, A. Alahi, and L. Fei-Fei,  “Perceptual losses for real-time style transfer and super-resolution,”  in Computer Vision – ECCV 2016, 2016.
  * [9] Q. Wu, C. Fan, Y. Li, et al.,  “A novel perceptual loss function for single image super-resolution,”  Multimedia Tools and Applications, 2020.
  * [10] X. Wang, L. Xie, C. Dong, et al.,  “Real-esrgan: Training real-world blind super-resolution with pure synthetic data,”  in International Conference on Computer Vision Workshops (ICCVW), 2021.
  * [11] Z. Wang, B. Alan, H. Sheikh, et al.,  “Image quality assessment: from error visibility to structural similarity,”  IEEE Transactions on Image Processing, 2004.
  * [12] R. Zhang, P. Isola, A. Efros, et al.,  “The unreasonable effectiveness of deep features as a perceptual metric,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [13] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter,  “Gans trained by a two time-scale update rule converge to a local nash equilibrium,”  in Proceedings of the 5th International Conference on Learning Representations (ICLR), 2017.
  * [14] N. Wang, C. Gary, R. Anderson, et al.,  “Accelerating quantitative susceptibility imaging acquisition using compressed sensing,”  Physics in Medicine Biology, 2018.
  * [15] N. Wang, J. Zhuang, H. Wei, et al.,  “Probing demyelination and remyelination of the cuprizone mouse model using multimodality mri,”  Journal of Magnetic Resonance Imaging, 2019.
  * [16] S. Crater, S. Maharjan, Y. Qi, et al.,  “Resolution and b value dependent structural connectome in ex vivo mouse brain,”  NeuroImage, 2022.
  * [17] D. Marcus, T. Wang, J. Parker, et al.,  “Open Access Series of Imaging Studies (OASIS): Cross-sectional MRI Data in Young, Middle Aged, Nondemented, and Demented Older Adults,”  Journal of Cognitive Neuroscience, 2007.
  * [18] M. Glasser, S. Sotiropoulos, J. Wilson, et al.,  “The minimal preprocessing pipelines for the human connectome project,”  NeuroImage, 2013.
  * [19] A. Simpson, M. Antonelli, S. Bakas, et al.,  “A large annotated medical image dataset for the development and evaluation of segmentation algorithms,”  CoRR, 2019.
  * [20] M. Cardoso, W. Li, R. Brown, et al.,  “Monai: An open-source framework for deep learning in healthcare,”  arXiv preprint arXiv:2211.02701, 2022.