11institutetext: Department of Electrical and Electronic Engineering, Southern
University of Science and Technology, Shenzhen, China
11email<EMAIL_ADDRESS>22institutetext: Department of Electrical and
Electronic Engineering, The University of Hong Kong, Hong Kong SAR, China

# DS3-Net: Difficulty-perceived Common-to-T1ce Semi-Supervised Multimodal MRI
Synthesis Network

Ziqi Huang 11 Li Lin 1122 Pujin Cheng 11 Kai Pan 11 Xiaoying
Tang${}^{1(\textrm{{\char 0\relax})}}$

###### Abstract

Contrast-enhanced T1 (T1ce) is one of the most essential magnetic resonance
imaging (MRI) modalities for diagnosing and analyzing brain tumors, especially
gliomas. In clinical practice, common MRI modalities such as T1, T2, and fluid
attenuation inversion recovery are relatively easy to access while T1ce is
more challenging considering the additional cost and potential risk of
allergies to the contrast agent. Therefore, it is of great clinical necessity
to develop a method to synthesize T1ce from other common modalities. Current
paired image translation methods typically have the issue of requiring a large
amount of paired data and do not focus on specific regions of interest, e.g.,
the tumor region, in the synthesization process. To address these issues, we
propose a Difficulty-perceived common-to-T1ce Semi-Supervised multimodal MRI
Synthesis network (DS3-Net), involving both paired and unpaired data together
with dual-level knowledge distillation. DS3-Net predicts a difficulty map to
progressively promote the synthesis task. Specifically, a pixelwise constraint
and a patchwise contrastive constraint are guided by the predicted difficulty
map. Through extensive experiments on the publicly-available BraTS2020
dataset, DS3-Net outperforms its supervised counterpart in each respect.
Furthermore, with only 5% paired data, the proposed DS3-Net achieves
competitive performance with state-of-the-art image translation methods
utilizing 100% paired data, delivering an average SSIM of 0.8947 and an
average PSNR of 23.60. The source code is available at
https://github.com/Huangziqi777/DS-3_Net.

###### Keywords:

Multimodal MRI synthesis Difficulty-perceived guidance Semi-supervised
learning Contrastive learning

##  1 Introduction

Magnetic resonance imaging (MRI) is a non-radioactive clinical imaging method
that is widely used in gliomas diagnosis [14, 10]. In clinical practice, there
are some relatively easy-to-obtain MRI modalities such as T1, T2 and fluid
attenuation inversion recovery (FLAIR), since they can be simultaneously
acquired in a single routine MRI scanning process by adjusting specific
parameters. Contrarily, there are some difficult-to-obtain MRI modalities such
as contrast-enhanced T1 (T1ce), since they require external intervention
(e.g., injection of a contrast agent) exerted to the patient of interest.
Compared with common MRI modalities, T1ce can clearly distinguish the
parenchymal part of brain tumor, which is of great guiding significance for
the diagnosis of tumor-related diseases. However, the contrast agent is
usually radioactive and some patients are allergic to it, and there is also an
increase in the scanning cost. As such, paired MRI data of common and T1ce
modalities are very scarce but indispensable in clinical practice, especially
in tumor diagnosis. In such context, there has been great interest in research
topics related to missing modality [22, 28, 29, 25].

Deep learning (DL) has been widely employed in image synthesis tasks. For
natural images, Pix2pix [11] achieves good performance on paired image
translation, while CycleGAN [31] and CUTGAN [19] perform well on unpaired
image translation. For medical images, pGAN [4] incorporates a content-
preserving perceptual loss into Pix2pix and applies it to translation across
2D brain MRI. MedGAN [1] takes into account image style transformation while
maintaining content consistency, which performs well on the PET-versus-CT
conversion task. MM-GAN [21] randomly drops out several input modalities and
makes use of selective discriminators for the loss calculation, synthesizing
high-quality images under a missing-modality setting. However, these existing
DL methods rely heavily on a large number of annotated or paired images in the
training phase for supervision. Recently, semi-supervised learning which can
fully exploit unlabeled data gains increasing interest, especially in the
medical image analysis realm which severely suffers from scarce data [9, 30,
3].

Synthesizing MRI containing brain tumor is a very challenging task.
Conventional image synthesis frameworks [13, 26] for synthesizing glioma-
contained MRI, especially for synthesizing T1ce and FLAIR, are usually
unsatisfactory, mainly because of the following reasons: 1) Huge domain gaps
exist across different glioma-contained MRI modalities, manifesting in the
intensity distribution. 2) A large amount of paired data are required for well
training a pixel-level image reconstruction network, which contradicts the
real-life clinical situation of scarce paired data. 3) The image pattern of
the glioma region varies significantly from those of other brain structures,
which makes it difficult to be synthesized even if other brain structures can
be well reconstructed.

In this work, we propose a novel framework for synthesizing the difficult-to-
obtain MRI modality T1ce, employing semi-supervised learning with limited
paired data and a larger number of unpaired data. Our main contributions are
four-fold: 1) To the best of our knowledge, this is the first semi-supervised
framework applied to multimodal MRI synthesis for gliomas. 2) In light of the
teacher-student network, we make full use of unpaired multimodal MRI data
through maintaining consistency in spaces of both high and low dimensions. 3)
We innovatively estimate a difficulty-perceived map and adopt it to
dynamically weigh both pixelwise and patchwise constraints according to the
difficulty of model learning, and thus the model can assign the difficult-to-
learn parts (such as the glioma region) more attention. 4) Extensive
comparison experiments are conducted, both quantitatively and qualitatively.

Figure 1: Flowchart of our proposed DS3-Net. For simplicity, the GAN losses
for the teacher network and the student network, and the patchwise difficulty-
perceived contrastive loss which is similar between the teacher network and
the student network, are not included in this figure.

##  2 Methodology

Let
$D_{p}=\left\\{\left(x_{i}^{\mathcal{T}},y_{i}^{\mathcal{T}}\right)\right\\}_{i=1}^{N}$
denote the paired image dataset including the common MRI modalities
$x_{i}^{\mathcal{T}}=\left\\{\left({}_{1}x_{i}^{\mathcal{T}},_{2}x_{i}^{\mathcal{T}},..._{n}x_{i}^{\mathcal{T}}\right)\right\\}_{n=1}^{C}$,
where $C$ is set as three including T1, T2 and FLAIR and $y_{i}$ is the target
modality which is T1ce in our case. Let
$D_{u}=\left\\{\left(x_{i}^{\mathcal{S}}\right)\right\\}_{i=1}^{M}$ denote the
unpaired image dataset, with no corresponding T1ce. Note that the sample size
of the unpaired datset $M$ is much larger than that of the paired datset $N$,
with no patient-level overlap.

Our proposed Difficulty-perceived common-to-T1ce Semi-Supervised multimodal
MRI Synthesis network (DS3-Net) is demonstrated in Fig. 1, including a teacher
model for learning expertise knowledge from paired data and a student model
for imitating the teacher’s decision. The student model has the same
architecture as the teacher model and the two models are encouraged to output
consistent predictions.

###  2.1 Difficulty-perceived Attention Map

One of the main challenges in conducting image synthesis to brain MRI
containing glioma is that different regions have different synthesizing
difficulties. For instance, normal brain tissues have relatively more
straightforward intensity mapping relationships, while brain tumor regions
often suffer from diverse patterns between common MRI and T1ce. Introducing an
attention map to assess the synthesizing difficulty for each region is a
feasible solution [16]. Therefore, we intuitively utilize a predicted map
$\mathcal{M}$ from the discriminator to dynamically guide the synthesis
process, as the well-generated regions can easily fool the discriminator while
the poorly-generated regions can not. In our case, the discriminator is a
70$\times$70 PatchGAN [12], leading to patch-level difficulty scores.
Specifically, the difficulty map $\mathcal{M}^{\mathcal{T}}$ from the teacher
model is defined as

$\mathcal{M}^{\mathcal{T}}=\left\\{\begin{array}[]{lr}\left\|1-\mathcal{D}_{\mathcal{T}}\left(\mathcal{G}_{\mathcal{T}}\left(x_{i}^{\mathcal{T}}\right)\right)\right\|,&x_{i}^{\mathcal{T}}>0\\\
0.2,&x_{i}^{\mathcal{T}}=0.\end{array}\right.$ (1)

$\mathcal{D}_{\mathcal{T}}$ and $\mathcal{G}_{\mathcal{T}}$ respectively refer
to the teacher network’s discriminator and generator. For background pixels
without any brain tissue, we set a small constant 0.2 to ensure that their
synthesis results are normal.

###  2.2 Difficulty-perceived Pixelwise and Patchwise Constraints

To fully utilize the multimodal information, we use three encoders each of
which has nine ResNet blocks. We also employ fusion block modified from the SE
module [8] by replacing sigmoid with softmax to better fuse the three common
modalities before being inputted to the decoder.

L1 loss is adopted to provide a pixelwise constraint between the real T1ce
$y_{i}^{\mathcal{T}}$ and the synthesized T1ce
$\mathcal{G}_{\mathcal{T}}(x_{i}^{\mathcal{T}})$. We use the difficulty map
$\mathcal{M}^{\mathcal{T}}$ to guide the L1 loss by assigning each pixel a
difficulty score, allowing the poorly synthesized regions to gain more
attention. Specifically, we introduce a pixelwise difficulty-perceived L1 loss
$\mathcal{L}_{pid}^{\mathcal{T}}$ for the teacher network

$\mathcal{L}_{pid}^{\mathcal{T}}=\mathcal{M}^{\mathcal{T}}\circ\left\|y_{i}^{\mathcal{T}}-\mathcal{G}_{\mathcal{T}}(x_{i}^{\mathcal{T}})\right\|_{1}.$
(2)

For the purpose of maintaining the same semantic information at the same
location in a single sample, we use patchwise contrastive learning to
constrain the parameters of different hidden layers. Specifically, we
construct a positive pair $\left(z_{l}^{s},z_{l}^{s+}\right)$ and a negative
pair $\left(z_{l}^{s},z_{l}^{s-}\right)$, where
$z_{l}^{s}=\mathcal{H}_{(l)}(\mathcal{G}_{\mathcal{T}(l)}(x_{i}^{s}))$ refers
to features extracted from patch $s$ of the input $x_{i}$ at the $l^{th}$
layer of the teacher network’s generator $\mathcal{G}_{\mathcal{T}}$. Next, we
map the selected features into a high-dimension space by an additional MLP
$\mathcal{H}_{(l)}$. Similarly,
$z_{l}^{s+}=\mathcal{H}_{(l)}(\mathcal{G}_{\mathcal{T}(l)}(\hat{y}_{i}^{s}))$
refers to features from the synthesized T1ce $\hat{y}_{i}$ at patch $s$,
regarded as the positive sample, and $z_{l}^{s-}$ represents features from
patches different from $s$. We then pull the positive pair together and push
the negative pair away via the InfoNCE loss [18]. The difficulty map
$\mathcal{M}^{\mathcal{T}}$ is down-sampled to match the resolution of each
hidden layer $l^{th}$, defined as $m_{l}^{\mathcal{T}}$, and is multiplied
with each patch’s features to re-distribute the patchwise importance. To be
specific, the patchwise difficulty-perceived contrastive loss
$\mathcal{L}_{pad}^{\mathcal{T}}$ is calculated as

$\mathcal{L}_{pad}^{\mathcal{T}}=-\sum_{l\in
L}m_{l}^{\mathcal{T}}\circ\log\left[\frac{\exp\left(z_{l}^{s}\cdot
z_{l}^{s+}/\tau\right)}{\left.\exp\left(z_{l}^{s}\cdot
z_{l}^{s+}/\tau\right)+\sum\exp\left(z_{l}^{s}\cdot
z_{l}^{s-}/\tau\right)\right)}\right],$ (3)

where $L=\left\\{0,4,8,12,16\right\\}$ are the selected hidden layers and the
temperature $\tau$ for the InfoNCE loss is empirically set as 0.07.

Furthermore, an LSGAN loss [17] is adopted, encouraging the synthesized
distribution to be close to the real one. In summary, the total loss function
$\mathcal{L}^{\mathcal{T}}$ of the teacher network is

$\mathcal{L}^{\mathcal{T}}=\lambda_{pid}\mathcal{L}_{pid}^{\mathcal{T}}+\lambda_{pad}\mathcal{L}_{pad}^{\mathcal{T}}+\lambda_{GAN}\mathcal{L}_{GAN}^{\mathcal{T}},$
(4)

where $\lambda_{pid}$, $\lambda_{pad}$ and $\lambda_{GAN}$ are empirically set
as 100, 1, 1.

###  2.3 Difficulty-perceived Dual-level Distillation

We feed the unpaired dataset $D_{u}$ into the student network
$\mathcal{N}_{\mathcal{S}}$ which is initialized with pre-trained weights from
the teacher network. For semi-supervised learning, knowledge distillation [5,
30] between the teacher and student models is one of the most common
procedures to expand the dataset and perform parameter regularization. In our
DS3-Net pipeline, both feature-level and image-level distillations are
performed to ensure the consistency of the two models in both high-dimension
and low-dimension spaces.

To be specific, we first input $x_{i}^{\mathcal{S}}\in D_{u}$ into both
$\mathcal{N}_{\mathcal{S}}$ and $\mathcal{N}_{\mathcal{T}}$ to obtain a coarse
prediction $\mathcal{G}_{\mathcal{S}}(x_{i}^{\mathcal{S}})$ and a fine
prediction $\mathcal{G}_{\mathcal{T}}(x_{i}^{\mathcal{S}})$. Given that the
paired ground truth for $x_{i}^{s}$ is not available, we treat the fine
decision $\hat{q}_{i}^{\mathcal{T}}$ as the pseudo ground truth for the
student network, and then conduct image-level distillation through an L1 loss
function.

We expect the student model to imitate the teacher model in terms of
representation at the high-dimension space to better handle the unpaired
samples. Therefore, when one sample goes through the two networks, we extract
the $k^{th}$ layer’s features $f_{k}^{\mathcal{T}}$ and $f_{k}^{\mathcal{S}}$,
and then perform feature-level distillation between the two networks by
minimizing the feature discrepancy.

Apparently, the student discriminator $\mathcal{D}_{\mathcal{S}}$ can also
produce a difficulty map $\mathcal{M}^{\mathcal{S}}$ for each unpaired sample
and we employ the same difficulty guided strategy towards the distillation
process. We thus have the image-level distillation loss

$\mathcal{L}_{id}^{\mathcal{S}}=\mathcal{M}^{\mathcal{S}}\circ\left\|\mathcal{G}_{\mathcal{T}}(x_{i}^{\mathcal{S}})-\mathcal{G}_{\mathcal{S}}(x_{i}^{\mathcal{S}})\right\|_{1},$
(5)

and the feature-level distillation loss

$\mathcal{L}_{fd}^{\mathcal{S}}=\frac{1}{K}\sum_{k\in
L_{distill}}m_{k}^{\mathcal{S}}\circ\left\|f_{k}^{\mathcal{T}}-f_{k}^{\mathcal{S}}\right\|_{1},$
(6)

where $L_{distill}=\left\\{4,8,12,16,21\right\\}$, $K$ denotes the number of
the selected layers, and $m_{k}^{\mathcal{S}}$ represents the difficulty map
down-sampled to the same size as that of the feature map of the $k^{th}$
layer.

Similar to the teacher model, the student model also has an LSGAN loss
$\mathcal{L}_{GAN}^{\mathcal{S}}$ and a patchwise difficulty-perceived
contrastive loss $\mathcal{L}_{pad}^{\mathcal{S}}$. Collectively, the loss
function $\mathcal{L}^{\mathcal{S}}$ for the student network is defined as

$\mathcal{L}^{\mathcal{S}}=\lambda_{id}\mathcal{L}_{id}^{\mathcal{S}}+\lambda_{fd}\mathcal{L}_{fd}^{\mathcal{S}}+\lambda_{pad}\mathcal{L}_{pad}^{\mathcal{S}}+\lambda_{GAN}\mathcal{L}_{GAN}^{\mathcal{S}}.$
(7)

Here we empirically set $\lambda_{id}$, $\lambda_{fd}$, $\lambda_{pad}$ and
$\lambda_{GAN}$ as 100, 1, 1 and 1.

##  3 Experiments and Results

###  3.1 Dataset

BraTS2020 [2] (https://www.med.upenn.edu/cbica/brats2020/) is a large
publicly-available dataset, consisting in total 369 paired MRI data of T1, T2,
FLAIR, and T1ce from gliomas patients. We divide each 3D image into 2D slices
at the axial view. Each 2D slice is center-cropped to 224 $\times$ 224\. We
divide the entire dataset into training, validation, and test sets according
to a ratio of 7:1:2 at the patient level, and use slices that contain tumor
during the training phase. Since the tumor core (TC) containing the enhancing
tumor (ET) is the key region to distinguish T1ce from other modalities while
it usually constitutes only a small portion of the whole tumor, we adjust the
proportion to be 1:1 for the numbers of slices with and without TC.

###  3.2 Training Strategy

At the first training stage, the paired dataset are utilized to pre-activate
the teacher network, and we take the weights of the 5th epoch before
convergence as the initialization weights for the next stage of training to
prevent overfitting. Specifically, each difficulty map score is set to be the
constant 1 in the first stage which ensures a stable output from the initial
network. In the second stage, we start with the difficulty maps delivered by
the discriminators and train the two networks together by optimizing the final
loss function
$\mathcal{L}=\mathcal{L}^{\mathcal{T}}+(1-\frac{t}{T})\mathcal{L}^{\mathcal{S}}$,
where $t\in[0,T)$ represents the current epoch counting and $T$ is the total
number of the training epochs. All methods are implemented with Pytorch using
NVIDIA RTX 3090 GPUs. AdamW [15] is chosen to be the optimizer and the batch
size is set as 6 with equivalent numbers of paired and unpaired samples. The
learning rates in the first stage for the generator
$\mathcal{G}^{\mathcal{T}}$, the MLP
$\left\\{\mathcal{H}_{(l)}|l\in\mathcal{L}\right\\}$ and the discriminator
$\mathcal{D}^{\mathcal{T}}$ are respectively set as 0.0006, 0.0006 and 0.0003,
and these values are attenuated to a fifth of the original numbers in the
second training phase. For the student network training, we have the same
learning rate setting as that for the initial teacher network and train for a
total of 100 epochs. All of the learning rate settings follow the linear decay
strategy.

Table 1: Quantitive evaluations for DS3-Net with different percentages of paired data. Bold metric represents the best performance. Paired percentage | Method | Image quality | Segmentation quality
---|---|---|---
SSIM$\uparrow$ | PSNR$\uparrow$ | MSE$\downarrow$ | ET Dice$\uparrow$ | TC Dice$\uparrow$
0% | CUTGAN [19] | 0.7982 | 17.63 | 0.020 | 33.9% | 56.1%
5% | DS3-Net (paired only) | 0.8691 | 21.97 | 0.008 | 39.5% | 63.7%
DS3-Net | 0.8947 | 23.60 | 0.009 | 43.3% | 66.0%
10% | DS3-Net (paired only) | 0.8763 | 23.01 | 0.007 | 43.6% | 65.3%
DS3-Net | 0.8943 | 23.90 | 0.008 | 43.6% | 66.9%
50% | DS3-Net (paired only) | 0.8895 | 23.73 | 0.006 | 44.1% | 67.9%
DS3-Net | 0.8893 | 23.79 | 0.005 | 44.9% | 68.1%
100% | pGAN [4] | 0.8846 | 23.56 | 0.010 | 36.7% | 61.8%
MedGAN [1] | 0.8900 | 24.34 | 0.011 | 37.8% | 58.3%
Pix2pix [11] | 0.9007 | 25.02 | 0.004 | 41.7% | 65.0%
DS3-Net | 0.8959 | 24.26 | 0.005 | 47.8% | 67.2%

###  3.3 Results

In the training phase, we consider a few percentages (e.g., 5%, 10% and 50%)
of the full training set as our paired dataset and the rest as the unpaired
dataset. To better evaluate our method, we compare DS3-Net with several state-
of-the-art (SOTA) methods including two for natural images (Pix2pix [11] and
CUTGAN [19]) and two for medical images (pGAN [4] and MedGAN [1]).

The structure similarity index measure (SSIM), peak signal-to-noise ratio
(PSNR) [6] and mean squared error (MSE) are employed to evaluate the
effectiveness of DS3-Net. Besides, because T1ce plays a crucial role in
identifying ET and TC [7, 27, 24], we perform tumor segmentation using the
synthesized T1ce together with the three common modalities and use the Dice
scores [23] of ET and TC to assess the quality of the synthesized T1ce. The
entire test set is divided into five folds, four of which are used for
training the segmentation network and one for testing. UNet [20] is employed
as our segmentation network. During the testing phase, we use the student
network to deliver the prediction. All comparison results are listed in Table
1.

Table 2: Ablation studies for each component of DS3-Net.  Component | SSIM$\uparrow$ | PSNR$\uparrow$ | MSE$\downarrow$
---|---|---|---
DS3-Net w.o. $\mathcal{M}_{\mathcal{T}}/\mathcal{M}_{\mathcal{S}}$ | 0.8884 | 21.38 | 0.012
DS3-Net w.o. $\mathcal{L}_{fd}$ | 0.8883 | 21.33 | 0.012
DS3-Net w.o. $\mathcal{L}_{id}$ | 0.7424 | 13.43 | 0.063
DS3-Net teacher network $\mathcal{N}_{T}$ | 0.8715 | 20.67 | 0.013
DS3-Net student network $\mathcal{N}_{S}$ | 0.8947 | 23.60 | 0.009

We observe that the semi-supervised training strategy can effectively improve
the synthesized T1ce’s quality and the segmentation performance of ET and TC
compared to the vanilla supervised method (namely DS3-Net (paired only)), as
demonstrated in Table 1. Apparently, DS3-Net achieves competitive performance
with full-data training methods, even with only 5% paired data for training,
especially in segmenting ET and TC. To qualitatively illustrate the
performance, we display visualizations of the synthesized T1ce from different
methods in Fig. 2. It can be easily observed that DS3-Net performs well in
synthesizing tumor regions and achieves competitive performance with other
SOTA methods trained with 100% paired data. Error maps are also presented in
Fig. 2, revealing that the erroneous regions mainly focus around the tumor,
but our proposed DS3-Net can effectively alleviate this dilemma to some extent
(also see Fig. A1 of the appendix). Additionally, by zooming the tumor regions
for visualization, DS3-Net is found to deliver T1ce with a clearer boundary
for ET compared with other SOTA methods.

Figure 2: Visualization of synthesized T1ce and the corresponding error maps
from different methods. Each percentage represents DS3-Net trained with
different paired percentage.

Ablation studies are conducted with 5% paired data training, to assess each
component’s importance in the proposed DS3-Net. As shown in Table 2, removing
any of the proposed components deteriorates the synthesization performance in
terms of all evaluation metrics. In addition, it can be found that without
$\mathcal{L}_{id}$, DS3-Net suffers dramatic degradation, revealing that
image-level distillation is an indispensable component in our framework. Since
the student network inputs more data and receives reliable guidance from the
teacher network, the image quality is effectively improved compared with that
from the teacher network.

##  4 Conclusion

In this work, we propose a novel difficulty-perceived semi-supervised
multimodal MRI synthesis pipeline to generate the difficult-to-obtain modality
T1ce from three common easy-to-obtain MRI modalities including T1, T2, and
FLAIR. Difficulty-perceived maps are adopted to guide the synthesization
process of important regions, and dual-level distillation enables the model to
train a well-performimg network with limited paired data. Extensive
experiments are conducted on a large publicly-available dataset, and the
effectiveness of DS3-Net is successfully identified.

## References

  * [1] Armanious, K., Jiang, C., Fischer, M., Küstner, T., Hepp, T., Nikolaou, K., Gatidis, S., Yang, B.: Medgan: Medical image translation using gans. Computerized Medical Imaging and Graphics 79, 101684 (2020)
  * [2] Bakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., Kirby, J.S., Freymann, J.B., Farahani, K., Davatzikos, C.: Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features. Scientific Data 4(1), 1–13 (2017)
  * [3] Chen, X., Yuan, Y., Zeng, G., Wang, J.: Semi-supervised semantic segmentation with cross pseudo supervision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2613–2622. (2021)
  * [4] Dar, S.U., Yurt, M., Karacan, L., Erdem, A., Erdem, E., Çukur, T.: Image synthesis in multi-contrast mri with conditional generative adversarial networks. IEEE Transactions on Medical Imaging 38(10), 2375–2388 (2019)
  * [5] Hinton, G., and Vinyals. O., and Dean. J.: Distilling the Knowledge in a Neural Network. arXiv e-prints (2015). arXiv:1503.02531
  * [6] Hore, A., Ziou, D.: Image quality metrics: Psnr vs. ssim. In: 2010 20th International Conference on Pattern Recognition. pp. 2366–2369 (2010)
  * [7] Huang, Z., Lin, L., Cheng, P., Peng, L., Tang, X.: Multi-modal brain tumor segmentation via missing modality synthesis and modality-level attention fusion. In: 2022 26th International Conference on Pattern Recognition. (under review) (2022)
  * [8] Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition. pp. 7132–7141 (2018)
  * [9] Ibrahim, M.S., Vahdat, A., Ranjbar, M., Macready, W.G.: Semi-supervised semantic image segmentation with self-correcting networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12715–12725 (2020)
  * [10] Işın, A., Direkoğlu, C., Şah, M.: Review of mri-based brain tumor image segmentation using deep learning methods. Procedia Computer Science 102, 317–324 (2016)
  * [11] Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1125–1134 (2017)
  * [12] Li, C., Wand, M.: Precomputed real-time texture synthesis with markovian generative adversarial networks. In: European Conference on Computer Vision. pp. 702–716. Springer (2016). 10.1007/978-3-319-46487-9_43
  * [13] Li, H., Paetzold, J.C., Sekuboyina, A., Kofler, F., Zhang, J., Kirschke, J.S., Wiestler, B., Menze, B.: Diamondgan: unified multi-modal generative adversarial networks for mri sequences synthesis. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 795–803. Springer (2019). 10.1007/978-3-030-32251-9_87
  * [14] Liu, S., Cai, W., Liu, S., Zhang, F., Fulham, M., Feng, D., Pujol, S., Kikinis, R.: Multimodal neuroimaging computing: a review of the applications in neuropsychiatric disorders. Brain Informatics 2(3), 167–180 (2015)
  * [15] Loshchilov, I., Hutter, F.: Fixing Weight Decay Regularization in Adam. arXiv e-prints (2017). arxiv:1711.05101
  * [16] Ma, B., Zhao, Y., Yang, Y., Zhang, X., Dong, X., Zeng, D., Ma, S., Li, S.: Mri image synthesis with dual discriminator adversarial learning and difficulty-aware attention mechanism for hippocampal subfields segmentation. Computerized Medical Imaging and Graphics 86, 101800 (2020)
  * [17] Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z. and Paul Smolley, S.: Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision. pp. 2794–2802. (2017)
  * [18] Van den Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv e-prints (2018). arXiv:1807.03748
  * [19] Park, T., Efros, A.A., Zhang, R., Zhu, J.Y.: Contrastive learning for unpaired image-to-image translation. In: European Conference on Computer Vision. pp. 319–345. Springer (2020). 10.1007/978-3-030-58545-7_19
  * [20] Ronneberger O., Fischer P., Brox T.: U-Net: Convolutional Networks for Biomedical Image Segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 234–241. Springer (2015). 10.1007/978-3-319-24574-4_28
  * [21] Sharma, A., Hamarneh, G.: Missing mri pulse sequence synthesis using multi-modal generative adversarial network. IEEE Transactions on Medical Imaging 39(4), 1170–1183 (2019)
  * [22] Shen, Y., Gao, M.: Brain tumor segmentation on mri with missing modalities. In: International Conference on Information Processing in Medical Imaging. pp. 417–428. Springer (2019). 10.1007/978-3-030-20351-1_32
  * [23] Taha, A.A., Hanbury, A.: Metrics for evaluating 3d medical image segmentation: analysis, selection, and tool. BMC Medical Imaging 15(1), 1–28 (2015)
  * [24] Wu, J., Tang, X.: Brain segmentation based on multi-atlas and diffeomorphism guided 3d fully convolutional network ensembles. Pattern Recognition 115, 107904 (2021)
  * [25] Yang, J., Li, X.X., Liu, F., Nie, D., Lio, P., Qi, H., Shen, D.: Fast t2w/flair mri acquisition by optimal sampling of information complementary to pre-acquired t1w mri, arXiv e-prints (2021). arXiv:2111.06400
  * [26] Yurt, M., Dar, S.U., Erdem, A., Erdem, E., Oguz, K.K., Çukur, T.: mustgan: Multi-stream generative adversarial networks for mr image synthesis. Medical Image Analysis 70, 101944 (2021)
  * [27] Zhang, Y., Wu, J., Liu, Y., Chen, Y., Wu, E.X., Tang, X.: Mi-unet: Multi-inputs unet incorporating brain parcellation for stroke lesion segmentation from t1-weighted magnetic resonance images. IEEE Journal of Biomedical and Health Informatics 25(2), 526–535 (2021)
  * [28] Zhou, T., Canu, S., Vera, P., Ruan, S.: Brain tumor segmentation with missing modalities via latent multi-source correlation representation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 533–541. Springer (2020). /10.1007/978-3-030-59719-1_52
  * [29] Zhou, T., Canu, S., Vera, P., Ruan, S.: Latent correlation representation learning for brain tumor segmentation with missing mri modalities. IEEE Transactions on Image Processing 30, 4263–4274 (2021)
  * [30] Zhou, Y., Chen, H., Lin, H., Heng, P.A.: Deep semi-supervised knowledge distillation for overlapping cervical cell instance segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 521–531. Springer (2020). /10.1007/978-3-030-59710-8_51
  * [31] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2223–2232 (2017)