# Convolutional Neural Network-Based Image Watermarking using Discrete Wavelet
Transform

Alireza Tavakoli  Zahra Honjani  Hedieh Sajedi

###### Abstract

With the growing popularity of the Internet, digital images are used and
transferred more frequently. Although this phenomenon facilitates easy access
to information, it also creates security concerns and violates intellectual
property rights by allowing illegal use, copying, and digital content theft.
Using watermarks in digital images is one of the most common ways to maintain
security. Watermarking is proving and declaring ownership of an image by
adding a digital watermark to the original image. Watermarks can be either
text or an image placed overtly or covertly in an image and are expected to be
challenging to remove. This paper proposes a combination of convolutional
neural networks (CNNs) and wavelet transforms to obtain a watermarking network
for embedding and extracting watermarks. The network is independent of the
host image resolution, can accept all kinds of watermarks, and has only 11
layers while keeping performance. Performance is measured by two terms; the
similarity between the extracted watermark and the original one and the
similarity between the host image and the watermarked one.

Keywords— watermarking, convolutional neural networks, wavelet transform,
neural networks

##  1 Introduction

Protecting art ownership was always an issue since painters used to sign their
works, which was not safe. Due to widespread Internet use, artistic works are
becoming digital, so the threat of privacy, copyright, and illegal usage has
increased. Authentication is not limited to artistic works but includes
identity document (ID) cards and source tracking. In order to solve these
issues, digital watermarking is an effective method of hiding information that
can be used for authentication purposes.

The digital watermarking technique involves embedding information (watermark)
into the content and extracting it when needed. Various attacks could be
applied to the watermarked content to damage or remove the embedded watermark,
which shows the importance of robust watermarking. Even though watermarking
can be applied to a wide range of content, including video [1], audio [2], or
images, in this paper, we will focus on images.

Until recently, digital watermarking was based on deterministic algorithms.
Typical methods are based on embedding the watermark in one of the discrete
cosine transform (DCT) [3], discrete wavelet transform (DWT) [4, 5, 6],
discrete Fourier transform (DFT) [7], or quantization index modulation [8], or
a combination of these [9, 10].

Due to the inflexibility of deterministic algorithms, small changes in
watermarked images lead to inaccurate extracted watermarks. The inflexibility
leads to using neural networks as a robust technique for digital watermarking
[11, 12]. Even though these methods solve the robustness problem, some non-
deterministic networks still perform poorly, and others have complex
structures.

This paper investigates the combination of deterministic and neural network
methods to improve accuracy and performance. As the results suggest, we have
improvements in some attacks. In addition, the proposed network enhances the
quality of watermarked images by using DWT as a preprocessing technique.

The rest of the paper is organized as follows: In Section 2, we discuss
relevant previous studies. Then in Section 3, we outline the basic concept,
wavelet transform, used for the proposed algorithm. The proposed network
structure is explained in Section 4. Section 5 discusses the training
technique, the experimental results, and the comparison with literature [12],
and [13]. Moreover, this paper is concluded in Section 6.

##  2 Related Works

Various image watermarking schemes have previously been done to provide
robustness, authenticity, and imperceptibility. Earlier, we mentioned some
papers proposing watermarking algorithms which use DWT, DFT, and neural
networks. Here we will explore some of them in more detail.

In paper [4], original images and watermarks are split into RGB channels to be
used as inputs in the other embedding process. In the original image, the blue
channel has been subjected to DWT and Singular Value Decomposition (SVD). A
first Arnold transformation is applied to the blue channel of the watermark
image, which acts as a key. In addition to Arnold’s transformation, SVD is
applied to the watermark image, and then the image is embedded with the
watermark. In addition to being semi-blind, this method is also robust against
attacks due to the use of the key during the watermark extraction.

As another deterministic approach, Xiangui Kang proposed a blind DWT-DFT
composite image watermarking algorithm that is robust against both affine
transformation and Joint Photographic Experts Group (JPEG) compression [10].
This paper used watermark structure, two-dimensional interleaving, and
synchronization techniques to improve the robustness.

In recent years, residual blocks, fully connected layers, DCT layers, and CNN
have been used as watermarking networks. An end-to-end autoencoder in ReDMark
[12] simulates a DCT layer in its deep network. The concatenation of the input
image and watermark is used as input to the DCT layer, and after
transformation, the embedding is performed. The proposed structure is robust
against JPEG attacks since the DCT layer is used.

Jae-Eun Lee proposed a robust watermarking neural network that included an
attack simulation and was adjusted to the resolution of the host image and the
watermark [13]. It is composed of convolutional layers with average pooling
layers. This method also includes a strength factor to adjust the tradeoff
between invisibility and robustness. This method uses the images’ luma
component (Y) to hide the watermark. A watermark is converted into a layer the
same size as the original image. The watermark is then merged with the
processed image in the leading network. Despite the impressive results, the
proposed network is relatively complex. In contrast to mentioned articles,
this paper presented a non-deterministic method with higher robustness than
discussed papers.

##  3 Discrete Wavelet Transform

Each image has smooth regions interrupted by edges. Edges provide much
information about the image. In the Fourier transform, a signal is decomposed
into sin waves that are not localized in time. Therefore, Fourier transforms
cannot efficiently represent sudden changes (particularly edges), and wavelet
transforms are preferred. A wavelet transform is a way of representing a
function by a specific orthonormal series. A wavelet is a wave-like
oscillation that is localized in time. Different types of wavelet transform
exist based on the mother function $\psi(t)$ and the scaling function
$\varphi(t)$. One of the most widely-used wavelets is Haar whose $\psi(t)$ and
$\varphi(t)$ are shown in Figure 1.

(a) Scaling function

(b) Mother function

Figure 1: Haar’s functions.

In addition, the discrete wavelet transform decomposes an image into four
wavelet subbands, LL, LH, HL, and HH, which were examined independently to
compare their capacity for hiding information. The results are demonstrated in
Section 5 [14].

##  4 Proposed Method

The proposed network consists of 5 main parts; watermark preprocess network,
image preprocess network, embedding network, attack layer, and extraction
network. Throughout this paper, we will discuss each part in detail, but
Figure 2 shows the overall structure of the network. In addition to these
parts, we also use the wavelet transform for embedding and extraction.
Different shapes of images can be accepted through this structure because it
is based on CNN layers. The codes of the proposed method are available on
GitHub111https://github.com/alirezatwk/Convolutional-Neural-Network-Based-
Image-Watermarking-using-Discrete-Wavelet-Transform. Since DWT must be
implemented in the model, we have used [15] to implement DWT layers that can
run on the same hardware as the rest of the machine learning pipeline without
impacting performance.

Figure 2: Network structure.

###  4.1 Host Image Preprocessing Network

We select the first channel of the original image and scale it into the
$[0,1]$ interval. Next, it is decomposed by the Haar wavelet. Then the LL
layer is used as the input of the host image preprocessing network. The LL
layer is fed into a convolutional layer with 3x3 filters whose strides equal
one. According to Figure 3, the layer consists of 64 filters with the same
padding so that the output shape is the same as the host image.

Figure 3: Host image preprocessing network structure.

###  4.2 Watermark Preprocessing Network

As shown in Figure 4, the first step is to reshape array-like watermarks to 2D
watermarks so that applying convolutional layers will be possible. The
preprocessing network includes three network blocks. The blocks comprise
convolutional transpose, batch normalization, ReLU activation, and average
pooling. Convolutional transposes have 512, 128, and 1 filters, respectively.
The output’s data type and shape are the same as the host image, so we can
concatenate them in the following networks.

Figure 4: Watermark preprocessing network structure.

###  4.3 Embedding Network

First, the preprocessed image is concatenated with the preprocessed watermark.
The concatenation is then used as input for the four-block embedding network,
as shown in Figure 5. The first three blocks include a convolutional layer
with 64 filters, a batch normalization, and a ReLU activation function. The
last block consists of a convolutional layer with one filter and a Tanh
activation function to achieve an appropriate output size.

As mentioned earlier, in the host image preprocessing network, the LL channel
was passed to the embedding network. Now we change the main LL with the
watermarked one and apply a wavelet inverse transform which results in the
watermarked image.

Figure 5: Embedding network structure.

###  4.4 Attack Simulator

Watermarked images might suffer from malicious attacks. In order to increase
its robustness, an attack simulator was added to the network. This simulator
executes different types of attacks with the ratio in Table 1 on each mini-
batch. Figure 6 shows how explained attacks affect the original image.

Table 1: Attacks for attack simulator. Attack | Info | Ratio
---|---|---
No attack | - | 1/3
Salt and Pepper | $p=0.1$ | 1/6
Gaussian noise | $\sigma=0.15$ | 1/6
JPEG |  quality$=50$ | 1/6
Dropout | $p=0.3$ | 1/6

(a) Original image

(b) Gaussian’s attack

(c) Salt & pepper attack

Figure 6: Examples of attacks.

###  4.5 Extraction Network

The watermark is embedded in the LL channel, so the first step is to extract
the LL channel. The extracted channel is then used as input for the three-
block extracting network, as shown in Figure 7. The first two blocks include
two convolutional layers with 128 and 256 filters, a batch normalization, and
a ReLU activation function. The last block consists of a convolutional layer
with one filter and a Sigmoid activation function. All convolutional layers in
this section will have a stride of 2 to adjust the output shape to the
watermark shape.

Figure 7: Extraction network structure.

###  4.6 Loss Function

The two-term loss function is used for training the network. The similarity
between a watermarked image and the host image is measured by the mean square
error (MSE) loss function shown in Equation 1.

$L_{1}=\frac{1}{MN}\sum_{i,j}^{MN}[I_{H}(i,j)-I_{WI}(i,j)]^{2}$ (1)

where $M$ and $N$ are the resolution of the host image, $I_{H}$ is the host
image, and $I_{WI}$ is the watermarked one.

Also, the mean absolute error (MAE) function is used as the loss function
between the extracted watermark and input watermark displayed in Equation 2.

$L_{2}=\frac{1}{X}\sum_{i}^{X}|WM_{O}(i)-WM_{E}(i)|$ (2)

where X is the size of the watermark, $WM_{O}$ is the original watermark, and
$WM_{E}$ is the extracted one.

As shown in Equation 3, a combination of these functions determines the loss
function of the whole network.

$L_{3}=\lambda_{1}L_{1}+\lambda_{2}L_{2}$ (3)

coefficients ($\lambda_{1}$ and $\lambda_{2}$) are hyper-parameters that
adjust invisibility and robustness.

##  5 Experiments

Previously, we described the network in detail. This section explains training
materials, such as datasets, training facilities, and evaluation metrics.

###  5.1 Dataset

COCO dataset222http://images.cocodataset.org/zips/train2017.zip was used
containing 328,000 color images in different sizes [16]. Randomly, 40000
images were chosen as training data and 1000 as testing dataset. All images in
the dataset were resized to (256, 256, 3) to prepare for feeding into the
network. For the watermark, a random generator was used to generate 256-bit
watermarks for each instance. By generating random watermarks for each
instance, the network became more generalized.

###  5.2 Training

The proposed watermarking network was trained in a Laptop with an Intel(R)
Core(TM) i7-9750H CPU @ 2.60 GHz, 32 GB RAM, and the NVIDIA GeForce GTX 1650
GPU. Every mini-batch consists of 10 images and 10 new watermarks. There were
60 epochs. 33.0 for image loss weight ($\lambda_{1}$) and 0.2 for watermark
loss weight ($\lambda_{2}$). The learning rate was 0.001.

###  5.3 Metrics

Two metrics were used to estimate robustness and watermark invisibility. Most
previous works used peak-signal-to-noise ratio (PSNR) to evaluate the
invisibility of the watermarked image. PSNR measures the similarity between
the watermarked image and the host image by Equation 4.

$PSNR=10\log_{10}(\frac{MAX_{I}^{2}}{MSE})$ (4)

here, $MAX_{I}$ is the maximum possible pixel value of the image. Also, $MSE$
is the mean square difference between the host image and the watermarked one.

Due to the importance of the extracted watermark, another metric is required
to assess the robustness of the network. In this study, bit error ratio (BER)
was used, which is shown in Equation 5.

$\begin{split}BER&=\frac{100}{X}\sum_{1\leq i\leq X}D(WM_{O}(i),WM_{E}(i))\\\
D(x,y)&=\begin{cases}0,&x=y\\\ 1,&x\neq y\\\ \end{cases}\end{split}$ (5)

where $X$ is the size of the watermark, $WM_{O}$ is the original watermark and
$WM_{E}$ is the extracted one.

###  5.4 Results

In this section, the algorithm described above is examined from various
perspectives. First, the effect of the attack simulator is described. Then
there is a comparison between the accuracy of the wavelet channels. In the
end, it will be compared to state-of-art methods.

####  5.4.1 Attack Simulator Analysis

In order to evaluate the functionality of the attack simulator, the model was
trained without it, and table 2 shows a significant difference which suggests
the critical role of the attack simulator in network robustness.

Table 2: Effect of the attack simulator on robustness. Attack |  | Attack-simulator
---
free model
| Model with
---
attack simulator
PSNR | 50.9 | 40.1
No attack (BER) | 0.014 | 0.0003
Salt & pepper (BER) | 46.44 | 1.4231
Gaussian noise (BER) | 43.55 | 7.5382

####  5.4.2 Wavelet Channels Analysis

DWT bands have different metrics and efficiency values, as suggested by Shaila
R Hallur in [17] The purpose of this section is to analyze the results of each
band in order to choose the most proficient model. These experiments are based
on the same parameters setting with different DWT bands. According to Figure
8, the HH band is preferred for the PSNR criterion.

Figure 8: PSNR comparison of wavelet layers.

Nevertheless, in the BER comparison, LL has superior performance, as shown in
Figure 9.

Figure 9: BER comparison of wavelet layers.

Further investigations are conducted on the HH band in the remainder of the
paper.

####  5.4.3 Comparison with State-of-the-Arts Methods

As mentioned, for evaluating the invisibility of the watermarked images, the
PSNR of the watermarked images and the host images on the test dataset are
calculated, and the average of these numbers results in 40.1 dB. As shown in
Figure 10, the difference between the first and second columns is quite
negligible.

(a) host image

(b) watermarked image

(c) 10 times magnified difference of host and watermarked image

Figure 10: Examples of input images, output images, and their differences.

The measured BER on output watermarks without any attack is 0.0003. However,
applying the attacks, this number increased to 9.58.

Table 3 shows the network’s robustness under each attack. These performances
are compared with state-of-art methods. Regarding Table 3, using wavelet
transform in the structure keeps PSNR, but there is an improvement in the BER
metric. Additionally, the proposed network has performed better under Gaussian
noise and Salt and Pepper attacks.

Table 3: Comparing proposed and state-of-art methods. Attack | ReDMark | [13] | Proposed
---|---|---|---
PSNR | 40.24 | 40.58 | 40.1
No attack (BER) | - | 0.7015 | 0.0003
Salt and pepper (BER) | 9.1 | 3.1888 | 1.4231
Gaussian noise (BER) | 14.5 | 27 | 7.5382
Drop out (BER) | 8 | 4.7194 | 11.7051
JPEG (BER) | 25.4 | 0.6696 | 27.2482

##  6 Conclusion & Future Work

In this paper, we propose a watermarking neural network that can work well for
attacked images and has fewer convolutional layers than previous networks due
to wavelet transform. Compared with [13], the overall BER has decreased. The
PSNR metric of the proposed model is slightly higher than this one; in
addition, the model is smaller in size and works better in attacked images.

According to Arpit Bansal in [18], considering some special attacks in the
network leads to poor performance under new attacks. We claimed deterministic
methods are inflexible, and thus we used a neural network. However, we are
still using deterministic methods in the attack simulator. As a result, the
attack simulator can be updated to a more general form in future efforts. It
is also possible to investigate famous neural network architectures in order
to improve results.

## References

  * [1] Rachna Patel, Kalpesh Lad, Mukesh Patel, and Madhavi Desai.  An efficient dct-sbpm based video steganography in compressed domain.  International Journal of Information Technology, 13(3):1073–1078, 2021.
  * [2] Saurav Joshi, Jeetendra Pande, and B. K. Singh.  Watermarking of audio signals using iris data for protecting intellectual property rights of multiple owners.  International Journal of Information Technology, 10(4):559–566, 2018.
  * [3] Z.J. Xu, Z.Z. Wang, and Q. Lu.  Research on image watermarking algorithm based on dct.  Procedia Environmental Sciences, 10:1129–1135, 2011.  2011 3rd International Conference on Environmental Science and Information Application Technology ESIAT 2011.
  * [4] Jeril George, Satishkumar Varma, and Madhumita Chatterjee.  Color image watermarking using DWT-SVD and Arnold transform.  In 2014 Annual IEEE India Conference (INDICON), pages 1–6, 2014\.
  * [5] Chittaranjan Pradhan.  Robust and blind watermarking using arnold 4d cat map in discrete wavelet.  International Journal of Information Technology, 12(2):593–597, 2020.
  * [6] Kaiser J. Giri and Rumaan Bashir.  A block based watermarking approach for color images using discrete wavelet transformation.  International Journal of Information Technology, 10(2):139–146, 2018.
  * [7] Chunlei Li, Zhaoxiang Zhang, Yunhong Wang, Bin Ma, and Di Huang.  Dither modulation of significant amplitude difference for wavelet based robust watermarking.  Neurocomput., 166(C):404–415, October 2015.
  * [8] Hwai-Tsu Hu, Yu-Jung Chang, and Szu-Hong Chen.  A progressive QIM to cope with SVD-based blind image watermarking in DWT domain.  In 2014 IEEE China Summit & International Conference on Signal and Information Processing (ChinaSIP), pages 421–425, 2014.
  * [9] Fatemeh Nejati, Hedieh Sajedi, and Maryam Mohammadi.  Fragile watermarking for image authentication using QR factorization and fourier transform.  In 2019 5th International Conference on Web Research (ICWR), pages 45–49, 2019.
  * [10] Xiangui Kang, Jiwu Huang, Yun Q Shi, and Yan Lin.  A DWT-DFT composite watermarking scheme robust to both affine transform and JPEG compression.  IEEE Transactions on Circuits and Systems for Video Technology, 13(8):776–786, 2003.
  * [11] Haribabu Kandi, Deepak Mishra, and Subrahmanyam R.K. Sai Gorthi.  Exploring the learning capabilities of convolutional neural networks for robust image watermarking.  Comput. Secur., 65(C):247–268, March 2017.
  * [12] Mahdi Ahmadi, Alireza Norouzi, Nader Karimi, Shadrokh Samavi, and Ali Emami.  ReDMark: Framework for residual diffusion watermarking based on deep networks.  Expert Syst. Appl., 146(C), May 2020.
  * [13] Jae-Eun Lee, Young-Ho Seo, and Dong-Wook Kim.  Convolutional neural network-based digital image watermarking adaptive to the resolution of image and watermark.  Applied Sciences, 10(19), 2020.
  * [14] Karamjeet Singh, Sukhjeet Kaur Ranade, and Chandan Singh.  Comparative performance analysis of various wavelet and nonlocal means based approaches for image denoising.  Optik, 131:423–437, 2017.
  * [15] Francesco Versaci.  Wavetf: A fast 2d wavelet transform for machine learning in keras.  In Pattern Recognition. ICPR International Workshops and Challenges, pages 605–618. Springer International Publishing, 2021.
  * [16] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: Common objects in context.  In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision – ECCV 2014, pages 740–755, Cham, 2014. Springer International Publishing.
  * [17] Shaila R Hallur, Suresh Kuri, Giridhar S. Sudi, and G.H.Kulkarni.  A robust digital watermarking for gray scale image.  International Journal For Technological Research In Engineering, 2(10), June 2015.
  * [18] Arpit Bansal, Ping-Yeh Chiang, Michael J Curry, Rajiv Jain, Curtis Wigington, Varun Manjunatha, John P Dickerson, and Tom Goldstein.  Certified neural network watermarks with randomized smoothing.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 1450–1465. PMLR, July 2022.


Uncertainty-driven and Adversarial Calibration Learning for Epicardial Adipose
Tissue Segmentation

Kai Zhao, Zhiming Liu, Jiaqi Liu, Jingbiao Zhou, Bihong Liao, Huifang Tang,

Qiuyu Wang, Chunquan Li$\ast$

Abstract Epicardial adipose tissue (EAT) is a type of visceral fat that can
secrete large amounts of adipokines to affect the myocardium and coronary
arteries. EAT volume and density can be used as independent risk markers
measurement of volume by noninvasive magnetic resonance images is the best
method of assessing EAT. However, segmenting EAT is challenging due to the low
contrast between EAT and pericardial effusion and the presence of motion
artifacts. we propose a novel feature latent space multilevel supervision
network (SPDNet) with uncertainty-driven and adversarial calibration learning
to enhance segmentation for more accurate EAT volume estimation. The network
first addresses the blurring of EAT edges due to the medical images in the
open medical environments with low quality or out-of-distribution by modeling
the uncertainty as a Gaussian distribution in the feature latent space, which
using its Bayesian estimation as a regularization constraint to optimize
SwinUNETR. Second, an adversarial training strategy is introduced to calibrate
the segmentation feature map and consider the multi-scale feature differences
between the uncertainty-guided predictive segmentation and the ground truth
segmentation, synthesizing the multi-scale adversarial loss directly improves
the ability to discriminate the similarity between organizations. Experiments
on both the cardiac public MRI dataset (ACDC) and the real-world clinical
cohort EAT dataset show that the proposed network outperforms mainstream
models, validating that uncertainty-driven and adversarial calibration
learning can be used to provide additional information for modeling multi-
scale ambiguities.

KeyWord: Epicardial adipose tissue segmentation, Feature latent space,
Uncertainty-driven, Adversarial calibration, Bayesian estimation

## Introduction

Epicardial adipose tissue (EAT) resides within both the atrioventricular and
interventricular grooves, situated proximal to the coronary arteries and
myocardium, enveloping a considerable portion of the cardiac vasculature. As
the volume of EAT increases, adipose deposits commence extending towards the
ventricular free wall, potentially enshrouding the entirety of the cardiac
organ. EAT functions as a metabolically active endocrine and paracrine organ,
releasing significant quantities of both anti-inflammatory and pro-
inflammatory adipokines. These bioactive molecules exert influence over the
dynamic equilibrium of energy-glycolipid metabolism, as well as cardiac
structure and function[1, 2]. EAT is closely associated with the development
of coronary heart disease, thus it is considered to be a biomarker for a class
of risk factors. With advances in imaging technology, it has been possible in
recent years to determine the natural evolution of its volume and density in
at-risk populations, suggesting that EAT might undergo substantial alterations
in plaque calcification well in advance of its maturation. More findings
propose that EAT serves as an independent parameter that is expected to be a
surrogate biomarker for cardiovascular risk estimation[3, 4].

Nevertheless, the quantitative imaging assessment of EAT with intricate
topology is a challenging task. Since EATs have different physiologic and
pathologic properties at different locations, this leads to two major problems
for the imaging physician in accurately quantifying their statistical
differences: Primarily, the heart is a continually pulsating and contracting
muscular organ that plays a pivotal role in pumping blood to sustain proper
bodily function. This constant motion produces artifacts, leading to the
indistinct delineation of EAT edges from the surrounding tissues, as
illustrated in Fig. 1a. Secondly, there exists a challenge in distinguishing
EAT from pericardial effusion, as both may manifest very similar signal
intensities in MRI[5], as depicted in Fig. 1b. Consequently, this makes the
difference in their grayscale values small and increases increasing the
difficulty of visual differentiation. In addition, the morphology and
distribution of EAT may exhibit variations based on individual
characteristics, further increasing the difficulty of accurately identifying
tissue edges. The above factors lead to inefficient and cognitively subjective
manual labeling assessments. Thus, to reduce the burden of quantitative
assessment of EAT using image processing techniques for automated operations
can provide objective and rapid quantitative assessment, which greatly
improves the efficiency of imaging physicians.

Fig1. The first row shows four slices of the heart at different times during
the MRI scan, and the second row of images marks the region of epicardial fat
in yellow. The first two samples show the situation when motion blur is
present, and the last two samples show the situation when both pericardial
effusion and epicardial fat are present. We have pointed out some of the
corresponding regions with blue arrows.

In the field of medical image segmentation, most of the current mainstream
semantic segmentation methods use Vision Transformer[6] or U-Net[7]
architectures, focusing on the learning of deterministic features, e.g.[8, 9,
10]. Regrettably, these methodologies overlook the influence of data or model
uncertainty issues on segmentation outcomes, encompassing factors like motion
artifacts and cognitive ambiguities. So, there may be certain model
performance bottlenecks for segmentation problems with such uncertainties.
Although there have been some approaches that consider the reliability of
segmentation results and introduce uncertainty estimation, where uncertainty
is directly or indirectly represented as network parameter distributions or
feature latent space distributions in the model, these models are based on
learning strategies for a parameter or feature distributions do not consider
calibrating the modeled uncertainty[11, 12, 13], and lack the capability to
further utilize uncertainty to guide the segmentation performance improvement.
In this study, an uncertainty quantification and calibration framework for
feature latent space distribution representation is directly constructed based
on the above methods: First, the segmentation model adopts SwinUNETR as its
backbone network. Subsequently, a Probabilistic Net based on Bayesian
estimation is constructed to drive the uncertainty representation of the
model. Finally, a Discriminator Net is constructed to perform adversarial
calibration of the segmentation feature maps and to enhance the network’s
utilization of multi-scale features. Our objective is to investigate an
efficient and automated method for EAT segmentation.

The main contributions of this study are as follows:(1) The Probabilistic Net
have engineered adeptly models data uncertainty by employing a Gaussian
distribution within the feature latent space. Feature vectors sampled from
this distribution are incorporated into the segmentation feature maps layer by
layer and by using quantized uncertainty constraints to suppress segmentation
performance degradation due to motion artifacts. (2) The Adversarial training
strategy was used to calibrate the segmentation feature maps, and a multi-
scale loss optimization model of Discriminator Net was introduced into the
objective function to enhance the model’s ability to capture the feature
differences at each level, to better distinguish similar tissues with low
contrast. (3) The model proposed in this study was validated on an open data
set for real-world applications, and the efficacy of uncertainty-driven and
adversarial calibration methodologies underwent evaluation in publicly
available datasets and self-constructed clinical cohort segmentation tasks.
The results show that the proposed method achieves significant improvements in
EAT segmentation, which provides strong support for early diagnosis and
treatment of coronary artery disease.

## Related work

Probability Model for Images Segmentation: Image segmentation tasks aim to
assign a category label to each pixel in an image. However, in the real-world
implementation, a large amount of information is inherently uncertain, leading
to potential inconsistencies in the distribution of training and test data.
Deterministic segmentation methods encounter difficulties in resolving such
uncertainties. Rahman introduced approximate Bayesian inference to model this
inconsistency in their study, demonstrating its effectiveness, particularly
with smaller datasets[14]. Kohl proposed the Probabilistic U-Net[11], which
integrates a conditional variational autoencoder (CVAE)[15] with a U-Net. This
innovative approach addresses the inconsistency issue by leveraging the
stochastic nature of the conditional variational autoencoder to generate
multiple segmentation hypotheses. However, this method’s introduction of
uncertainty information solely in the topmost layer of the U-Net encoder leads
to limitations in explaining features at different scales. To address this,
PHi-Seg improve the model structure, acquiring a hierarchical latent space by
the feature of the image at various scales, and infusing feature sampled from
this space into each layer of the encoder[16]. This adaptation enables the
latent space to adapt to changing image dimensions, resulting in a improvement
in the model’s performance, particularly in capturing intricate details. These
methods mentioned above have significant advantages when faced with
uncertainty in segmentation problems.

Adversarial Training Strategies: The adversarial training strategy was
initially introduced in Generative Adversarial Network (GAN)[17] by Goodfellow
et al. This strategy enhances the generalization ability of the model by
optimizing the adversarial game between the discriminator and the generator to
generate more realistic samples. In the field of image segmentation, models
employing this strategy typically consist of two network modules: a
segmentation network (Generator) dedicated to image segmentation and a
discriminative network (Discriminator) tasked with discerning between real and
generated labels. For instance, Xu et al. introduced a semi-supervised
adversarial approach, leveraging an adversarial training strategy to enhance
segmentation prediction accuracy by enabling the network to learn the data
distribution of labels[18]. Hung et al. combined cross-entropy loss with
adversarial loss, resulting in a substantial improvement in the semantic
segmentation accuracy of the model[19]. The findings presented in[20]
demonstrate that the adversarial training strategy proves effective in
mitigating labeling noise in real-world data and performs admirably in
scenarios with limited dataset sizes. Furthermore, in[21] noteworthy is the
incorporation of multi-scale loss functions in discriminator networks,
seamlessly integrated into segmentation networks through adversarial training.
This enables them to grasp both global and local features, leading to
heightened model accuracy. These research outcomes unequivocally underscore
the validity of adversarial training strategies within the realm of image
segmentation, offering substantial validation for our research endeavors.

## Materials and methods

Fig. 2 shows the model structure of the methodology of this study: (1) the
SwinUNETR-based backbone network, denoted as Segmentor Net, (2) the
uncertainty-driven Probabilistic Net, and (3) Discriminator Net for
calibrating uncertainty representations of feature maps and computing multi-
scale loss.

The proposed methods are slated for validation using two distinct datasets:
the public ACDC MICCAI 2017 dataset[22] and a real-world clinical cohort
dataset on EAT. Detailed elucidation of every network and their optimization
algorithms that exploit the uncertainty representation of the feature latent
space and adversarial training calibration are described in detail below.

Fig2. The network is divided into three modules in total: (1) Segmentor Net is
used to extract features from the input image and can fuse random feature
vectors sampled from (2) Probabilistic Net, and finally get a coarse
segmentation. (3)Discriminator Net fuses the coarse segmentation and the true
labeling with the input image, respectively, and then computes the loss from
these two different inputs.

### Materials

Cohort study of EAT Dataset: The real-word EAT MRI dataset was obtained at the
First Affiliated Hospital of South China University by serial image
acquisition during end-expiratory breath-holding on a 3.0T scanner (Siemens,
Germany). The dataset has been de-identified and was approved by Review Board
of Clinical Research 4310 Program of the University of South China (2021KS-
XN-12-02) on November 12, 2021. This study adhered to the principles of the
Declaration of Helsinki.

Employing a balanced steady-state free precession (bSSFP) sequence, we
acquired 9 to 13 consecutive short-axis cine images of the left ventricle,
ranging from the base to the apex. The scanning parameters were as follows:
repetition time (TR) / echo time (TE) of 3.30/1.45MS, a field of view of
325×400 mm², flip angle of 45-50°, slice thickness of 8 mm, and a matrix size
of 256×166. The raw images were loaded into the commercial software CVI 42
(Circle Cardiovascular Imaging Inc., Calgary, Alberta, Canada). Image post-
processing was then performed by an experienced radiologist and two
experienced graduate students. Epicardial fat, which is the adipose tissue
located between the outer wall of the myocardium and the visceral layer of the
pericardium, was manually outlined on left ventricular end-diastolic short-
axis images from the base of the left ventricle to the apex. A total of 61
patients’ MRI were collected as a training set(about 730 images), and the test
set(about 120 images) contained another 10 patients not present in the
training set to validate the reliability and effectiveness of our method.

ACDC Dataset: The dataset used in this study is a publicly accessible resource
designed for diagnosing cardiac diseases. It encompasses cardiac MRI from 150
patients. Each patient’s image sequence covers a complete cardiac cycle,
comprising between 28 to 40 images, with a spatial resolution ranging from
1.37 to 1.68 mm²/pixel. Alongside the raw MRI, the ACDC dataset also offers
segmentation labels for manual annotation of cardiac structures, encompassing
regions of the left ventricle (LV), right ventricle (RV), and interventricular
septum (Myo). The training set incorporates data from 100 patients, while the
test set includes data from an additional 50 patients. The dataset’s richness
and diversity render it a pivotal resource for advancing research in the field
of cardiac imaging. Moreover, it furnishes a robust experimental foundation
for our study. The data can be accessed from https://www.creatis.insa-
lyon.fr/Challenge/acdc/.

### The SPDNet for EAT Segmentation

Due to SwinUNETR’s[23] exceptional performance in medical image segmentation
tasks, our method uses it as the Segmentor Net, to achieve more accurate
segmentation in MRI-scanned images. Building upon this foundation, we
amalgamated the feature information extracted from the Segmentor Net with
uncertainty information sampled from the a posteriori feature latent space
different scales of the Probabilistic Net.

The approach is shown in Fig. 2, where we combine Probabilistic Net with
Segmentor Net. The Probabilistic Net is adept at modeling intricate
distributions, whereas the Segmentor Net demonstrates excellent performance in
various medical applications. Image and Image + Ground Truth are first modeled
by Priori Net and Posteriori Net in Probabilistic Net to construct a priori
feature latent space and a posteriori feature latent space, respectively.
Next, we sample a random feature vector Z from the a posteriori feature latent
space and inject it into the Segmentor Net, which is fused with the Image
features extracted by the Segmentor Net to each layer to generate the
corresponding segmentation maps and compute the reconstruction loss. Lastly,
we employ an adversarial training strategy to train the model through a
Discriminator Net with multiscale loss to compensate for the lack of ability
to perform direct learning of multiscale space constraints during the end-to-
end training process to calibrate the segmentation feature maps.

#### Segmentor Net

The backbone network SwinUNETR used an encoder-decoder paradigm and has an
overall U-shaped structure, which is derived from the classical model U-Net in
the field of medical image segmentation. The Encoder in SwinUNETR is composed
of a Patch Partition layer and four stages. The Patch Partition layer divides
the image into individual 4$\times$4 sized patches, with each patch treated as
a token. These tokens are then processed through Merging and Linear Embed
Blocks (comprising a linear embedding layer and Swin Transformer Blocks),
denoted as "Stage 1".Swin Transformer Blocks [24] can the capability to
partition the input into non-overlapping windows and perform local self-
attention within each window region. The Merging process is employed to
amalgamate the patches and augment the dimensionality of the output. In “Stage
1”, there is a reduction in the number of tokens in a single dimension
(downsampling by a factor of 2 in resolution), with the output dimension set
to 48. Subsequently, the Swin Transformer Blocks and Merging process are
reiterated in Stage 2, with the output dimension doubled in comparison to
“Stage 1”. This process is repeated twice more in “Stage 3” and “Stage 4”,
resulting in output resolutions of $\frac{H}{8}\times\frac{W}{8}$ and
$\frac{H}{16}\times\frac{W}{16}$, respectively. The features derived from each
“Stage” undergo transmission to the CNN-based decode block through skip
connections, and “Concat” (the features are spliced together according to the
dimensions of the channels) with the features passed from “Deconv”. In the
last layer, the features from the Patch Partition are concatenated with the
transposed convolution features and features sampled from the Probabilistic
Net feature latent space. This concatenation of the features together is based
on the dimensions of the channels. Subsequently, a SoftMax is applied to
obtain the final segmentation map. Tiny Swin Transformer Blocks (the number of
Swin Transformer Blocks for the four stages is [2,2,6,2]) and convolutional
layers with the number of channels [48,96,192,384] are used in our approach.
This block yields an initial coarse segmentation result, and based on this
result, we calculate the reconstruction loss once. The original SwinUNTER loss
function comprises the cross-entropy loss function, denoted as
$L_{ce}\left(\hat{y}_{i},y_{i}\right)=\sum^{N}_{i=0}y_{i}\ln\hat{y}_{i}+\left(1-y_{i}\right)\ln\left(1-\hat{y}_{i}\right)$.To
capture small lesion features, we introduced the Dice loss,
$L_{dice}\left(\hat{y}_{i},y_{i}\right)=1-\frac{2\ast\sum^{N}_{i=0}\hat{y}_{i}\ast
y_{i}}{\sum^{N}_{i=0}\hat{y}_{i}+\sum^{N}_{i=0}y_{i}}$. These two losses are
combined using an adjustable parameter to form a new loss function, denoted as
$L_{rec}$:

$\begin{split}L_{rec}\left(\hat{y}_{i},y_{i}\right)=-\alpha
L_{ce}\left(\hat{y}_{i},y_{i}\right)+\left(1-\alpha\right)L_{dice}\left(\hat{y}_{i},y_{i}\right)\\\
\end{split}$ (1)

In the above equation, where $i\in\left\\{0,1\right\\}$ represents the
category (0 for background, 1 for epicardial fat), $\hat{y}_{i}$ denotes the
prediction for category $i$, and $y_{i}$ denotes the ground truth of class.
The parameter $\alpha$ is artificially set to regulate the ratio between the
cross-entropy loss and the Dice loss. In experiments, we find the best
performance at $\alpha$ $=$ 0.6.

#### Probabilistic Net

Within this module, two U-Net networks with the same structure, namely the a
priori net and the posterior net, are incorporated. The primary function of
the a priori net is to model the image $X$. At the decoding stage, multiple
low-dimensional latent spaces of different scales, denoted as multiple a
priori probability distributions $p_{i}\left(z_{i}|x_{i}\right)$, are
obtained. Each position in these distributions corresponds to a segmentation
variable. These are Gaussian distributions with mean $\mu_{i}^{prior}$ and
variance $\sigma_{i}^{prior}\in\mathbf{\mathbb{R}}^{H_{i}\times W_{i}}$ (where
$i\leq L$, and $L$) is the number of layers of the decoder). We compute the
$\mu_{i}^{prior}$ and $\sigma_{i}^{prior}$ for the feature latent space at
each scale by 1$\times$1 convolution. The a priori net comprises a
deterministic feature extractor, U-Net, responsible for computing spatial
features of a given input image $X$ at different levels. In a conventional
U-Net, the U-Net decoder features at each scale are upsampled and then
connected to the U-Net encoder features at the corresponding scale. However,
in a priori net, the latent hierarchy at each scale undergoes an additional
step: sampling $z_{i}\in\mathbb{R}^{H_{i}\times W_{i}}$ from the low-
dimensional latent space constructed via $\mu_{i}^{prior}$ and
$\sigma_{i}^{prior}$, which is connected to the features of the input U-Net
decoder and then up-sampled in the usual way. By combining the deterministic
feature extractor and latent space sampling, we achieve effective utilization
of uncertainty at different scales, laying a solid foundation for the accuracy
of the segmentation task. The latent variables $z_{i}$ sampled from the
corresponding Gaussian distribution in the latent space and the prior
distribution $P$ are formulated as follows:

$\begin{split}z_{i}\mathcal{\sim
N}\left(\mu_{i}^{prior}\left(x_{i}\right),\sigma_{i}^{prior}\left(x_{i}\right)\right)=p\left(z_{i}|x_{i}\right).\end{split}$
(2)

$\begin{split}P\left(z_{0},\cdots,z_{i}|x_{0},\cdots,x_{i}\right)=p\left(z_{0}|x_{0}\right)\cdot\cdot\cdot
p\left(z_{i}|x_{i}\right).\end{split}$ (3)

The posterior net is almost identical in structure to the priori net,
differing solely in the number of channels in the input section. It models the
variational a posteriori $Q\left(.|X,Y\right)$ by requiring inputs of both $X$
and $Y$. In our case, $Y$ is the segmentation label, and $X$ is the input
image. Its role is intended to help maximize the evidence lower bound (ELBO)
of the likelihood $p\left(Y|X\right)$.

In the posterior net, we compute the feature mean $\mu_{i}^{post}\mathbb{\in
R}^{H_{i}\times W_{i}}$ and variance $\sigma_{i}^{post}\mathbb{\in
R}^{H_{i}\times W_{i}}$ at each scale and model a Gaussian distribution with
these means and variances. During the training period, the sampled samples
$z_{i}$ are fed into the Segmentor Net to be fused with the features extracted
by the Segmentor Net encoder in the jump connection part. The purpose of this
step is to reduce the interference of motion artifacts on the network by using
uncertainty constraints to guide the loss function $L_{rec}$ during the
training phase. In the posterior net, the posterior latent variable $z_{i}$
and the posterior distribution $P$ are formulated as follows:

$\begin{split}z_{i}\mathcal{\sim
N}\left(\mu_{i}^{post}\left(x_{i},y_{i}\right),\sigma_{i}^{post}\left(x_{i},y_{i}\right)\right)=q\left(z_{i}|x_{i},y_{i}\right)\end{split}$
(4)

$\begin{split}Q\left(z_{0},\cdots,z_{i}|\left(x_{0},\cdots,x_{i}\right),\left(y_{0},\cdots,y_{i}\right)\right)=q\left(z_{0}|x_{0},y_{0}\right)\cdot\cdot\cdot
q\left(z_{i}|x_{i},y_{i}\right).\end{split}$ (5)

In addition, there is a Kullback-Leibler divergence
$D_{KL}\left(Q||P\right)=E_{z\sim Q}\left[logQ-logP\right]$,which brings $P$
and $Q$ closer together. We also use a weight parameter $\beta$ to participate
in the computation of the reconstruction loss during the coarse segmentation
loss computation (We found that the proposed method works robustly if $\beta\
=10$), and our loss function formula is shown below:

$\begin{split}L_{ELBO}={E_{z\sim
Q}}\left[L_{rec}\left(S\left(X,z\right),Y\right)\right]+\beta\cdot\sum_{i=0}^{L}{E_{z_{i}\sim
Q}D_{KL}}\left(q_{i}\left(z_{i}|x_{i},y_{i}\right)\parallel
p_{i}\left(z_{i}|x_{i}\right)\right).\end{split}$ (6)

#### Discriminator Net

Discriminator Net contains multilayer convolutional layers and leaky
activation functions, while batch normalization is introduced in the last two
convolutional layers to normalize the batch data.

It has two different inputs: the coarse segmented image $S\left(X,z\right)$
fused with the input image $X$ and the segmented label $Y$ fused with the
input image $X$ denoted as $D\left(X\ast S\left(X,z\right)\right)$ and
$D\left(X\ast Y\right)$. The Discriminator Net employs a multilayer structure
to extract hierarchical features and compute a multiscale loss $L_{1}$. This
loss makes full use of the different scale of features, allowing the model
i.e. to learn global and local features more efficiently and to calibrate the
segmented feature map further. Training of the Discriminator Net occurs in an
alternating manner with the Segmentor Net through backpropagation of the
multiscale loss $L_{1}$. During training, the parameters of the Segmentor Net
are initially fixed to update the parameters of the Discriminator Net.
Subsequently, the parameters of the Discriminator Net are fixed, and the
parameters of the Segmentor Net are updated by computing the gradient using
the losses propagated from the Discriminator Net. This training approach is
akin to a minimax game: the objective of the Segmentor Net is to minimize the
multi-scale feature loss during training, whereas the Discriminator Net
endeavors to maximize the multi-scale feature loss.

Through continual training and parameter updates, the Discriminator Net and
Segmentor Net will become more and more powerful and gradually reach a
balance. In Discriminator Net we use Mean Absolute Error (MAE) $L_{mse}$ for
multi-scale feature loss, so our Discriminator Net loss is as follows:

$\begin{split}L_{mse}\left(D\left(X\ast Y\right),D\left(X\ast
S\left(X,z\right)\right)\right)=\frac{1}{C}\sum_{i=1}^{C}||\left(d_{i}\left(X\ast
Y\right),d_{i}\left(X\ast S\left(X,z\right)\right)\right)||_{1}.\end{split}$
(7)

In the above equation, $C$ is the total number of layers of the Discriminator
Net, and $d_{i}$ represents the features extracted from the $i$th
convolutional layer.

In the following expression, $\theta_{S}$ and $\theta_{D}$ represent the
parameters of the generator and discriminator, respectively, and K is the
batch size of the input image. To better the optimization of our objective
function, we integrate the ELBO loss with the discriminator training loss.
Thus, the total loss of our model is expressed as follows:

$\begin{split}\min_{\theta_{S}}\max_{\theta_{D}}L_{total}\left(\theta_{S},\theta_{D}\right)=\frac{1}{K}\sum_{i=1}^{K}L_{mse}\left(D\left(X\ast
Y\right),D\left(X\ast S\left(X,z\right)\right)\right)+L_{ELBO}.\end{split}$
(8)

## Experiments and Results

In this study, we conducted a comprehensive performance evaluation of the
model using the ACDC MICCAI 2017 cardiac dataset and the real-world epicardial
fat dataset. We deliberately selected six contemporary and leading medical
image segmentation methods: SegAN [21], FCT[25], nnFormer[26], SwinUnet[27],
TransUNet[28], and U-Net[7]for comparative experiments. To assess the
segmentation performance, three metrics were employed: Dice score, Jaccard
(Jaccard similarity coefficient), and Harsdorff distance (HD). The model
underwent training for 100 epochs using Adam’s optimization function, with a
batch size set to 8 and an initial learning rate of 0.0001. Throughout the
training process, we utilized an NVIDIA 3090 graphics card and enhanced the
model’s performance with data augmentation techniques, including random
rotations, flipping samples on either the x- or y-axis, and introducing random
angular skews. Such experimental design ensures the reliability and rigor of
our comparison tests and provides a solid foundation for the robustness of the
method in practical applications.

Fig3. The visualized of the epicardial fat segmentation results, with each row
in the first column being a different MRI slice of the patient’s heart. We
compared several other different methods and compared the model predictions
with the Ground Truth, with the last column being GT.

Fig4. The segmentation performance of our proposed method as well as other
methods is compared in the test set based on Dice score, Jaccard (Jaccard
similarity coefficient) and Hausdorff Distance (HD) scores and visualized in
the form of box plots. The x and y axes of each plot indicate the name of the
model and the evaluation scores for each metric, respectively.

### Comparative experiments with other segmentation models

According to Table 1, we conducted a comprehensive comparison of our model
with six other models on the private dataset. The results demonstrate that our
model achieves optimal scores on all three evaluation metrics. Specifically,
the Dice score reaches 0.933, the Jaccard coefficient is 0.875, and the
Harsdorff distance is 9.696mm. Additionally, we visually represented the
distribution of the evaluation metric scores through box-and-line plots in
Fig. 4. The results clearly indicate that, compared to the other methods, the
model significantly reduces the number of outliers predicted for the three
metrics, further confirming the stability and superiority of our method. These
results fully showcase the excellent performance of our model in medical image
segmentation tasks.

Observing Fig. 3, we note a significant segmentation deficiency in other
methods, particularly in cases of motion blur. Additionally, in images where
epicardial fat and pericardial effusion coexist, other methods encounter
difficulty in differentiation. This underscores the model’s effective
mitigation of the effects of motion blur and its enhanced ability to
accurately distinguish epicardial fat from pericardial effusion.

To further validate the segmentation performance, we conducted experiments on
the public dataset ACDC MICCAI 2017, with the results presented in Table 2. On
this dataset, the model also achieves the best performance across various
metrics. These experiments demonstrate that the method is not only feasible in
the cardiac binary classification task but also effective in the cardiac
multiclassification task. We visualized the segmentation results by
differentiating the categories through different colors, as shown in Fig. 5.
And we represented the distribution of the evaluation metric scores in Fig. 6.
This series of results clearly demonstrate the excellent performance of our
model, providing a reliable solution for cardiac image segmentation.

Fig5. Visualization of segmentation results in the ACDC dataset compared to
other methods, with the left ventricle in blue, the left ventricular
myocardium in green, and the right ventricle in pink.

Fig6. Boxplot of each indicator in the ACDC test set.

Methods Dice score Jaccard HD(mm) U-Net[7] 0.866 0.771 13.721 TransUNet[28]
0.899 0.822 12.617 SwinUNet[27] 0.906 0.831 12.503 SegAN[21] 0.917 0.849
11.368 nnFormer[26] 0.921 0.855 10.683 FCT[25] 0.927 0.865 10.356 SPDNet(ours)
0.933 0.875 9.696

Table 1 Comparative experiment results with the segmentation methods in the
table were evaluated on the epicardial fat dataset. The evaluation metrics
used are Dice score, Jaccard (Jaccard similarity coefficient) and Hausdorff
Distance (HD), with black bolding representing the best performance.

Methods Dice score Jaccard HD(mm) U-Net[7] 0.876 0.786 14.174 TransUNet[28]
0.897 0.824 12.429 SwinUNet[27] 0.901 0.831 11.792 SegAN[21] 0.912 0.854
10.968 nnFormer[26] 0.921 0.864 10.053 FCT[25] 0.931 0.873 9.818 SPDNet(ours)
0.939 0.881 9.049

Table2 Comparative experiment results with other methods on the ACDC public
dataset.

Fig7. Results from ablation experiments were visually presented in case
sections containing either motion artifacts or pericardial effusions. The
first row of case sections illustrates instances with motion artifacts, while
the second row showcases sections where pericardial effusion is present in
proximity to epicardial adipose tissue. Relevant areas have been highlighted
with green arrows for clarity.

### Ablation experiments

The performance improvement of our algorithm lies in the introduction of two
key nets: the Probabilistic Net and the Discriminator Net. To assess the
effectiveness of these nets, we conducted exhaustive ablation experiments,
except for adding or subtracting net, we did not modify other aspects (in Fig.
7).

The results of the experiments, as presented in Table 3, indicate a
significant enhancement in the model’s performance with the expansion of each
net. Introducing the Probabilistic Net results in a 0.03 improvement in the
Dice coefficient score, a 0.03 improvement in the Jaccard index, and a 0.39 mm
improvement in the Harsdorff distance index. This experiment substantiates the
efficacy of our enhancements. Similarly, the addition of the Discriminator Net
further enhances the performance across each metric. The improvements observed
in these quantitative metrics underscore the effectiveness of our enhancement
measures. These results robustly support the superior performance of our model
and the validity of our methods.

Probabilistic Net Segmentor Net Dice score Jaccard HD(mm) 0.921 0.857 10.679
$\surd$ 0.925 0.863 10.287 $\surd$ $\surd$ 0.933 0.875 9.696

Table3 Performance of ablation experiments with different modules.

### Limitations and Discussion

This study focuses on solving the problem of motion artifacts appearing in MRI
and quantifying them as uncertainty. However, our approach is limited by the
specific problem and application scenarios and thus has some limitations in
considering uncertainty. Furthermore, we used a Bayesian-based approach to
construct a Probabilistic Net, which learns the distribution of network
weights with uncertainty. While this provides us with more comprehensive
information, it also comes with a larger computational cost.

In future endeavors, we aim to enhance our methods to cater to a broader
spectrum of application scenarios and streamline computational efficiency by
optimizing our algorithms. This involves investigating more efficient
approaches to quantify uncertainty and ensuring scalability and practicality.
Additionally, we are committed to designing a series of new experiments to
comprehensively evaluate the effectiveness of uncertainty and validate the
performance of our methods across diverse contexts.

## Conclusions

In this study, epicardial fat segmentation faced two major challenges: the
difficulty in distinguishing the presence of motion artifacts and EAT from
pericardial effusion. In addition, manual labeling of epicardial fat is a
time-consuming and laborious task. Therefore, our study aims to propose an
efficient automated segmentation model to assist clinicians in risk assessment
and thus improve their productivity. Our approach optimizes the Segmentor Net
by transforming the blurring of epicardial adipose tissue (EAT) edges induced
by motion artifacts into a regularization constraint via the Probabilistic
Net. This optimization enhances segmentation performance through the
utilization of uncertainty representation. Subsequently, an adversarial
training strategy is employed to learn the predictive segmentation maps and
ground truth multiscale feature differences by constructing a Discriminator
Net with multiscale loss to calibrate the predictive segmentation maps and
better differentiate pericardial effusion from epicardial fat.

The proposed model was experimentally validated on the public cardiac dataset
ACDC and the real-world epicardial fat dataset. The experimental results show
that the proposed model has achieved significant performance improvement,
which provides a reliable auxiliary tool for clinicians to accurately segment
epicardial fat and provides strong support for risk assessment and diagnosis.

## Acknowledgement

This work was supported by National Natural Science Foundation of China
[62171166,62272212]; Research Foundation of the First Affiliated Hospital of
University of South China for Advanced Talents [20210002-1005 USCAT-2021-01];
Natural Science Foundation of Hunan Province [2023JJ30536,2023JJ30547];
Scientific Research Fund Project of Hunan Provincial Health Commission
[20201920]; Special Funds for the Construction of Innovative Provinces in
Hunan [2020SK4008]; Clinical Research 4310 Program of the University of South
China [20224310NHYCG05,20214310NHYCG03].

## References

  * [1] Gianluca Iacobellis, Domenico Corradi, and Arya M Sharma.  Epicardial adipose tissue: anatomic, biomolecular and clinical relationships with the heart.  Nature clinical practice Cardiovascular medicine, 2(10):536–543, 2005.
  * [2] Milton Packer.  Epicardial adipose tissue may mediate deleterious effects of obesity and inflammation on the myocardium.  Journal of the American College of Cardiology, 71(20):2360–2372, 2018.
  * [3] Alexios S Antonopoulos, Fabio Sanna, Nikant Sabharwal, Sheena Thomas, Evangelos K Oikonomou, Laura Herdman, Marios Margaritis, Cheerag Shirodaria, Anna-Maria Kampoli, Ioannis Akoumianakis, et al.  Detecting human coronary inflammation by imaging perivascular fat.  Science translational medicine, 9(398):eaal2658, 2017.
  * [4] Gianluca Iacobellis.  Epicardial adipose tissue in contemporary cardiology.  Nature reviews cardiology, 19(9):593–606, 2022.
  * [5] Gijs van Woerden, Thomas M Gorter, B Daan Westenbrink, Tineke P Willems, Dirk J van Veldhuisen, and Michiel Rienstra.  Epicardial fat in heart failure patients with mid-range and preserved ejection fraction.  European journal of heart failure, 20(11):1559–1566, 2018.
  * [6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [7] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
  * [8] Zhongxiang Zhang, Guangmin Sun, Kun Zheng, Jin-Kui Yang, Xiao-rong Zhu, and Yu Li.  Tc-net: A joint learning framework based on cnn and vision transformer for multi-lesion medical images segmentation.  Computers in Biology and Medicine, 161:106967, 2023.
  * [9] Yang Lei, Yifu Ding, Richard LJ Qiu, Tonghe Wang, Justin Roper, Yabo Fu, Hui-Kuo Shu, Hui Mao, and Xiaofeng Yang.  Hippocampus substructure segmentation using morphological vision transformer learning.  arXiv preprint arXiv:2306.08723, 2023.
  * [10] Xiaomeng Feng, Taiping Wang, Xiaohang Yang, Minfei Zhang, Wanpeng Guo, and Weina Wang.  Convwin-unet: Unet-like hierarchical vision transformer combined with convolution for medical image segmentation.  Mathematical Biosciences and Engineering, 20(1):128–144, 2023.
  * [11] Simon Kohl, Bernardino Romera-Paredes, Clemens Meyer, Jeffrey De Fauw, Joseph R Ledsam, Klaus Maier-Hein, SM Eslami, Danilo Jimenez Rezende, and Olaf Ronneberger.  A probabilistic u-net for segmentation of ambiguous images.  Advances in neural information processing systems, 31, 2018.
  * [12] Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U Rajendra Acharya, et al.  A review of uncertainty quantification in deep learning: Techniques, applications and challenges.  Information fusion, 76:243–297, 2021.
  * [13] Zongbo Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou.  Trusted multi-view classification with dynamic evidential fusion.  IEEE transactions on pattern analysis and machine intelligence, 45(2):2551–2566, 2022.
  * [14] Aimon Rahman, Jeya Maria Jose Valanarasu, Ilker Hacihaliloglu, and Vishal M Patel.  Ambiguous medical image segmentation using diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11536–11546, 2023.
  * [15] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [16] Christian F Baumgartner, Kerem C Tezcan, Krishna Chaitanya, Andreas M Hötker, Urs J Muehlematter, Khoschy Schawkat, Anton S Becker, Olivio Donati, and Ender Konukoglu.  Phiseg: Capturing uncertainty in medical image segmentation.  In Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part II 22, pages 119–127. Springer, 2019.
  * [17] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27, 2014.
  * [18] Zihang Xu, Zhenghua Xu, Shuo Zhang, and Thomas Lukasiewicz.  Pca: Semi-supervised segmentation with patch confidence adversarial training.  arXiv preprint arXiv:2207.11683, 2022.
  * [19] Wei-Chih Hung, Yi-Hsuan Tsai, Yan-Ting Liou, Yen-Yu Lin, and Ming-Hsuan Yang.  Adversarial learning for semi-supervised semantic segmentation.  arXiv preprint arXiv:1802.07934, 2018.
  * [20] Manoj Kumar Kanakasabapathy, Prudhvi Thirumalaraju, Hemanth Kandula, Fenil Doshi, Anjali Devi Sivakumar, Deeksha Kartik, Raghav Gupta, Rohan Pooniwala, John A Branda, Athe M Tsibris, et al.  Adaptive adversarial neural networks for the analysis of lossy and domain-shifted datasets of medical images.  Nature biomedical engineering, 5(6):571–585, 2021.
  * [21] Yuan Xue, Tao Xu, Han Zhang, L Rodney Long, and Xiaolei Huang.  Segan: Adversarial network with multi-scale l 1 loss for medical image segmentation.  Neuroinformatics, 16:383–392, 2018.
  * [22] Olivier Bernard, Alain Lalande, Clement Zotti, Frederick Cervenansky, Xin Yang, Pheng-Ann Heng, Irem Cetin, Karim Lekadir, Oscar Camara, Miguel Angel Gonzalez Ballester, et al.  Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved?  IEEE transactions on medical imaging, 37(11):2514–2525, 2018.
  * [23] Yucheng Tang, Dong Yang, Wenqi Li, Holger R Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh.  Self-supervised pre-training of swin transformers for 3d medical image analysis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20730–20740, 2022.
  * [24] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021.
  * [25] Athanasios Tragakis, Chaitanya Kaul, Roderick Murray-Smith, and Dirk Husmeier.  The fully convolutional transformer for medical image segmentation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3660–3669, 2023.
  * [26] Hong-Yu Zhou, Jiansen Guo, Yinghao Zhang, Lequan Yu, Liansheng Wang, and Yizhou Yu.  nnformer: Interleaved transformer for volumetric segmentation.  arXiv preprint arXiv:2109.03201, 2021.
  * [27] Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang.  Swin-unet: Unet-like pure transformer for medical image segmentation.  In European conference on computer vision, pages 205–218. Springer, 2022.
  * [28] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou.  Transunet: Transformers make strong encoders for medical image segmentation.  arXiv preprint arXiv:2102.04306, 2021.