# DU-GAN: Generative Adversarial Networks with Dual-Domain U-Net Based
Discriminators for Low-Dose CT Denoising

Zhizhong Huang, , Junping Zhang, ,
Yi Zhang, , and Hongming Shan © 2021 IEEE. Personal use of this material is
permitted. Permission from IEEE must be obtained for all other uses, in any
current or future media, including reprinting/republishing this material for
advertising or promotional purposes, creating new collective works, for resale
or redistribution to servers or lists, or reuse of any copyrighted component
of this work in other works. (Corresponding author: Hongming Shan) Z. Huang
and J. Zhang are with the Shanghai Key Lab of Intelligent Information
Processing and the School of Computer Science, Fudan University, Shanghai
200433, China (email<EMAIL_ADDRESS>jpzhang@fudan.edu.cn). Y. Zhang
is with the College of Computer Science, Sichuan University, Chengdu 610065,
China (e-mail: yzhang@scu.edu.cn). H. Shan is with the Institute of Science
and Technology for Brain-inspired Intelligence and MOE Frontiers Center for
Brain Science, Fudan University, Shanghai, 200433, China, and also with the
Shanghai Center for Brain Science and Brain-inspired Technology, Shanghai
201210, China (e-mail: hmshan@fudan.edu.cn).

###### Abstract

Low-dose computed tomography (LDCT) has drawn major attention in the medical
imaging field due to the potential health risks of CT-associated X-ray
radiation to patients. Reducing the radiation dose, however, decreases the
quality of the reconstructed images, which consequently compromises the
diagnostic performance. Over the past few years, various deep learning
techniques, especially generative adversarial networks (GANs), have been
introduced to improve the image quality of LDCT images through denoising,
achieving impressive results over traditional approaches. GANs-based denoising
methods usually leverage an additional classification network, i.e.
discriminator, to learn the most discriminate difference between the denoised
and normal-dose images and, hence, regularize the denoising model accordingly;
it often focuses either on the global structure or local details. To better
regularize the LDCT denoising model, this paper proposes a novel method,
termed DU-GAN, which leverages U-Net based discriminators in the GANs
framework to learn both global and local difference between the denoised and
normal-dose images in both image and gradient domains. The merit of such a
U-Net based discriminator is that it can not only provide the per-pixel
feedback to the denoising network through the outputs of the U-Net but also
focus on the global structure in a semantic level through the middle layer of
the U-Net. In addition to the adversarial training in the image domain, we
also apply another U-Net based discriminator in the image gradient domain to
alleviate the artifacts caused by photon starvation and enhance the edge of
the denoised CT images. Furthermore, the CutMix technique enables the per-
pixel outputs of the U-Net based discriminator to provide radiologists with a
confidence map to visualize the uncertainty of the denoised results,
facilitating the LDCT-based screening and diagnosis. Extensive experiments on
the simulated and real-world datasets demonstrate superior performance over
recently published methods both qualitatively and quantitatively. Our source
code is made available at https://github.com/Hzzone/DU-GAN.

###### Index Terms:

Convolutional neural network, generative adversarial network, image
translation, low-dose CT denoising, U-Net, uncertainty estimation, artifact
removal.

##  I Introduction

Computed tomography (CT) can provide the cross-sectional images of the
internal body by the x-ray radiation, which is one of the most important
imaging modalities in clinical diagnosis. Although CT plays an essential role
in diagnosing diseases, the widespread use of CT is raising more and more
public concerns towards its safety since CT-related X-ray radiation may cause
unavoidable damage to the health of humans and induce cancers. Consequently,
reducing the radiation dose of CT as low as reasonably achievable (_a.k.a._
ALARA) is a well-accepted principle in CT-related research over the past
decades [1]. The reduction of radiation dose, however, inevitably brings the
noise and artifacts into the reconstructed images, severely compromising the
subsequent diagnosis and other tasks such as LDCT-based lung nodule
classification [2].

A straightforward way to address this issue is to reduce the noise in the LDCT
image [3, 4]. However, it remains a challenging problem due to its ill-posed
nature. In recent years, various deep learning based methods have been
proposed for LDCT denoising [5, 6, 7, 8, 9, 10, 11], achieving impressive
results. There are two key components in designing a denoising model: network
architecture and loss function; the former one can determine the capacity of
the denoising model while the latter one can control how the denoised images
visually look like [7]. Although different network architectures such as 2D
convolutional neural networks (CNNs) [5], 3D CNNs [7, 10], and residual
encoder-decoder CNNs (RED-CNN) [12] have been explored for LDCT denoising,
literature has shown that the loss function is relatively more important than
the network architecture as it has a direct impact on the image quality [7,
13].

One of the most popular loss functions is the mean-squared error (MSE), which
computes the average of the squares of the per-pixel errors between the
denoised and normal-dose images. Although gaining impressive performance in
terms of peak signal-to-noise (PSNR), MSE usually leads to over-smoothened
images, which has been proven to poorly correlate with the human perception of
image quality [14, 15]. In view of this observation, alternative loss
functions such as perceptual loss, $\ell_{1}$ loss, and adversarial loss have
been investigated for LDCT denoising. Among them, adversarial loss has been
shown to be a powerful one as it can dynamically measure the similarity
between the denoised and normal-dose images during the training, which enables
the denoised images to preserve more texture information from normal-dose one.
The computation of the adversarial loss is based on the discriminator, which
is a classification network to learn a representation differentiating the
denoised images from the normal-dose images; it can measure the most
discriminant difference either in a global or local level, depending on that
one unit of the output of discriminator corresponds to the whole image or a
local region. Such a discriminator is prone to forgetting previous difference
because the distribution of synthetics samples shifts as the generator
constantly changes through training, failing to maintain a powerful data
representation to characterize the global and local image difference [16]. As
a result, it often results in the generated images with discontinued and
mottled local structures [17] or images with incoherent geometric and
structural patterns [18]. In addition to the noise, LDCT images may contain
severe streak artifacts caused by photon starvation, which may not be
effectively removed through the loss function solely in the image domain.

To learn a powerful data representation to regularize the denoising model in
the adversarial training, we propose a U-Net [19] based discriminator in the
GANs framework for LDCT denoising, termed DU-GAN, which can simultaneously
learn the global and local difference between the denoised and normal-dose
images in image and gradient domains. More specifically, our proposed
discriminator follows the U-Net architecture including an encoder and a
decoder network, where the encoder encodes the input to a scalar value
focusing on the global structures while the decoder reconstructs a per-pixel
confidence map capturing the changes of local details between the denoised and
normal-dose images. In doing so, it can provide not only the per-pixel
feedback but also the global structural difference to the denoising network.
In addition to the adversarial training in the image domain, we also apply
another U-Net based discriminator in the image gradient domain to alleviate
the artifacts caused by photon starvation and enhance the edge of the denoised
images. Moreover, to regularize the U-Net based discriminator, we introduce
the CutMix data augmentation to mix the denoised and normal-dose images.
Consequently, the U-Net based discriminator can provide radiologists with the
per-pixel outputs as a confidence map to visualize the uncertainty of the
denoised results, which can facilitate radiologists’ screening and diagnosis
when using the denoised LDCT images.

The benefits of the proposed DU-GAN are as follows.

  1. 1.

Unlike existing GAN-based denoising methods that use a classification as the
discriminator, the proposed DU-GAN utilizes a U-Net based discriminator for
LDCT denoising, which can simultaneously learn global and local difference
between the denoised and normal-dose images. Consequently, it can provide not
only the per-pixel feedback but also the global structural difference to the
denoising model.

  2. 2.

In addition to adversarial training in the image domain, the proposed DU-GAN
also performs adversarial training in the image gradient domains, which can
alleviate the streak artifacts caused by photon starvation and enhance the
edge of the denoised images.

  3. 3.

The proposed DU-GAN can provide radiologists with a confidence map visualizing
the uncertainty of the denoised results through the CutMix technique, which
could facilitate radiologists’ screening and diagnosis when using the denoised
LDCT images.

  4. 4.

Extensive experiments on simulated and real-world datasets demonstrate the
effectiveness of the proposed method through both qualitative and quantitative
comparisons.

The remainder of this paper is organized as follows. We briefly survey the
developments of the LDCT denoising methods and generative adversarial networks
in Section II. We present our LDCT denoising framework DU-GAN with dual-domain
U-Net based discriminators, and then introduce the CutMix regularization
technique as well as the network architectures and loss functions in our
framework in Section III, followed by both qualitative and quantitative
comparisons with the state-of-the-art methods on the simulated and real-world
datasets in Section IV. Finally, we conclude this paper in Section V.

##  II Related Work

This section briefly surveys the development of LDCT denoising and generative
adversarial networks.

###  II-A LDCT Denoising

The noise reduction algorithms for LDCT can be summarized into three
categories: 1) sinogram filtration; 2) iterative reconstruction; and 3) image
post-processing. As a significant difference from routine CT, the LDCT
acquires noisy sinogram data from scanner. A straightforward solution is to
perform the denoising process on the sinogram data before image
reconstruction, i.e. sinogram filtration-based methods [20, 21, 22]. Iterative
reconstruction methods combine the statistics of raw data in the sinogram
domain [23, 24] and the prior information in the image domain such as total
variation [25] and dictionary learning [26]; these pieces of generic
information can be effectively integrated in the maximum likelihood and
compressed sensing frameworks. These two categories, however, require the
access to raw data that are typically unavailable from commercial CT scanner.

Figure 1: Overall framework of our proposed DU-GAN. The generator produces
denoised LDCT images, and two independent branches with U-Net based
discriminators perform at the image and gradient domains. The U-Net based
discriminator provides both global structure and local per-pixel feedback to
the generator. Furthermore, the image discriminator encourages the generator
to produce photo-realistic CT images while the gradient discriminator is
utilized for better edge and alleviating streak artifacts caused by photon
starvation.

Different from the previous two categories, image post-processing methods
directly operate on the reconstructed images that are publicly available after
removing patient privacy. Traditional methods such as non-local means [27] and
block-matching 3D [28], however, lead to the loss of some critical structural
details and result in over-smoothened denoised LDCT images. The rapid
development of deep learning techniques has advanced many medical
applications. In LDCT denoising, deep-learning-based models have achieved
impressive results [5, 7, 9, 10, 12, 29]. There are two critical components in
designing a deep-learning-based denoising model: network architecture and loss
function; the former one determines the capacity of a denoising model while
the later one controls how the denoised images visually look like. Although
the literature has proposed several different network architectures for LDCT
denoising such as 2D CNNs [5], 3D CNN [7, 10], RED-CNN [5], and cascaded CNN
[12], the literature has shown that the loss function plays a relatively more
important role than network architecture as it has a direct impact on the
image quality [7, 13]. The simplest loss function is the MSE, which however
has been shown to poorly correlate with the human perception of image quality
[14, 15]. In view of this observation, alternative loss functions such as
perceptual loss, $\ell_{1}$ loss, adversarial loss, or mixed loss functions
have been investigated for LDCT denoising. Among them, adversarial loss has
been shown to be a powerful one as it can dynamically measure the similarity
between the denoised and normal-dose images during the training, which enables
the denoised images to preserve more texture information from normal-dose one.
Adversarial loss reflects either global or local similarity, depending on the
design of discriminator.

Unlike the conventional adversarial loss, the adversarial loss used in this
study is based on a U-Net based discriminator, which can simultaneously
characterize global and local difference between the denoised and normal-dose
images, better regularizing the denoising model. That is, DU-GAN enjoys both
advantages of the per-pixel discriminator capturing the changes at pixel level
and traditional classification discriminator focusing on global structures. In
addition to the adversarial loss in the image domain, the adversarial loss in
the image gradient domain proposed in this paper can alleviate the streak
artifacts caused by photon starvation and enhance the edge of the denoised
images.

###  II-B Generative Adversarial Networks (GANs)

As one of the most hot research topics in recent years, GANs [14] and their
variants have been successfully applied to various tasks [30, 31, 32]. They
typically consist of two networks: 1) a generator learning to capture the data
distribution of training data and produce new samples that are
indistinguishable from the real ones, and 2) a discriminator attempting to
distinguish real samples from fake ones produced by the generator. These two
networks are trained alternatively, ending once the balance is achieved. In
the context of LDCT denoising, the generator aims to produce photo-realistic
denoised results to fool the discriminator while the discriminator tries to
distinguish the real normal-dose CT (NDCT) images and denoised ones. To foster
the stability of training GANs, various variants of GANs have been proposed,
such as Wasserstein GAN (WGAN) [33], WGAN with gradient penalty (WGAN-GP)
[34], and least-squares GANs [35].

In this paper, we adopt the least-squares GANs [35], spectral normalization
[36], and U-Net based discriminator [16] to form the GANs framework for LDCT
denoising. As a significant difference, our DU-GAN performs adversarial
training in both image and gradient domains, which can reduce noise and
alleviate streak artifacts simultaneously. We note that the proposed DU-GAN is
also suitable for other variants of GAN such as WGAN and WGAN-GP.

##  III Methodology

Fig. 1 presents the proposed DU-GAN for LDCT denoising, which contains a
denoising model as generator, and two U-Net based discriminators in both image
and gradient domains. We highlight that the U-Net based discriminator is able
to learn the global and local difference between denoised and normal-dose
images. Next, we present all components, network architecture, and loss
functions in detail, followed by its complexity.

###  III-A The Denoising Process

The denoising process is to learn a generative model $G$ that maps an LDCT
image $\boldsymbol{I}_{\mathrm{LD}}\in\mathbb{R}^{w\times h}$ of size $w\times
h$ to its normal-dose CT (NDCT) counterpart
$\boldsymbol{I}_{\mathrm{ND}}\in\mathbb{R}^{w\times h}$ by removing the noise
in LDCT image. Formally, it can be written as:

$\displaystyle\boldsymbol{I}_{\mathrm{den}}=G(\boldsymbol{I}_{\mathrm{LD}})\approx\boldsymbol{I}_{\mathrm{ND}},$
(1)

where $\boldsymbol{I}_{\mathrm{den}}$ denotes the denoised LDCT image.
Typically, LDCT denoising can be seen as a specific image translation problem.
Therefore, the GANs-based methods [7, 8, 9, 37] utilize the GANs to improve
the visual quality of denoised LDCT images thanks to its strong capability of
GANs in generating high-quality images. Different from the conventional GANs
that take a noise vector to generate an image, our denoising model serves as
the generator that only takes the LDCT image as the input. In this study, we
used the RED-CNN [6] as the denoising model to demonstrate the effectiveness
of the dual-domain U-Net based discriminators in the adversarial training.

Figure 2: The difference between a) traditional classification discriminator,
and b) U-Net based discriminator. U-Net based discriminator extends
traditional one to capture the global and local information simultaneously.

###  III-B Dual-Domain U-Net Based Discriminator

The GANs-based methods [7, 8, 9, 37] for LDCT denoising usually maintain the
competition of GANs under the structural level, whose discriminator
progressively downsamples the input into a scalar value and are trained with
Wasserstein GANs [33, 34], as shown in Fig. 2(a). However, the discriminator
is prone to forgetting previous samples because the distribution of synthetics
samples shifts as the generator constantly changes during training, failing to
maintain a powerful data representation to characterize the global and local
image difference [16, 38].

To address the problems above, we introduce the U-Net based discriminators in
both image and gradient domains.

####  III-B1 U-Net based discriminator in the image domain

To learn a powerful data representation that can characterize both global and
local difference, we design an LDCT denoising framework based on GANs to deal
with LDCT denoising. Traditionally, U-Net contains an encoder, a decoder, and
several skip connections copying the feature-maps from the encoder to the
decoder to preserve high-resolution features, which has demonstrated its
state-of-the-art performance in many semantic segmentation tasks [39, 40] and
image translation tasks [16, 31]. In the context of LDCT denoising, we
highlight that U-Net and its variants are only used as the denoising model,
which have not been explored as the discriminator. We adopt the U-Net to
replace the standard classification discriminator in GANs to have a U-Net
style discriminator that allows the discriminator to maintain both global and
local data representation. Fig. 2(b) details the architecture of U-Net based
discriminator.

Figure 3: Gradients of horizontal and vertical directions from a pair of LDCT
and NDCT images. The streak artifacts can be easily captured in the gradient
domain.

Here, we use $D^{\mathrm{img}}$ to denote the U-Net based discriminator in the
image domain. The encoder of $D^{\mathrm{img}}$,
$D^{\mathrm{img}}_{\mathrm{enc}}$, follows the traditional discriminator that
progressively downsamples the input using several convolutional layers,
capturing the global structure context. On the other hand, the decoder
$D^{\mathrm{img}}_{\mathrm{dec}}$ performs progressive upsampling with skip
connections from encoder $D^{\mathrm{img}}_{\mathrm{enc}}$ in a reverse order,
further enhancing the ability of discriminator to draw the local details of
real and fake samples. Furthermore, the discriminator loss is computed from
the outputs of both $D^{\mathrm{img}}_{\mathrm{enc}}$ and
$D^{\mathrm{img}}_{\mathrm{dec}}$, while the traditional discriminator used in
previous works [7, 8, 37] _only_ classifies the inputs into being real and
fake from the encoder. In doing so, the U-Net based discriminator can provide
more informative feedback to the generator including both local per-pixel and
global structural information. In this paper, we employ the least-squares GANs
[35] rather than conventional GANs [14] for the discriminators to stabilize
the training process and improve the visual quality of denoised LDCT.
Formally, the discriminator loss for $D^{\mathrm{img}}$ from both
$D^{\mathrm{img}}_{\mathrm{enc}}$ and $D^{\mathrm{img}}_{\mathrm{dec}}$ can be
written as:

$\displaystyle\mathcal{L}_{D^{\mathrm{img}}}=$
$\displaystyle\underbrace{\mathbb{E}_{\boldsymbol{I}_{\mathrm{ND}}}\Big{[}D^{\mathrm{img}}_{\mathrm{enc}}\big{(}\boldsymbol{I}_{\mathrm{ND}}\big{)}-1\Big{]}^{2}+\mathbb{E}_{\boldsymbol{I}_{\mathrm{LD}}}\Big{[}D^{\mathrm{img}}_{\mathrm{enc}}\big{(}\boldsymbol{I}_{\mathrm{den}}\big{)}\Big{]}^{2}}_{\text{global\
adversarial}}+$
$\displaystyle\underbrace{\mathbb{E}_{\boldsymbol{I}_{\mathrm{ND}}}\Big{[}D^{\mathrm{img}}_{\mathrm{dec}}\big{(}\boldsymbol{I}_{\mathrm{ND}}\big{)}-1\Big{]}^{2}+\mathbb{E}_{\boldsymbol{I}_{\mathrm{LD}}}\Big{[}D^{\mathrm{img}}_{\mathrm{dec}}\big{(}\boldsymbol{I}_{\mathrm{den}}\big{)}\Big{]}^{2}}_{\text{local\
adversarial}}$ (2)

where 1 is the decision boundary of least-squares GANs.

####  III-B2 U-Net based discriminator in the gradient domain

However, the competition in the image domain alone is only able to force the
generator towards generating photo-realistic denoised LDCT images; it is
insufficient to encourage better edge for keeping the pathological changes of
original NDCT images and alleviate the streak artifacts caused by photon
starvation in LDCT. Previous methods such as [9] measure the different MSE in
the gradient domain, which may be insufficient to enhance the edge as MSE
tends to blur image. To this end, we propose to perform an additional GANs
competition in the gradient domain, where our motivation is presented in Fig.
3. Specifically, the streaks and edge in CT images are highlighted in their
horizontal and vertical gradient magnitudes. Therefore, another branch of the
gradients estimated by a Sobel operator [41] is performed aside the image
branch, which encourages better edge information and alleviates streak
artifacts. Similar to (2), we can define the discriminator loss in the
gradient domain $\mathcal{L}_{D^{\mathrm{grd}}}$, where $D^{\mathrm{grd}}$
represents the discriminator in the gradient domain.

####  III-B3 Dual-domain U-Net based discriminators

Combining the U-Net based discriminators in the image and gradient domains,
two independent GANs competitions are maintained during training. The overall
framework of our proposed LDCT denoising model is shown in Fig. 1. In detail,
the generator is to denoise an LDCT image, which is then fed into two
independent discriminators operating in the image and gradient domains. The
discriminator $D^{\mathrm{img}}$ in the image domain branch penalizes the
generator generating photo-realistic denoised LDCT while the discriminator
$D^{\mathrm{grd}}$ in the gradient domain branch encourages better edge while
alleviating streak artifacts caused by photon starvation. Additionally, the
discriminator in each branch employs a U-Net based architecture to encourage
the generator focusing both global structure and local details, which can also
boost the interpretability of the denoising process with the per-pixel
confidence map output by $D^{\mathrm{img}}_{\mathrm{dec}}$ and
$D^{\mathrm{grd}}_{\mathrm{dec}}$. Finally, the dual-domain U-Net based
discriminator loss can be defined as follows:

$\displaystyle\mathcal{L}_{D^{\mathrm{dud}}}=\mathcal{L}_{D^{\mathrm{img}}}+\mathcal{L}_{D^{\mathrm{grd}}}.$
(3)

###  III-C CutMix Regularization

The discriminator $D^{\mathrm{img}}$ suffers from the decreasing capability in
recognizing the local differences between real and fake samples as the
training goes, which may unexpectedly harm the denoising performance. Besides,
the discriminator is supposed to focus on structure change at the global level
and local details at the per-pixel level. To address these issues, we adopt
the CutMix augmentation technique to regularize the discriminator inspired by
[16, 42], which can empower the discriminator to learn the intrinsic
difference between real and fake samples. Specifically, CutMix technique
generates a new training image from two images by cutting patches from the one
and pasting them to another. We define this augmentation technique in the
context of LDCT denoising as follows:

$\displaystyle\boldsymbol{I}_{\mathrm{mix}}=$
$\displaystyle\mathrm{mix}(\boldsymbol{I}_{\mathrm{ND}},\boldsymbol{I}_{\mathrm{den}},\boldsymbol{M})$
$\displaystyle=$
$\displaystyle\boldsymbol{M}\odot\boldsymbol{I}_{\mathrm{ND}}+(1-\boldsymbol{M})\odot\boldsymbol{I}_{\mathrm{den}},$
(4)

where $\boldsymbol{M}\in\\{0,1\\}^{w\times h}$ is a binary mask controlling
how to mix the NDCT and denoised images, and $\odot$ represents the element-
wise multiplication.

Figure 4: Illustration of the CutMix regularization for the U-Net based
discriminator, i.e., $D^{\mathrm{img}}$. We randomly sample the ratio $r$ and
top-left coordinates of the bounding box to form the mask $\boldsymbol{M}$
controlling where to crop. $D^{\mathrm{img}}_{\mathrm{dec}}$ is able to
effectively capture the pixel differences between NDCT and denoised LDCT
images while $D^{\mathrm{img}}_{\mathrm{enc}}$ can predict the mixed ratio.
Note that the blue color of $D^{\mathrm{img}}_{\mathrm{dec}}$ indicates the
lower confidence score. Therefore, a well-trained discriminator can provide
radiologists with a confidence map showing the uncertainty of the denoised
results.

The mixed samples should be regarded as fake samples globally by the encoder
$D^{\mathrm{img}}_{\mathrm{enc}}$ since the CutMix operation has destroyed the
global context of NDCT image; otherwise the CutMix may be introduced to
denoised LDCT images during the training of GANs, causing undesirable
denoising. Similarly, the $D^{\mathrm{img}}_{\mathrm{dec}}$ should be able to
recognize the mixed area to provide the generator with accurate per-pixel
feedback. Therefore, the regularization loss of CutMix can be formulated as:

$\displaystyle\mathcal{L}_{\mathrm{reg}}=\mathbb{E}_{\boldsymbol{I}_{\mathrm{mix}}}\Big{[}[D^{\mathrm{img}}_{\mathrm{enc}}(\boldsymbol{I}_{\mathrm{mix}})]^{2}+[D^{\mathrm{img}}_{\mathrm{dec}}(\boldsymbol{I}_{\mathrm{mix}})-\boldsymbol{M}]^{2}\Big{]},$
(5)

where $\boldsymbol{M}$ used in CutMix also serves as the ground truth for
$D^{\mathrm{img}}_{\mathrm{dec}}$.

Furthermore, to penalize the outputs of discriminator to be consistent with
the per-pixel predictions after the CutMix operation, we further introduce
another consistency loss following [16] to regularize the discriminator with
CutMix operation, which can be written as:

$\displaystyle\mathcal{L}_{\mathrm{con}}=\|D^{\mathrm{img}}_{\mathrm{dec}}(\boldsymbol{I}_{\mathrm{mix}})-\mathrm{mix}(D^{\mathrm{img}}_{\mathrm{dec}}(\boldsymbol{I}_{\mathrm{ND}}),D^{\mathrm{img}}_{\mathrm{dec}}(\boldsymbol{I}_{\mathrm{den}}),\boldsymbol{M})\|_{F}^{2},$
(6)

where $\|\cdot\|_{F}$ represents the Frobenius norm.

During training, the binary mask $\boldsymbol{M}$ is generated following the
same pipeline as [42, 43]. Specifically, we first sample the combination ratio
$r$ from beta distribution $\mathrm{Beta}(1,1)$ and then uniformly sample the
top-left coordinates of the bounding box of cropping regions from
$\boldsymbol{I}_{\mathrm{ND}}$ to $\boldsymbol{I}_{\mathrm{den}}$, with
preserving the $r$ ratio. Similar to [42, 43], we employ a probability
$p_{\mathrm{mix}}$ to control whether to apply the CutMix regularization
technique for each mini-batch samples, which is empirically set to $0.5$. Fig.
4 presents the visual results of $D^{\mathrm{img}}$ with CutMix regularization
technique. It can be observed that the outputs of
$D^{\mathrm{img}}_{\mathrm{dec}}$ are the spatial combination of the real and
generated patches with respect to the real/fake classification score.
Therefore, the results have demonstrated the strong discriminative capability
of the U-Net based discriminator in accurately learning per-pixel differences
between real and generated samples, even though they are cut and mixed
together to fool the discriminator. Besides learning the per-pixel local
details, $D^{\mathrm{img}}_{\mathrm{enc}}$ can accurately predict the
proportion of real patches, i.e., the mixed ratio, as it is to focus on the
global structures.

###  III-D Network Architecture

As we described above, our proposed method follows the GANs framework to
optimize the generator effectively for LDCT denoising, with the U-Net based
discriminator focusing on both global structures and local details, and an
extra gradient branch encouraging better boundaries and details. In this
subsection, we describe the network architectures of the generator and U-Net
based discriminator.

####  III-D1 RED-CNN based generator

In this paper, we employ RED-CNN [6] as the generator of our framework for
LDCT denoising since this paper mainly focuses on the adversarial loss from
dual-domain U-Net based discriminators. The main difference from [6] is that
our framework is optimized in GANs manner, while the vanilla RED-CNN suffers
the problem of over-smoothened LDCT images with MSE. Specifically, RED-CNN
employs the U-Net architecture but removes the downsampling/upsampling
operations to prevent information loss. We stack 10 (de)convolutional layers
at both encoder and decoder, each of which has 32 filters for the sake of the
computation cost, followed by a ReLU activation function. There are in total
10 residual skip connections. It is important to note that although RED-CNN is
adopted as the generator in our framework, the proposed method can be also
adapted to other GANs-based methods such as CPCE [7] and WGAN-VGG [8] with
only changing the discriminators.

####  III-D2 U-Net based discriminator

As detailed in Section III-B, there are two independent discriminators in both
image and gradient domains, each of which follows a U-Net architecture.
Specifically, $D_{\mathrm{enc}}$ has 6 downsampling ResBlocks [44] with
increasing number of filters; i.e. 64, 128, 256, 512, 512, and 512. At the
bottom of $D_{\mathrm{enc}}$, a fully-connected layer is used to output the
global confidence score. Similarly, $D_{\mathrm{dec}}$ used the same number of
ResBlocks in a reverse order to process the bilinearly upsampled features and
the skip residuals of the same resolution, followed by a $1\times 1$
convolutional layer to output the per-pixel confidence map. Most importantly,
a spectral normalization layer [36] and a Leaky ReLU activation with a slope
of 0.2 for negative input follow each convolutional layer of $D$ except the
last one.

We note that the network architectures of the generator and discriminator were
proposed in literature; we did not propose a new network architecture to
achieve the performance gain. One of our key contributions is to use the U-net
as the discriminator in dual-domain to capture both local details and global
structures for LDCT denoising.

###  III-E Loss Functions

####  III-E1 Adversarial loss

Here we employ the sum of these two branches as the adversarial loss, which is
defined in the context of least-squares GANs as follows:

$\displaystyle\mathcal{L}_{\mathrm{adv}}=\mathbb{E}\Big{[}$
$\displaystyle\underbrace{[D^{\mathrm{img}}_{\mathrm{enc}}(\boldsymbol{I}_{\mathrm{den}}\big{)}-1]^{2}+[D^{\mathrm{img}}_{\mathrm{dec}}(\boldsymbol{I}_{\mathrm{den}})-1]^{2}}_{\text{image\
domain}}+$ (7)
$\displaystyle\underbrace{[D^{\mathrm{grd}}_{\mathrm{enc}}(\nabla(\boldsymbol{I}_{\mathrm{den}}))-1]^{2}+[D^{\mathrm{grd}}_{\mathrm{dec}}(\nabla(\boldsymbol{I}_{\mathrm{den}}))-1]^{2}}_{\text{gradient\
domain}}\Big{]},$

where $\nabla$ denotes the Sobel operator to obtain the image gradient.

####  III-E2 Pixel-wise loss

To encourage the generator output the denoised LDCT images that match the NDCT
images with both pixel level and gradient level, we adopt an pixel-wise loss
between the NDCT images and denoised LDCT images, which includes a pixel loss
and gradient loss for each branch as shown in Fig. 1. The additional gradient
loss can encourage to better preserve edge information at the pixel level. The
two losses can be written as:

$\displaystyle\mathcal{L}_{\mathrm{img}}$
$\displaystyle=\mathbb{E}_{(\boldsymbol{I}_{\mathrm{LD}},\boldsymbol{I}_{\mathrm{ND}})}\Big{\|}\boldsymbol{I}_{\mathrm{den}}-\boldsymbol{I}_{\mathrm{ND}}\Big{\|}^{2}_{F},$
(8) $\displaystyle\mathcal{L}_{\mathrm{grd}}$
$\displaystyle=\mathbb{E}_{(\boldsymbol{I}_{\mathrm{LD}},\boldsymbol{I}_{\mathrm{ND}})}\Big{|}\nabla(\boldsymbol{I}_{\mathrm{den}})-\nabla(\boldsymbol{I}_{\mathrm{ND}})\Big{|}.$
(9)

Note that we employ the mean squared error in pixel level rather than the
feature level using pretrained model [7, 8] for the sake of computation cost,
and the absolute mean error in gradient level as the gradients is much sparser
than pixels.

####  III-E3 Final loss

To encourage the generator to generate photo-realistic denoised LDCT images
with better edge information and alleviate streak artifacts, the final loss
function to optimize the generator $G$ is expressed as:

$\displaystyle\mathcal{L}_{G}=\lambda_{\mathrm{adv}}\mathcal{L}_{\mathrm{adv}}+\lambda_{\mathrm{img}}\mathcal{L}_{\mathrm{img}}+\lambda_{\mathrm{grd}}\mathcal{L}_{\mathrm{grd}},$
(10)

where $\lambda_{\mathrm{adv}}$, $\lambda_{\mathrm{img}}$ and
$\lambda_{\mathrm{grd}}$ are the weights for $\mathcal{L}_{\mathrm{adv}}$,
$\mathcal{L}_{\mathrm{img}}$ and $\mathcal{L}_{\mathrm{grd}}$, respectively.
Here, we empirically determine the hyper-parameters in a sequential way.
First, with only pixel-wise loss, our proposed DU-GAN reduces to RED-CNN since
discriminators are not included during training. Although fast convergence,
only optimizing the MSE loss leads to over-smoothing and blurred results,
causing the loss of structural details. We set $\lambda_{\mathrm{img}}$ to be
1. Second, we tune the $\lambda_{\mathrm{adv}}$ to control the importance of
adversarial loss to capture the texture details. We start from a small value
for $\lambda_{\mathrm{adv}}$, and then gradually increase the importance of
the adversarial loss, and visualize the denoising results. Finally, we tune
$\lambda_{\mathrm{grd}}$ to capture edge information with a large value as the
gradients are much sparser than pixels.

The discriminators $D^{\mathrm{img}}$ and $D^{\mathrm{grd}}$ are optimized by
minimizing the following mixed loss:

$\displaystyle\mathcal{L}_{D}=\mathcal{L}_{D^{\mathrm{dud}}}+\mathcal{L}_{\mathrm{reg}}+\mathcal{L}_{\mathrm{con}}.$
(11)

Note that we employ the same loss function in (11) to optimize both
$D^{\mathrm{img}}$ and $D^{\mathrm{grd}}$ but they are independent to each
other and $D^{\mathrm{grd}}$ has an additional Sobel operator to compute the
gradients.

###  III-F Complexity of DU-GAN

Next, we discuss the complexity of the DU-GAN in terms of hyper-parameters and
computational costs. First, compared to MSE-based methods that directly
optimize mean-squared loss, DU-GAN is a GANs-based method that introduces an
additional adversarial loss to the training process. Compared to vanilla GANs-
based methods with a traditional classification discriminator, DU-GAN proposed
to use the U-Net based discriminator to focus on both local details and global
structures. Furthermore, DU-GAN also introduces another gradient branch along
the original pixel branch to encourage clear boundaries. Therefore, there is
only one extra hyper-parameter to control the importance of the gradient
branch. Second, the main computational costs of DU-GAN come from the proposed
U-Net based discriminator and gradient branch. However, such computational
costs are affordable considering the better denoising quality and performance
for our DU-GAN and only happen during the training stage. That is, the
inference efficiency is still the same as the traditional ones.

##  IV Experiments

This section presents the datasets, implementation details, qualitative and
quantitative evaluations, uncertainty visualization, and ablation study.

Figure 5: Transverse CT images from the Mayo-10%, Mayo-25%, and Piglet-5%: (a)
LDCT; (b) NDCT; (c) RED-CNN; (d) WGAN-VGG; (e) CPCE-2D; (f) Q-AE; (g) CNCL;
and (h) DU-GAN (ours). Zoomed ROI of the red rectangle is shown below the
full-size one. The display window is [-160, 240] HU for better visualization.
Red arrow indicates low attenuation lesion. Green arrow indicates the white
edge artifacts shown in some baseline algorithms while not shown in our
method.

###  IV-A Datasets

####  IV-A1 Simulated dataset

The LDCT dataset used in this study was originally for  _the 2016 NIHAAPM-Mayo
Clinic Low-Dose CT Grand Challenge_ , and lately released in [45]. It provides
scans from three regions of the body with different _simulated_ low doses;
i.e., head with $25\%$ of normal-dose, abdomen with $25\%$, and chest with
$10\%$. In our experiments, we used the 25% abdomen and 10% chest datasets,
named Mayo-25% and Mayo-10%, respectively. We evaluated our method on abdomen
scans for comparisons with most previous works, and conducted experiments on
chest scans since $10\%$ of normal-dose at chest is rather challenging
compared to the $25\%$ of normal-dose at abdomen. For each dataset, we
randomly select 20 patients for training and another 20 patients for testing;
no identity overlapping between training and testing. In detail, 300K and 64K
image patches were randomly selected from each set. For more information about
this dataset, please refer to [45].

Figure 6: Transverse neck CT images from the Mayo-10%: (a) LDCT; (b) NDCT; (c) RED-CNN; (d) WGAN-VGG; (e) CPCE-2D; (f) Q-AE; (g) CNCL; and (h) DU-GAN (ours). Zoomed ROI of the red rectangle is shown in the second row. The display window is [-160, 240] HU for better visualization. Red arrow indicates bone area while green arrow indicates a small structure. TABLE I: Quantitative comparisons of different methods on the testing sets of two simulated datasets and one real-world dataset. The best results among MSE- and GAN-based methods are marked in bold. | Method | Mayo-10% |  | Mayo-25% |  | Piglet-5%
---|---|---|---|---|---|---
| PSNR$\uparrow$ | RMSE$\downarrow$ | SSIM$\uparrow$ |  | PSNR$\uparrow$ | RMSE$\downarrow$ | SSIM$\uparrow$ |  | PSNR$\uparrow$ | RMSE$\downarrow$ | SSIM$\uparrow$
| LDCT | 14.6382 | 0.1913 | 0.6561 |  | 31.5517 | 0.0283 | 0.8639 |  | 28.7279 | 0.0395 | 0.8587
MSE-based | RED-CNN [6] | 23.1388 | 0.0721 | 0.7249 |  | 34.5740 | 0.0196 | 0.9236 |  | 26.9691 | 0.0450 | 0.9318
Q-AE [46] | 21.3149 | 0.0884 | 0.7045 |  | 34.6477 | 0.0197 | 0.9215 |  | 29.7081 | 0.0331 | 0.9317
GAN-based | WGAN-VGG [8] | 20.3922 | 0.0992 | 0.7029 |  | 33.2910 | 0.0226 | 0.9092 |  | 30.3787 | 0.0318 | 0.9232
CPCE-2D [7] | 20.1435 | 0.0899 | 0.7295 |  | 33.0612 | 0.0232 | 0.9125 |  | 28.5329 | 0.0379 | 0.9211
CNCL [47] | 21.8964 | 0.0852 | 0.7110 |  | 32.4967 | 0.0243 | 0.9048 |  | 28.5673 | 0.0383 | 0.9132
| DU-GAN (ours) | 22.3075 | 0.0802 | 0.7489 |  | 34.6186 | 0.0196 | 0.9196 |  | 29.8598 | 0.0325 | 0.9345

####  IV-A2 Real-world dataset

The real-world dataset from [37] includes 850 CT scans of a deceased piglet
obtained by a GE scanner (Discovery CT750 HD). The dataset provides CT scans
of the normal-dose, 50%, 25%, 10% and 5% dose with a size of $512\times 512$,
708 of which is served for training while the left for testing. We evaluated
our method on 5% low-dose CTs as it is the most challenging dose, where the
dataset is named Piglet-5%. We randomly selected 60K and 12K image patches
from training and testing sets, respectively. For more information about this
dataset, please refer to [37].

###  IV-B Implementation Details

Following [7, 8, 46], we employed the image patches with a size of $64\times
64$ and a window of $[-300,300]$ to train all models with emphasis on tissue
CT window, which are then directly applied to the whole image for
visualization and testing. Note that we excluded those image patches that were
mostly air. During training, all images are linearly normalized to $[0,1]$.

During training, we trained the model with a maximum of 100K iterations and
with a mini-batch of size 64 on one NVIDIA V100 GPU. All networks in the
proposed framework are initialized with He initialization [48], and optimized
by Adam optimization method [49] with a fixed learning rate of $10^{-4}$. The
hyperparameters in the loss functions were empirically set as follows:
$\lambda_{\mathrm{adv}}$ was $0.1$; $\lambda_{\mathrm{img}}$ was $1$; and
$\lambda_{\mathrm{grd}}$ was $20$. We implemented four deep-learning-based
methods including RED-CNN [6], WGAN-VGG [8], CPCE-2D [7], Q-AE [46], and CNCL
[47] with the reference of official source code.

###  IV-C Qualitative Evaluations

To demonstrate the effectiveness of the proposed method in generating photo-
realistic denoised results with faithful details, Fig. 5 showcases the
representative results from three different datasets while Fig. 6 presents the
results of one neck CT slice with strong streak artifacts. The regions-of-
interest (ROIs) marked by the red rectangles are zoomed below, respectively.

Figure 7: Uncertainty visualization of applying the trained discriminator to
the outputs of different methods: (a) LDCT; (b) NDCT; (c) RED-CNN; (d) WGAN-
VGG; (e) CPCE-2D; (f) Q-AE; (g) CNCL; and (h) DU-GAN (ours). The display
window is [-160, 240] HU for better visualization. Note that the blue color of
$D^{\mathrm{img}}_{\mathrm{dec}}$ indicates the lower confidence score while
red color indicates higher confidence score.

All methods present visually well denoised results to some degrees. However,
RED-CNN and Q-AE over-smoothed and blurred the LDCT images as they are
optimized by the MSE loss that tends to average the results, causing the loss
of structural details. Although WGAN-VGG and CPCE-2D have greatly improved the
visual fidelity, as expected, due to the use of adversarial loss, minor streak
artifacts can still be observed since their traditional classification
discriminator only provide the generator with global structure feedback.
Besides, they employed the perceptual loss in the high-level feature space to
suppress the blurriness resulting from MSE loss. The perceptual loss, however,
can only preserve the structures of NDCT images since some local details may
be lost after processed by a pre-trained model. For example, the low
attenuation lesions in Fig. 5, and the bones in Fig. 6 are less clear by WGAN-
VGG and CPCE-2D while they can be easily observed in NDCT as well as the
results of our methods. Most importantly, the small structures with their
boundaries are consistently preserved with a clear visual fidelity. This
benefits from the well-designed dual-domain U-Net based discriminators, which
can provide feedback of both global structures and local details to the
generator, compared to the traditional classification discriminator used in
WGAN-VGG and CPCE-2D with only structure information. Besides, the gradient
domain branch can also encourage the denoising model to better preserve edge
information.

Beyond encouraging better edge, Fig. 6 also demonstrates its impressive
performance in dealing with the LDCT images with strong streak artifacts
caused by photon starvation. Compared to the baseline methods that produce
strongly blurry and ghosted denoised results, our method can effectively
address this problem in the following aspects:

  1. 1.

streak artifacts can be easily detected by the gradient domain branch; and

  2. 2.

once detected, the dual-domain U-Net discriminators can fill the occlusion
area by adversarial training to alleviate the impact of streak artifacts.

In summary, all of these results further validate the superiority of our
methods.

###  IV-D Quantitative Evaluations

For quantitative evaluations, we adopted three widely-used metrics including
peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and root mean
square error (RMSE). More specifically, PSNR and RMSE measure the denoising
performance at pixel level while SSIM computes the structural similarity
within a window. Table I presents the results of different methods. First,
RED-CNN and Q-AE are MSE-based denoising methods as they are directly trained
with solely MSE loss. Although they achieve better PSNR and RMSE results, the
visual results in Figs. 5 and 6 confirm that MSE-based methods produce over-
smoothed results compared to the NDCT images, leading to lose of structural
information [7, 8, 50]. Note that the over-smoothed denoising results lead to
a lower SSIM score. Second, WGAN-VGG, CPCE-2D, and our DU-GAN are GAN-based
methods. CPCE-2D performed better than WGAN-VGG due to the conveying path
since WGAN-VGG has to reconstruct the denoised results from the input LDCT
images. Obviously, our method performs the best in terms of SSIM score with
high visual fidelity while the PSNR and RMSE are also better than WGAN-VGG and
CPCE-2D, indicating the superior denoising performance of our method while
better structural fidelity.

Though DU-GAN used the same network architecture of the RED-CNN as the
denoising model, their qualitative and quantitative difference directly comes
from the adversarial training and dual-domain U-Net based discriminators. The
results of DU-GAN preserve more structural details that are important for
diagnosis, at the cost of compromising the quantitative metrics such as PSNR
and RMSE. We note that PSNR and RMSE are pixel-wise metrics, poorly
correlating with human perception of image quality [14].

###  IV-E Uncertainty Visualization

Fig. 4 shows the proposed discriminator with the U-Net architecture and CutMix
regularization can robustly learn the per-pixel differences of local details
between NDCT and denoised LDCT images by the decoder and also focus on the
global structures by the encoder. With this well-trained discriminator, we can
provide radiologist with a confidence map showing the uncertainty of the
denoised results, since it is to learn the distribution of real samples, i.e.,
NDCT images. Therefore, we directly applied the trained discriminator
$D^{\mathrm{img}}$ to the LDCT, NDCT, and the denoised LDCT images of
different methods.

Fig. 7 shows the uncertainty visualization. Obviously, the discriminator can
accurately distinguish the LDCT from NDCT images, on both global score and
per-pixel confidence. As both RED-CNN and Q-AE over-smoothen the LDCT images,
the abdomen area of transverse CT image becomes better than LDCT images on the
confidence map, according to the results of $D^{\mathrm{img}}_{\mathrm{dec}}$.
This also explains why RED-CNN and Q-AE have the lowest global score of
$D^{\mathrm{img}}_{\mathrm{enc}}$, which indicates that the discriminator can
robustly detect the blurriness in the CT images. Furthermore, although CPCE-2D
can produce more clear denoised results than RED-CNN, the streak artifacts
significantly compromised the quality of the denoised results. Similarly, the
WGAN-VGG has learned more local details than CPCE-2D but it still cannot
handle the impact of the streak artifacts. On the contrary, the proposed
method can produce the most photo-realistic denoised results with the highest
global score. Compared to the traditional classification discriminator used in
CPCE-2D and WGAN-VGG, our DU-GAN can provide the generator with the per-pixel
feedback by learning the local detail differences. It can be seen from the
per-pixel of $D^{\mathrm{img}}_{\mathrm{dec}}$. In other words, we achieve a
more smooth per-pixel confidence, indicating that the discriminator cannot
distinguish the real and fake samples at the per-pixel level.

###  IV-F Ablation Study

In this subsection, we conducted the ablation study of our method to fully
explore the proposed method in terms of the importance of different
components, the architectures of discriminator, and the different patch sizes.
The ablation study was done on the testing set of Mayo-10% dataset, which
includes a total of 6,590 slices from 20 patients.

####  IV-F1 Components analysis

We investigate the impact of the U-Net based discriminator in the image
domain, CutMix regularization, and dual-domain training (i.e., with gradient
branch) by gradually applying them to the baseline method. Similar to WGAN-VGG
and CPCE-2D, the baseline method only includes the traditional classification
discriminator with the same hyperparameters for fair comparison.

TABLE II: Ablation study of component analysis. Our method is the baseline method with U-Net discriminator in the image domain, CutMix regularization, and the U-Net discriminator in the gradient domain (dual-domain). The best results are marked in bold. Method | PSNR$\uparrow$ | RMSE$\downarrow$ | SSIM$\uparrow$
---|---|---|---
Baseline | 21.4988 | 0.0871 | 0.7365
\+ U-Net Based Discriminator | 22.1214 | 0.0816 | 0.7454
\+ CutMix Regularization | 21.7894 | 0.0844 | 0.7477
Ours (+ Dual-Domain) | 22.3075 | 0.0802 | 0.7489

Table II presents the quantitative results for ablation study. First,
replacing the traditional classification discriminator with a U-Net based
discriminator can simultaneously provide the generator with both global
structure and local per-pixel feedback, which leads to a significant increase
in terms of SSIM. Second, when we further use CutMix technique to regularize
the U-Net based discriminator, the mixed samples can boost the discriminant
capacity of discriminator and make discriminator more focus on the local
details, leading to the increased SSIM score and a slightly decreased PSNR and
RMSE. Last, further adding the U-net based discriminator in the gradient
domain into the method above forming the dual-domain training yields our
method. Specifically, the additional gradient domain training can help our
method remove the streak artifacts and encourage more clear edge in the
denoised LDCT images. As a result, it can effectively improve all metrics
including the PSNR and RMSE in pixel space and SSIM in visual similarity.

TABLE III: Ablation study of different discriminators on testing set of Mayo-10% dataset. The best results are marked in bold. Method | PSNR$\uparrow$ | RMSE$\downarrow$ | SSIM$\uparrow$
---|---|---|---
Patch | 21.4988 | 0.0871 | 0.7365
Global | 22.6810 | 0.0760 | 0.7262
Pixel | 23.1102 | 0.0724 | 0.7343
U-Net | 22.1214 | 0.0816 | 0.7454

####  IV-F2 Architectures of discriminator

Since the architectures of discriminator play a critical role in the training
of GANs, it is worthwhile studying the advantage of the U-Net based
discriminator over other classical discriminator architectures such as patch
discriminator [31], pixel discriminator [31], and traditional global
discriminator. Compared to traditional classification discriminator that
classifies the real and fake samples at image level, patch discriminator
focuses on the image patches. Due to the patch training of low-dose CT
denoising, this discriminator architecture can be seen as the patch
discriminator. The discriminator with seven convolutional layers and one
fully-connected layer is regarded as the global discriminator. On the other
hand, the pixel discriminator [31] contains 7 $1\times 1$ convolutional layers
to penalizes the generator at per-pixel level. For fair comparisons, we
trained patch and pixel discriminator with image patches and trained the
global discriminator with the whole images with the size of $512\times 512$,
respectively. Table III shows that the combination of global and pixel
information in U-Net based discriminator produces the best SSIM score. This
indicates the advantage of U-net-based discriminator for LDCT denoising over
other classical discriminator architectures such as patch discriminator, pixel
discriminator, and traditional global discriminator. Instead of pixel
discriminator only capturing per-pixel difference and traditional
classification discriminator only focusing on global structure, the U-Net
based discriminator has the advantages of both worlds, yielding better
quantitative results and denoising quality.

TABLE IV: Ablation study of patch sizes on the testing set of Mayo-10% dataset. The best results are marked in bold. Size | PSNR$\uparrow$ | RMSE$\downarrow$ | SSIM$\uparrow$
---|---|---|---
64$\times$64 | 22.3075 | 0.0802 | 0.7489
128$\times$128 | 22.0218 | 0.0826 | 0.7479
256$\times$256 | 21.8478 | 0.0843 | 0.7467
512$\times$512 | 21.7254 | 0.0855 | 0.7441

####  IV-F3 Patch size

Due to the U-Net architecture of the discriminator, it is also important to
analyze the influence of the patch size during training. However, it is very
difficult to directly train the denoising model from scratch. Therefore, we
trained our model with the image size of $64\times 64$, $128\times 128$,
$256\times 256$, and $512\times 512$, and we fine-tuned the generator based on
the model trained on previous smaller size. Table IV shows that a small patch
size can achieve better performance because the larger patch sizes may
introduce training difficulties with less training samples.

##  V Discussion and Conclusion

In this paper, we proposed a novel DU-GAN for LDCT denoising. The introduced
U-Net based discriminator can not only provide the per-pixel feedback to the
denoising network but also focus on the global structure. We further add an
extra U-Net based discriminator into the gradient domain, which can enhance
the edge information and alleviate the streak artifacts caused by photon
starvation. We also examined that the CutMix technique can boost the training
of discriminator, which can provide the radiologists with a confidence map on
the uncertainty of the denoised results. Extensive experiments demonstrated
the effectiveness of the proposed method through visual comparison and
quantitative comparison.

Although a different architecture or model size could certainly affect
results, our DU-GAN has demonstrated its generalization ability on two
simulated low-dose CT datasets of different doses and one real-world dataset,
with the same network architecture and hyperparameters. Our ablation study
validates each component and their relative importance of all components
should be consistent on a new dataset, which indicates that our DU-GAN can be
easily adapted to different scenarios.

We acknowledge some limitations in this work. First, we used the qualitative
and quantitative comparisons to evaluate the image quality. A human reader
study may be needed to further validate its potential in clinical diagnosis,
though there are significant difference between the proposed and other
baseline methods. Second, the U-net based discriminator can provide
radiologists with a confidence map of the denoised images. How this helps
radiologists in clinical routine could be examined with specific tasks such as
liver lesion diagnosis, which can be further studied as a future direction.
Third, DU-GAN could introduce slightly more computational cost during training
since it employs the U-Net as the discriminator and adopts a dual-domain
training strategy. However, we emphasize that the extra computational cost is
relatively affordable as DU-GAN trains the whole framework based on the
$64\times 64$ image patches instead of the original image size $512\times
512$. Therefore, we believe that the computational cost can be significantly
reduced. We note that the extra computational cost only happens for the
training stage. That is, the inference efficiency is still the same as the
traditional one as the the dual-domain discriminators are not involved during
testing stage. Finally, we only validated DU-GAN with two dose levels in this
paper and it is worth further validating DU-GAN with lower radiation dose.

In conclusion, the proposed DU-GAN achieves better denoising performance than
other GAN-based models and has great potential for clinical use with
uncertainty visualization.

## References

  * [1] N. B. Shah and S. L. Platt, “ALARA: is there a cause for alarm? reducing radiation risks from computed tomography scanning in children,” _Current Opinion Pediatrics_ , vol. 20, no. 3, pp. 243–247, 2008.
  * [2] Y. Lei, Y. Tian, H. Shan, J. Zhang, G. Wang, and M. K. Kalra, “Shape and margin-aware lung nodule classification in low-dose CT images via soft activation mapping,” _Med. Image Anal._ , vol. 60, p. 101628, 2020.
  * [3] F. Attivissimo, G. Cavone, A. M. L. Lanzolla, and M. Spadavecchia, “A technique to improve the image quality in computer tomography,” _IEEE Trans. Instrum. Meas._ , vol. 59, no. 5, pp. 1251–1257, 2010.
  * [4] G. Wang, J. C. Ye, and B. De Man, “Deep learning for tomographic image reconstruction,” _Nat. Mach. Intell._ , vol. 2, no. 12, pp. 737–748, 2020\.
  * [5] H. Chen, Y. Zhang, W. Zhang, P. Liao, K. Li, J. Zhou, and G. Wang, “Low-dose CT via convolutional neural network,” _Biomed. Opt. Express_ , vol. 8, no. 2, pp. 679–694, 2017.
  * [6] H. Chen _et al._ , “Low-dose CT with a residual encoder-decoder convolutional neural network (RED-CNN),” _IEEE Trans. Med. Imaging_ , vol. 36, no. 12, pp. 2524–2535, 2017.
  * [7] H. Shan _et al._ , “3-D convolutional encoder-decoder network for low-dose CT via transfer learning from a 2-D trained network,” _IEEE Trans. Med. Imaging_ , vol. 37, no. 6, pp. 1522–1534, 2018.
  * [8] Q. Yang _et al._ , “Low dose CT image denoising using a generative adversarial network with Wasserstein distance and perceptual loss,” _IEEE Trans. Med. Imaging_ , vol. 37, no. 6, pp. 1348–1357, 2018.
  * [9] H. Shan _et al._ , “Competitive performance of a modularized deep neural network compared to commercial algorithms for low-dose CT image reconstruction,” _Nat. Mach. Intell._ , vol. 1, no. 6, pp. 269–276, 2019\.
  * [10] J. M. Wolterink, T. Leiner, M. A. Viergever, and I. Isgum, “Generative adversarial networks for noise reduction in low-dose CT,” _IEEE Trans. Med. Imaging_ , vol. 36, no. 12, pp. 2536–2545, 2017.
  * [11] G. Wang, J. C. Ye, K. Mueller, and J. A. Fessler, “Image reconstruction is a new frontier of machine learning,” _IEEE Trans. Med. Imaging_ , vol. 37, no. 6, pp. 1289–1296, 2018.
  * [12] D. Wu, K. Kim, G. E. Fakhri, and Q. Li, “A cascaded convolutional nerual network for x-ray low-dose CT image denoising,” _arXiv preprint arXiv:1705.04267_ , 2017.
  * [13] H. Zhao, O. Gallo, I. Frosio, and J. Kautz, “Loss functions for image restoration with neural networks,” _IEEE Trans. Comput. Imaging_ , vol. 3, no. 1, pp. 47–57, 2016.
  * [14] I. Goodfellow _et al._ , “Generative adversarial nets,” in _Proc. Adv. Neural Inf. Process. Syst._ , 2014, pp. 2672–2680.
  * [15] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE Trans. Image Process._ , vol. 13, no. 4, pp. 600–612, 2004.
  * [16] E. Schonfeld, B. Schiele, and A. Khoreva, “A U-Net based discriminator for generative adversarial networks,” in _Proc. IEEE Conf. Comp. Vis. Patt. Recogn._ , 2020, pp. 8207–8216.
  * [17] C. H. Lin, C.-C. Chang, Y.-S. Chen, D.-C. Juan, W. Wei, and H.-T. Chen, “COCO-GAN: generation by parts via conditional coordinating,” in _Proc. IEEE Int. Conf. Comput. Vis._ , 2019, pp. 4512–4521.
  * [18] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena, “Self-attention generative adversarial networks,” in _Proc. Int. Conf. Mach. Learn._ PMLR, 2019, pp. 7354–7363.
  * [19] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _Int. Conf. Med. Image Comput. Comput. Assist. Interv._ Springer, 2015, pp. 234–241.
  * [20] J. Wang, H. Lu, T. Li, and Z. Liang, “Sinogram noise reduction for low-dose CT by statistics-based nonlinear filters,” in _Proc. of SPIE_ , vol. 5747, 2005, p. 2059.
  * [21] J. Wang, T. Li, H. Lu, and Z. Liang, “Penalized weighted least-squares approach to sinogram noise reduction and image reconstruction for low-dose X-ray computed tomography,” _IEEE Trans. Med. Imaging_ , vol. 25, no. 10, pp. 1272–1283, 2006.
  * [22] A. Manduca _et al._ , “Projection space denoising with bilateral filtering and CT noise modeling for dose reduction in CT,” _Med. Phys._ , vol. 36, no. 11, pp. 4911–4919, 2009.
  * [23] S. Ramani and J. A. Fessler, “A splitting-based iterative algorithm for accelerated statistical x-ray ct reconstruction,” _IEEE Trans. Med. Imaging_ , vol. 31, no. 3, pp. 677–688, 2011.
  * [24] W. Wu, J. Shi, H. Yu, W. Wu, and V. Vardhanabhuti, “Tensor gradient L0-norm minimization-based low-dose CT and its application to COVID-19,” _IEEE Trans. Instrum. Meas._ , vol. 70, pp. 1–12, 2021.
  * [25] X. Zheng, S. Ravishankar, Y. Long, and J. A. Fessler, “PWLS-ULTRA: An efficient clustering and learning-based approach for low-dose 3d ct image reconstruction,” _IEEE Trans. Med. Imaging_ , vol. 37, no. 6, pp. 1498–1510, 2018.
  * [26] Q. Xu, H. Yu, X. Mou, L. Zhang, J. Hsieh, and G. Wang, “Low-dose x-ray CT reconstruction via dictionary learning,” _IEEE Trans. Med. Imaging_ , vol. 31, no. 9, pp. 1682–1697, 2012.
  * [27] J. Ma _et al._ , “Low-dose computed tomography image restoration using previous normal-dose scan,” _Med. Phys._ , vol. 38, no. 10, pp. 5713–5731, 2011.
  * [28] P. F. Feruglio, C. Vinegoni, J. Gros, A. Sbarbati, and R. Weissleder, “Block matching 3D random noise filtering for absorption optical projection tomography,” _Phys. Med. Biol._ , vol. 55, no. 18, p. 5401, 2010.
  * [29] M. Li, W. Hsu, X. Xie, J. Cong, and W. Gao, “SACNN: self-attention convolutional neural network for low-dose CT denoising with self-supervised perceptual loss network,” _IEEE Trans. Med. Imaging_ , vol. 39, no. 7, pp. 2289–2301, 2020.
  * [30] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of GANs for improved quality, stability, and variation,” in _Proc. Int. Conf. Learn Represent._ , 2018, pp. 1–26.
  * [31] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _Proc. IEEE Conf. Comp. Vis. Patt. Recogn._ , 2017, pp. 1125–1134.
  * [32] A. Guo, L. Fang, M. Qi, and S. Li, “Unsupervised denoising of optical coherence tomography images with nonlocal-generative adversarial network,” _IEEE Trans. Instrum. Meas._ , vol. 70, pp. 1–12, 2020.
  * [33] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein generative adversarial networks,” in _Proc. Int. Conf. Mach. Learn._ , 2017, pp. 214–223.
  * [34] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville, “Improved training of Wasserstein GANs,” in _Proc. Adv. Neural Inf. Process. Syst._ , 2017, pp. 5769–5779.
  * [35] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. Paul Smolley, “Least squares generative adversarial networks,” in _Proc. IEEE Int. Conf. Comp. Vis._ , 2017, pp. 2794–2802.
  * [36] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida, “Spectral normalization for generative adversarial networks,” in _Proc. Int. Conf. Learn Represent._ , 2018, pp. 1–26.
  * [37] X. Yi and P. Babyn, “Sharpness-aware low-dose CT denoising using conditional generative adversarial network,” _J. Digit. Imaging_ , vol. 31, no. 5, pp. 655–669, 2018.
  * [38] T. Chen, X. Zhai, M. Ritter, M. Lucic, and N. Houlsby, “Self-supervised GANs via auxiliary rotation loss,” in _Proc. IEEE/CVF Conf. Comp. Vis. Patt. Recogn._ , 2019, pp. 12 154–12 163.
  * [39] Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, and J. Liang, “Unet++: A nested U-net architecture for medical image segmentation,” in _Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support_. Springer, 2018, pp. 3–11.
  * [40] Y. Han and J. C. Ye, “Framing U-Net via deep convolutional framelets: Application to sparse-view CT,” _IEEE Trans. on Med. Imaging_ , vol. 37, no. 6, pp. 1418–1429, 2018.
  * [41] N. Kanopoulos, N. Vasanthavada, and R. L. Baker, “Design of an image edge detection filter using the Sobel operator,” _IEEE J. Solid-State Circuits_ , vol. 23, no. 2, pp. 358–367, 1988.
  * [42] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, “CutMix: Regularization strategy to train strong classifiers with localizable features,” in _Proc. IEEE Int. Conf. Comp. Vis._ , 2019, pp. 6023–6032.
  * [43] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” _Proc. Int. Conf. Learn Represent._ , 2018.
  * [44] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proc. IEEE Conf. Comp. Vis. Patt. Recogn._ , 2016, pp. 770–778.
  * [45] T. R. Moen _et al._ , “Low-dose CT image and projection dataset,” _Med. Phys._ , vol. 48, no. 2, pp. 902–911, 2021.
  * [46] F. Fan _et al._ , “Quadratic autoencoder (Q-AE) for low-dose CT denoising,” _IEEE Trans. Med. Imaging_ , vol. 39, no. 6, pp. 2035–2050, 2019\.
  * [47] M. Geng _et al._ , “Content-noise complementary learning for medical image denoising,” _IEEE Trans. Med. Imaging_ , 2021.
  * [48] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in _Proc. IEEE Int. Conf. Comput. Vis._ , 2015, pp. 1026–1034.
  * [49] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [50] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in _Eur. Conf. Comp. Vis._ Springer, 2016, pp. 694–711.


# CogSense: A Cognitively Inspired Framework for Perception Adaptation

Hyukseong Kwon
<EMAIL_ADDRESS>Amir Rahimi
<EMAIL_ADDRESS>Kevin G. Lee
<EMAIL_ADDRESS>Amit Agarwal
<EMAIL_ADDRESS>Rajan Bhattacharyya
<EMAIL_ADDRESS>
(HRL Laboratories, LLC., 3011 Malibu Canyon Road, Malibu, CA 90265, USA)

###### Abstract

This paper proposes the CogSense system, which is inspired by sense-making
cognition and perception in the mammalian brain to perform perception error
detection and perception parameter adaptation using probabilistic signal
temporal logic. As a specific application, a contrast-based perception
adaption method is presented and validated. The proposed method evaluates
perception errors using heterogeneous probe functions computed from the
detected objects and subsequently solves a contrast optimization problem to
correct perception errors. The CogSense probe functions utilize the
characteristics of geometry, dynamics, and detected blob image quality of the
objects to develop axioms in a probabilistic signal temporal logic framework.
By evaluating these axioms, we can formally verify whether the detections are
valid or erroneous. Further, using the CogSense axioms, we generate the
probabilistic signal temporal logic-based constraints to finally solve the
contrast-based optimization problem to reduce false positives and false
negatives.

_Keywords_ Perception error $\cdot$ Temporal logic $\cdot$ Perception
adaptation

##  1 Introduction

Perception errors are still challenging issues despite performance
improvements in perception systems in the last decade. In autonomous driving
or navigation systems, incorrect detections threaten the safe and robust
performance of fully autonomous systems. Most modern perception systems
utilize a pipeline of algorithms to combine functions of object detection and
tracking, however they perform their inference in a purely feedforward manner.
Perceptual processing in the mammalian brain, however, has significant
feedback, as evidenced in the presence of dense connectivity from higher-order
visual areas encoding object categories to early visible areas encoding
features [1]. This principle of feedback extends to intelligence after
perception, in cognition, where the brain moves far beyond simply binding
visual features and matching to the most likely feature distribution to
recognize an object, but instead engages in cycles of attention, hypothesis
revision, and decision making about conflicting features to infer the true
identity of an object under uncertain conditions. The cognitive capability of
the human brain to make inferences under uncertainty by foraging, adaptation,
evaluation, and decision making is known as sense-making and has been studied
using behavioral experiments and neurocognitive models [2, 3, 4]. There have
also been approaches attempting to embed this feedback into deep neural
networks [5, 6, 7].

Evidence from neuroimaging illustrates the rich top-down feedback connections
that provide the platform for attentive perceptual cognition in the brain.
These connections extend from areas that perform decision-making in the
prefrontal cortex back to modulate activity in the sensory cortical areas that
provide the visual features themselves [8, 9]. Further, diverse cortical areas
that process the semantic properties of objects feed activity to the
heteromodal association areas that bind these semantic properties with their
sensory properties (including visual, auditory, tactile features) to combine
perception and semantics in cognition [10, 11].

Adaptation in the brain occurs in many ways under a multitude of mechanisms,
from local to global. For instance, when we receive feedback that demonstrates
significant surprise, or unexpected uncertainty of our model of the world, the
brain adapts its processing for a remarkable ecological endeavor: controlling
attention to acquiring new statistics of the world around it to make better
inferences by adapting at multiple levels through a neuromodulatory cascade of
norepinephrine [12]. At the decision-making level, neurobiological circuits
that maintain the inference model are directed to release the current model
from working memory in the prefrontal cortex. Feedback to the sensor occurs at
the sensory level: the pupils in the eye dilate to acquire more stimulus. In
turn, experimental evidence also supports mid-level cognitive adaptation in
the level of alertness and orienting behavior to reacquire new statistics of
the world [13]. Indeed, such models have been used in artificial neural
networks for adapting behavior in robots [14, 15].

To this end, we are inspired by sense-making cognition as a model for
intelligence in perception, using feedback, adaptation, and semantics. To
enable fully autonomous systems, sense-making cognition for intelligent
perception is necessary under uncertain conditions, including corner cases. In
this paper, we demonstrated an approach to intelligent perception in a system
that detects perception errors, and uses the principles of feedback and
adaptation to correct them.

##  2 Related Work

In the context of robotics, Perception systems need to have strong confidence
about their environment. Perception errors can lead to disastrous consequences
for the robot or its surroundings, for example, autonomous car crashes. Today,
Deep Neural Networks (DNN) are commonly used for perception; however, they are
notoriously easy to fool [16] and do not output calibrated confidences [17].
To prevent this vulnerability, researchers have investigated a more global
measure called uncertainty. There are two kinds of uncertainty: aleatoric and
epistemic. Aleatoric uncertainty is the uncertainty of our input, and
epistemic uncertainty is the uncertainty of our model. Some methods to
estimate epistemic uncertainty are Bayesian Neural Networks [18], ensemble
methods [19, 20], Monte Carlo dropout [21], sampling-free methods [22], or
directly from the input [23]. Uncertainty can be used as an indicator of the
likelihood of an error. If the uncertainty associated with output is high, the
network is not very confident in its output, and that an error is likely.
However, most of these methods do not study how to reduce the uncertainty
associated with an input to improve performance.

Other approaches to detect perception errors and fix them have focused on
formally verifying the systems using temporal logic [24, 25, 26, 27]. This
approach has the advantage of mathematically proven performance guarantees, an
essential aspect of autonomous systems operating under uncertain conditions.
However, most of these systems apply motion control to the autonomous
platforms themselves rather than fix the perception systems. Unlike previous
works, our work is the first to apply temporal logic to directly modulate the
image contrast to the best of our knowledge.

More direct methods to predict and fix errors use sensor cues from LIDAR or
additional cameras to detect errors [31, 32]. In contrast to these works, our
method requires no additional sensors besides a single camera and can be
applied on top of any off-the-shelf object detector.

Our approach builds upon contrast enhancement techniques. Early work in the
perceptual domain has explored improving upon conventional contrast
enhancement techniques by using adaptive histogram equalization [28].
Subsequent work has applied contrast enhancement to object detection, where it
helps to accentuate features to detect objects more robustly [29, 30].
However, corresponding conventional methods use the image contrast information
of the entire image. So, if some non-object areas cause high contrast,
contrast adaptation cannot improve object detection.

In our proposed method, we draw upon temporal logic-based methods and contrast
enhancement techniques. We improve the perception system using feedback
control of a contrast parameter only from detected objects within a formally
verified system.

The rest of the paper will be presented as follows: In the next section, we
describe the proposed perception error detection/correction system. In Section
4, the probes converted from the perception data are explained. Then the
detailed approach for error correction is presented in Section 5. With that
mechanism, Section 6 goes into the details of the contrast-based perception
adaptation. And Section 7 validates the proposed methods, and finally Section
8 concludes the paper.

##  3 Overview

The overall structure of the CogSense error detection and perception
adaptation system is shown in Figure 1. Here, we review the construction of
the CogSense system, from training data to characterizing the operational
ranges of the perception system. First, from the perception data, we generate
probes that describe detections’ characteristics, such as the size of objects,
their type, tracking deviations from a motion tracker, and image statistics
such as contrast and entropy. Using the probes, we set up ‘Probabilistic
Signal Temporal Logic (PSTL) [27],’ and PSTL provides axioms, each of which is
constructed with a single or multiple probes with the corresponding
statistical analyses. As an intermediate process, those axioms provide the
error analysis on detections. This is the cognitively inspired process
described earlier, in which heterogeneous sources of information from multiple
modalities, including physical, semantic, and image based statistics provide
checks and balances on the plausibility of output detections from the
perception system that are orthogonal to the measures of uncertainty described
above. Finally, with these axiom-based constraints, we solve an optimization
problem to synthesize controls for the perception modules to reduce perception
errors and improve valid detection rates. This paper first mathematically
describes the CogSense system for the heterogeneous probes for error detection
using PSTL and then the PSTL-based optimization framework for controlling
parameters to correct perception errors. The following section describes our
experimental results that utilize heterogeneous probes in CogSense to detect
errors and an image contrast optimization for perception adaptation to correct
errors.

Figure 1: Overview architecture. We perform object detection on an input image
and generate probes containing the statistics of the detections. Next we
construct PTSL axioms and adapt the contrast of the input image by optimizing
on the PTSL axioms. Finally we perform object detection again on the contrast-
adjusted image.

In more detail, our approach is as follows:

  1. 1.

Through the perception module, we receive input images of the scene.

  2. 2.

Objects in the image are detected and recognized.

  3. 3.

This module converts the perception data into probes that we use for signal
temporal logic.

  4. 4.

Probes are converted into the axioms under the probabilistic signal temporal
logic structure.

  5. 5.

The axioms are evaluated to verify if the corresponding observations are valid
or erroneous based on the constraints using the statistically analyzed probe
bounds.

  6. 6.

If the axioms are invalid within certain probabilities, estimate the optimal
contrast bound and entropy bound as perception module parameters to apply by
solving the image contrast/entropy based optimization problem. Finally, this
estimated parameters are delivered back to the perception module to adjust its
contrast and entropy.

##  4 Perception probes generation and error evaluation

The first step in the process is to obtain the perception data along with
characteristics of detections and recognitions. To get different types of
characteristics efficiently, we used YOLOv3 [33] as shown in Figure 2.

Figure 2: Sample image of person detection using YOLOv3

We can extract image-based bounding box information from detected object boxes
in a single frame and the corresponding tracking information along with the
image sequences. We call this information "probes." In our system, we have
multiple probes such as detected object sizes, aspect ratios, recognition ID
consistency, tracking deviations, and so on. The following are the sample
probes we use in this paper.

  * •

Object size (in the image plane and in the world coordinate frame)

  * •

Aspect ratio of the detected objects

  * •

Localization and tracking performance

  * •

Recognition confidence

  * •

Contrast of the detected boxes

  * •

Entropy of the detected boxes

Each probe will be used to build constraints for perception adaptation using
the "probabilistic signal temporal logic (PSTL) [27]." PSTL uses probabilistic
and temporal aspects on real-valued signals. Let us assume that the signal
$\xi$ at time $t$ satisfies a probabilistic atomic predicate $\lambda$. If we
briefly describe its formula,

$(\xi,t)\models\lambda_{\alpha_{t}}^{\epsilon_{t}}\iff
P(\lambda_{\alpha_{t}}(\xi(t))<0)>1-\epsilon_{t}.$ (1)

In this predicate, $\lambda_{\alpha_{t}}^{\epsilon_{t}}$, $\alpha_{t}$ is a
time-varying random variable and $1-\epsilon_{t}$ is the tolerance level in
satisfying the probabilistic properties. With this formula, we convert our
probes into axioms.

All of our detections are divided into true positives and false positives.
From the true positive and false positive detections, we can perform
statistical analysis for each probe. Figure 3 shows a descriptive example of a
probe. For a detected object, $x$, we assume that we generate a probe, $f(x)$.
By analyzing the values from true positives and also those from false
positives, we can obtain probabilistic distributions of true positives and
false positives as shown in the figure. We can define upper and lower bounds
for true positives from the intersections between two different distribution
graphs. And the shaded area presents the confidence probability, $P_{TP}$, of
the probe.

Figure 3: Example of probability distribution of a probe. There is some domain
[a,b] where the probability of true positives is optimized.

If we describe this relation in a mathematical form (axiom) with the
probabilistic inequality from the probabilistic signal temporal logic, it
becomes as follows:

$\forall x,Pr(a\leq f(x,t_{s}:t_{e})\leq b\rightarrow y)\geq P_{TP}$ (2)

where $Pr(\cdot)$ is the predicate and y is the true detection or recognition.
And $t_{s}:t_{e}$ means the time sequence between $t_{s}$ and $t_{e}$, so
$f(x,t_{s}:t_{e})$ is the probe sequence in the time frame of $t_{s}:t_{e}$.

Depending on the probe dimensions, the probabilistic function can also be
multi-dimensional. By integrating all the available axioms from $x$, we can
have a multi-dimensional range of the corresponding detection or recognition.
When a new probe violates the corresponding axioms more than a certain
probabilistic threshold, we can verify that the corresponding detection is
considered as erroneous, and the probability of an error is higher. If it is
categorized as erroneous, we apply the perception adaptation as shown in the
next section.

##  5 Perception error correction using the PSTL-constraint-based optimization

Detecting perception errors is not sufficient to recover the perception
quality in the following image sequences. Therefore, we also want to adjust
perception modules to have more accurate and robust detections with that
knowledge. In this paper, we propose a new optimization technique using the
PSTL-constraint-based optimization with the following format:

$u_{opt}=\arg\min_{u_{t}}J(x_{t},u_{t})$ (3)

where $x_{t}$ is the probe state at time $t$ and $u_{t}$ is the control input
to the perception module. And $J(\cdot)$ is the cost function of estimating
perception errors. Our goal is to achieve the optimal $u_{t}$ to reduce
perception errors. Therefore, minimizing $J(\cdot)$ can achieve the optimal
perception module control input. Eventually, the final optimization formula
with the two or more PSTL-based constraints for probes, $f(x_{t})$,
$g(z_{t})$, etc. becomes,

$\displaystyle\begin{gathered}u_{opt}=\arg\min_{u_{t}}J(x_{t},z_{t},u_{t})\\\
s.t.\\\ \forall x_{t},Pr(a\leq f(x_{t},t_{s}:t_{e})\leq b\rightarrow y)\geq
P_{TP_{x}}\\\ \forall z_{t},Pr(g(x_{t},t_{s}:t_{e})\leq c\rightarrow w)\geq
P_{TP_{z}}\\\ \cdots\end{gathered}$ (4)

where $y$ and $w$ are the true positive labels for the probes, $x_{t}$ and
$x_{t}$, respectively. $a$, $b$, and $c$ are probabilistic error bounds
acquired from the true positive / false positive distribution described in
Section 5. And $P_{TP_{x}}$ and $P_{TP_{z}}$ are the probability thresholds
for $x_{t}$ and $x_{t}$, respectively.

##  6 Contrast-based perception adaptation

To achieve the contrast-based perception adaptation, we first set up the
object detection constraints using five different types of constraints: (1)
Detection ID consistency (tracking of the same object); (2) Localization
consistency within the expected trajectory; (3) Bounding box size consistency
in the image plane; (4) Contrast matching in the desired range; and (5)
Entropy matching in the desired range. Details for each constraint are
presented below. $t_{k}$ is the current time and $t_{(k-M)}$ is the time that
the temporal logic window starts.

  * •

Consistent detection

$Pr({\sum_{t=t_{k}-t_{k-M}}^{t_{k}}det_{ID=X}})>P_{X}$ (5)

where $P_{X}$ is the probabilistic threshold for consistent ID detections. In
this constraint, the detection ID is checked to be consistent. If the IDs keep
changing, the detection process cannot be robust.

  * •

Bounding box size deviation over time

$Pr(|BB_{D}-BB_{t}|_{t=t_{k}-t_{k-M}}^{t_{k}})<P_{BB}$ (6)

where $BB_{t}$ is the bounding box size at time $t$ and $BB_{D}$ is the
desired bounding box size from its history. And $P_{BB}$ is the probabilistic
threshold for consistent bounding box size. Usually, highly varying bounding
box sizes for the same object (unless it does not approach or move away
abruptly) indicates unreliable bounding box estimation (e.g. too sensitive
with respect to lighting condition changes).

  * •

Localization deviation from the desired tracking trajectory

$Pr(|Path_{Desired}-loc_{t}|_{t=t_{k}-t_{k-M}}^{t_{k}})<P_{loc}$ (7)

where $loc_{t}$ is the detected object’s location at time $t$ and
$Path_{Desired}$ is its expected path from the history, and $P_{l}oc$ is the
probabilistic threshold for consistent localization. Localization deviation
usually comes from unreliable bounding box estimation, which indicates
perception errors.

  * •

Contrast

$Pr(|C_{D}-C_{t}|_{t=t_{k}-t_{k-M}}^{t_{k}})<P_{C}$ (8)

where $C_{t}$ is the contrast of the bounding box at time $t$, $C_{D}$ is the
desired contrast from the training phase, and $P_{C}$ is the probabilistic
threshold for contrast. Highly varying contrast also modifies the details of
the same object information over time. Contrast consistency is one of the
factors that we need to keep for more robust perception.

  * •

Entropy

$Pr(|E_{D}-E_{t}|_{t=t_{k}-t_{k-M}}^{t_{k}})<P_{E}$ (9)

where $E_{t}$ is the entropy of the bounding box at time $t$ and $E_{D}$ is
the desired entropy from the training phase. And $P_{E}$ is the probabilistic
threshold for entropy. Too blurred images or too sharpened images ruin even
the state of the art deep-learning based object detection methods. So, the
entropy deviates from the desired value, it is highly possible that we have
perception errors.

Then the corresponding optimization formula to control contrast
$(c_{i}(t)+\Delta c)$ with the cost function $J(c,\Delta c)$ is defined as,

$\arg\min_{\Delta c}J(c,\Delta c)=\arg\min_{\Delta
c}\sum_{i=1}^{n}|c_{i}(t)-c_{D}-\Delta c|$ (10)

where $c_{i}(t)$ is the contrast value of the $i^{th}$ detected object at time
$t$, $c_{D}$ is the desired contrast value from the procedure of finding the
probabilistic distributions of the probes, and $\Delta c$ is the system
control input for contrast (which is the same as estimated deviation to apply
to the perception module).

For contrast control, once the desirable contrast deviation is acquired, we
set up the expansion of histogram ranges to achieve that contrast changes
using the peak-to-peak contrast (Michelson contrast) [34]. The peak-to-peak
contrast is defined in the following way:

$C(t)=\frac{I_{max}-I_{min}}{I_{max}+I_{min}}$ (11)

where $I_{max}$ is the maximum image intensity value and $I_{min}$ is the
minimum image intensity value. From this definition, we can expect a new
contrast is supposed to be:

$C(t)=\frac{I_{max}-I_{min}+2B}{I_{max}+I_{min}}$ (12)

where $B$ is the expanded histogram range to achieve the new contrast. Since
$\Delta c=C_{desired}-C(k)$, the histogram changing range changing amount will
be

$B=\frac{\Delta c\cdot(I_{max}+I_{min})}{2}$ (13)

##  7 Experiments

In this section, we present test results with our contrast-based PSTL
perception adaptation. First, we present a test result on a video from the
Multiple Object Tracking(MOT) Benchmark dataset [35]. Through our proposed
method, we improve detection results as shown in Figure 4. The red box in the
upper figure is a erroneous person detection from the original image, and the
green boxes in the lower figure are correct person detection newly added from
the CogSense system.

Figure 4: Sample of error reduction and detection improvement. On top are
unique detections from a baseline YOLOv3. Below are unique detections using
our proposed CogSense method. We intentionally remove common detections from
both to increase readability.

For supporting the proposed method with quantified performances, we also
provide a precision-recall graph and a ROC curve in Figs. 5 and 6,
respectively. As shown in the graphs, the newly proposed approach presents
better recall rates with the same precision, and also less false positive
rates while achieving the same true positive rates. Especially, in the ROC
curve, with 10% detection confidence thresholding, the false positive rate is
reduced by 41.48%.

Figure 5: Performance Comparison - Precision vs Recall Figure 6: Performance
Comparison - ROC curve

In addition to the above, we also tested more challenging video clips
collected from [36, 37, 38, 39, 40]. Then we compare the detection results
from the original video sequences with those from the conventional Contrast
Limited Adaptive Histogram Equalization (CLAHE) [28] and those from our
proposed CogSense method. Tables 1 and 2 show the true positive rates (the
number of true positives over the sum of true positive and false positive
detections) and false positive counts, respectively. As shown in Table 1,
although the improvement rates are not that huge, the true positive rates are
actually improved overall compared to the original method and the CLAHE
method. And if we look into the false positive counts in Table 2, false
positive counts with the proposed method are reduced by $12.5\%$ comparing to
the original video processing, and by $34.75\%$ comparing to the CLAHE-applied
video processing. The reason of CLAHE-applied videos’ high false positive
rates is that CLAHE-based image enhancement improves visual aspects to human,
not for object detection.

Table 1: True positive rates from the original, CLAHE and the proposed CogSense approach Video | Original | CLAHE | CogSense
---|---|---|---
Clip 1 [36] | 0.9804 | 0.9602 | 0.9817
Clip 2 [37] | 0.9931 | 0.9931 | 0.9904
Clip 3 [38] | 0.9829 | 0.9860 | 0.9835
Clip 4 [38] | 0.9964 | 0.9977 | 0.9993
Clip 5 [39] | 0.9822 | 0.9796 | 0.9834
Clip 6 [40] | 0.9495 | 0.9065 | 0.9527
Table 2: False positive counts from the original, CLAHE and the proposed CogSense approach Video | Original | CLAHE | CogSense
---|---|---|---
Clip 1 [36] | 22 | 43 | 20
Clip 2 [37] | 14 | 8 | 12
Clip 3 [38] | 13 | 11 | 13
Clip 4 [38] | 5 | 3 | 1
Clip 5 [39] | 17 | 19 | 16
Clip 6 [40] | 17 | 34 | 15
Figure 7: Qualitative results of our proposed CogSense method. From left to
right, Clip 1, Clip 2, Clip 3, Clip 4, Clip 5, and Clip 6. From top to bottom:
Original, CLAHE, and CogSense.

Figure 7 shows sample image triples from the three methods. In the figure,
each row corresponds to the individual clip number in order. In Clip 1, the
original video processing missed one person and the CLAHE-applied video
processing still missed the same person, and even provided multiple detections
from the same person. On the other hand, our proposed method detects all three
objects. Also in Clip 2 and Clip 6, the proposed method detects more true
positives. And finally, the result of Clip 4 shows two different detections
from the original video processing and CLAHE-applied video processing,
however, our proposed method detected them both.

As shown through the test results, our proposed CogSense method using PSTL-
based constraints and detected-bounding-box-based optimization provides more
robust detection outputs.

##  8 Conclusion and Discussion

This paper presents CogSense, a new perception error detection and perception
parameter adaptation method using the probabilistic signal temporal logic, and
a contrast-based perception adaption approach as a specific case. The proposed
method evaluates perception errors using heterogeneous probes of the detected
objects and subsequently correct perception errors by solving a contrast-based
optimization problem and other objected detection-based constraints generated
from the probabilistic signal temporal logic system. Our proposed contrast-
based perception adaptation uses information only from the detected bounding
boxes, and specifically our approach applies adaption to these boxes to
improve object detection rather than unnecessarily applying adaptation to the
entire image enhancement, which could have unintended consequences to other
detections. In future work, we will increase the heterogeneity of probes to
extend to other domains such as semantics and context, and pursue multi-modal
perception parameter adaption by optimizing parameters to simultaneously
control entities that extend beyond the current paper’s demonstration of image
based contrast to enhance the CogSense system’s perception correction
capabilities.

## References

  * [1] S. Gabay, Y. Pertzov, and A. Henik, "Orienting of attention, pupil size, and the norepinephrine system," Attention, Perception, and Psychophysics 73, 123–129, 2011 https://doi.org/10.3758/s13414-010-0015-4.
  * [2] G. A. Ascoli, M. M. Botvinick, R. J. Heuer, and R. Bhattacharyya, “Neurocognitive models of sense-making,” Biol. Inspired Cogn. Archit., vol. 8, pp. 82–89, 2014.
  * [3] C. Lebiere et al., “A Functional Model of Sensemaking in a Neurocognitive Architecture,” Comput. Intell. Neurosci., vol. 2013, Article ID 921695, doi: 10.1155/2013/921695.
  * [4] M. D. Howard et al., “The Neural Basis of Decision-Making During Sensemaking: Implications for Human-System Interaction,” Proc. IEEE Aerosp. Conf. March 10 2015.
  * [5] C. J. Spoerer, P. McClure and N. Kriegeskorte, "Recurrent convolutional neural networks: a better mode of biological object recognition," Frontiers of Psychology, 2017.
  * [6] Y. Huang, S. Dai, T. Nguyen, P. Bao, D. Tsao, R. G. Barniuk, and A. Anandkumar, "Brain-inspired Robust Vision using Convolutional Neural Networks with Feedback," 2019 Conference on Neural Information Processing Systems NeuroAI Workshop, 2019.
  * [7] J. Kubilius, M. Schrimpf, K. Kar, R. Rajalingham, H. Hong, N. J. Majaj, E. B. Issa, P. Bashivan, J. Prescott-Roy, K. Schmidt, A. Nayebi, D. Bear, D. L. K. Yamins, and J. J. DiCarlo, "Brain-Like Object Recognition with High-Peforming Shallow Recurrent ANNs," 2018 Conference on Neural Information Processing Systems, 2018.
  * [8] T. P. Zanto, M. T. Rubens, A. Thangavel, and A. Gazzaley, “Causal role of the prefrontal cortex in top-down modulation of visual processing and working memory,” Nat. Neurosci., vol. 14, no. 5, pp. 656–661, 2011.
  * [9] R. F. Helfrich, M. Huang, G. Wilson, and R. T. Knight, “Prefrontal cortex modulates posterior alpha oscillations during top-down guided visual perception,” Proc. Natl. Acad. Sci., vol. 114, no. 35, pp. 9457–9462, Aug. 2017, doi: 10.1073/pnas.1705965114.
  * [10] M. F. Bonner, J. E. Peelle, P. A. Cook, and M. Grossman, “Heteromodal conceptual processing in the angular gyrus,” Neuroimage, vol. 71, pp. 175–186, 2013.
  * [11] G. A. Calvert, R. Campbell, and M. J. Brammer, “Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex,” Curr. Biol., vol. 10, no. 11, pp. 649–657, 2000.
  * [12] A. J. Yu and P. Dayan, “Uncertainty, neuromodulation, and attention,” Neuron, vol. 46, pp. 681–692, 2005.
  * [13] G. Aston-Jones and J. D. Cohen, “An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance,” Annu Rev Neurosci, vol. 28, pp. 403–450, 2005.
  * [14] J. L. Krichmar, “The Neuromodulatory System - A Framework for Survival and Adaptive Behavior in a Challenging World,” Adapt. Behav., vol. 16, pp. 385–399, 2008.
  * [15] X. Zou, S. Kolouri, P. K. Pilly, and J. L. Krichmar, “Neuromodulated attention and goal-driven perception in uncertain domains,” Neural Netw., 2020.
  * [16] A. Nguyen, J. Yosinski, and J. Clune, "Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images," 2015 IEEE Proceedings on Computer Vision, 2015.
  * [17] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, "On Calibration of Modern Neural Networks," 2017 International Conference on Machine Learning, 2017.
  * [18] A. Kendall and Y. Gal, "What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?," 2017 Conference on Neural Information Processing Systems, 2017.
  * [19] H. Grimmett, R. Paul, R. Triebel, and I. Posner, "Knowing When We Don’t Know: Introspective Classification for Mission-Critical Decision Making," 2013 IEEE International Conference on Robotics and Automation, 2013.
  * [20] C. G. Blair, J. Thompson, and N. M. Robertson, "Introspective classification for pedestrian detection," 2014 Sensor Signal Processing for Defense, 2014.
  * [21] Y. Gal and Z. Ghahramani, "Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning," 2016 International Conference on Machine Learning, 2016.
  * [22] J. Postels, F. Ferroni, H. Coskun, N. Navab, and F. Tombari, "Sampling-free Epistemi Uncertainty Estimation Using Approximated Variance Propagation," 2019 IEEE International Conference on Computer Vision, 2019.
  * [23] S. Daftry, S. Zeng, J. A. Bagnell, and M. Herbert, "Introspective Perception: Learning to Predict Failures in Vision Systems," 2016 IEEE International Conference on Intelligent Robotics and Systems, 2016.
  * [24] A. Dokhanchi, H.B. Amor, J.V. Deshmukh, and G. Fainekos, “Evaluating perception systems for autonomous vehicles using quality temporal logic,” International Conference on Runtime Verification, 2018.
  * [25] R.R. da Silva, V. Kurtz, and M. Hebert, “Active Perception and Control from Temporal Logic Specifications,” arXiv:1905.03662, 2019.
  * [26] S. Jha, V. Raman, D. Sadigh, and S.A. Seshia, “Safe Autonomy Under Perception Uncertainty Using Chance-Constrained Temporal Logic,” Journal of Automated Reasoning, 2018.
  * [27] D. Sadigh and A. Kapoor, “Safe control under uncertainty with Probabilistic Signal Temporal Logic,” in Proc. Of Robotics: Science and Systems, 2016.
  * [28] J. A. Stark, “Adaptive Image Contrast Enhancement Using Generalizations of Histogram Equalization,” IEEE Transactions on Image Processing, Vol. 9, No. 5, pp.889-896, 2000.
  * [29] D. Holz and H. Yang, “Enhanced Contrast for Object Detection and Characterization by Optical Imaging,” US9626591B2.
  * [30] V. Vonikakis, D. Chrysostomou, R. Kouskouridas and A. Gasteratos, “Improving the Robustness in Feature Detection by Local Contrast Enhancement,” 2012 IEEE International Conference on Image Systems and Techniques Proceedings, July 2012.
  * [31] D. Barnes, W. Maddern, and I. Posner, "Find Your Own Way: Weakly-Supervised Segmentation of Path Proposals for Urban Autonomy," 2017 IEEE International Conference on Robotics and Automation, 2017.
  * [32] M. S. Ramanagopal, C. Anderson, R. Vasudevan, and M. Johnson-Roberson, "Failing to Learn: Autonomously Identifying Perception Failures for Self-driving Cars," 2018 IEEE International Conference on Intelligent Robotics and Systems, 2018.
  * [33] https://pjreddie.com/darknet/yolo/.
  * [34] A. Michelson, “Studies in Optics,” University of Chicago Press, 1927.
  * [35] Multiple Object Tracking Benchmark (https://motchallenge.net).
  * [36] https://www.youtube.com/watch?v=0xSSnfRYBQY
  * [37] https://www.youtube.com/watch?v=BOa0zQBRs_M
  * [38] https://www.youtube.com/watch?v=NjRnTmUr_kg
  * [39] https://www.youtube.com/watch?v=cGjWbGgDaCc
  * [40] https://www.youtube.com/watch?v=99wezqewopU