# Continuous Conversion of CT Kernel using Switchable CycleGAN with AdaIN

Serin Yang, Eung Yeop Kim, and Jong Chul Ye This work was supported by the
National Research Foundation (NRF) of Korea grant NRF-2020R1A2B5B03001980. S.
Yang and J.C. Ye are with the Department of Bio and Brain Engineering, Korea
Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic
of Korea (e-mail: yangsr, jong.ye@kaist.ac.kr). E.Y. Kim is with the
Department of Radiology, Samsung Medical Center, Sungkyunkwan University
School of Medicine, 81 Irwon-ro Gangnam-gu, Seoul, 06351, Republic of Korea
(e-mail<EMAIL_ADDRESS>
###### Abstract

X-ray computed tomography (CT) uses different filter kernels to highlight
different structures. Since the raw sinogram data is usually removed after the
reconstruction, in case there are additional need for other types of kernel
images that were not previously generated, the patient may need to be scanned
again. Accordingly, there exists increasing demand for post-hoc image domain
conversion from one kernel to another without sacrificing the image quality.
In this paper, we propose a novel unsupervised continuous kernel conversion
method using cycle-consistent generative adversarial network (cycleGAN) with
adaptive instance normalization (AdaIN). Even without paired training data,
not only can our network translate the images between two different kernels,
but it can also convert images along the interpolation path between the two
kernel domains. We also show that the quality of generated images can be
further improved if intermediate kernel domain images are available.

Experimental results confirm that our method not only enables accurate kernel
conversion that is comparable to supervised learning methods, but also
generates intermediate kernel images in the unseen domain that are useful for
hypopharyngeal cancer diagnosis.

###### Index Terms:

Computed tomography, reconstruction kernels, cycle-consistent adversarial
networks, style transfer, adaptive instance normalization (AdaIN)

##  I Introduction

In computed tomography (CT) images, raw sinogram data are collected from
detectors, from which tomographic images are reconstructed using algorithms
such as filtered backprojection (FBP). Depending on the structural property of
the object, a specific reconstruction fitter kernel is selected which affects
the range of features that can be seen[1]. For example, high pass filters,
which are often used to bone and tissues with high CT contrast, preserve
higher spatial frequencies while reducing lower spatial frequencies. On the
other hand, low pass filters compromise spatial resolution but significantly
reduces the noises, which are adequate for brain or soft tissue imaging [1],
[2].

Figure 1: (a) Vanilla cycleGAN for conversion of two different kernels. (b)
Switchable cycleGAN for two domains, and (c) switchable cycleGAN with split
AdaIN for three domains.

Although different kernels would be required in order to examine different
structures, the size of the required storage can become quickly too large to
accommodate various types of kernel images [3, 4]. Therefore, a routine
practice is just to save a particular kernel images. Accordingly, kernel
conversion is expected to be useful in case when additional kernel images are
required.

For example, in case of paranasal sinuses CT, only one type of kernel images
are usually reconstructed, either sharp or soft kernels. Sharp kernels are
applicable to check abnormality of bone but hard to evaluate soft tissue. On
the other hand, it is relatively easy to evaluate abnormality in the soft
tissue with soft kernels but difficult to see bone details. Also, in case of
temporal bone CT, soft kernel images are occasionally required although sharp
kernel images are only acquired in routine clinical practice. In these cases,
kernel conversion would be useful for both healthcare experts and patients
because it reduces additional burden of extra CT scanning. Through kernel
conversions, both sharp and soft kernel images can be obtained and utilized
adequately in various situations.

Although many researchers have investigated this issue [3, 4, 5], the image
domain filter kernel conversion is still seen as a difficult task since the
relationship between different kernel images needs to be found out and a new
texture should be synthesized for the target domain.

Recently, a deep learning approach was explored for kernel conversion [6].
This approach is based on the supervised training, so that it can only be
applied when there exists paired dataset from the different kernels. Although
paired data sets for training could be generated from the same sinogram with
different filter kernels, collecting all paired CT kernel image data sets for
different CT scanners and acquisition conditions like KVp, mAs, etc. would be
a daunting task that could only be accomplished by carefully planning the data
collection in advance.

Therefore, one of the most important motivations of this work is to use an
unsupervised deep learning approach for kernel conversion that can be trained
without paired data set. In particular, we consider the kernel conversion
problem as an unsupervised image style transfer problem, and develop an
unsupervised deep learning approach.

In fact, cycle-consistent adversarial network (cycleGAN) [7] is one of the
representative unsupervised image style transfer methods that can learn to
translate between two different domains. Furthermore, our recent theoretical
work [8] shows that the cycleGAN can be interpreted as an optimal transport
between two probabilistic distributions [9, 10] that simultaneously minimizes
the statistical distances between the empirical data and synthesized data in
two domains. Therefore, we are interested in using cycleGAN as our model, with
each domain consisting of distinct kernel images.

One important contribution is that unlike the conventional cycleGAN that uses
two distinct generators, we employ the switchable cycleGAN architecture [11]
for kernel conversion so that a single conditional generator with adaptive
instance normalization (AdaIN) [12] can be used for both forward and backward
kernel conversion. In addition to reducing the memory requirement for the
cycleGAN training [11], the proposed cycleGAN generator with AdaIN layer can
generate every interpolating path along an optimal transport path between two
target domains at the inference phase, which brings us an opportunity to
discover new objects or tissues which could not have been observed with the
currently given two reconstruction kernels.

Unfortunately, the existing switchable CycleGAN [11] proves difficult to
utilize the intermediate kernel images if available. This is because the
kernel interpolation is only unidirectional from soft or sharp kernels to the
intermediate kernel, but the conversion from the intermediate kernel to the
other kernels is not possible. To remedy this, here we propose a novel split
AdaIN code generator architecture in which two AdaIN code generators are used
independently of each other for encoder and decoder. One of the major
innovations in this novel architecture is that it now enables bidirectional
kernel conversion between any pairs of kernel domains along the interpolation
path, thus enabling effective use of the intermediate domain kernel images to
greatly improve the overall kernel conversion performance.

This paper is organized as follows. CT conversion and image style transfers
are first reviewed in Section II, after which we explain how these techniques
can be synergistically combined for our kernel conversion algorithm in Section
III. Section IV describes the data set and experimental setup. Experimental
results are provided in Section V, which is followed by Discussion and
Conclusions in Section VI and Section VII.

##  II Related works

###  II-A CT Kernel Conversion

In classical CT kernel conversion approaches, one common way is to combine two
different kernel images into one for better diagnostic purpose [3, 4, 5]. For
example, for hybrid kernel combination, the upper and lower thresholds for
pixel intensities are chosen, and then if the pixel values of the images
reconstructed with low pass filters are outside those thresholds, the values
are replaced with those from the high pass filters. Unfortunately, the
optimality of the combined filter kernel is a subjective matter, which depends
on the clinical applications.

In this regard, deep learning approaches for CT kernel conversion [6, 13] do
not interfere with the existing clinical workflows, as the generated images
are still in the conventional kernel sets. Furthermore, a radiomic study
revealed that the generated images do not sacrifice the accuracy of the
diagnosis [13]. Nonetheless, this method does not generate new kinds of hybrid
informations that could be obtained in the aforementioned kernel combination
methods.

###  II-B Deep Learning for Image Style Transfer

Currently, two types of approaches are available for image style transfer.
First, a content image and a style reference image are passed to a neural
network, and the goal is to convert the content image to imitate styles from
the style reference. For example, Gatys et al. [14] solves an optimization
problem in the feature spaces between the content and style images. However,
the iterative optimization process takes significant amount of time and the
results are easily overly stylized. Instead, the adaptive instance
normalization (AdaIN) was proposed as a simple alternative [12]. Specifically,
AdaIN layer estimates the means and variances of referenced style features and
uses them to correct the mean and variance of content features. Despite the
simplicity, a recent theoretical works [15] showed that the style transfer by
AdaIN is a special case of optimal transport [9, 10] between two Gaussian
distributions.

Another type of style transfer is performed as a distribution matching
approach. Specifically, let the target style images lie in the domain
$\mathcal{X}$ equipped with a probability measure $\mu$, whereas the input
content images lie in $\mathcal{Y}$ with a probability measure $\nu$. Then,
the image style transfer is to transport the content distribution $\nu$ to the
style image distribution $\mu$, and vice versa. In our recent theoretical work
[8], we show that this type of style transfer problem can be solved through
optimal transport [9, 10]. In particular, if we define the transportation cost
as the sum of the statistical distances between the empirical distribution and
the generated distribution in $\mathcal{X}$ and $\mathcal{Y}$, respectively,
and try to find the joint distribution that minimizes the sum of the
distances, then the Kantorovich dual formulation [9, 10] leads to the cycleGAN
formulation. This justifies why cycleGAN has become a representative style
transfer method.

Figure 2: Generator architecture in two-domain cycleGAN with one AdaIN code
generator.

Figure 3: Generator architecture in multi-domain cycleGAN with two different
AdaIN code generators.

By synergistically combining the two ideas, we recently proposed switchable
cycleGAN [11] that combines AdaIN into cycleGAN so that only a single
generator can be used for style transfer between two domains. Although the
original motivation of [11] was to reduce the memory requirement for the
cycleGAN training by eliminating additional generator, the cycleGAN with AdaIN
has an important advantage that cannot be achieved using the conventional
cycleGAN. Specifically, the switchable cycleGAN with AdaIN can generate
intermediate images on the interpolating path between two domains in the
training data set, leading to diverse kernel conversions.

Unfortunately, the direct use of switchable cycleGAN turns out difficult to
utilize the intermediate kernel images if available. As will be described
later, the kernel interpolation by the existing switchable cycleGAN [11] works
only unidirectional from soft or sharp kernels to the intermediate kernel, but
the conversion from the intermediate kernel data to the other kernel images,
which is essential to fully utilize the intermediate kernel images, is not
possible. This is one of the motivation for the new split AdaIN architecture,
which will be also introduced in this paper.

##  III Theory

###  III-A Instance Norm, AdaIN and Optimal Transport

Suppose that a multi-channel feature tensor at a specific layer is represented
by

$\displaystyle{\bm{X}}=$
$\displaystyle\begin{bmatrix}{\bm{x}}_{1}&\cdots&{\bm{x}}_{P}\end{bmatrix}\in{\mathbb{R}}^{HW\times
P},$ (1)

where $P$ is the number of channel in the feature tensor ${\bm{x}}$, and
${\bm{x}}_{i}\in{\mathbb{R}}^{HW\times 1}$ refers to the $i$-th column vector
of ${\bm{x}}$, which represents the vectorized feature map of size of $H\times
W$ at the $i$-th channel.

Then, instance normalization [16] and AdaIN [12] convert the feature data at
each channel using the following transform:

$\displaystyle{\bm{z}}_{i}$
$\displaystyle=\mathcal{T}({\bm{x}}_{i},{\bm{y}}_{i}),\quad i=1,\cdots,P$ (2)

where

$\displaystyle\mathcal{T}({\bm{x}},{\bm{y}}):=\frac{\sigma({\bm{y}})}{\sigma({\bm{x}})}\left({\bm{x}}-m({\bm{x}})\bm{1}\right)+m({\bm{y}})\bm{1},\quad$
(3)

where $\bm{1}\in{\mathbb{R}}^{HW}$ is the $HW$-dimensional vector composed of
1, and $m({\bm{x}})$ and $\sigma({\bm{x}})$ are the mean and standard
deviation of ${\bm{x}}\in{\mathbb{R}}^{HW}$; $m({\bm{y}})$ and
$\sigma({\bm{y}})$ refer to the target style domain mean and standard
deviation, respectively.

Specifically, the instance norm uses $(\sigma({\bm{y}}),m({\bm{y}}))=(1,0)$,
whereas the $(\sigma({\bm{y}}),m({\bm{y}}))$ are computed from the style
features for the case of AdaIN. Eq. (3) implies that the mean and variance of
the feature in the content image are normalized so that they can match the
mean and variance of the style image feature. In fact, the transform (3) is
closely related to the optimal transport between two probabilistic
distributions [10, 9].

Specifically, let the two probability spaces
$\mathcal{U}\subset{\mathbb{R}}^{HW}$ and
$\mathcal{V}\subset{\mathbb{R}}^{HW}$ be equipped with the Gaussian
probability measure
$\mu\sim\mathcal{N}({\bm{m}}_{U},{\boldsymbol{\Sigma}}_{U})$ and
$\nu\sim\mathcal{N}({\bm{m}}_{V},{\boldsymbol{\Sigma}}_{V})$, respectively,
where ${\bm{m}}_{U}$ and ${\boldsymbol{\Sigma}}_{U}$ denote the mean vector
and the covariance matrix, respectively. Then, a closed form optimal transport
map from the measure $\mu$ to the measure $\nu$ with respect to Wasserstein-2
distance can be obtained as follows [15]:

$\displaystyle
T_{\mu\rightarrow\nu}({\bm{x}})={\bm{m}}_{V}+{\boldsymbol{\Sigma}}_{U}^{-\frac{1}{2}}\big{(}{\boldsymbol{\Sigma}}_{U}^{\frac{1}{2}}{\boldsymbol{\Sigma}}_{V}{\boldsymbol{\Sigma}}_{U}^{\frac{1}{2}}\big{)}^{\frac{1}{2}}{\boldsymbol{\Sigma}}_{U}^{-\frac{1}{2}}({\bm{x}}-{\bm{m}}_{U})$
(4)

In particular, if we assume the independent and identically distributed
feature spaces, i.e.

$\displaystyle{\bm{m}}_{U}=m({\bm{x}})\bm{1},{\boldsymbol{\Sigma}}_{U}=\sigma({\bm{x}}){\bm{I}},{\bm{m}}_{V}=m({\bm{y}}),{\boldsymbol{\Sigma}}_{V}=\sigma({\bm{y}}){\bm{I}}$

then AdaIN transform in (3) can be obtained as a special case of (4) as
follows:

$\displaystyle
T_{\mu\rightarrow\nu}({\bm{x}})=m({\bm{y}})\bm{1}+\frac{\sigma({\bm{y}})}{\sigma({\bm{x}})}({\bm{x}}-m({\bm{x}})\bm{1})$
(5)

Accordingly, we can also see that instance norm is a special case of optimal
transport that transports the feature to the normalized Gaussian distribution
with zero mean and unit variance.

###  III-B Switchable CycleGAN with AdaIN

Suppose that the domain $\mathcal{S}$ is composed of CT images from soft
tissue kernel, whereas the images in the domain $\mathcal{H}$ are generated by
sharp bone kernel. Then, as shown in Fig. 1(a), a standard cycleGAN framework
for kernel conversion would require two generators: the forward generator from
sharp kernel to soft kernel $(G_{S})$, the backward generator from soft kernel
to sharp kernel images ($G_{H}$).

On the other hand, our switchable cycleGAN implements two generators using a
single baseline autoencoder network $G$ followed by AdaIN-based optimal
transport layers. Specifically, to generate images in the target domain from
source domain inputs, we use an autoencoder as the baseline network and then
use the AdaIN transform to transport the autoencoder features to the target
kernel features.

More specifically, the AdaIN codes for signaling $\mathcal{H}$ and
$\mathcal{S}$ domains are generated as:

$\displaystyle\left(m({\bm{y}}),\sigma({\bm{y}})\right)=\begin{cases}(1,0),&\mbox{domain
$\mathcal{H}$}\\\ (\sigma_{S},m_{S}),&\mbox{domain $\mathcal{S}$}\end{cases}$
(6)

Using this, AdaIN code generator $F$ is defined as follows:

$\displaystyle F(\beta):=$ $\displaystyle\begin{bmatrix}\sigma(\beta)\\\
m(\beta)\end{bmatrix}=(1-\beta)\begin{bmatrix}1\\\
0\end{bmatrix}+\beta\begin{bmatrix}\sigma_{S}\\\ m_{S}\end{bmatrix}$ (7)

where $(\sigma_{S},m_{S})$ are learnable parameters during the training, and
$\beta$ is a variable that represents the domain.

Using this, we can consider two generator architectures. First, when the
training data are only from two domains such as $\mathcal{S}$ and
$\mathcal{H}$ domains, the AdaIN generator only needs to specify the target
domain, since the source domain is automatically specified. For example, the
conversion to the $\mathcal{H}$ and $\mathcal{S}$ can be done by

$\displaystyle G_{H}({\bm{x}})$
$\displaystyle=G_{1,0}({\bm{x}}):=G({\bm{x}};F(0))$ (8) $\displaystyle
G_{S}({\bm{y}})$ $\displaystyle=G_{0,1}({\bm{y}}):=G({\bm{y}};F(1))$ (9)

where the first and second subscripts of the generator $G$ denote the source
and target domains, respectively. Accordingly, this can be implemented using a
single AdaIN code generator as shown in Fig. 2.

On the other hand, if there are additional middle domain data and we consider
the conversion from them, we should specify the source domain. In this case,
the generator should be implemented in the form

$\displaystyle
G_{\alpha,\beta}({\bm{z}}):=G({\bm{z}};F_{e}(\alpha),F_{d}(\beta))$ (10)

which implies the conversion from the source in an intermediate domain
$\alpha\mathcal{H}+(1-\alpha)\mathcal{S}$ to a target domain
$\beta\mathcal{H}+(1-\beta)\mathcal{S}$. Here,
$\alpha\mathcal{H}+(1-\alpha)\mathcal{S}$ represents a domain that lies
between $\mathcal{S}$ and $\mathcal{H}$ domains, whose conceptual distance is
determined by $\alpha$. For example, the middle kernel domain between
$\mathcal{S}$ and $\mathcal{H}$ can be referred to as
$0.5\mathcal{H}+0.5\mathcal{S}$.

In (10), to specify the source and target domains, the autoencoder is equipped
with the encoder AdaIN code $F_{e}(\alpha)$ and decoder code $F_{d}(\beta)$,
which are similar to (7) but independently implemented. Therefore, the split
AdaIN code generator architecture in Fig. 3 is necessary. Accordingly, with a
slight abuse of notation, two generators $G_{S}$ and $G_{H}$ can be
implemented using a single generator with split AdaIN:

$\displaystyle
G_{H}({\bm{x}})=G_{1,0}({\bm{x}}):=G({\bm{x}};F_{e}(1),F_{d}(0))$ (11)
$\displaystyle
G_{S}({\bm{y}})=G_{0,1}({\bm{y}}):=G({\bm{y}};F_{e}(0),F_{d}(1))$ (12)

As will become clear, the split AdaIN is essential to utilize the intermediate
kernel domain data for training, if available. Moreover, split AdaIN increases
the flexibility so that it allows conversion between any pairs of domains
along the interpolation path between the $\mathcal{H}$ and $\mathcal{S}$.

###  III-C Network Training

####  III-C1 When two domain training data are available

This corresponds to the scenario in [11]. In this case, the training can be
done by solving the following min-max optimization problem:

$\displaystyle\min\limits_{G,F}\max\limits_{D_{H},D_{S}}\ell_{total}(G,F,D_{S},D_{H})$
(13)

where the total loss is given by

$\begin{split}&\ell_{total}(G,F,D_{S},D_{H})=\\\
&-\lambda_{disc}\ell_{disc}(G,F,D_{S},D_{H})\\\
&+\lambda_{cyc}\ell_{cycle}(G,F)\\\ &+\lambda_{id}\ell_{id}(G,F)\end{split}$
(14)

where $\lambda_{disc}$, $\lambda_{cyc}$ and $\lambda_{id}$ denote the
weighting parameters for the discriminator, cycle and the identity loss terms.

Here, the discriminator loss $\ell_{disc}(G,F,D_{S},D_{H})$ is composed of
LSGAN loss [17]:

$\begin{split}&\ell_{disc}(G,F,D_{S},D_{H})=\\\ &\mathbb{E}_{{\bm{y}}\sim
P_{\mathcal{H}}}\left[|D_{H}({\bm{y}})\|^{2}_{2}\right]+\mathbb{E}_{{\bm{x}}\sim
P_{\mathcal{S}}}\left[\|1-D_{H}(G_{1,0}({\bm{x}}))\|^{2}_{2}\right]\\\
&+\mathbb{E}_{{\bm{x}}\sim
P_{\mathcal{S}}}\left[\|D_{S}({\bm{x}})\|^{2}_{2}\right]+\mathbb{E}_{{\bm{y}}\sim
P_{\mathcal{H}}}\left[\|1-D_{S}(G_{0,1}({\bm{y}}))\|^{2}_{2}\right]\end{split}$

where $\|\cdot\|_{2}$ is the $l_{2}$ norm, $G_{1,0}({\bm{x}})$ and
$G_{0,1}({\bm{y}})$ are defined in (8) and (9), respectively, and $D_{S}$
(resp. $D_{H}$) is the discriminator that tells the fake soft kernel (resp.
sharp kernel) images from real soft kernel (resp. sharp kernel) images.

The cycle loss $\ell_{cyc}(G,F)$ in (14) is defined as:

$\begin{split}\ell_{cyc}(G,F)=&\mathbb{E}_{{\bm{y}}\sim
P_{\mathcal{H}}}[\|G_{1,0}(G_{0,1}({\bm{y}}))-{\bm{y}}\|_{1}]\\\
&+\mathbb{E}_{{\bm{x}}\sim
P_{\mathcal{S}}}[\|G_{0,1}(G_{1,0}({\bm{x}}))-{\bm{x}}\|_{1}]\end{split}$ (15)

In addition, the identity loss in (14) is given by

$\begin{split}\ell_{id}(G,F)=&\mathbb{E}_{{\bm{y}}\sim
P_{\mathcal{H}}}[\|G_{1,0}({\bm{y}})-{\bm{y}}\|_{1}]\\\
&+\mathbb{E}_{{\bm{x}}\sim
P_{\mathcal{S}}}[\|G_{0,1}({\bm{x}})-{\bm{x}}\|_{1}]\end{split}$ (16)

Since $G_{1,0}$ is the network that converts input images to the $\mathcal{H}$
domain, $G_{1,0}({\bm{y}})$ is in fact the autoencoder for
${\bm{y}}\in\mathcal{H}$. Similar explanation can be applied to
$G_{0,1}({\bm{x}})$ for ${\bm{x}}\in\mathcal{S}$. Accordingly, the identity
loss in (16) can be interpreted as the auto-encoder loss when the target
domain images are used as inputs, and this observation is used when expanding
to three domains.

####  III-C2 When three domain training data are available

This scenario is unique which cannot be dealt with the architecture in [11].
Suppose that the intermediate kernel domain $\mathcal{M}$ is given by we
assume

$\displaystyle\mathcal{M}=0.5\mathcal{H}+0.5\mathcal{S}$

implying that the $\mathcal{M}$ is in the middle of the interpolation path
between $\mathcal{H}$ and $\mathcal{S}$ domain.

In this case, the training can be done by solving the following min-max
optimization problem:

$\displaystyle\min\limits_{G,F_{e},F_{d}}\max\limits_{D_{H},D_{S},D_{M}}\ell_{total}(G,F_{e},F_{d},D_{S},D_{H},D_{M})$
(17)

where the total loss is given by

$\begin{split}&\ell_{total}(G,F_{e},F_{d},D_{S},D_{H},D_{M})\\\
=&-\lambda_{disc}\ell_{disc}(G,F_{e},F_{d},D_{S},D_{H},D_{M})\\\
&+\lambda_{cyc}\ell_{cycle}(G,F_{e},F_{d})\\\
&+\lambda_{AE}\ell_{AE}(G,F_{e},F_{d})\\\
&+\lambda_{sc}\ell_{sc}(G,F_{e},F_{d})\end{split}$ (18)

Thanks to the availability of the intermediate domain $\mathcal{M}$, the
discriminator loss should be modified to include the additional domain
information:

$\begin{split}&\ell_{disc}(G,F_{e},F_{d},D_{S},D_{H},D_{M})=\\\
&\mathbb{E}_{{\bm{y}}\sim
P_{\mathcal{H}}}\left[|D_{H}({\bm{y}})\|^{2}_{2}\right]+\mathbb{E}_{{\bm{x}}\sim
P_{\mathcal{S}}}\left[\|1-D_{H}(G_{1,0}({\bm{x}}))\|^{2}_{2}\right]\\\
&+\mathbb{E}_{{\bm{x}}\sim
P_{\mathcal{S}}}\left[\|D_{S}({\bm{x}})\|^{2}_{2}\right]+\mathbb{E}_{{\bm{y}}\sim
P_{\mathcal{H}}}\left[\|1-D_{S}(G_{0,1}({\bm{y}}))\|^{2}_{2}\right]\\\
&+\mathbb{E}_{{\bm{x}}\sim
P_{\mathcal{S}}}\left[\|D_{S}({\bm{x}})\|^{2}_{2}\right]+\mathbb{E}_{{\bm{z}}\sim
P_{\mathcal{M}}}\left[\|1-D_{S}(G_{0.5,1}({\bm{z}}))\|^{2}_{2}\right]\\\
&+\mathbb{E}_{{\bm{z}}\sim
P_{\mathcal{M}}}\left[\|D_{M}({\bm{z}})\|^{2}_{2}\right]+\mathbb{E}_{{\bm{x}}\sim
P_{\mathcal{S}}}\left[\|1-D_{M}(G_{1,0.5}({\bm{x}}))\|^{2}_{2}\right]\\\
&+\mathbb{E}_{{\bm{y}}\sim
P_{\mathcal{H}}}\left[\|D_{H}({\bm{y}})\|^{2}_{2}\right]+\mathbb{E}_{{\bm{z}}\sim
P_{\mathcal{M}}}\left[\|1-D_{H}(G_{0.5,0}({\bm{z}}))\|^{2}_{2}\right]\\\
&+\mathbb{E}_{{\bm{z}}\sim
P_{\mathcal{M}}}\left[\|D_{M}({\bm{z}})\|^{2}_{2}\right]+\mathbb{E}_{{\bm{y}}\sim
P_{\mathcal{H}}}\left[\|1-D_{M}(G_{0,0.5}({\bm{y}}))\|^{2}_{2}\right]\end{split}$

where $G_{\alpha,\beta}$ is now defined by (10), and $D_{M}$ is the
discriminator that tells the fake intermediate kernel images from real
intermediate kernel images. The cycle loss in (18) should be also modified to
include the intermediate domain:

$\begin{split}\ell_{cyc}(G,F_{e},F_{d})=&\mathbb{E}_{{\bm{y}}\sim
P_{\mathcal{H}}}[\|G_{1,0}(G_{0,1}({\bm{y}}))-{\bm{y}}\|_{1}]\\\
&+\mathbb{E}_{{\bm{x}}\sim
P_{\mathcal{S}}}[\|G_{0,1}(G_{1,0}({\bm{x}}))-{\bm{x}}\|_{1}]\\\
&+\mathbb{E}_{{\bm{y}}\sim
P_{\mathcal{H}}}[\|G_{0.5,0}(G_{0,0.5}({\bm{y}}))-{\bm{y}}\|_{1}]\\\
&+\mathbb{E}_{{\bm{x}}\sim
P_{\mathcal{S}}}[\|G_{0.5,1}(G_{1,0.5}({\bm{x}}))-{\bm{x}}\|_{1}]\\\
&+\mathbb{E}_{{\bm{z}}\sim
P_{\mathcal{M}}}[\|G_{0,0.5}(G_{0.5,0}({\bm{z}}))-{\bm{z}}\|_{1}]\\\
&+\mathbb{E}_{{\bm{z}}\sim
P_{\mathcal{M}}}[\|G_{1,0.5}(G_{0.5,1}({\bm{z}}))-{\bm{z}}\|_{1}]\end{split}$
(19)

The auto-encoder loss in (18), which is an extension of the identity loss in
(16), is given by

$\begin{split}\ell_{AE}(G,F_{e},F_{d})=&\mathbb{E}_{{\bm{y}}\sim
P_{\mathcal{H}}}[\|G_{0,0}({\bm{y}})-{\bm{y}}\|_{1}]\\\
&+\mathbb{E}_{{\bm{x}}\sim
P_{\mathcal{S}}}[\|G_{1,1}({\bm{x}})-{\bm{x}}\|_{1}]\\\
&+\mathbb{E}_{{\bm{z}}\sim
P_{\mathcal{M}}}[\|G_{0.5,0.5}({\bm{z}})-{\bm{z}}\|_{1}]\end{split}$ (20)

Finally, one unique loss which was not present in the two domain training is
the the self-consistency loss, which imposes the constraint that the style
transfer through intermediate domain should be the same as the direct
conversion. This can be represented by

$\begin{split}&\ell_{sc}(G,F_{e},F_{d})=\\\ &\mathbb{E}_{{\bm{y}}\sim
P_{\mathcal{H}}}[\|G_{0.5,1}(G_{0,0.5}({\bm{y}})-G_{0,1}({\bm{y}})\|_{1}]\\\
&+\mathbb{E}_{{\bm{x}}\sim
P_{\mathcal{S}}}[\|G_{0.5,0}(G_{1,0.5}({\bm{x}})-G_{1,0}({\bm{x}})\|_{1}]\end{split}$
(21)

These loss functions are illustrated in Fig. 1.

##  IV Methods

###  IV-A Data Acquisition

To verify the proposed continuous kernel conversion, we use the following
dataset provided by Gachon University Gil Hospital, Korea.

####  IV-A1 Head dataset

Head images from 11 patients were obtained (SOMATOM Definition Edge, Siemens
Healthineers, Germany). Each patient had two sets of images, each of which was
generated with sharp and soft kernels, respectively. Specific details of
kernels will be explained later. Each patient data consists of around 50
slices. Accordingly, total 540 slices were achieved. One patient was excluded
because of its different image matrix size ($571\times 512$ vs. $512\times
512$). We used seven patients for training, two for validation, the other one
patient for test (44 slices). Totally, we used ten patients consisting of
three men and seven women with mean age $39.7\pm 19.1$ years.

####  IV-A2 Facial bone dataset

Facial bone images of 12 patients were obtained. Similar to the head dataset,
each patient data was composed of reconstructed images with sharp and soft
kernels for bone and brain, respectively. Each patient data involved different
number of slices from 48 to 189 slices, producing 1683 slices of facial bone
images for each kernel. One patient was excluded because of its different
image matrix size ($512\times 534$ vs. $512\times 512$). We used eight
patients for training, two for validation, the remaining one patient for test
(165 slices). Totally, we used 11 patients consisting of four men and seven
women with mean age $32.3\pm 15.4$ years.

Additional face bone images of eight patients were obtained that include three
kernel domain data. Different from the dataset above, each patient data was
composed of reconstructed images with sharp, soft, and middle kernels. Each
patient data includes from 167 to 209 slices. The dataset has totally 1491
slices of facial bone images for each kernel. We used five patients for
training, two for validation, the other one patient for test (209 slices).
Totally, the dataset was composed of five men and three women with mean age
$38.6\pm 24.5$ years.

####  IV-A3 Hypopharyngeal cancer dataset

The dataset involved CT images of one patient suffering from hypopharyngeal
cancer, which led to the demand for cartilage abnormality detection. The
images covered from head to chest of its patient. The single volume was
composed of 110 slices. We used a half of the slices for fine-tuning the model
after training, and the other half for inference.

###  IV-B Kernels

For head dataset, J30s and J70h kernels were used. For facial bone dataset,
J40s and J70h kernels were applied to reconstruct CT images. J30s and J40s
kernels are low pass filters which are adequate for soft tissue. J70h kernel
is a representative high pass filter and usually used to observe bones. In
this paper, we refer to low pass filters such as J30s and J40s kernels as soft
kernels, and high pass filters such as J70h kernel as sharp kernels.

Different from the two datasets, hypopharyngeal cancer images were
reconstructed with Br44 kernel, which has similar property as a soft kernel.
In facial bone dataset for multi-domain learning, Hr40, Hr49, and Hr68 kernels
were utilized to reconstruct CT images. Hr40 and Hr68 kernels correspond to
soft and sharp kernels, respectively. The Hr49 kernel has intermediate
properties of Hr40 and Hr68 kernels. Therefore, we refer to Hr49 kernel as the
intermediate kernel in this paper.

###  IV-C Network Architecture

####  IV-C1 Autoencoder

The autoencoder used in this paper is based on the U-Net architecture with
pooling layer implemented by polyphase decomposition [18]. It has skip
connections between encoder and decoder parts, which enable inputs and outputs
to share information. Also, the input is added pixel-wise to the output at the
end of the network. The network, therefore, learns residuals.

In conventional encoder-decoder networks, the input images go through several
down-sampling layers until a bottleneck layer, in order for a network to
extract low frequency information. Through the down-sampling layers, the
network necessarily loses significant amount of information which plays a
crucial role in auto-encoder learning. Therefore, instead of using pooling
layers, we used all the given information which can be achieved by lossless
decomposition using polyphase decomposition as shown in Fig. 4(a).
Specifically, at the layers where a pooling operation is required, we arranged
all the pixels into four groups. It can be thought as having a 2$\times$2
filter with stride of 2 in order not to make overlapping. The first pixels of
each sub-region gather together to form new output. Also, the second pixels
assemble another output. In the same manner, the third and fourth pixels make
the third and fourth outputs, respectively. These four outputs are stacked
along a channel direction. After this sub-pixel pooling operation, the size of
final output would be reduced by half while the number of channels would
increase fourfold.

Figure 4: (a) Pooling layer using polyphase decomposition process. Pixels are
divided into four groups and each group composes one image of a reduced size.
Then, the groups are stacked along channel direction. (b) Unpooling layer
using polyphase recomposition.

Figure 5: Architecture of the discriminator.

Unpooling operation using polyphase recomposition can be done in exactly the
opposite way of polyphase pooling operation as shown in Figure 4(b).
Interestingly, unpooling operation in a decoder part can be extended to a
transposed convolution. Transposed convolution involves both polyphase
recomposition and filtering operation. While polyphase unpooling requires
fixed position for each pixel, transposed convolution does not have any
positional condition to follow. When using transposed convolution, the network
parameters are, therefore, chosen to be optimal without any specific
restriction. This results in experimentally better performance of the network
with transposed convolution compared to unpooling operation using polyphase
recomposition. Thus, we used transposed convolution in our proposed Polyphase
U-Net.

####  IV-C2 Discriminator

As a discriminator, we used PatchGAN [19] which shows good performance at
capturing high frequency information because it focuses only on the scale of
its patches, not the entire image. The illustration of the discriminator is
shown in Figure 5. The input whose size is $128\times 128$ with one channel
passes through a convolution layer with stride two. Then, next convolution
layer with stride two gets the feature map of size $62\times 62$. The feature
maps go through two successive convolution layers with stride one. Finally,
the output is convolved with the last convolution layer. Kernel size of all
the convolution layers in the discriminator is $5\times 5$. Final output size
is $24\times 24$ which was chosen empirically.

####  IV-C3 AdaIN Code Generator

Details about the architecture of AdaIN code generator are illustrated in
Figs. 2 and 3. In the two-domain switchable cycleGAN (Fig. 2), only one shared
code generator is implemented and connected to both encoder and decoder parts
of the generator. One vector of size $128$ is input to the shared code
generator. The shared code generator includes four fully connected layers with
output size of 64. The final output code is given to 10 convolution blocks in
the generator of cycleGAN. Since these convolution blocks have different
number of channels, mean and variance code vectors are generated separately
for each convolution block. For the mean code vectors, one fully connected
layer is applied. ReLU activation layer is applied in addition to one fully
connected layer for the variance code vector because of non-negative property
of variance. Meanwhile, in case of three domain conversion, we utilized
switchable cycleGAN with split AdaIN, which requires source domain and target
domain code generators for encoder and decoder parts of the U-net,
respectively, as shown in Figure 3. All the other details are same as in two-
domain switchable cycleGAN.

###  IV-D Training detail

The input images were randomly cropped into small patches of size $128\times
128$ during the training. They were also randomly flipped both horizontally
and vertically. Despite small size of dataset, patch-based training with
random flipping provided data augmentation and enabled more stable training
[20]. The learning rate was set as $10^{-4}$ and $10^{-5}$ for Head and Facial
bone dataset, respectively. The Adam optimization algorithm [21] was used, and
the momentum parameters $\beta_{1}=0.9,\beta_{2}=0.999$. We saved models with
the best quantitative results with validation data set and tested the models
using 5-fold cross validation scheme. In case of facial bone dataset for
multi-domain learning, 3-fold cross validation was conducted. We implemented
the networks using PyTorch library [22]. We trained the networks on two NVIDIA
GeForce GTX 2080 Ti.

###  IV-E Comparative studies

First, we compared our algorithm with classical kernel conversion approaches.
Classical methods considered kernel conversion as kernel smoothing and
sharpening. Specifically, to convert sharp kernel images into soft kernel
ones, frequency band decomposition and alteration of energy in each band were
performed [23]. For the conversion from soft to sharp kernel, we sharpened the
soft kernel images using Laplacian filter kernel and Wiener-Hunt deconvolution
method with the estimated point spread function [24].

Moreover, we compared our proposed model with supervised learning method. For
supervised learning method, the same generator architecture as our method was
used [18] and the mean squared error loss between output and ground truth
images was used. Two different networks were trained separately for two
opposite directional kernel translation. Batch size was 8 and 32 for Head and
Facial bone dataset, respectively. Other settings for training were same as
the proposed model.

In addition to the supervised learning method, we conducted comparative
studies using vanilla cycleGAN [7]. Again, same U-Net architecture with
polyphase decomposion was used for two generators. Due to the need to train
two distinct generators, the total number of trainable weights increased
compared to our method (see Figure 1). Batch size was 8 and 40 for Head and
Facial bone dataset, respectively. The other training details were same as the
proposed model.

###  IV-F Evaluation metrics

Since ground-truth data are available for the two types of dataset, we used
the peak signal to noise ratio (PSNR) as our quantitative metric, which is
defined as follows:

$PSNR=10\log_{10}\left(\frac{MAX_{x}^{2}}{MSE}\right)$ (22)
$MSE=\frac{1}{N_{1}N_{2}}\sum_{i=0}^{N_{1}-1}\sum_{j=0}^{N_{2}-1}[x_{i,j}-\hat{x}_{i,j}]$
(23)

where $N_{1}$ and $N_{2}$ are row and column dimensions of the images,
$x_{i,j}$ denotes the $(i,j)$-th pixel, and $MAX_{x}$ is the maximum possible
pixel value of image $x$. We also used structural similarity (SSIM) index [25]
which is defined as

$SSIM(x,\hat{x})=\frac{(2m_{x}m_{\hat{x}}+c_{1})(2\sigma_{x\hat{x}}+c_{2})}{(m_{x}^{2}+m_{\hat{x}}^{2}+c_{1})(\sigma_{x}^{2}+\sigma_{\hat{x}}^{2}+c_{2})}.$
(24)

where $m$ is average of the image, $\sigma$ is variance of the image, and
$\sigma_{x\hat{x}}$ is covariance of the images $x$ and $\hat{x}$. The two
variables $c_{1}=(k_{1}L)^{2}$ and $c_{2}=(k_{1}L)^{2}$ are used to stabilize
the division where $L$ is the dynamic range of the pixel intensities and
$k_{1}=0.01,k_{2}=0.03$ by default.

TABLE I: Quantitative comparison of various methods in two-domain learning Head | PSNR | SSIM
---|---|---
sharp | soft | sharp | soft
Classical method | 12.4335 | 14.4620 | 0.6633 | 0.7401
Supervised (MSE Loss) | 32.6624 | 24.2981 | 0.9022 | 0.8336
Vanilla CycleGAN | 30.8758 | 21.9194 | 0.8913 | 0.8161
Switchable CycleGAN | 31.6671 | 23.9193 | 0.8922 | 0.8766
Facial Bone | PSNR | SSIM
sharp | soft | sharp | soft
Classical method | 13.1227 | 10.4685 | 0.5198 | 0.5369
Supervised (MSE Loss) | 28.3261 | 21.1531 | 0.8079 | 0.8088
Vanilla CycleGAN | 25.5337 | 17.4077 | 0.7135 | 0.7966
Switchable CycleGAN | 26.6328 | 19.5712 | 0.7563 | 0.8336

Figure 6: Kernel conversion results in two-domain by the classical method,
supervised learning, vanilla cycleGAN, and the proposed switchable cycleGAN.
The first two rows are from Head dataset and the last two rows are from Facial
bone dataset.

##  V Experimental results

###  V-A Comparison with previous methods

First, we compared our proposed model with classical kernel conversion method,
supervised learning method, and vanilla cycleGAN [7]. In Figure 6 and Table I,
the images generated from the classical method showed the worst performance in
both quantitative and qualitative perspectives. Although higher PSNR and SSIM
values were obtained for supervised learning in Table I, visual investigation
shows that some blurring artifacts are presented with the supervised learning
as shown in Figure 6. More details are as follows:

Figure 7: Continuous interpolation of kernel images along an optimal transport
path.

Figure 8: Continuous interpolation of kernel images along an optimal transport
path.

####  V-A1 Soft kernel from sharp kernel

Image translation from sharp kernel images to soft kernel images is shown in
the first and third rows of Figure 6. Soft kernel images should clearly
identify soft tissues such as blood vessels, not bones. It is shown that
performance of our proposed method is obviously better than those of
supervised and vanilla cycleGAN methods. In particular, as can be seen in the
first row of Fig. 6, the grid-like artifacts appeared on the result of the
supervised model. The images generated from cycleGAN could not follow the data
distribution and the pixel intensities around the bone were significantly
increased. However, these artifacts were not shown in the results of our
method. We believe that the artifacts for the supervised learning and vanilla
cycle GAN may occur because of the limited training data set. Nonetheless, we
have found that our proposed model has its strength in capturing the data
distribution much more effectively.

The third row of Fig. 6 also shows that the bone shapes reconstructed by the
conventional method are obviously different from the target images. In
addition, the synthesized soft kernel images by the traditional method and the
supervised learning model show significantly more noise compared to the other
unsupervised learning methods. The results clearly show that our model was
better at translating sharp kernel images into soft kernel images than the
supervised or vanilla cycleGAN.

####  V-A2 Sharp kernel from soft kernel

Sharp kernel images are usually used to gather information about clear bone
outlines. The images generated from soft kernel images should contain
information about a clear delineation of the bone.

The results of the image translation from soft kernel images to sharp kernel
images are shown in the second and fourth rows of Figure 6. The traditional
method sharpened the input soft kernel images, but their sharpness was far
below that of the target images. In the meantime, the results of all three
deep-learning methods were similar. Not only could they enhance sharp bones
outlines from blurry, soft kernel images, but they could also follow the
texture of sharp kernel domain. However, some artifacts occurred along bones
in the supervised and cycleGAN methods (Figure. 6).

Figure 9: Cartilage abnormality is clearly seen as $\beta$ value varies. The
left most column is MR images with contrast enhancement. Right thyroid
cartilage tumor infiltration can be observed from MR images whereas it is hard
to detect on CT images which are on the right most column. They were
reconstructed with Br44 kernel which has properties of the soft kernel. The
images in the middle were generated from the images in the right most column.
Compared to MR images, the generated CT images showed a comparable level of
diagnostic ability to check cartilage abnormality. With input kernel images on
the right most column, sharp kernel images were generated using $\beta=0$, and
soft kernel images using $\beta=1$

.

Figure 10: Interpolation from sharp to soft kernel results in two-domain and
three-domain learning. The first row is the switchable cycleGAN with single
AdaIN code generator while the second row is the switchable cycleGAN with
split AdaIN. The last row is real CT images. The right and middle ones
correspond to target images and the left one is the input image.

####  V-A3 Contributing factors of AdaIN to improved quality

Image translation by switchable cycleGAN from soft kernel to sharp kernel
images is actually same as vanilla cycleGAN since it utilizes mean and
variance of its own instead of those estimated from AdaIN. Thus, the sharp
kernel images that were generated with our model showed in qualitative and
quantitative results an almost equivalent quality to that with the vanilla
cycle GAN. For SSIM results on the head data, paired $t$-test was performed on
the sharp images generated by vanilla and switchable cycleGAN. The $p$-value
was larger than $0.05$, which means that our switchable cycleGAN generated
sharp kernel images that were not significantly different from those by
vanilla cycleGAN in terms of SSIM metric (Table I).

On the other hand, the model in the other direction, which translates sharp
kernel images into soft kernel images, contains AdaIN, so that the mean value
and variance of the sharp kernel domain are scaled or added. It should be
noted that the image quality of our proposed model has been significantly
improved in relation to that of vanilla CycleGAN. It is supported in both
qualitative and quantitative results.

In particular, artifacts could be observed in soft kernel images generated
from the vanilla cycleGAN method, and this made it difficult to accurately
distinguish the boundaries between bone and soft tissue. In addition, the
pixel intensities around the bone have been increased in the vanilla cycleGAN
images. However, these artifacts disappeared in the generated soft kernel
images of our model (Figure 6). In addition, the evaluation metrics when
generating soft kernel images have been significantly improved. For $t$-test
on the PSNR and SSIM results between the vanilla cycleGAN and the proposed
method, the $p$-values were less than 0.01 when generating soft kernel images
from the head dataset (Table I). Furthermore, when comparing PSNR results from
the soft kernel images of the facial bone dataset, the $p$-value was less than
$0.05$. From these analyses, we could see that switchable cycleGAN showed
significantly improved performance in generating soft kernel images compared
to the vanilla cycleGAN. The effectiveness of AdaIN was clearly demonstrated
by this different increase in image quality between two opposite directions.

Although the small data set can be a major factor in overfitting the network,
thanks to AdaIN the number of model parameters in our model has been halved
and model learning has been proven to be more stable [11].

###  V-B Interpolation between two kernel images

During the training phase, we set $\beta$ as one if soft kernel images are
generated from sharp kernel images, and $\beta$ as zero if sharp kernel images
are generated from soft kernel images. Once the training was done, at the
inference phase different $\beta$ values can be applied to generate
interpolating kernel images along an optimal transport path (Fig. 7 and 8).
Interestingly, we were able to see objects that could not be observed with
previous kernels.

Figure 11: Interpolation from middle to sharp or soft kernel images in three-
domain learning. The window level and width of the first row is 400 and 1500
and those of the second row are 50 and 120 (HU).

In particular, once our proposed method is applied, cartilage abnormality can
be detected as shown in Figure 9. For hypopharyngeal cancer case, it is
important to check whether thyroid cartilage is invaded or not. This
identification usually relies on magnetic resonance imaging since it is hard
to observe the abnormality from CT images. Since the cartilage has high
attenuation like bone, sharp kernel images could be potentially useful in
determining if the cancer has penetrated or not. Unfortunately, CT images are
not reconstructed with sharp kernels to evaluate this type of cancer. Instead,
soft-kernel CT images are usually collected for other diagnostic purpose. With
the help of our proposed method, we conjectured that discontinuity in outer
border of cartilage could be discovered from the generated images from soft
kernel images.

Specifically, the hypopharyngeal cancer data contains only Br44 kernel images
which have soft kernel properties, which means there exists no target domain
images. Therefore, we trained the model with some slices of these Br44 kernel
images and J70h kernel images from Facial bone dataset as target domain data.
Specifically, our model is trained to generate J70h kernel images when
$\beta=0$, and Br44 kernel images when $\beta=1$.

In Figure 9, for $\beta=0$, overshoots and undershoots occurred around the
bone outlines, which hindered close observation on bone shapes. However, more
accurate examination could become possible by adjusting $\beta$ values to be
larger than zero. Specifically, the images from $\beta$ larger than zero could
be generated with smoother texture compared to those from $\beta=0$. This
resulted in clear delineation with no more undershoot or overshoot around the
bone but maintaining its shape as shown Figure 9. This example clearly shows
the clinical usage of the continuous conversion of the filter kernels along
the optimal transport path. It is reminded that the simple image domain
interpolation does not provide such synergistic information as ours since it
is a simple mixing of image domain textures. See Section VI for more details.

Figure 12: (a) AdaIN-based feature domain interpolation. (b) Image domain
interpolation. The experiment was conducted on facial bone dataset for multi-
domain learning. The left-most column is sharp kernel images used as input and
the other columns are the generated images. The second and the fourth rows
correspond to difference images between the generated images and real soft
kernel images. The difference image window is (-200, 200) HU.

###  V-C Three domain learning

One of the limitations of switchable CycleGAN with two domains is that the
effect of $\beta$ is not gradual. Although the generated soft kernel images
are subject to gradual changes in the range from 0.9 to 1.0 in Fig. 8, outside
of this range, the generated images showed no noticeable changes.

Therefore, another important contribution of this paper is to show that this
non-uniform dependency on $\beta$ can be addressed if the intermediate domain
data is available. Specifically, by utilizing the middle kernel images during
the training, the generated images with $\beta=0.5$ could have similar
property of that of real middle kernel images. This can be confirmed in Figure
10. Here, the images at $\beta=0.5$ from three-domain switchable cycleGAN were
similar to real images of Hr49 kernel, whereas the two domain switchable
cycleGAN produces no noticeable changes between 0$\sim$ 0.7. In fact, the
generated images changed gradually from $\beta=0$ to $\beta=1$ in the three-
domain learning.

In addition, generating sharp or soft kernel images from middle ones is only
possible by using three domain data (Figure 11). Here, intermediate kernel
images can be utilized to generate other kernel images, which further helped
to enhance the performance of the network with self-consistency loss.
Specifically, the middle kernel images generated from sharp kernel images can
be used as input to generate soft kernel images. Then these soft kernel images
are forced to be identical to the soft kernel images generated directly from
sharp kernel ones. This constraint helped to improve the three domain
conversion of the proposed model. The role of the self-consistency loss is
described in more detail later.

Additionally, the performance of the model slightly enhanced with the three
domain learning, as can be seen in Table II in terms of PSNR and SSIM values.
Accordingly, we can conclude that three domain conversion would be preferable
with respect to the quality of the generated images and the effective range of
interpolation.

TABLE II: Quantitative comparison of various methods with facial bone dataset for multi-domain learning Domain |  Method | PSNR | SSIM
---|---|---|---
sharp | soft | sharp | soft
3 |  Switchable cycleGAN with self-consistency loss (Ours) | 25.7611 | 19.2320 | 0.7137 | 0.8524
Switchable cycleGAN without self-consistency loss | 24.2031 | 16.6467 | 0.6658 | 0.8194
2 |  Switchable cycleGAN | 25.3679 | 18.8973 | 0.7345 | 0.8139
Vanilla cycleGAN | 25.2707 | 17.6606 | 0.7084 | 0.8240

##  VI Discussion

###  VI-A Strengths of Switchable cycleGAN

####  VI-A1 Difference from image domain interpolation

One of the strengths of switchable cycleGAN is that it can generate images
between two image domains through feature domain interpolation. Specifically,
the feature domain interpolation between two domains can be performed using
AdaIN code vector interpolation. Figure 12 illustates the difference between
the feature domain and image domain interpolation. When generating soft kernel
images from sharp kernel ones, we could not get middle kernel images from
image domain interpolation. The noise level was not reduced enough to get
smooth texture of middle kernel. Also, these noises were spread on the entire
area until $\beta$ values became one to generate soft kernel images. On the
other hand, as can be seen from the difference images in Figure 12, our
feature domain interpolation gradually changes the texture of the images.

####  VI-A2 Robustness to dataset size

Since switchable cycleGAN shares a single generator, the number of parameters
of the model is reduced, which results in robust training even on small
dataset. Similar to the experiments conducted in [11], we also investigate the
robustness to the training dataset size using Head dataset. In Figure 13,
vanilla cycleGAN showed sudden decreases in both PSNR and SSIM performance as
dataset size decreased. However, the switchable cycle GAN showed a moderate
drop in performance with the reduced training data set. In particular, our
method trained with only a quarter of the data set leads to better performance
compared to the vanilla cycleGAN trained with the entire data set.

Figure 13: Comparison between vanilla cycleGAN and switchable cycleGAN according to dataset size. The two methods have similar number of trainable parameters.  TABLE III: Quantitative comparison on the effect of the number of parameters of the generator Head | PSNR | SSIM
---|---|---
sharp | soft | sharp | soft
Vanilla CycleGAN (half parameters) | 31.3088 | 23.3729 | 0.8909 | 0.8461
Switchable CycleGAN | 31.6671 | 23.9193 | 0.8922 | 0.8766

Figure 14: Comparison between U-Net [26] and Polyphase U-Net.

####  VI-A3 The number of parameters of the generator

We could expect the performance improvement thanks to the shared generator of
the the switchable cycleGAN. This improvement results from the smaller number
of parameters in the network as well as from the shared generator. Though the
network size reduction provided robustness to the training dataset size, we
found that the benefit from the shared network architecture is more profound
from following experiments.

Specifically, as can be seen from the Table III, switchable cycleGAN showed
significantly increased performance over the vanilla cycleGAN. From the paired
$t$-test between the vanilla cycleGAN and the proposed method, $p$-value was
less than $0.05$ on both PSNR and SSIM results of the soft kernel images.
Also, when comparing PSNR results on the sharp kernel images between the
vanilla cycleGAN and the proposed method, $p$-value was less than $0.05$. In
this comparative experiment, the vanilla cycleGAN involves two different
generators with the number of parameters reduced by half so that the total
number of trainable parameters are about the same as our switchable cycleGAN.
Nonetheless, our switchable cycleGAN outperformed vanilla cycleGAN. This
suggests that shared generator is a key that has contributed to learning the
common representation of data and improving the performance of the model.

###  VI-B Effect of Polyphase U-Net

Here, we investigated the differences between U-Net and Polyphase U-Net as a
generator in our switchable cycleGAN. For a fair comparison with the original
U-Net [26], we reduced the number of channels so that the total number of
parameters in the Polyphase U-Net is around 30 millions. As shown in Figure
14, Polyphase U-Net showed enhanced performance compared to the U-Net [26].
The images generated from the Polyphase U-Net were very similar to the target
images with a sharp kernel in terms of outlines of bones, but also textures.

Specifically, the shapes of the bones followed the target images well and were
clear enough to identify the small structures. In addition, it was difficult
to identify a broken nasal bone with the image reconstructed with a soft
kernel (Figure 14, third row). Only the existence of the bone could be
confirmed. To our expectation, the shape of the broken bone is clearly shown
in the kernel conversion by our method using the Polyphase U-Net architecture.
However, the first and the third row of Figure 14 show that some irrelevant
patterns around bones appeared when the conventional U-Net [26] is used as a
generator. In addition, in the generation of soft kernel images, Polyphase
U-Net also provides slightly better results than the conventional U-Net. As
shown in the fourth row of Figure 14, the texture of the images generated by
U-Net, which should be as smooth as the target image, is noisy.

The improved performance of the Polyphase U-Net was also reflected in the
quantitative results (Table IV). Since the problem of information loss during
pooling operation was resolved in the Polyphase U-Net, it showed better
performance compared to the original U-Net.

TABLE IV: Quantitative Comparison between U-Net and PolyPhase U-Net Head | PSNR | SSIM
---|---|---
sharp | soft | sharp | soft
CycleGAN with U-Net | 31.2706 | 23.2257 | 0.8887 | 0.8615
CycleGAN with PolyPhase U-Net | 31.6671 | 23.9193 | 0.8922 | 0.8766
Facial Bone | PSNR | SSIM
sharp | soft | sharp | soft
CycleGAN with U-Net | 26.1938 | 18.7120 | 0.7490 | 0.7912
CycleGAN with PolyPhase U-Net | 26.6328 | 19.5712 | 0.7563 | 0.8336

###  VI-C Role of self-consistency loss in multi-domain learning

The self-consistency loss was unique for three-domain learning of switchable
cycleGAN (see (14)). As shown in Figure 15, various artifacts appeared when
training the model without this term. Specifically, in the generation of sharp
kernel images, artifacts occurred along the outlines of bones, and irrelevant
patterns appeared at the bottom of the generated images. However, these
artifacts are no more present when using the self-consistency loss. This
qualitative observation can be also supported by quantitative results in Table
II. Compared to the model trained without self-consistency loss, all the PSNR
and SSIM values were higher in the model trained with self-consistency loss.

Figure 15: Ablation study results on self-consistency loss. The first row
images are generated images without the self-consistency loss, and the second
row images are generated with the self-consistency loss. The last row shows
the real images. The right most real image is input, and the left and middle
ones are target sharp and middle kernel images, respectively.

###  VI-D Autoencoder and Cycle-Consistency

Note that the supervised learning should be able to preserve the fidelity of
the structures since it has pixelwise constraint. By contrast, the
unsupervised learning methods lacks this constraint. As a result, one may
wonder whether our cycleGAN method may generate fake structures. However, our
results show an opposite phenomenon. This interesting phenomenon is due to the
use of identity-loss and cycle-consistency loss. As reported in our prior work
[27], when the cycle-consistency and identity losses were missing, spurious
artifacts occured in the low-dose CT denoising, but proper use of both cycle-
consistency and identity loss prevented this from happening.

Similar to [27], in this paper we used both identity and cycle-consistency
loss. The identity loss, which is equal to the auto-encoder loss for the case
of two-domain learning, plays a role to preserve the structure by providing
pixel-wise constraint. Additionally, the cycle-consistency loss in cycleGAN
also poses strong pixel-wise constraint in that it ensures self-consistency
when reverting to the original domain, which prevent from fake structures
being created. In contrast to the CycleGAN, the standard GAN lacks these two
constraint, which leads to the creation of a falsified structure as reported
in [27].

###  VI-E Limitation

Our method is not free of limitations. Though the proposed method could
generate kernel images better than other methods and even comparable with the
target images, fine details were not perfectly preserved in some cases such as
trabecular bones. Also, as shown in Figure 8, the pixel value has increased
around the brain-bone interface. For clinical applications such as cancer
diagnosis, it is not a serious problem, but it could be misunderstood as
bleeding in some applications. Although one could confirm whether it is a true
bleeding or an artifacts in this by adjusting $\beta$ values, this needs to be
verified in real clinical scenario. In addition, overshoot and undershoot
around the bones prevented an accurate assessment of the thickness of the
bones or nasal cavities (the first row of Figure 9 and the third row of Figure
14). This edge enhancement usually occurred when generating sharp kernel
images using $\beta=0$. Though the edge enhancement problem can be relieved by
using $\beta$ values greater than zero, more systematic study may be
necessary.

##  VII Conclusion

Different properties of two kernels, sharp and soft kernels, produce two types
of reconstructed images. Here, we have proposed a post-hoc image domain
translation between these two kernels to generate one kernel image from the
other. Our proposed method was based on a switchable cycleGAN in combination
with the adaptive instance normalization. Thanks to AdaIN, the conversion of
kernels could be carried out with a single generator, and different images
were generated along the optimal transport path by the combination of the two
given kernels. Furthermore, intermediate domain kernel images can be
effectively utilized during the training by introducing split AdaIN code
generators, which significantly improved the feature domain interpolation
performance. The improved performance of our proposed model has been proven
with extensive experimental results.

## References

  * [1] P. A. Setiyono, D. Rochmayanti, A. Nino Kurniawan, A. Nugroho Setiawan _et al._ , “The optimization of mastoid CT image using windows and kernel reconstructions,” _JPhCS_ , vol. 1471, no. 1, p. 012015, 2020.
  * [2] K. L. Boedeker, M. F. McNitt-Gray, S. R. Rogers, D. A. Truong, M. S. Brown, D. W. Gjertson, and J. G. Goldin, “Emphysema: effect of reconstruction algorithm on CT imaging measures,” _Radiology_ , vol. 232, no. 1, pp. 295–301, 2004.
  * [3] K. L. Weiss, R. S. Cornelius, A. L. Greeley, D. Sun, I.-Y. J. Chang, W. O. Boyce, and J. L. Weiss, “Hybrid convolution kernel: optimized CT of the head, neck, and spine,” _American Journal of Roentgenology_ , vol. 196, no. 2, pp. 403–406, 2011.
  * [4] S. Takagi, H. Nagase, T. Hayashi, T. Kita, K. Hayashi, S. Sanada, and M. Koike, “Combined multi-kernel head computed tomography images optimized for depicting both brain parenchyma and bone,” _Journal of X-Ray Science and Technology_ , vol. 22, no. 3, pp. 369–376, 2014.
  * [5] A. D. Missert, L. Yu, S. Leng, J. G. Fletcher, and C. H. McCollough, “Synthesizing images from multiple kernels using a deep convolutional neural network,” _Medical physics_ , vol. 47, no. 2, pp. 422–430, 2020.
  * [6] S. M. Lee, J.-G. Lee, G. Lee, J. Choe, K.-H. Do, N. Kim, and J. B. Seo, “CT image conversion among different reconstruction kernels without a sinogram by using a convolutional neural network,” _Korean journal of radiology_ , vol. 20, no. 2, pp. 295–303, 2019.
  * [7] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2223–2232.
  * [8] B. Sim, G. Oh, K. Jeongsol, J. Chanyong, and J. C. Ye, “Optimal transport driven cyclegan for unsupervised learning in inverse problems,” _SIAM Jour. Imaging Sciences (to appear), Also available as arXiv preprint arXiv:1909.12116_ , 2020.
  * [9] C. Villani, _Optimal transport: old and new_. Springer Science & Business Media, 2008, vol. 338.
  * [10] G. Peyré, M. Cuturi _et al._ , “Computational optimal transport,” _Foundations and Trends in Machine Learning_ , vol. 11, no. 5-6, pp. 355–607, 2019.
  * [11] J. Gu and J. C. Ye, “Adain-based tunable cyclegan for efficient unsupervised low-dose ct denoising,” _IEEE Transactions on Computational Imaging_ , vol. 7, pp. 73–85, 2021.
  * [12] X. Huang and S. Belongie, “Arbitrary style transfer in real-time with adaptive instance normalization,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 1501–1510.
  * [13] J. Choe, S. M. Lee, K.-H. Do, G. Lee, J.-G. Lee, S. M. Lee, and J. B. Seo, “Deep learning–based image conversion of CT reconstruction kernels improves radiomics reproducibility for pulmonary nodules or masses,” _Radiology_ , vol. 292, no. 2, pp. 365–373, 2019.
  * [14] L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2414–2423.
  * [15] Y. Mroueh, “Wasserstein style transfer,” _arXiv preprint arXiv:1905.12828_ , 2019.
  * [16] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” _arXiv preprint arXiv:1607.08022_ , 2016\.
  * [17] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. Paul Smolley, “Least squares generative adversarial networks,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 2794–2802.
  * [18] B. Kim and J. C. Ye, “Mumford–shah loss functional for image segmentation with deep learning,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 1856–1866, 2019.
  * [19] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1125–1134.
  * [20] Y. Oh, S. Park, and J. C. Ye, “Deep learning COVID-19 features on CXR using limited training data sets,” _IEEE Transactions on Medical Imaging_ , vol. 39, no. 8, pp. 2688–2700, 2020.
  * [21] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [22] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” 2017.
  * [23] L. Gallardo-Estrella, D. A. Lynch, M. Prokop, D. Stinson, J. Zach, P. F. Judy, B. van Ginneken, and E. M. van Rikxoort, “Normalizing computed tomography data reconstructed with different filter kernels: effect on emphysema quantification,” _European radiology_ , vol. 26, no. 2, pp. 478–486, 2016\.
  * [24] Z. Al-Ameen, G. Sulong, M. Gapar, and M. Johar, “Reducing the gaussian blur artifact from CT medical images by employing a combination of sharpening filters and iterative deblurring algorithms,” _Journal of Theoretical and Applied Information Technology_ , vol. 46, no. 1, pp. 31–36, 2012.
  * [25] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE transactions on image processing_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [26] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [27] E. Kang, H. J. Koo, D. H. Yang, J. B. Seo, and J. C. Ye, “Cycle-consistent adversarial denoising network for multiphase coronary CT angiography,” _Medical physics_ , vol. 46, no. 2, pp. 550–562, 2019.


# On the Robustness of Deep Reinforcement Learning in IRS-Aided Wireless
Communications Systems

Amal Feriani, Amine Mezghani, Member, IEEE, and Ekram Hossain, Fellow, IEEE
The authors are with the Department of Electrical and Computer Engineering at
the University of Manitoba, Canada (emails<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>}

###### Abstract

We consider an Intelligent Reflecting Surface (IRS)-aided multiple-input
single-output (MISO) system for downlink transmission. We compare the
performance of Deep Reinforcement Learning (DRL) and conventional optimization
methods in finding optimal phase shifts of the IRS elements to maximize the
user signal-to-noise (SNR) ratio. Furthermore, we evaluate the robustness of
these methods to channel impairments and changes in the system. We demonstrate
numerically that DRL solutions show more robustness to noisy channels and user
mobility.

Keywords: Intelligent Reflecting Surface (IRS), downlink transmission, optimal
phase shift, Deep Reinforcement Learning (DRL), imperfect Channel State
Information (CSI), Vector Approximate Message Passing (VAMP) algorithm,
Alternating Direction Method of Multipliers (ADMM) algorithm

##  I Introduction

Recently, Deep Reinforcement Learning (DRL) has attracted a lot of interest to
solve a wide range of wireless communications problems [1, 2]. Thanks to its
learning capabilities and inference speed, DRL has shown competitive
performance compared to traditional optimization methods. This is why DRL-
based algorithms are considered as one of the key technologies for future
wireless generations. In the same vein, Intelligent Reflecting Surfaces (IRS)
represent a promising pillar of the future wireless evolution since they are
able to provide additional gain in wireless system performance at a low cost.

DRL has been already applied to solve IRS-related problems such as
passive/active beamforming. In [3], the authors considered a Multiple-Input
Single-Output (MISO) system and used DRL to find optimal passive phase shifts.
The DRL agent achieves a better performance than the fixed-point iteration
algorithm and results close to a SemiDefinite Relaxation (SDR) upper bound.
The joint optimization of transmit beamforming and phase shifts in multi-user
MISO network was proposed in [4] where the authors used a unified DRL
algorithm to solve it. Furthermore, [5] approached the problem of minimization
of the Base Station (BS) transmit power in MISO system using DRL for passive
beamforming and traditional optimization-based method for active beamforming.
An IRS-assisted downlink NOMA multi-user system was studied in [6] and DRL was
employed to predict the IRS phase shifters. DRL showcases better sum-rate
performance compared to the OMA scheme.

The objective of this paper is twofold: First, we provide a comparison between
DRL methods and the best performing traditional optimization methods such as
Vector Approximate Message Passing (VAMP) [7] and Alternating Direction Method
of Multipliers (ADMM) [8] in solving the phase shift optimization problem for
IRS-aided wireless communications systems. This work will help to build a
benchmark to compare DRL to other exiting techniques. Furthermore, unlike the
works mentioned above, our aim is not to solve passive beamforming with DRL
but to motivate the use of DRL over other conventional methods in practical
settings. For instance, we focus on the robustness of the optimal solution to
channel impairments. To do so, we compare the decrease in the achieved SNR
when the Channel State Information (CSI) is perturbed (i.e. additional noise)
and the user location changes. Our work is novel since we consider that the
agent’s knowledge of the environment is not perfect and we test the obtained
solution under different perturbation scenarios. Our objective through this
work is to bring a more practical perspective to the proposed algorithms for
phase shift optimization and to wireless problems in general. Simulation
results show that, compared to other optimization methods, DRL is more robust
to changes in the system. This is another advantage of DRL methods along with
the inference speed that motivates the use of learning-based techniques to
solve wireless problems.

The rest of the paper is organized as follows. The system model, assumptions,
and the problem formulation are presented in Section II. The DRL-based
approach for phase-shift optimization is presented in Section III. The
performance evaluation and benchmarking results are presented in Section IV
before the paper is concluded in Section V.

##  II System Model and Problem formulation

Consider a BS equipped with $N$ antenna elements serving a single-antenna user
in the downlink. An IRS, with $M=M_{x}\times M_{y}$ reflective elements, is
deployed to assist the BS. $M_{x}$ and $M_{y}$ refer to the number of IRS
elements in each row and column, respectively. We denote by
$\mathbf{H}_{bu}\in\mathbb{C}^{N\times 1}$,
$\mathbf{H}_{ru}\in\mathbb{C}^{M\times 1}$, and
$\mathbf{H}_{br}\in\mathbb{C}^{M\times N}$ the direct channel between the BS
and the user, the channel between the IRS and the user and the MIMO channel
linking the BS and the IRS (Fig. 1).

Figure 1: IRS-aided single-user MISO system.

Henceforth, the received signal at the user is given by:

$\displaystyle
y=(\mathbf{H}_{bu}^{H}+\mathbf{H}_{ru}^{H}\mathbf{\Phi}\mathbf{H}_{br})\mathbf{F}s+n$

where $s$ is a zero-mean and unit-variance transmit symbol,
$n\sim\mathcal{CN}(0,\sigma^{2}_{n})$ is the additive white Gaussian noise.
$\mathbf{F}\in\mathbb{C}^{N\times 1}$ is the beamforming vector satisfying the
constraint $||\mathbf{F}||^{2}=P_{max}$ with $P_{max}$ being the BS maximum
transmit power.
$\mathbf{\Phi}=Diag(\phi_{1},\dots,\phi_{M})\in\mathbb{C}^{M\times M}$ is the
IRS phase shift matrix such that $\phi_{i}=e^{j\theta_{i}}$ with
$\theta_{i}\in[0,2\pi]$ representing the phase shift of the $i$th IRS
reflective element for $i=1,\dots,M$. Consequently, the user’s effective
channel is expressed as follows:

$\displaystyle\mathbf{H}^{H}=\mathbf{H}_{bu}^{H}+\mathbf{H}_{ru}^{H}\mathbf{\Phi}\mathbf{H}_{br}.$

Furthermore, we assume that the channels are quasi-static frequency flat-
fading and the CSI is available at the BS. The received SNR at the user is
given by

$\displaystyle\rho=\frac{|\mathbf{H}^{H}\mathbf{F}|^{2}}{\sigma_{n}^{2}}.$ (1)

We seek to optimize the phase shift matrix $\mathbf{\Phi}$ and the beamforming
vector to maximize the user SNR. The optimization problem can be stated as
follows:

$\displaystyle\max_{\mathbf{\Phi},\mathbf{F}}$
$\displaystyle\quad||(\mathbf{H}_{bu}^{H}+\mathbf{H}_{ru}^{H}\mathbf{\Phi}\mathbf{H}_{br})\mathbf{F}||_{2}^{2}$
s.t. $\displaystyle\quad||\mathbf{F}||^{2}=P_{max}$
$\displaystyle\quad|\mathbf{\Phi}_{i,i}|^{2}=1.$

For this MISO system, we know that, for a fixed $\mathbf{\Phi}$, the optimal
beamforming vector that maximizes the SNR is given by the maximum-ratio
transmission (MRT) method [9] as follows:

$\displaystyle\mathbf{F}^{*}=\frac{\sqrt{P_{max}}\mathbf{H}}{||\mathbf{H}||_{2}}.$
(2)

Hence, the optimization problem can be simplified as:

$\displaystyle(P_{1}):\max_{\mathbf{\Phi}}$
$\displaystyle\quad||\mathbf{H}_{bu}^{H}+\mathbf{H}_{ru}^{H}\mathbf{\Phi}\mathbf{H}_{br}||_{2}^{2}$
(3) s.t. $\displaystyle\quad|\mathbf{\Phi}_{i,i}|^{2}=1,\quad i=1,\dots,M.$
(4)

The optimization problem as formulated in ($P_{1}$) is a non-convex problem
due to the unimodular constraints.

Several works attempted to solve this optimization problem. In particular,
[10] focused on minimizing the Minimum Mean Square Error (MMSE) criterion
which is equivalent to maximizing the user SNR according to the I-MMSE
relationship. The authors derived a closed-form solution for the beamforming
vector and proposed a modified algorithm based on VAMP algorithm. This work
shows better performance compared to ADMM and has a complexity of
$O(NM+M^{2}+N^{3})$. The disadvantages of this method are two-fold: (i) it is
an iterative algorithm, and (ii) it requires the perfect knowledge of CSI. In
[11], the authors used an SDR method which is characterized by a high
complexity (i.e. $O((M+1)^{6})$).

To overcome the above-mentioned shortcomings of traditional optimization
methods, DRL is used to learn the optimal phase shifts. The advantage of DRL
compared to the previous methods is, it can be queried in real-time and
necessitates less CPU time. A comparison between SDR and DRL-based solutions
for ($P_{1}$) was presented in [3]. In this work, we provide a comparison
between VAMP, ADMM, and DRL in addition to a performance analysis under
different channel impairments.

##  III DRL-Based Approach

###  III-A Background

The standard reinforcement learning framework consists of an agent learning to
maximize its expected cumulative rewards through interacting with an
environment. At each interaction or timestep $t$, given an observation
$s_{t}$, the agent chooses an action $a_{t}$ and receives an instantaneous
reward $r_{t}$, and the system transit to a new state $s_{t+1}$. The tuples
$(s_{t},a_{t},r_{t},s_{t+1})$ constitute the agent experiences used for
learning the optimal policy $\pi:S\mapsto\mathcal{P}(A)$. This learning
problem is often modeled as a Markov Decision Process (MDP), described by the
tuple $(S,A,P,R,\gamma)$, when the state space is fully-observable. $S$ and
$A$ define the state and the action spaces, respectively; $P:=S\times
A\mapsto[0,1]$ denotes the transition probability function, $R:=S\times
A\times S\mapsto R$ is the reward function, and $\gamma\in[0,1]$ is a discount
factor that trades-off the immediate and upcoming rewards.

We define the agent’s expected return as the sum of the _discounted_ future
rewards
$R_{t}=\mathbb{E}\Big{[}\sum_{i=t}^{T}\gamma^{(i-t)}R(s_{i},a_{i},s_{i+1})|a_{i}\sim\pi(.|s_{i})\big{]}$.
Thus, the agent’s objective is to maximize the expected long-term reward
$J=\mathbb{E}_{\pi}\Big{[}\sum_{t=0}^{\infty}\gamma^{t}R_{t}\Big{]}.$

Another well-known function used to measure the agent’s returns is the action-
value function $Q$. It measures the expected accumulated rewards after
executing an action $a_{t}$ at a state $s_{t}$ and following the policy $\pi$
thereafter:

$\leavevmode\resizebox{432.17372pt}{}{$Q^{\pi}(s,a)=\mathbb{E}\Bigg{[}\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t},s_{t+1})|a_{t}\sim\pi(.|s_{t}),s_{0}=s,a_{0}=a\Bigg{]}$}.$

The famous $Q-$learning algorithm makes use of the _Temporal Difference_ (TD)
formulation where the $Q$ function is learned using the following recursive
expression

$Q(s_{t},a_{t})=(1-\alpha)Q(s_{t},a_{t})+\alpha\big{[}r(s_{t},a_{t})+\gamma\max_{a^{\prime}}Q(s_{t+1},a^{\prime})\big{]}$

(5)

in which $\alpha$ is a learning rate.

The main disadvantage of the $Q$-learning method is the $\max$ operator in (5)
which limits it to discrete action spaces. In this context, Deterministic
Policy Gradient (DPG) algorithm [12] can be considered as an extension of
$Q$-learning to continuous action spaces by replacing the
$\max_{a^{\prime}}Q(s_{t+1},a^{\prime})$ in (5) by $Q(s_{t+1},\mu(s_{t+1}))$,
where
$\mu(s_{t+1})=\operatorname*{arg\,max}_{a^{\prime}}Q(s_{t+1},a^{\prime})$.
Thus, DPG algorithms concurrently learn a $Q$-function $Q_{\phi}$ and a policy
$\mu_{\psi}$, where $\phi$ and $\psi$ are the parameters of the $Q$-function
and the policy, respectively. The $Q$-function is learned by minimizing the
_Bellman error_ in (6). As for the policy, the objective is to learn a
_deterministic policy_ that outputs the action corresponding to
$\max_{a}Q_{\phi}(s,a)$. The policy is called deterministic because it gives
the exact action to take at each state $s$. Hence, the learning process
consists in performing gradient ascent with respect to $\theta$ to solve the
objective in (8):

$\displaystyle\phi^{*}$
$\displaystyle=\operatorname*{arg\,min}_{\phi}\frac{1}{2}\sum_{(s,a,r,s^{\prime})}||Q_{\phi}(s,a)-y||^{2}$
(6) $\displaystyle y$
$\displaystyle=R(s,a)+\gamma\hat{Q}_{\phi^{\prime}}(s^{\prime},\hat{\mu_{\psi^{\prime}}}(s^{\prime}))$
(7) $\displaystyle J(\theta)$ $\displaystyle=\mathbb{E}_{s\sim
D}\bigg{[}Q_{\phi}(s,\mu_{\theta}(s))\bigg{]}$ (8)
$\displaystyle\nabla_{\theta}J(\theta)$ $\displaystyle=\mathbb{E}_{s\sim
D}\bigg{[}\nabla_{\theta}\mu_{\theta}(s)\nabla_{a}Q_{\phi}(s,a)|_{a=\mu_{\theta}(s)}\bigg{]}$
(9)

where $\hat{Q}$ and $\hat{\mu}$ are called target networks which are copies of
the $Q$ and $\mu$.These target networks are used to compute the target values
[7]. The target parameters are updated using a Polyak update with a parameter
$\tau$.

Deep Deterministic Policy Gradient (DDPG) [13] has been widely used to solve
wireless communication problems. It is an extension of the DPG method where
the policy $\mu_{\psi}$ and the critic $Q_{\phi}$ are both deep neural
networks.

###  III-B MDP Formulation

The optimization problem in ($P_{1}$) is modeled as an MDP where the agent is
the BS. In what follows, we define the state and action spaces and the reward
function:

  * •

State space $S$: consists of the user SNR and action
$s_{t}=[\rho_{t-1},a_{t-1}]$. The dimension of state space is $M+1$ and spans
$\mathbb{R}_{+}$. Using the past actions helps mimicking the recurrent version
of RL.

  * •

Action space $A$: at each timestep, the agent chooses the phase shifts
$\theta_{i},i=1\dots M$. Thus, the dimension of the action space is $M$ and
$a_{t}=[\theta_{1},\dots,\theta_{M}]\in[0,2\pi]^{M}$.

  * •

Reward: at each time step, the agent’s reward is the received SNR as defined
in (1).

Technically, this formulation can be considered as a partially observable MDP
since the agent observes only its SNR and does not have access to the CSI
which is used to compute the next state. So, the agent needs to infer the CSI
in order to recover the full state. In this paper, we will solve this problem
by using a model-free RL algorithm. In future works, other methods can be
considered to deal with the problem of partial observability of the
environment.

We assume that the agent interacts with the environment in an episodic manner.
An episode ends when the number of time-steps exceeds a fixed horizon $T$. At
the beginning of each episode, we sample new channel realizations and randomly
choose the phase shifts to compute the first state $s_{0}$.

The pseudo-code in Algorithm 1 summarizes the DDPG algorithm to find the
optimal phase shifts. To ensure a better exploration, we add an adaptive
parameter space noise [14] in addition to the action space noise. The
parameter noise is applied to the weights of the actor network. The
traditional action noise makes the policy stochastic and enables the
perturbation of action likelihoods independently from the state. However, the
parameter space noise incorporates the noise directly in the parameters
yielding a _state-dependent_ exploration [15]. For more details, we refer the
reader to [14].

Algorithm 1 DDPG with Parameter Noise Exploration

1:Input: Initialize actor parameters $\psi$, critic parameters $\phi$, empty
memory buffer $\mathcal{D}$, parameter space noise standard deviation
$\sigma_{0}$, intervals for training and noise adaptive scaling,
$T_{\text{train}}$, $T_{\text{adapt}}$, number of training steps
$N_{\text{train}}$, an adaption update coefficient $\alpha$ and threshold
$\delta$

2:Initialize the target networks parameters $\phi^{\prime}\leftarrow\phi$ and
$\psi^{\prime}\leftarrow\psi$

3:Set $\tilde{\mu}=\mu_{\tilde{\psi}}$ for exploration and
$\bar{\mu}=\mu_{\bar{\psi}}$ for noise adaption, $t=0$

4:repeat

5: Apply noise perturbation to actor parameters
$\tilde{\psi}\leftarrow\psi+\mathcal{N}(0,\sigma_{k}^{2}\mathbf{I})$ to obtain
$\tilde{\mu}$

6: Receive state $\bm{s_{t}}$ and choose an action
$\bm{a_{t}}=\text{clip}(\tilde{\mu}(\bm{s_{t}})+\epsilon,A_{\text{low}},A_{\text{high}})$
according to the perturbed policy $\tilde{\mu}$ where
$\epsilon\sim\mathcal{N}(0.\sigma_{a})$

7: Execute $\bm{a_{t}}$ and observe the receive reward $r$, the next state
$\bm{s^{t+1}}$. Store $(\bm{s_{t}},\bm{a_{t}},r,\bm{s_{t+1}})$ in
$\mathcal{D}$

8: If $\bm{s^{t+1}}$ is terminal, reset the environment and update the
perturbed actor
$\tilde{\psi}\leftarrow\psi+\mathcal{N}(0,\sigma_{k}^{2}\mathbf{I})$

9: $t=t+1$

10: if $t$ mod $T_{\text{train}}$ then

11: for $i=0,\dots,N_{\text{train}}$ do

12: Sample a minibatch $\mathcal{B}$ from $\mathcal{D}$

13: Update the actor and the critic parameters as specified by (6), (9)

14: Update target networks

15: ${\phi^{\prime}}\leftarrow\tau\phi+(1-\tau)\phi^{\prime}$

16: $\psi^{\prime}\leftarrow\tau\psi+(1-\tau)\psi^{\prime}$

17: if $i$ mod $T_{\text{adapt}}$ then

18: Perturb $\bar{\psi}\leftarrow\psi+\mathcal{N}(0,\sigma^{2}_{k}\mathbf{I})$
and obtain $\bar{\mu}$

19: Estimate the standard deviation between $\mu$ and $\bar{\mu}$

20:
$d_{k}=\mathbb{E}_{\bm{s}}\big{[}\sqrt{\frac{1}{|\mathcal{A}|}\sum_{j=}^{|\mathcal{A}|}(\mu(\bm{s})_{j}-\bar{\mu}(\bm{s})_{j})^{2}}\big{]}$

21: Adapt the parameter noise scale

22: $\sigma_{k+1}\leftarrow\left\\{\begin{matrix}\alpha\,\sigma_{k}\quad\text{
if }d_{k}\leq\delta,\\\
\frac{1}{\alpha}\sigma_{k}\quad\text{otherwise}\end{matrix}\right.$

23: end if

24: end for

25: end if

26:until convergence

##  IV Numerical Results

###  IV-A Simulation Setup

In this section, we detail the experimental setup used to generate the
results.

Channel models: The channel between the BS and the user is assumed to
experience Rayleigh fading which suggests that the line-of-sight (LOS) signal
between them is blocked:

$\displaystyle\bm{H}_{bu}=\sqrt{G_{bu}}\bm{h}_{bu}$

where $\bm{h}_{bu}\in\mathbb{C}^{N\times 1}$ and its elements are sampled
independently from $\mathcal{CN}(0,1)$. The channel between the BS and IRS as
well as the one between the IRS and the user are Ricean fading:

$\displaystyle\bm{H}_{br}$
$\displaystyle=\sqrt{G_{br}}\Big{(}\sqrt{\frac{K_{1}}{K_{1}+1}}\bm{H}_{br}^{LOS}+\sqrt{\frac{1}{K_{1}+1}}\bm{H}_{br}^{NLOS}\Big{)}$
$\displaystyle\bm{H}_{ru}$
$\displaystyle=\sqrt{G_{ru}}\Big{(}\sqrt{\frac{K_{2}}{K_{2}+1}}\bm{H}_{ru}^{LOS}+\sqrt{\frac{1}{K_{2}+1}}\bm{H}_{ru}^{NLOS}\Big{)}$

where $\bm{H}_{br}^{LOS}$ and $\bm{H}_{ru}^{LOS}$ are the deterministic LOS
components for IRS-user and BS-IRS links computed as shown [3].
$\bm{H}_{br}^{NLOS}$ and $\bm{H}_{ru}^{NLOS}$ are the NLOS components
independently distributed as $\mathcal{CN}(0,1)$. $K_{1}$ and $K_{2}$
represent the Ricean factors.

The path-loss model is given by $G(d)=G_{0}-10\alpha\log_{10}(d/d_{0})$ in dB,
where $G_{0}$ is the path-loss at the reference distance, $d_{0}$ is the
reference distance and $\alpha$ is the path loss exponent.

Simulation hyperparameters: Table I summarizes the simulation parameters. For
the DDPG algorithm, we use the hyperparameters reported in [3] in addition to
the parameter space noise. We find that using a long episode horizon (i.e
$1000$) is critical for learning.

Policy representation: For both actor and critic, the same architecture is
used: a feedforward deep neural network with two hidden layers with $300$ and
$200$ hidden units and Relu activations. Before applying the activation
function, we use a normalization layer [16] to stabilize the learning under
parameter noise.

Baselines: We compare the DRL agent with VAMP and ADMM algorithms due to their
convergence guarantees and speed.

For reproducibility purposes, we will release the code of the environment used
for the simulations below. [17] is used as an implementation of the DRL
algorithm.

TABLE I: Simulation parameters Parameter | Value
---|---
$K_{1}=K_{2}$ | $10$
$N$ | $10$
$M$ | $50$
$\sigma^{2}_{n}$ |  $-80$ dBm
$P_{\max}$ |  $5$ dBm
$d_{0}$ |  $1$ m
$G_{0}$ |  $-30$ dB
$\alpha_{br},\alpha_{bu},\alpha_{ru}$ | $2,2,2.8$
$d_{br},d_{bu},d_{ru}$ |  $51,48,1.5$ m
Hidden layers | $300,200$
learning rate | $1e^{-3}$
Episode horizon $H$ | $1000$
Replay buffer size | $1e^{6}$
$N_{train}=T_{adapt}$ | $50$
$T_{train}$ | $1000$
$\sigma_{a}=\sigma_{0}$ | $0.1$

###  IV-B Results and Discussion

#### Performance comparison

We first compare the three considered methods: VAMP, ADMM, and DRL. Here, we
set the parameters of the network as reported in Table I and vary the number
of IRS elements. In Fig. 2, we illustrate the achievable SNR as a function of
the number of IRS reflective elements for the different considered algorithms.
The achievable SNR increases as the number of IRS elements increases. With
perfect CSI, VAMP has the best achieved SNR compared to the other methods. We
observe that the DRL method is able to achieve good performance compared to
ADMM even without access to channel information. Fig. 3 shows that the agent’s
reward improves as the number of IRS elements increases. However, the SNR gain
becomes smaller as the number of IRS elements gets higher.

Furthermore, we examine the inference speed of each algorithm and summarize
them in Table II. We average the inference time over $1000$ realizations. VAMP
and ADMM algorithms are run for $30$ iterations to achieve convergence [10].
After training, the DRL agent is queried in real-time to get the optimal phase
shifts. Compared to traditional optimization techniques, DRL is superior in
terms of inference speed which makes it more appealing for practical
implementations.

Figure 2: Achieved SNR. Figure 3: DRL training curves for different number of IRS elements. TABLE II: Inference time in ms M | VAMP | ADMM | DRL
---|---|---|---
50 | 30 | 19 | 0.3
100 | 38 | 29 | 0.48
144 | 48 | 41 | 0.49
196 | 76 | 64 | 0.6
256 | 100 | 100 | 0.62

#### Performance under unreliable channel

Since in real-world scenarios, the CSI acquired by the transmitter is not
perfect and subject to errors, in this section we investigate the impact of
CSI errors on the performance of the considered algorithms. As the first step,
we obtain the optimal phase shifts and beamforming vector using the available
CSI. Afterwards, we apply perturbations to the channels as shown in (10):

$\displaystyle\bm{H}[t]=\sqrt{\epsilon}\bm{H}[t-1]+\sqrt{\epsilon-1}\bm{w}[t]$
(10)

where $\bm{w}[t]$ contains independent and identically distributed
$\mathcal{CN}(0,1)$ elements. We compute the user SNR using the optimal phase
shifts and the perturbed channels. Note that the objective is to study the
impact of the channel errors on the optimal solutions. This means that we
verify if the obtained solutions remain optimal with small changes in the CSI.

Fig. 4 illustrates the decrease in the user SNR due to different levels of
noise. Since VAMP and ADMM rely on perfect CSI to find optimal solutions, the
drop in performance is severe compared to DRL. Indeed, the DRL agent is able
to learn a good policy without relying on the CSI directly which makes it more
robust to changes in the environment. Furthermore, we observe that VAMP is
more sensitive to changes in the channel than ADMM. For a high level of noise,
VAMP performance drops by more than $16$ dB whereas the performance of the DRL
agent decreases only by $3$ dB.

Figure 4: The loss in SNR due to channel errors.

#### Performance under user mobility

To further verify the robustness of the DRL to changes in its environment, we
increase the distance between the BS and the user to simulate a mobile target.
Originally, the distance between the user and BS is $48$ m. We assume that the
user is moving away from the BS which represents the worst-case scenario.
Hence, the distance is increased by a step-size in $[5,25]$m. Note that we
first compute the optimal phase shifts and beamforming vector using the
current CSI, then we increase the user distance and compute the SNR using the
optimal solutions. A similar pattern is observed in Fig. 5 where VAMP and ADMM
suffer dramatic drops in the user SNR. Surprisingly, the DRL performance stays
approximately the same.

Figure 5: The loss in SNR due to user mobility.

##  V Conclusion

In this paper, we aimed to investigate the robustness of DRL against changes
in CSI. This is a practical scenario where the CSI at the transmitter is
subject to errors and noise. DRL shows an interesting resilience to different
channel impairments. This is advantageous compared to other
optimization/inference methods where perfect CSI is crucial to ensure good
performance. Furthermore, we provide a rigorous comparison between DRL, VAMP,
and ADMM to optimize the phase shift matrix for downlink MISO transmission.
Numerical results show that DRL is better than ADMM which fails to find a good
solution for the considered system. Although VAMP has better performance, the
DRL agent is capable of learning good policies without relying on any CSI.

In this work, we have only considered errors in the CSI; however, other
sources of perturbations can impact the performance of the DRL agent. In a
future work, we will do a more detailed investigation of the DRL methods for
wireless continuous control. Furthermore, only robustness during inference is
considered in this paper. Training under uncertainty can also be studied as an
extension.

## References

  * [1] Y. Al-Eryani, M. Akrout, and E. Hossain, “Multiple access in cell-free networks: Outage performance, dynamic clustering, and deep reinforcement learning-based design,” IEEE Journal on Selected Areas in Communications, vol. 39, no. 4, pp. 1028–1042, 2021.
  * [2] Y. Al-Eryani, M. Akrout, and E. Hossain, “Antenna clustering for simultaneous wireless information and power transfer in a mimo full-duplex system: A deep reinforcement learning-based design,” IEEE Transactions on Communications, vol. 69, no. 4, pp. 2331–2345, 2021.
  * [3] K. Feng, Q. Wang, X. Li, and C.-K. Wen, “Deep reinforcement learning based intelligent reflecting surface optimization for miso communication systems,” IEEE Wireless Communications Letters, vol. 9, no. 5, pp. 745–749, 2020\.
  * [4] C. Huang, R. Mo, and C. Yuen, “Reconfigurable intelligent surface assisted multiuser miso systems exploiting deep reinforcement learning,” IEEE Journal on Selected Areas in Communications, vol. 38, no. 8, pp. 1839–1850, 2020\.
  * [5] J. Lin, Y. Zou, X. Dong, S. Gong, D. T. Hoang, and D. Niyato, “Optimization-driven deep reinforcement learning for robust beamforming in irs-assisted wireless communications,” arXiv preprint arXiv:2005.11885, 2020.
  * [6] M. Shehab, B. S. Ciftler, T. Khattab, M. Abdallah, and D. Trinchero, “Deep reinforcement learning powered irs-assisted downlink noma,” arXiv preprint arXiv:2104.01414, 2021.
  * [7] S. Rangan, P. Schniter, and A. K. Fletcher, “Vector approximate message passing,” IEEE Transactions on Information Theory, vol. 65, no. 10, pp. 6664–6684, 2019.
  * [8] S. Boyd, N. Parikh, and E. Chu, Distributed optimization and statistical learning via the alternating direction method of multipliers.  Now Publishers Inc, 2011.
  * [9] Q. Wu and R. Zhang, “Beamforming optimization for intelligent reflecting surface with discrete phase shifts,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7830–7833, IEEE, 2019.
  * [10] H. U. Rehman, F. Bellili, A. Mezghani, et al., “Joint active and passive beamforming for irs-assisted multi-user mimo systems: A vamp approach,” arXiv preprint arXiv:2102.01232, 2021.
  * [11] X. Yu, D. Xu, and R. Schober, “Miso wireless communication systems via intelligent reflecting surfaces,” in 2019 IEEE/CIC International Conference on Communications in China (ICCC), pp. 735–740, IEEE, 2019.
  * [12] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller, “Deterministic policy gradient algorithms,” in Proceedings of the 31st International Conference on International Conference on Machine Learning, pp. 387–395, 2014.
  * [13] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” arXiv preprint arXiv:1509.02971, 2015.
  * [14] M. Plappert, R. Houthooft, P. Dhariwal, S. Sidor, R. Y. Chen, X. Chen, T. Asfour, P. Abbeel, and M. Andrychowicz, “Parameter space noise for exploration,” arXiv preprint arXiv:1706.01905, 2017.
  * [15] T. Rückstieß, M. Felder, and J. Schmidhuber, “State-dependent exploration for policy gradient methods,” in Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 234–249, Springer, 2008.
  * [16] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” arXiv preprint arXiv:1607.06450, 2016.
  * [17] A. Hill, A. Raffin, M. Ernestus, A. Gleave, A. Kanervisto, R. Traore, P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, and Y. Wu, “Stable baselines.” https://github.com/hill-a/stable-baselines, 2018.