# Prediction of low-keV monochromatic images from polyenergetic CT scans for
improved automatic detection of pulmonary embolism

###### Abstract

Detector-based spectral computed tomography is a recent dual-energy CT (DECT)
technology that offers the possibility of obtaining spectral information. From
this spectral data, different types of images can be derived, amongst others
virtual monoenergetic (monoE) images. MonoE images potentially exhibit
decreased artifacts, improve contrast, and overall contain lower noise values,
making them ideal candidates for better delineation and thus improved
diagnostic accuracy of vascular abnormalities.

In this paper, we are training convolutional neural networks (CNN) that can
emulate the generation of monoE images from conventional single energy CT
acquisitions. For this task, we investigate several commonly used image-
translation methods. We demonstrate that these methods while creating visually
similar outputs, lead to a poorer performance when used for automatic
classification of pulmonary embolism (PE). We expand on these methods through
the use of a multi-task optimization approach, under which the networks
achieve improved classification as well as generation results, as reflected by
PSNR and SSIM scores. Further, evaluating our proposed framework on a subset
of the RSNA-PE challenge data set shows that we are able to improve the Area
under the Receiver Operating Characteristic curve (AuROC) in comparison to a
naïve classification approach from 0.8142 to 0.8420.

Index Terms—  Image-to-Image Translation, Spectral Computer Tomography, Domain
Adaptation, Pulmonary Embolism Diagnosis

##  1 Introduction

In spectral computed tomography (DECT), projection data simultaneously
obtained from both detector layers is utilized to generate spectral images
such as virtual monoenergetic (monoE) scans. Next to the conventional
(polyenergetic) images, multiple spectrally distinct attenuation maps can be
obtained from a single scan and used to derive different types of images. The
clinical uses of the DECTs can be summarized with enhanced visualization of
intravascular contrast, reduction of artifacts such as calcium blooming,
material decomposition, and radiation dose reduction [1]. Therefore, in
comparison to conventional CT, DECT compares favorably for the diagnosis of
various diseases such as myocardial perfusion [2] or pulmonary embolisms [3].

We argue that similar to the expert radiologist, convolutional neural networks
(CNN) may benefit when trained on DECT data. However, as most currently
existing CT data sets were acquired with conventional CT scanners they do not
comprise monoE images. To bridge this gap, we investigate the use of existing
image-translation models such as Pix2Pix [4], which might be able to use the
underlying distributions in polyenergetic images to predict spectral images
akin to what was done to translate CT images to MRI scans [5]. In turn, these
generated synthetic monoE images might be used as input for CNNs potentially
facilitating their detection of pathologies. While existing image-translation
methods are able to generate visually appealing results they do not enforce
features that enable the correct identification of certain classes. For this
reason, we introduce a joint optimization between the generation of the
monoenergetic domain and the simultaneous identification of pathologies. This
leads to a network that learns to combine features necessary for a downstream
classification task as well as for synthetic image generation. In other words,
the proposed framework learns a suitable mapping on the basis of monoenergetic
images.

|
---|---
|
Fig. 1: Overview of the different approaches for the combination of domain
adaptation and classification. On the left, a) trains a generator on the
paired dataset, which would be followed by the training of a classifier on
annotated data in b). c) displays our approach of the joint optimization of
generator and classifier, where the generator learns the mapping for unlabeled
data, while adjusting its features in a way that it does not hinder the
classification network when given annotated data.

Our contributions can be summarized as (1) an extensive study comparing
various image-translation methods for the prediction of monoE images from
conventional polyenergetic scans, (2) evaluation of the classification
accuracy of predicted synthetic monoE images for the detection of PE, and (3)
proposal of a training regime, enabling to generate data that is not only
visually similar but also incorporates features necessary for the automatic
identification of pathologies.

##  2 Methods and Materials

Suppose, we are given two distinct data sets $D_{1}$ and $D_{2}$. $D_{1}$
consists of unannotated images with poly- and monoenergetic depictions.
$D_{2}$ describes a set of images with slice-level disease annotations without
corresponding monoenergetic representation. We now aim to design a unified
model that jointly optimizes disease identification and domain adaption most
fitting for the task. We have formulated these two tasks into the same
framework so that 1) it trains these tasks end-to-end and 2) the two tasks can
be mutually beneficial. The proposed architecture is displayed in Fig. 1 c).

Methodology: The proposed framework jointly optimizes two tasks in an end-to-
end manner. As one task, we consider the problem of translating between the
domain of polyenergetic $x\in X$ and monoE images $y\in Y$ as a paired image-
translation problem. Here, a generator aims to learn a mapping $G:x\rightarrow
y$, which minimizes the difference between the two paired images. This
objective can be expressed as

$\mathcal{L}_{L1}=\mathbb{E}_{x,y}[||G(x)-y||_{1}].$ (1)

We utilize the mean absolute error as it has been found to lead to less blurry
images [4].

Consecutively, the output of the generator is fed into a classification
network $C$, which attempts to predict the occurrence of a disease label $z$,
$C:G(x)\rightarrow z$ of the annotated data set. We utilize ResNet50 [6],
however our framework can be easily extended to employ any other existing CNN
model. We utilize a sigmoid activation $\sigma$ for making output predictions.

$\begin{split}\mathcal{L}_{cls}=&\mathbb{E}_{x,z}[-z\log\sigma(C(G(x)))\\\
&-(1-z)\log(1-\sigma(C(G(x)))]\end{split}$ (2)

To optimize both objectives during the training process, we construct our data
set as a combination of the two data sets (see below) and sample the batch in
a way such that on average it consists of 50% of either. Therefore, target
disease labels are only given for half of the batch and monoenergetic target
images for the other half. To accommodate this circumstance into the
optimization function we introduce a marker variable $m$, which switches
between $[0,1]$ depending on whether we are presented a target image $y$ or a
target label $z$. In this manner, the final loss can be formulated as

$\mathcal{L}=m*L_{cls}+(1-m)*L_{L1}.$ (3)

For backpropagation of the gradients one network is frozen, while the other is
updated similar to an adversarial training.

Implementation Details: We train our networks jointly in an end-to-end manner
by sequentially passing data through the generator and classification network.
Our generator network utilizes a fully convolutional 9-block ResNet encoder-
decoder network, however, similar to our classifier, the model can easily be
replaced by more advanced architectures. We use Adam for optimization with a
learning rate of 0.0002, $\beta_{1}=0.9$ and $\beta_{2}=0.99$ with a weight
decay of 0.00001. After training for 5 epochs on the joint data set, we decay
our learning rate to 0 over the following 5 epochs. We use an image-size of
$512\times 512$ with a batch size of 5 for all our experiments.

Input | L1 | SPL | Pix2Pix | CRN | Pix2PixHD | Ours | Target
---|---|---|---|---|---|---|---
|  |  |  |  |  |  |
|  |
---|---
|  |
---|---
|  |
---|---
|  |
---|---
|  |
---|---
|  |
---|---
|  |
---|---
|  |
---|---
|  |  |  |  |  |  |
|  |  |  |  |  |  |
Fig. 2: Qualitative comparison of different image translation methods on our
internal DE-CTPA dataset. Individual SSIM and PSNR values are shown on the
images. Areas around Pulmonary Embolisms are displayed seperately.

Experimental Setup: We utilize two data sets for our experiments. The private
dual-energy computed tomography pulmonary angiography (DE-CTPA) data set
$D_{1}$ was gathered during routine clinical workup of 27 consecutive patients
with suspected pulmonary embolism. The CT scans were performed on a dual-layer
detector (IQon Spectral CT, Philips Healthcare). Standard arterial series and
the corresponding monoenergetic images at low-energy levels (40 keV) were
reconstructed. The data set contains 7892 image pairs.

The second data set $D_{2}$ is a subset of the RSNA STR Pulmonary Embolism
Detection [7]. Out of the annotated 7279 subjects, we sample 10% of the
training data patient-wise. The sampled data set consists of a total of 161253
annotated slices with roughly the same label distribution as present in the
open training set. We further split the data patient-wise 50%/25%/25% into
train-, val- and test-sets, respectively.

For our experiments on our DE-CTPA, we perform 5-fold cross validation and
average our reconstruction results in terms of Peak-Signal-to-Noise-Ratio
(PSNR) and Structural Similarity Index Measure (SSIM). For the identification
of PE, we perform binary classification on slice level for each presented
image domain and report the area under the receiver-operating-characteristic
(AuROC) on the test split of the model which performed best on the validation
set. We validated our model after each epoch.

We compare against various image-translation models such as an L1-loss-based
generator as a baseline, Pix2Pix[4], Pix2PixHD[8], CRN[9], SPL[10]. We further
added L1-Losses to feature-loss based methods (CRN, Pix2PixHD), denoted by *.
All methods apart from CRN and Pix2PixHD, which use their originally proposed
architecture, are trained using the same 9-block ResNet architecture. Orig.
denotes the direct usage of conventional CT imagery for either the computation
of PSNR/SSIM or as input into a classification network. To evaluate
classification performance for different image translation methods, we train
all methods on the same split in the cross validation setting of our internal
data set.

Compliance with ethical standards: The first data set was gathered as part of
a retrospective single-centre HIPAA-compliant study, which was approved by the
local institutional review board (No. S-236/2020) with a waiver for written
informed consent. As the second data set is part of a public competition
ethical approval was not required as confirmed by the license attached with
the open access data.

Conflicts of Interest: No funding was received for conducting this study. The
authors have no relevant financial or non-financial interests to disclose.

##  3 Results

Method | SSIM | PSNR
---|---|---
Orig. | $0.945\pm 0.007$ | $30.189\pm 0.690$
L1 | $\textbf{ 0.984}\pm\textbf{0.002 }$ | $\textbf{ 42.365}\pm\textbf{0.642 }$
SPL[10] |  0.983 $\pm$ 0.002 | $40.888\pm 0.216$
Pix2Pix [4] | $0.978\pm 0.003$ | $40.897\pm 0.697$
Pix2PixHD [8] | $0.971\pm 0.004$ | $38.739\pm 0.624$
CRN [9] | $0.371\pm 0.551$ | $19.482\pm 16.033$
Pix2PixHD* | $0.971\pm 0.004$ | $38.415\pm 1.278$
CRN* | $0.976\pm 0.0045$ | $37.582\pm 1.574$
Ours | $\textbf{0.984}\pm\textbf{0.002}$ | $\textit{41.706}\pm\textit{0.547}$
Table 1: Reconstruction results of various Image-Translation methods. Best and
second-best result in bold and cursive.

Quantative Results of Translation Properties: The quantitative results on the
reconstruction ability are displayed in Table 1. Models optimized on image-
based comparison outperform feature-loss and adversarial methods for the
evaluated task. Our method achieves similar performance to the L1-based
generator. All methods apart from the feature loss based CRN model manage to
create high quality visual reconstructions of the monoE images. Qualitative
samples can be seen in Fig. 2. Areas around pulmonary embolisms are further
highlighted.

Impact on automatic PE diagnosis: The quantitative results on the
classification results of a ResNet50 network trained on various input image
domains are displayed in Table 2. Despite the similar SSIM/PSNR results, the
L1-loss-based model generates images, which slightly hamper the classification
ability of a model. The other compared models worsen the performance, while
our proposed method manages to generate visually fitting images as well as
improves classification results over the baseline.

Domain | Orig. | L1 | SPL | Pix2Pix | CRN* | P2PHD* | Ours
---|---|---|---|---|---|---|---
AuROC | 0.8142 | 0.8102 | 0.8061 | 0.8051 | 0.8038 | 0.8019 | 0.8420
Table 2: Pulmonary Embolism-Classification results of a ResNet-50 trained on
images from different image domains of various Image-Translation methods. Best
result in bold.

##  4 Discussion

We have investigated the potential use of the prediction of monoenergetic from
polyenergetic images for the automatic identification of pathologies in CTPAs.
We have displayed that most established image translation method either fail
to correctly reconstruct that domain or are dismissing features necessary for
classification. To offset these shortcomings of existing approaches we
introduce an end-to-end learnable framework which combines the training of the
classification and translation network. The reconstruction loss terms manage
to let the network predict the visual properties, while the classification
loss lets it enhance distinguishable features for the trained task. Results on
the RSNA STR Pulmonary Embolism Detection dataset indicate that our approach
provides a successful domain adaptation to monoenergetic imagery as it
outperforms existing image-translation methods for paired data, while using
the same or less parameters.

##  5 Conclusion

The proposed joint optimization strategy allows training of reconstruction of
monoenergetic images without losing features necessary for the classification
process. Our method, hereby, improves noticeably over straight forward
classification, while outperforming existing methods.

##  6 Acknowledgements

The present contribution is supported by the Helmholtz Association under the
joint research school “HIDSS4Health - Helmholtz Information and Data Science
School for Health”.

## References

  * [1] Prabhakar Rajiah, Suhny Abbara, and Sandra Simon Halliburton,  “Spectral detector ct for cardiovascular applications,”  Diagnostic and Interventional Radiology, vol. 23, no. 3, pp. 187, 2017.
  * [2] Rachid Fahmi, Brendan L Eck, Jacob Levi, Anas Fares, Amar Dhanantwari, Mani Vembar, Hiram G Bezerra, and David L Wilson,  “Quantitative myocardial perfusion imaging in a porcine ischemia model using a prototype spectral detector ct system,”  Physics in Medicine & Biology, vol. 61, no. 6, pp. 2407, 2016.
  * [3] Jakob Weiss, Mike Notohamiprodjo, Malte Bongers, Christoph Schabel, Stefanie Mangold, Konstantin Nikolaou, Fabian Bamberg, and Ahmed E Othman,  “Effect of noise-optimized monoenergetic postprocessing on diagnostic accuracy for detecting incidental pulmonary embolism in portal-venous phase dual-energy computed tomography,”  Investigative radiology, vol. 52, no. 3, pp. 142–147, 2017.
  * [4] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros,  “Image-to-image translation with conditional adversarial networks,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1125–1134.
  * [5] Jelmer M Wolterink, Anna M Dinkla, Mark HF Savenije, Peter R Seevinck, Cornelis AT van den Berg, and Ivana Išgum,  “Deep mr to ct synthesis using unpaired data,”  in International workshop on simulation and synthesis in medical imaging. Springer, 2017, pp. 14–23.
  * [6] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [7] “RSNA STR Pulmonary Embolism Detection, https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/rules, Accessed: 2020-10-25,” .
  * [8] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro,  “High-resolution image synthesis and semantic manipulation with conditional gans,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 8798–8807.
  * [9] Qifeng Chen and Vladlen Koltun,  “Photographic image synthesis with cascaded refinement networks,”  in Proceedings of the IEEE international conference on computer vision, 2017, pp. 1511–1520.
  * [10] M. Saquib Sarfraz, Constantin Seibold, Haroon Khalid, and Rainer Stiefelhagen,  “Content and colour distillation for learning image translations with the spatial profile loss,”  in Proceedings of the 30th British Machine Vision Conference (BMVC), 2019.