# Reference Based Color Transfer for Medical Volume Rendering

Sudarshan Devkota<EMAIL_ADDRESS>University of Central
Florida4000 Central Florida BlvdOrlandoFlorida32816 and  Sumanta Pattanaik
<EMAIL_ADDRESS>University of Central Florida4000 Central Florida
BlvdOrlandoFlorida32816

(2020)

###### Abstract.

The benefits of medical imaging are enormous. Medical images provide
considerable amounts of anatomical information and this facilitates medical
practitioners in performing effective disease diagnosis and deciding upon the
best course of medical treatment. A transition from traditional monochromatic
medical images like CT scans, X-Rays or MRI images to a colored 3D
representation of the anatomical structure further enhances the capabilities
of medical professionals in extracting valuable medical information. The
proposed framework in our research starts with performing color transfer by
finding deep semantic correspondence between two medical images: a colored
reference image, and a monochromatic CT scan or an MRI image. We extend this
idea of reference based colorization technique to perform colored volume
rendering from a stack of grayscale medical images. Furthermore, we also
propose to use an effective reference image recommendation system to aid for
selection of good reference images. With our approach, we successfully perform
colored medical volume visualization and essentially eliminate the painstaking
process of user interaction with a transfer function to obtain color and
opacity parameters for volume rendering.

Deep Learning, color transfer, volume rendering, medical imaging

††copyright: acmcopyright††journalyear: 2020††copyright: none††ccs: Human-
centered computing Scientific visualization Figure 1.  The proposed system
comprises of colorization of grayscale medical images followed by direct
volume rendering.

##  1\. Introduction

Medical imaging is an extremely important aspect in the medical field and has
revolutionized the health care sectors, allowing for earlier diagnoses of
diseases and better patient outcomes. It has brought a significant change in
the healthcare sector and has aided scientists and medical practitioners to
learn more about the human body than ever before. Medical images like X-rays,
CT scans and MRIs are a valuable diagnostic tools and contain a significant
amount of anatomical information for clinical procedures. However, these
images are generally gray-scale images and a colored image could provide
better anatomical information than an ordinary monochromatic image.

The idea of colorizing grayscale images to make them appear aesthetically
pleasing is not new. In the last decade, several techniques have been proposed
for colorization of images and videos. While some of these techniques involve
propagating colored scribbles taken from users to achieve the final colorized
result (Levin et al., 2004)(Huang et al., 2005)(Luan et al., 2007)(Qu et al.,
2006), other learning based techniques leverage large-scale image dataset to
achieve automatic and reliable colorization without any user effort (Deshpande
et al., 2015) (Cheng et al., 2015) (Zhang et al., 2016)(Larsson et al.,
2016)(Iizuka et al., 2016). Another category of colorization technique is
example based colorization that seeks to transfer color properties from a
reference image to a target image (Welsh et al., 2002)(Irony et al.,
2005)(Gupta et al., 2012)(Bugeau et al., 2014). In an example based technique,
establishment of semantic similarities between the images plays a crucial role
in performing accurate color transfer. Due to large variations in structural
content or appearance, establishing semantic similarities could be challenging
with matching methods based on low level hand-crafted features like intensity
patch features, Haar features, HOG features, SIFT features, etc. A recent work
performed by Liao et al.(Liao et al., 2017) demonstrated that multi-level
features obtained using deep neural networks are robust in finding high level
semantic correspondence between images, thus reducing the limitation of using
hand-crafted features.

In our work, we take an approach similar to the one used by Liao et al.,
leveraging a pretrained network like VGG-19(Simonyan and Zisserman, 2014) as
the feature extractor network and using multi level features to establish deep
correspondence between images. While most of their approach serves as a base
to our implementation, we extend it to color transfer for medical data and
medical volume rendering with direct volume rendering technique. Specifically,
we perform colorization of monochromatic MRI images and CT scans with a
specific set of reference images and perform medical volume visualization with
the colored version of medical images.

Medical Volume Visualization provides meaningful information from the rendered
three-dimensional structure and it allows medical professionals like surgeons,
physicians and radiologists to better visualize the anatomy and understand the
disease processes. Furthermore, the shift from grayscale volume visualization,
where each voxel constitutes a single intensity value, to a colored 3D volume,
where each voxel has RGB components, further provides improved visualization
of the anatomical structures. This also enables medical practitioners in
reviewing large medical dataset in an effective and comprehensive way.

To perform a colored 3D rendering, each voxel must have $RGBA$ value, where
$RGB$ defines the color channels and $A$ is the opacity channel. Voxels that
have a low alpha value will appear transparent while voxels with high alpha
value will appear opaque. One essential step in traditional volume rendering
technique is to design proper transfer functions, where a user interacts with
a volume renderer by modifying values in the transfer function in order to
obtain best color and opacity parameters in the final rendered volume. This
work essentially involves trial-and-error interaction with the transfer
function, and can be tedious and time consuming. With our proposed system, we
obtain the color values with reference based colorization technique, and for
the alpha channel, we use the luminance values of each voxel as the alpha
value during volume rendering.

In brief, our major technical contributions are:

  * •

We experimentally verify the effectiveness of multi-level feature extraction
to establish deep correspondence and to eventually perform color transfer
between two medical images, where one is a colored reference image, and other
is a CT scan or a MRI image.

  * •

We extend color transfer with deep image analogy for medical volume
visualization. Our approach essentially removes tedious user interaction with
transfer function to obtain color parameters for volume rendering.

The overall layout of the paper is organized as follows. In section 2, we
discuss some of the existing methods in the field of image colorization and
volume rendering. In section 3, we explain about the image dataset that we use
in our proposed system. In subsection 4.1, we explain visual attribute
transfer using a state of the art dense correspondence algorithm like Deep
Image Analogy. This is followed by colorization using guided image filters in
subsection 4.2, where we elaborate on the further processing that is needed on
the output of deep image analogy to preserve only color and disregard other
visual attributes that are transferred from reference image to target
monochromatic medical image. In section 4.3, we explain a reference image
recommendation engine that automatically finds good reference images for color
transfer. The recommendation algorithm uses deep-features obtained from a pre-
trained network like VGG-19 to select a top few reference images which are
semantically similar to the target grayscale image. Furthermore, we briefly
explain the approach we’ve taken for volume rendering in section 4.4. And
finally in section 5, we present the medical image colorization and final
volume rendering results.

##  2\. Related Work

With the recent advances in neural networks, there have been several works to
show that deep learning approaches can be successfully applied to various
fields, and in some applications, there are cases where the results obtained
using these networks have even been superior to what human experts can
achieve. In this section we review the work that has been performed in the
following two sectors: Color Transfer and Volume Rendering.

###  2.1. Colorization

One of the novel techniques on color transfer which was proposed by Levin et
al. required color scribbling by users on top of the grayscale images and then
using an optimization technique for colorization of the input image (Levin et
al., 2004). There have been several advancements on this pioneering work. Qu
et al. extended their idea for colorization of manga which contains different
pattern and intensity continuous regions(Qu et al., 2006). These work showed
some promising results but one of the drawbacks of these methods is that they
require proper scribbling with intense manual effort.

In recent years, with capabilities of Deep Neural Networks, different learning
based techniques have been proposed for automatic colorization, which do not
require any user effort. Cheng et al. (Cheng et al., 2015) used a deep neural
network to colorize an image where they pose the colorization problem as a
regression problem. With comparatively larger set of data, Zhang et al. (Zhang
et al., 2016), Larsson et al. (Larsson et al., 2016) and Iizuka et al. (Iizuka
et al., 2016) used Convolutional Neural Networks (CNNs) to perform automatic
colorization, where the three methods differ in terms of loss functions and
CNN architectures. The first two methods used classification loss, while the
third one used regression loss. While all of these networks are learned from
large-scale data, there have been other works (Liao et al., 2017) (He et al.,
2017) that perform color transfer from a single colored reference image to a
grayscale image. These methods utilize pretrained networks like VGG-19 for
extracting deep features from the images. These deep features are used to find
reliable correspondence between the images which is further leveraged to
perform attribute transfer like style and color. While the work we perform is
primarily based on Liao et al.’s (Liao et al., 2017) work on visual attribute
transfer, we extend their idea to perform color transfer with various guided
image filters, and focus our work towards medical dataset. We further extend
our work for medical volume rendering.

###  2.2. Volume Rendering

In the last decade, research in volume rendering has expanded to a wide
variety of applications. Conventional volume rendering pipeline implements
either one dimensional or multidimensional transfer functions (Kindlmann and
Durkin, 1998) (Kniss et al., 2002)(Roettger et al., 2005) for volume
rendering. Volume rendering with multidimensional transfer functions than 1D
transfer functions have been immensely successful in producing better results,
however, with the introduction of more features for classification of voxels,
it makes interaction between users and the multidimensional transfer function
more complex.

In an attempt to take a different approach to volume rendering, a certain
number of works (Taosong He et al., 1996)(Berger et al., 2019)(Hong et al.,
2019) have utilized the capabilities of machine learning techniques by
introducing a new pipeline for volume visualization. While these methods have
been successful in producing exceptional results, the majority of them require
a significantly large amount of training data. In our method, we require a
comparatively smaller set of reference images to obtain the color properties
for volume rendering. Furthermore, our work only focuses on coloring part of
1D transfer function by automatic mapping of intensity to color without any
user effort. Obtaining opacity parameters from a transfer function also plays
a significant role in direct volume rendering, but we are not addressing it
here.

##  3\. Dataset

The Visible Human Project (VHP) (Ackerman, 1998) provides a publicly available
dataset that contains cross-sectional cryosection, CT scans, and MRI images
from one male cadaver and one female cadaver. In our work, we take 33 MRI
samples of head, 328 CT scans of thorax and abdomen, and 696 color
cryosections from Visible Human Male to perform color transfer on the
monochromatic MRI images. Among the color cryosection images, 40 of them are
from male-head, and remaining 656 images are from male-thorax and abdomen.
Here, the monochromatic images are our target images, and all of the color
cryosection slices form the database of reference images. After color
transfer, we perform ray casting for medical volume rendering where we use the
stack of colored medical images as our volumetric data.

Figure 2. Our proposed system pipeline consists of two stages. In the first
stage, we perform deep image analogy on two input images $T_{L}$ and $R_{Lab}$
to obtain the reconstructed output $T’_{Lab}$. Then, in the second stage, we
use guided image based filtering technique to perform colorization of image
$T_{L}$ with the help of reconstructed output $T’_{Lab}$

Insert Description here

Caption 1

Figure 3. Images in first column are two MRI slices of Visible Human(Ackerman,
1998) Head which represent the target images $T$, images in second column are
the cross-sectional cryosections of Visible Human Head which represent the
reference images $R$, and images in third column are the reconstructed output
$T^{\prime}$ obtained after performing visual attribute transfer with Deep
Image Analogy on images in first and second column. We can see unwanted
distortions in the output image that are not present in the target images.
(Three such distortions are indicated by red squares connected with straight
lines)

Caption 1

Figure 4.  With the use of Fast Global Smoothing filter(Min et al., 2014), we
obtain the final colorized version (third column) of the target grayscale
medical images presented in the second column with the reference images
presented in the first column. Target images (from top to bottom): two MRI
scans of head, two CT-scans of thorax, and one CT-scan of abdomen. Reference
Images (from top to bottom): two cross-sectional cryosections of head, two of
thorax and one of abdomen from. Both of the target and reference images are
from the Visible Human dataset(Ackerman, 1998).

##  4\. Methodology

Our end goal is to automatically transfer color or tone to a stack of
monochromatic medical images from semantically similar reference images and to
finally render a 3D volume using the colored image stack. Therefore, there are
two parts to our problem: automatic reference based color transfer and medical
volume rendering.

In our approach to tackle the first problem, we use a state of the art dense
correspondence algorithm like Deep Image Analogy (Liao et al., 2017) which
finds a semantically meaningful dense correspondence between two input images.
The reason behind establishing a dense correspondence is to perform color
transfer between semantically similar regions of reference image and target
image. Since, the reference images that we are using here are frozen cross
sections of the human body, we can argue that the range of colors present in
the reference image will be sufficient for colorization of target images.

###  4.1.  Deep Image Analogy

Using Deep Image Analogy (Liao et al., 2017), given that there are two input
images $T$ and $R$, we can construct an image $T^{\prime}$, which is
semantically similar to $T$ but contains the visual attributes of $R$. $T’$ is
the reconstructed result after transfer of visual attributes like color,
texture, and style. Liao et al. performed two major steps in their work for
dense matching. In the first step, their implementation computes features of
two input images $T$ and $R$ at different intermediate layers of a pretrained
CNN that is trained on image recognition tasks. In the second step, they use
the PatchMatch algorithm (Barnes et al., 2010) in deep feature domain rather
than in image domain to find deep correspondence between those images. The
final output after these two steps is a bidirectional similarity mapping
function $\phi_{T\leftrightarrow R}$ between those two images. PatchMatch is a
randomized correspondence algorithm which finds an approximate nearest
neighbor field (NNF) between two images. The algorithm has three main
components: initialization, propagation and random search, where it first
initializes NNF as a function $f:\mathbb{R}^{2}\rightarrow\mathbb{R}^{2}$ of
random offsets. This is followed by an iterative process, where good patch
offsets are propagated to adjacent pixels and and finally, a random search is
performed in the neighborhood of the best offset.

The pretrained network used by Liao et al. (Liao et al., 2017) to obtain
feature representations of an image is the original VGG-19 (Simonyan and
Zisserman, 2014) network that has been pre-trained on the colored ImageNet
dataset (Russakovsky et al., 2015). However, finding good semantic similarity
between two images is difficult when one of them is grayscale and the other is
color. Furthermore, original VGG-19 has a comparatively poor performance when
it comes to image recognition on grayscale images (He et al., 2018). So, in
order to minimize the difference in performance, we use a gray-VGG-19 as the
feature extractor network. Gray-VGG19 was trained only using the luminance
channel of images in the ImageNet dataset and the performance evaluation of
gray-VGG-19 shows that it exhibits superior performance to original VGG-19
when both of these networks are tested on gray images. Original VGG-19 tested
on gray images achieves 83.63%, while gray-VGG-19 achieves 89.39% on the Top-5
Class accuracy(He et al., 2018).

The system pipeline to perform automatic color transfer is shown in figure 2.
We propose our system with CIE Lab color space which is perceptually linear
and it expresses color as three values $L$, $a$, and $b$. Here, $L$ denotes
Luminance, and $a$ and $b$ denote Chrominance values. Thus each image can be
divided into three channels: a single luminance channel and two chrominance
channels. Our system takes two images: a grayscale target image and a color
reference image, which can be denoted as $T_{L}\in\mathbb{R}^{h*w*1}$ and
$R_{Lab}\in\mathbb{R}^{h*w*3}$ respectively. Here, $h$ and $w$ are the height
and width of the input images. The output from deep image analogy is the
bidirectional mapping function between the two images which is a spatial
warping function defined with bidirectional correspondences. The bidirectional
mapping functions can be denoted as $\phi_{T\leftrightarrow R}$, which returns
the transformed pixel location for a given source location
${}^{\prime}p^{\prime}$. It consists of a forward mapping function
$\phi_{T\rightarrow R}$, that maps pixels from T to R, and a reverse mapping
function $\phi_{R\rightarrow T}$, that maps pixels from R to T. Using the
mapping function $\phi_{R\rightarrow T}$, we map the image $R_{Lab}$ to image
$T’_{Lab}$. In other words, $T’_{Lab}$ is the reconstructed result obtained
after warping the image $R_{Lab}$ with the mapping function
$\phi_{R\rightarrow T}$.

Figure 3c) shows the reconstructed output images that were obtained after
performing deep image analogy on the given target and reference images in
figure 3a) and figure 3b). We can observe that the reconstructed result
$T’_{Lab}$ not only transferred color but other attributes like style and
texture that were extracted from the reference image $R_{Lab}$. The results
also contain distortions that are unwanted in our final colored image. In a
field like medical imaging, edges and corners are key features and the overall
structural details must be preserved when performing any kind of image
manipulation.

In our approach to tackle this problem, where we need to maintain the semantic
structure from the target image $T_{L}$, preserve color, and disregard other
visual details from the reference image, we performed guided image based
filtering technique to obtain the final colored image
$T^{\prime\prime}_{Lab}$. Guided image based edge preserving filters like
Domain-Transform (DT) filter (Gastal and Oliveira, 2011), the Guided Filter
(GF) (He et al., 2010), the Weighted Least Square Filter (WLS) (Farbman et
al., 2008) and the Fast Global Smoother (FGS) filter (Min et al., 2014) are
designed to perform smoothing of an image while preserving the edges using the
content of the guide image. Specifically, we use the Fast Global Smoother(Min
et al., 2014) to smooth both input image $T_{L}$ and $T^{\prime}_{Lab}$ with
the guidance $T_{L}$. More on this is explained in the following section.

###  4.2. Colorization using Guided Image Filters

As seen in figure 3, the reconstructed output after performing Deep Image
Analogy contains unwanted structural distortions that were introduced from the
reference image. Since our goal is to only transfer color from the reference
image, we resorted to using guided image based edge preserving smoothing
filters to preserve the color and structural details of the target image while
removing other visual artifacts from image $T^{\prime}_{Lab}$. One such filter
is the Fast Global Smoother (Min et al., 2014), which performs spatially
inhomogeneous edge-preserving smoothing. To be more specific, we performed

(1)
$T^{\prime\prime}_{Lab}=fgs(T^{\prime}_{Lab},T_{L})-fgs(T_{Lab},T_{L})+T_{Lab}$

where function $fgs(.)$ , which is a short-hand notation for the Fast Global
Smoothing filter, takes two input images: a source image, and a guide image
whose edges guide the smoothing operation for source image. First we perform
the edge aware smoothing operation on images $T’_{Lab}$ and $T_{Lab}$ with
guide image $T_{L}$ to get smoothened output $fgs(T^{\prime}_{Lab},T_{L})$ and
$fgs(T_{Lab},T_{L})$, then we add the differences between these two filtered
output to image $T_{Lab}$ to get the final colored image $T”_{Lab}$. Here,
since the target image $T_{Lab}$ is a grayscale image, its chromatic color
channels $T_{a}$ and $T_{b}$ contain true neutral gray values, i.e zero
values.

Figure 4c) shows the colorization result obtained from equation 1. We can
observe that the final colorized version of the target images are free from
undesirable distortions and attributes that were previously being introduced
from the reference image.

Caption 1

Figure 5.  Pipeline of reference image retrieval system, where the
recommendation engine takes two inputs: a monochromatic target image, and 696
color cryosection images from Visible Human head, thorax and abdomen. The
output of the recommendation engine are the top three reference images which
share similar semantic content with the target image.

Figure 6.  Comparison of results obtained after performing Deep Image Analogy
followed by Colorization using guided image filters. First column: Target
images with reference image on the top right corner of each target image.
Second column: Colorized result with Guided Filter(He et al., 2010), where
radius $r=16$, regularization term $\epsilon=2$ Third column: Colorized result
with Domain Transform Filter(Gastal and Oliveira, 2011), where sigma spatial
$\sigma_{s}=8$, sigma range(color) $\sigma_{r}=200$ Fourth column: Colorized
result with Weighted Least Square filter(Farbman et al., 2008), where lambda
$\lambda=0.2$, alpha $\alpha=1.8$ Fifth column: Colorized result with Fast
Global Smoother filter (Min et al., 2014), where lambda $\lambda=32$, sigma
range $\sigma_{r}=200$. Note that the reconstructed output obtained after
performing Deep Image Analogy is not shown in the figure above.

Insert Description here

Figure 7.  Comparison of results obtained $a)$ without gamma correction and
$b)$ with gamma correction. The circular image in the right-side of each image
shows a closer view of the region present inside the blue circle. With gamma
correction, the colored image on the bottom appears more detailed and at the
same time color spilling out of the edges is also reduced.

###  4.3. Reference Image Retrieval

To aid for selection of good reference images, we propose to use a robust
image retrieval algorithm that automatically recommends the best reference
images for performing color transfer. Figure 5 shows the pipeline of the
reference image recommendation system. The inputs to the recommendation engine
are the dataset of our reference images which contain 696 frozen cross-section
slices of Visible Human Male and a single monochromatic query image. Ideally,
the image suggested by the recommendation engine is expected to have similar
semantic structure as the query image.

There have been several works regarding content based image retrieval with the
use of deep neural networks (Babenko et al., 2014)(Wan et al., 2014)(Ng et
al., 2015)(Razavian et al., 2014). Content based Image Retrieval techniques
that use deep neural networks exploit the property of pretrained CNN networks,
where the lower layers (layers closer to the input) of the network encode low-
level details of an image and the higher layers are more sensitive towards
high-level details, that is, content of an image. Since the features obtained
from the higher layers provide a good representation of the content of an
image, it is common to use these features, which are also called descriptors,
to evaluate semantic similarity between two images.

A similar work on image retrieval has been performed by (He et al., 2018)
where they obtain features of reference images and a query image from two
different layers of gray-VGG-19 network and finally compute a similarity score
to rank the images. We adopt a similar technique, where we first feed all of
our reference images to VGG-19 network and pre-compute their feature vector
from the first fully connected layer $F^{6}_{c}$. In the second step we feed
our target image through the network to obtain its features from the same
fully connected layer of VGG-19 network.

Let $f_{R_{i}}$ denote the feature vector of $i^{th}$ reference image and ,
$f_{T}$ denote the feature vector of the target image. To obtain the measure
of semantic similarity between two images, we compute cosine similarity, as
given by equation 2, between these feature vectors for each reference-target
image pair ${R_{i},T}$. We use this similarity score to obtain the best
reference image for performing color transfer.

(2) $d(f_{R_{i}},f_{t})=\frac{f_{R_{i}}.f_{t}}{||f_{R_{i}}||\times||f_{t}||}$

Figure 8.  Volume visualization using 3D slicer for top half of Visible Human
Head. First row: Volume is rendered with 33 monochromatic MRI images. Second
row: Volume is rendered with corresponding colored versions of medical images
that are used to render the volume displayed in the first row.  Figure 9.
Volume visualization using 3D slicer for Visible Human thorax and abdomen.
First row: volume is rendered using 328 monochromatic CT scans. Second Row:
Volume is rendered with corresponding colored versions of medical images that
are used to render the volume displayed in the first row.

###  4.4. Direct Volume Rendering

After obtaining the colored images, we perform volume visualization using
3D-slicer (Fedorov et al., 2012), which is an open source medical image
computing platform. 3D-slicer provides various tools for biomedical research
and GPU-based Direct Volume Rendering is one of them. Typically, the final
volume is rendered using a stack of medical images, where ray-casting is
performed on the volumetric medical data. The ray-casting module is freely
available on 3D slicer and has been used in many studies (Li and Zhang,
2010)(Zhang et al., 2010)(Schubert and Scholl, 2011)(Meyer-Spradow et al.,
2009).

##  5\. Results

As mentioned in section 4.2, to remove the unwanted artifacts introduced from
reference image to target image, we performed guided image filtering on the
reconstructed output from Deep Image Analogy. Figure 6 shows the comparison of
results that were obtained after performing colorization with different guided
image filters. We obtain the output in the same way as mentioned in equation 1
where the fast global smoothing filter (FGS) is replaced by other filters:
Guided filter (GF), Domain Transform filter (DT),and Weighted Least Square
filter (WLS). A comparison between the four filters provided by Min et al.(Min
et al., 2014) shows that fgs provides the speed advantage over the other three
filters, and unlike GF and DT filters, it is not prone to halo effect. Due to
these benefits, the FGS filter is the preferred filter in our colorization
pipeline.

###  5.1. Gamma Correction

With color transfer, in addition to ensuring that all the semantic structures
present in the grayscale image are preserved in the final colored image, we
also aim to have our colorized result look as natural as possible. In some of
the resulting images that we obtained after performing color transfer, the
image seemed to lack proper contrast making the image appear to have a washed-
out effect. Furthermore, it was noticed that, in some regions of those images,
color was spilling out of the edges. This may happen due to excessive
smoothing of image with guided image filters. An example of this can be seen
in figure 7. So, to improve the vividness of the result, we use gamma
correction which is simple non-linear operation defined by the power law
transformation as given in equation 3.

(3) $I_{in}=\alpha I^{\gamma}_{out}$

Specifically, after obtaining the reconstructed image from deep image analogy,
we perform gamma encoding on both target and the reconstructed image, where we
use a gamma value of $\gamma=1/2$. This is followed by colorization using
guided image filters, and finally we perform gamma decoding with gamma value
of $\gamma=2$ to obtain the resulting colored image. With gamma correction,
not only the color content spilling out of the edges was suppressed, but also
the vividness of the image was improved to a certain extent, making the image
look comparatively more sharp and detailed.

###  5.2. Volume Visualization

Since there is no ground-truth for this task, we only present qualitative
results here that we obtained in our work. After obtaining the colored version
of all the monochromatic medical images, we perform volume visualization using
3D slicer (Fedorov et al., 2012). Specifically, we use the GPU based ray-
casting module provided in 3D slicer for volume visualization. First row in
Figure 8 and 9 show the volume rendered using a stack of grayscale medical
images while the second row in those two figures show the volume rendered
using colored medical images.

Here, we present the rendering result with two different image sets. In figure
8, the first row displays the rendered volume for the top half of the human
head, where we use 33 MRI scans obtained from the Visible Human Dataset.
Similarly, the first row of figure 9 displays the rendered volume for Visible
Human thorax and abdomen, where we use 328 monochromatic CT scans. In the
second row of figure 8 and 9, we present a similar viewpoint of the volume
rendered in the first row, but in this case, the volume visualization is
rather performed with colored versions of MRI images and CT scans.

To perform visual inspection on the quality of the rendered volume we also
compare the transverse section, coronal section and sagittal section between
grayscale volume and colored volume as shown in figure 8c) , 8d), 9c) and 9d).
We can see that the structure of grayscale volume is well preserved in the
final colored volume, but at the same time, we also notice the presence of
some dark patches in the colored volume, figure 9b), c) and d). On inspection
of the grayscale viewpoint of the same volume, we can tell that these patches
are not desired in the final colored volume and they were most definitely
introduced because of unwanted structures present in the reference images.

##  6\. Conclusion

In our work, we have managed to introduce a new pipeline for medical volume
visualization. First, we explored the use of visual attribute transfer with
deep image analogy, and then with the introduction of guided image based
filtering technique, we were able to perform automatic colorization of
grayscale medical images. After that, we extended our work for direct volume
rendering, where we make use of the ray-casting module provided by 3D slicer
to perform volume visualization using colored medical images. Furthermore, we
also introduced a robust reference image retrieval system which aims to
recommend the best reference image in terms of semantic similarity with the
target image to perform color transfer.

With our proposed pipeline, we completely eliminate the tedious work of
interacting with a volume renderer where a user has to experiment with
different values on a transfer function to generate plausible color in the
rendered volume. We also believe that our approach will be very helpful for a
variety of applications in the medical community and prove to be more useful
in performing anatomical analysis during clinical procedures.

##  7\. Discussion

In our work, we are successfully able to extend color transfer with deep image
analogy for medical volume visualization. The core of our work is based on the
work performed by Liao et al. (Liao et al., 2017) and since visual attribute
transfer with deep image analogy has its own drawbacks , we implicitly inherit
all of those limitations in our work.

Transfer of color or other attributes by finding out semantic similarities
between two images is not always reliable. The overall success of our work
also relies on the performance of a pretrained VGG-19 network. Our system
pipeline starts with extracting features from the images and the network
itself may not always be successful in extracting feature representation of
all the content present in the image. An alternative approach that we can take
here is to experiment with other pre-trained networks that have superior
performance than VGG-19. Then we can retrain the network on a medical image
dataset before using it as a feature extractor network.

Furthermore, when the reference image has a significant amount of semantic
differences with the target image, the system fails to find good
correspondence and as a result we see unsmooth colored artifacts in the final
images. Examples of that are seen in the second row of figure 6, where the
resulting colored CT scans of Visible Human Abdomen contain unwanted dark
patches which were transferred from the reference images. In cases like these,
it is difficult to perform reliable color transfer from one image to another.

###### Acknowledgements.

We thank anonymous reviewers for their valuable feedback. We also acknowledge
the authors of (Liao et al., 2017) and (Farbman et al., 2008) for making their
work publicly available.

## References

  * (1)
  * Ackerman (1998) M. J. Ackerman. 1998\.  The Visible Human Project.  _Proc. IEEE_ 86, 3 (1998), 504–511.
  * Babenko et al. (2014) Artem Babenko, Anton Slesarev, Alexandr Chigorin, and Victor Lempitsky. 2014.  Neural Codes for Image Retrieval.  _LNCS_ 8689\.  https://doi.org/10.1007/978-3-319-10590-1_38
  * Barnes et al. (2010) Connelly Barnes, Eli Shechtman, Dan B Goldman, and Adam Finkelstein. 2010.  The Generalized PatchMatch Correspondence Algorithm. In _European Conference on Computer Vision_.
  * Berger et al. (2019) M. Berger, J. Li, and J. A. Levine. 2019.  A Generative Model for Volume Rendering.  _IEEE Transactions on Visualization and Computer Graphics_ 25, 4 (2019), 1636–1650.
  * Bugeau et al. (2014) Aurélie Bugeau, Vinh-Thong Ta, and Nicolas Papadakis. 2014\.  Variational Exemplar-Based Image Colorization.  _IEEE Transactions on Image Processing_ 23, 1 (2014).  https://doi.org/10.1109/TIP.2013.2288929
  * Cheng et al. (2015) Zezhou Cheng, Qingxiong Yang, and Bin Sheng. 2015.  Deep Colorization. In _Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV)_ _(ICCV ’15)_. IEEE Computer Society, USA, 415–423.  https://doi.org/10.1109/ICCV.2015.55
  * Deshpande et al. (2015) Aditya Deshpande, Jason Rock, and David Forsyth. 2015\.  Learning Large-Scale Automatic Image Colorization. In _Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV)_ _(ICCV ’15)_. IEEE Computer Society, USA, 567–575.  https://doi.org/10.1109/ICCV.2015.72
  * Farbman et al. (2008) Zeev Farbman, Raanan Fattal, Dani Lischinski, and Richard Szeliski. 2008.  Edge-Preserving Decompositions for Multi-Scale Tone and Detail Manipulation.  _ACM Trans. Graph._ 27, 3 (Aug. 2008), 1–10.  https://doi.org/10.1145/1360612.1360666
  * Fedorov et al. (2012) Andrey Fedorov, Reinhard Beichel, Jayashree Kalpathy-Cramer, Julien Finet, Jean-Christophe Fillion-Robin, Sonia Pujol, Christian Bauer, Dominique Jennings, Fiona Fennessy, Milan Sonka, John Buatti, Stephen Aylward, James Miller, Steve Pieper, and Ron Kikinis. 2012\.  3D Slicer as an Image Computing Platform for the Quantitative Imaging Network.  _Magnetic resonance imaging_ 30 (07 2012), 1323–41.  https://doi.org/10.1016/j.mri.2012.05.001
  * Gastal and Oliveira (2011) Eduardo S. L. Gastal and Manuel M. Oliveira. 2011.  Domain Transform for Edge-Aware Image and Video Processing.  _ACM Trans. Graph._ 30, 4, Article 69 (July 2011), 12 pages.  https://doi.org/10.1145/2010324.1964964
  * Gupta et al. (2012) Raj Kumar Gupta, Alex Yong-Sang Chia, Deepu Rajan, Ee Sin Ng, and Huang Zhiyong. 2012.  Image Colorization Using Similar Images. In _Proceedings of the 20th ACM International Conference on Multimedia_ (Nara, Japan) _(MM ’12)_. Association for Computing Machinery, New York, NY, USA, 369–378.  https://doi.org/10.1145/2393347.2393402
  * He et al. (2010) Kaiming He, Jian Sun, and Xiaoou Tang. 2010.  Guided Image Filtering. In _Proceedings of the 11th European Conference on Computer Vision: Part I_ (Heraklion, Crete, Greece) _(ECCV’10)_. Springer-Verlag, Berlin, Heidelberg, 1–14.
  * He et al. (2018) Mingming He, Dongdong Chen, Jing Liao, Pedro V. Sander, and Lu Yuan. 2018.  Deep Exemplar-Based Colorization.  _ACM Trans. Graph._ 37, 4, Article 47 (July 2018), 16 pages.  https://doi.org/10.1145/3197517.3201365
  * He et al. (2017) Mingming He, Jing Liao, Lu Yuan, and Pedro V. Sander. 2017\.  Neural Color Transfer between Images.  _ArXiv_ abs/1710.00756 (2017).
  * Hong et al. (2019) Fan Hong, Can Liu, and Xiaoru Yuan. 2019.  DNN-VolVis: Interactive Volume Visualization Supported by Deep Neural Network. 282–291.  https://doi.org/10.1109/PacificVis.2019.00041
  * Huang et al. (2005) Yi-Chin Huang, Yi-Shin Tung, Jun-Cheng Chen, Sung-Wen Wang, and Ja-Ling Wu. 2005.  An Adaptive Edge Detection Based Colorization Algorithm and Its Applications. In _Proceedings of the 13th Annual ACM International Conference on Multimedia_ (Hilton, Singapore) _(MULTIMEDIA ’05)_. Association for Computing Machinery, New York, NY, USA, 351–354.  https://doi.org/10.1145/1101149.1101223
  * Iizuka et al. (2016) Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2016\.  Let There Be Color! Joint End-to-End Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification.  _ACM Trans. Graph._ 35, 4, Article 110 (July 2016), 11 pages.  https://doi.org/10.1145/2897824.2925974
  * Irony et al. (2005) Revital Irony, Daniel Cohen-Or, and Dani Lischinski. 2005\.  Colorization by Example. In _Proceedings of the Sixteenth Eurographics Conference on Rendering Techniques_ (Konstanz, Germany) _(EGSR ’05)_. Eurographics Association, Goslar, DEU, 201–210.
  * Kindlmann and Durkin (1998) Gordon Kindlmann and James W. Durkin. 1998.  Semi-Automatic Generation of Transfer Functions for Direct Volume Rendering. In _Proceedings of the 1998 IEEE Symposium on Volume Visualization_ (Research Triangle Park, North Carolina, USA) _(VVS ’98)_. Association for Computing Machinery, New York, NY, USA, 79–86.  https://doi.org/10.1145/288126.288167
  * Kniss et al. (2002) Joe Kniss, Gordon Kindlmann, and Charles Hansen. 2002\.  Multidimensional Transfer Functions for Interactive Volume Rendering.  _IEEE Transactions on Visualization and Computer Graphics_ 8, 3 (July 2002), 270–285.  https://doi.org/10.1109/TVCG.2002.1021579
  * Larsson et al. (2016) Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. 2016\.  Learning Representations for Automatic Colorization. In _European Conference on Computer Vision (ECCV)_.
  * Levin et al. (2004) Anat Levin, Dani Lischinski, and Yair Weiss. 2004\.  Colorization Using Optimization. In _ACM SIGGRAPH 2004 Papers_ (Los Angeles, California) _(SIGGRAPH ’04)_. Association for Computing Machinery, New York, NY, USA, 689–694.  https://doi.org/10.1145/1186562.1015780
  * Li and Zhang (2010) Zhenwei Li and Jianguo Zhang. 2010.  Study on volume rendering of CT slices based on ray casting.  _Proceedings - 2010 3rd IEEE International Conference on Computer Science and Information Technology, ICCSIT 2010_ 7 (07 2010).  https://doi.org/10.1109/ICCSIT.2010.5564768
  * Liao et al. (2017) Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, and Sing Bing Kang. 2017.  Visual Attribute Transfer through Deep Image Analogy.  _ACM Trans. Graph._ 36, 4, Article 120 (July 2017), 15 pages.  https://doi.org/10.1145/3072959.3073683
  * Luan et al. (2007) Qing Luan, Fang Wen, Daniel Cohen-Or, Lin Liang, Ying-Qing Xu, and Heung-Yeung Shum. 2007\.  Natural Image Colorization. In _Proceedings of the 18th Eurographics Conference on Rendering Techniques_ (Grenoble, France) _(EGSR’07)_. Eurographics Association, Goslar, DEU, 309–320.
  * Meyer-Spradow et al. (2009) Jennis Meyer-Spradow, Timo Ropinski, Jörg Mensmann, and Klaus Hinrichs. 2009.  Voreen: A Rapid-Prototyping Environment for Ray-Casting-Based Volume Visualizations.  _IEEE Comput. Graph. Appl._ 29, 6 (Nov. 2009), 6–13.
  * Min et al. (2014) Dongbo Min, Sunghwan Choi, Jiangbo Lu, Bumsub Ham, Kwanghoon Sohn, and Minh Do. 2014\.  Fast Global Image Smoothing Based on Weighted Least Squares.  _IEEE transactions on image processing : a publication of the IEEE Signal Processing Society_ 23 (10 2014).  https://doi.org/10.1109/TIP.2014.2366600
  * Ng et al. (2015) Joe Ng, Fan Yang, and Larry Davis. 2015.  Exploiting local features from deep networks for image retrieval. 53–61.  https://doi.org/10.1109/CVPRW.2015.7301272
  * Qu et al. (2006) Yingge Qu, Tien-Tsin Wong, and Pheng-Ann Heng. 2006\.  Manga Colorization.  _ACM Trans. Graph._ 25, 3 (July 2006), 1214–1220.  https://doi.org/10.1145/1141911.1142017
  * Razavian et al. (2014) Ali Razavian, Josephine Sullivan, Atsuto Maki, and Stefan Carlsson. 2014.  A Baseline for Visual Instance Retrieval with Deep Convolutional Networks.  _ITE Transactions on Media Technology and Applications_ 4 (12 2014).  https://doi.org/10.3169/mta.4.251
  * Roettger et al. (2005) Stefan Roettger, Michael Bauer, and Marc Stamminger. 2005\.  Spatialized Transfer Functions. In _EUROVIS 2005: Eurographics / IEEE VGTC Symposium on Visualization_ , Ken Brodlie, David Duke, and Ken Joy (Eds.). The Eurographics Association.  https://doi.org/10.2312/VisSym/EuroVis05/271-278
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015\.  ImageNet Large Scale Visual Recognition Challenge.  _International Journal of Computer Vision (IJCV)_ 115, 3 (2015), 211–252.  https://doi.org/10.1007/s11263-015-0816-y
  * Schubert and Scholl (2011) Nicole Schubert and Ingrid Scholl. 2011.  Comparing GPU-based multi-volume ray casting techniques.  _Computer Science - Research and Development_ 26 (02 2011), 39–50.  https://doi.org/10.1007/s00450-010-0141-1
  * Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014.  Very Deep Convolutional Networks for Large-Scale Image Recognition.  _arXiv 1409.1556_ (09 2014).
  * Taosong He et al. (1996) Taosong He, Lichan Hong, A. Kaufman, and H. Pfister. 1996.  Generation of transfer functions with stochastic search techniques. In _Proceedings of Seventh Annual IEEE Visualization ’96_. 227–234.
  * Wan et al. (2014) Ji Wan, Dayong Wang, Steven Chu Hong Hoi, Pengcheng Wu, Jianke Zhu, Yongdong Zhang, and Jintao Li. 2014.  Deep Learning for Content-Based Image Retrieval: A Comprehensive Study. In _Proceedings of the 22nd ACM International Conference on Multimedia_ (Orlando, Florida, USA) _(MM ’14)_. Association for Computing Machinery, New York, NY, USA, 157–166.  https://doi.org/10.1145/2647868.2654948
  * Welsh et al. (2002) Tomihisa Welsh, Michael Ashikhmin, and Klaus Mueller. 2002\.  Transferring Color to Greyscale Images.  _ACM Trans. Graph._ 21, 3 (July 2002), 277–280.  https://doi.org/10.1145/566654.566576
  * Zhang et al. (2010) Qi Zhang, Roy Eagleson, and Terry M. Peters. 2010.  Volume Visualization: A Technical Overview with a Focus on Medical Applications.  _Journal of Digital Imaging_ 24 (2010), 640–664.
  * Zhang et al. (2016) Richard Zhang, Phillip Isola, and Alexei A Efros. 2016\.  Colorful Image Colorization. In _ECCV_.