plus1sp

[1]Marian Himstedt

# Automatic Generation of Synthetic Colonoscopy Videos for Domain
Randomization

Abhishek Dinkar Jagtap  Institute of Medical Informatics, University of
Lübeck, Germany, e-mail<EMAIL_ADDRESS>Mattias Heinrich
Institute of Medical Informatics, University of Lübeck, Germany, e-mail:
<EMAIL_ADDRESS>
###### Abstract

An increasing number of colonoscopic guidance and assistance systems rely on
machine learning algorithms which require a large amount of high-quality
training data. In order to ensure high performance, the latter has to resemble
a substantial portion of possible configurations. This particularly addresses
varying anatomy, mucosa appearance and image sensor characteristics which are
likely deteriorated by motion blur and inadequate illumination. The limited
amount of readily available training data hampers to account for all of these
possible configurations which results in reduced generalization capabilities
of machine learning models. We propose an exemplary solution for synthesizing
colonoscopy videos with substantial appearance and anatomical variations which
enables to learn discriminative domain-randomized representations of the
interior colon while mimicking real-world settings. 111Software components and
video material will be made publicly available upon publication.

##  1 Introduction

A common problem faced in machine learning is the lack of sufficient training
data. For colonoscopy the majority of readily-available public image data is
limited to individual frames or short sequences for benchmarking CAD-based
polyp detection. Public colonoscopy videos of the entire colon structure are
limited to rather low-quality capsule endoscopy video footage. The lack of
ground truth camera poses further hampers the training of models for
applications different from polyp detection such as: anatomical segment
classification, visual place recognition (VPR), simultaneous localization and
mapping (SLAM) and structure from motion (SfM). These applications require
high-quality colonoscopy videos of the entire examinations covering all phases
of the intervention.

Fig. 1: Examples of synthetic colonoscopy images.

A common solution to this is the rendering of virtual endoscopy (VE) videos
based on CT colonography data. VE provides both, image sequences and ground
truth poses of varying anatomy, but (without further investigation) differs
substantially from the visual appearance of real colonoscopy images. This
entails gaps that have to be addressed by proper domain adaptation methods as
demonstrated in [mathew2020augmenting]. This, however, implies that
synthesized images resemble colonoscopy images (and their anatomical
locations) of small datasets which likely do not generalize well to unseen or
less observed colon regions. Domain randomization, in contrast, utilizes a
large amount of data which is randomly sampled over the entire configuration
space with the variables being carefully predefined. It is important to note
that domain randomization is practically applicable to only simulated data as
some of the parameters such as textures, materials, occlusions and coat masks
have to be properly controlled in a simulated environment have to be more
elaborated than for generating VE images in order to enable visual appearance
close to real colonoscopy images (see Fig. 1). Powerful engines such as Unity
have gained particular interest in the computer vision and robotics
communities [1, tremblay2018training], but have been rarely investigated in
medical imaging [2, billot2021synthseg].

Fig. 2: Overview of the utilized processing pipeline for generating synthetic
images.

Given sufficient capabilities of simulation, models can solely be trained on
domain-randomized data while still achieving high generalization performance
for inference on real-world test data.

This paper presents an exemplary implementation of domain randomization for
colonoscopy with all required algorithmic components. It is built up on prior
work [2] and supplements the latter by automated domain-randomized video
recording through following waypoints along the interior colon's centerline.

##  2 Material and Methods

###  2.1 Colon segmentation

At first, a CT colonography (CT with radiocontrast material) obtained from
TCIA is imported in 3D Slicer for semi-automatic colon segmentation which is
carried out as follows. A ROI around the colon is set manually with its image
content being thresholded. Subsequently we apply region-based segmentation on
the (thresholded) mask to further delineate the colon structure. The
segmentation mask is manually curated to ensure optimal results for successive
steps.

###  2.2 Centerline extraction

For automated image collection we require an appropriate camera path through
the interior colon. For this purpose, we estimate the centerline within the
colon structure based on the prior work of [6]. The key idea is to plan an
obstacle-free (w.r.t colon wall) path from the anus (colon entry) to the
caecum. Since the intuitive approach based on the shortest path estimation
tends to get too close to corners in turns, Wan et al. propose to explicitly
incorporate the inversed map of distances to the colon wall [6] which was
demonstrated to achieve optimal results with paths being exactly centered.
Subsequently we sample equidistant waypoints along the extracted centerline
which will be utilized within the simulation. Currently, we manually pick
start and end points of the centerline extraction which, however, could be
replaced by automatic anatomical landmark prediction through heatmap
regression.

###  2.3 3D model preparation

Next, the colon segmentation is imported in Blender for UV editing. Generally,
a mesh is created surrounding the organ that can be edited along the vertices
of the object. This mesh allows UV mapping which is a method for projecting a
3D model surface onto a 2D plane for texture mapping. An UV editing tool as
part of Blender offers the possibility of unwrapping the 3D object onto a 2D
plane where textures can be applied seamlessly throughout the region of the
colon. This texture gives a realistic pattern to the object. Default shaders
in Blender enable to change material properties corresponding to colon such as
surface IOR, secular tint and anisotropy to further enhance the realism.

###  2.4 Photorealistic rendering

The 3D model prepared in Blender is subsequently imported in Unity which
provides high definition render pipelines for our simulation environment that
can produce photorealistic colonoscopy images. This virtual engine is commonly
used for game development and has drawn particular interest in computer vision
research due to its powerful graphical simulation platform for generating
synthetic images. Using Unity we are able to synthesize images where
parameters such as lighting, materials, occlusions, transparency and coat mask
are altered to give it a more realistic appearance. These parameters are
carefully selected such that real-world characteristics are optimally
mimicked. As a starting base we utilize parts of the VR-Caps project
simulating a capsule endoscopic camera within Unity [2]. A 3D model of this
capsule with predefined attributes of an attached camera is placed inside the
colon which is used for data collection. Adjusting these parameters is crucial
for both mimicking real endoscopy and augmenting the data. The table below
shows the camera parameters and post-processing effects required to achieve a
fully synthetic model of the colon. For potential navigation tasks it is
possible to additionally store corresponding depth images.

Attributes | Values
---|---
Surface Metallic | 0.3
Surface Smoothness | 0.7
Lens Intensity | 0.1
Chromic Abberation | 0.5
Coat Mask | 0.435
Camera's Field of View | 91.375
Focal length | 159.45
ISO | 200
Aperture | 16
Anisotropy | 1
Tab. 1: Camera parameters and Post-processing Effects Fig. 3: Synthesized,
domain-randomized images captured at the same pose inside the colon. Textures
are obtained as from random patterns as well as synthetic patterns mimicking
mucosa appearances.

###  2.5 Automated Video Rendering

Manually collecting data for endoscopy becomes highly time-consuming when
creating synthetic datasets consisting of all the required variations and
diversity. For domain randomization we need to record sequences of images each
time with different textures and materials which entails substantial
individual setup. Thus, an approach for automating the process of data
collection is introduced, which allows us to collect numerous samples inside
the colon with different parameters. For this purpose we make use of the
scripting API offered by Unity which gives access to the simulation
environment and interactive components via executable scripts. Firstly, the
simulated capsule is introduced to the colon and then automatically steered
along the waypoints of the centerline (see Fig. 4). The Unity engine is setup
appropriately such that it enables smooth camera motion while following the
waypoints. Our path following script consist of two parts: an initialization
function which runs all required initial setups (parameter setup, initial
capsule pose) and an update function which constantly controls the movement of
the capsule (along the waypoints) and triggers actions such as changing
parameters (e.g. lightning, texture). All images captured by the camera of the
capsule are recorded. The parameters can either be adjusted on the fly
allowing to capture images at the same pose with varying conditions or
alternatively it is possible to alter the parameter set only for entire
traversals. Unity also enables to configure the capsule's speed and camera's
field of view and targeted frame rate (FPS).

##  3 Results and Discussion

We evaluate our simulation qualitatively based on image renderings for varying
parameters which becomes particularly apparent when randomizing surface
material and textural patterns. This is illustrated by Fig.3 which shows
different renderings from the same captured from the same camera pose inside
the colon. Fig. 4 shows an example of an extracted centerline and generated
waypoints being followed for automated video recording. For comparison, Fig. 5
shows exemplary real and synthetic images respectively.

Fig. 4: Path following the centerline of the colon. The green line visualize
the path and red circles waypoints being traced by the simulated capsule. Fig.
5: Comparison of synthetic (top) and real (bottom) colonoscopy images.

##  4 Conclusion

This paper presented a pipeline for generating synthetic colonoscopy videos
that can be used to improve training of deep learning models. By controlling
environment (e.g. texture, reflectance) as well as virtual camera (e.g.
lightning) properties we are able to simulate conditions being observed in
inference but hardly ever presented in training data which is particularly the
case for small-scale datasets. Inspired by substantive improvements reported
on computer vision and robotics applications and limited prior work (VR-Caps)
[2] we motivate to utilize domain-randomized synthesization for video
colonoscopy. In our future work, we will incorporate this additional data for
training deep learning-based approaches to SfM, SLAM and 3D reconstruction. In
order to further simplify the variation in patient anatomy, we will
investigate (fully) automatic segmentation of the colon in CT scans as well as
an alternative to the 3D model preparation in Blender.

## Bibliography

  * [1] S. Borkman, A. Crespi, S. Dhakad, S. Ganguly, J. Hogins, Y.-C. Jhang, M. Kamalzadeh, B. Li, S. Leal, P. Parisi _et al._ , ``Unity perception: Generate synthetic data for computer vision,'' _arXiv preprint arXiv:2107.04259_ , 2021.
  * [2] K. Incetan, I. O. Celik, A. Obeid, G. I. Gokceler, K. B. Ozyoruk, Y. Almalioglu, R. J. Chen, F. Mahmood, H. Gilbert, N. J. Durr, and M. Turan, ``Vr-caps: A virtual environment for capsule endoscopy,'' 2020.
  * [3] A. Choudhary, L. Tong, Y. Zhu, and M. D. Wang, ``Advancing medical imaging informatics by deep learning-based domain adaptation,'' _Yearbook of medical informatics_ , vol. 29, no. 01, pp. 129–138, 2020.
  * [4] A. Juliani, V.-P. Berges, E. Teng, A. Cohen, J. Harper, C. Elion, C. Goy, Y. Gao, H. Henry, M. Mattar _et al._ , ``Unity: A general platform for intelligent agents,'' _arXiv preprint arXiv:1809.02627_ , 2018.
  * [5] P. J. Reynisson, M. Scali, E. Smistad, E. F. Hofstad, H. O. Leira, F. Lindseth, T. A. Nagelhus Hernes, T. Amundsen, H. Sorger, and T. Langø, ``Airway segmentation and centerline extraction from thoracic ct–comparison of a new method to state of the art commercialized methods,'' _PloS one_ , vol. 10, no. 12, p. e0144282, 2015.
  * [6] M. Wan, Z. Liang, Q. Ke, L. Hong, I. Bitter, and A. Kaufman, ``Automatic centerline extraction for virtual colonoscopy,'' _IEEE transactions on medical imaging_ , vol. 21, no. 12, pp. 1450–1460, 2002.
  * [7] S. I. Nikolenko, ``Synthetic data for deep learning,'' 2021.
  * [8] V. Seib, B. Lange, and S. Wirtz, ``Mixing real and synthetic data to enhance neural network training–a review of current approaches,'' _arXiv preprint arXiv:2007.08781_ , 2020.