#  Remark on using quantum states prepared by the adiabatic quantum
computation

Kazuto Oshima

National Institute of Technology, Gunma College,Maebashi 371-8530, Japan
E-mail<EMAIL_ADDRESS>
###### Abstract

We indicate that there are points to keep in mind in utilizing quantum states
prepared by the adiabatic quantum computation. Even if an instantaneous
expectation value of a physical quantity for the adiabatically prepared
quantum state is close to an expectation value for the true vacuum, this does
not assure us that the prepared vacuum is close to the true vacuum. In general
time average of the expectation value tend to systematically differ from the
true value. Using a simple model we discuss how to diminish this systematic
difference.

PACS numbers:03.65.-w, 03.67.-a

Recently, several quantum systems are analyzed by quantum computers or quantum
simulators with not so many qubits[1]. The quantum annealing[2] gives a
fundamental principle of the D-wave. The adiabatic quantum computation,
advocated by Farhi et.al.[3] more than two decades ago, can also be carried
out for quantum systems with not so many number of qubits. For some quantum
field theories the adiabatic quantum computation have been used for preparing
ground states[4, 5]. After preparing ground states, it has been observed that
an expectation value of certain physical quantity varies significantly under a
constant Hamiltonian[4]. This oscillation originates from the deviation of the
prepared vacuum from the true vacuum. It also has been observed that a time
average of the expectation value of the physical quantity systematically
differs from the exact value computed by another method.

The purpose of this paper is to indicate that, even if an instantaneous
expectation value of a physical quantity for the approximate vacuum prepared
by the adiabatic method is close to the expectation value for the true vacuum
by chance, it is inevitable in general that an expectation value of a physical
quantity for the approximate vacuum significantly oscillates in time around a
point that slightly differs from an expectation value for the true vacuum. We
also discuss how to diminish this systematic difference.

According to the adiabatic quantum computation[3], we start from a simple
Hamiltonian $H_{0}$ that has a non-generate trivial vacuum. We gradually
change the Hamiltonian in time to a target Hamiltonian $H_{T}$ that we should
analyze. The quantum adiabatic theorem[6, 7] assures us that the trivial
vacuum of the initial Hamiltonian $H_{0}$ approaches the vacuum of the target
Hamiltonian $H_{T}$ if the change of the Hamiltonian is very moderate and
there is a sufficient energy gap between the vacuum and excited states of the
time varying Hamiltonian. We simulate quantum adiabatic computation using the
quantum simulator by IBM for the simplest one-qubit case. We examine the
quantum state prepared by the quantum adiabatic computation.

First, we choose the initial Hamiltonian ${\hat{H}}_{0}=-JZ,J>0$, and the
target Hamiltonian ${\hat{H}}_{T}=-JX$. The initial ground state is
$|0\rangle$ and the desired final state is $|+\rangle$. The adiabatic
Hamiltonian ${\hat{H}}_{A}(s)$ that connects ${\hat{H}}_{0}$ and
${\hat{H}}_{T}$ is given by

${\hat{H}}_{A}(s)=(1-s){\hat{H}}_{0}+s{\hat{H}}_{T},\qquad 0\leq s\leq 1,$ (1)

where for example $s={t\over T},0\leq t\leq T$ for an adequate time period
$T$. The quantum adiabatic computation starts at the time $t=0$ and finishes
at the time $t=T$. For the ideal case, the ground state of the target
Hamiltonian ${\hat{H}}_{T}$ has been prepared at the time $t=T$. After the
time $t=T$, we observe a physical quantity under the constant Hamiltonian
${\hat{H}}_{T}$. If the true ground state $|+\rangle$ has been prepared, an
expectation value of the physical quantity is constant. In the actual
adiabatic quantum computation we have a quantum state that is slightly
different from the true vacuum $|+\rangle$. Let us represent the state we have
at the time $t=T$ as

$|\psi(t=T)\rangle=\alpha|+\rangle+\beta|-\rangle,\quad|\alpha|^{2}+|\beta|^{2}=1,$
(2)

where $|\beta|^{2}$ is supposed to be small. Under the total Hamiltonian
${\hat{H}}_{T}=-JX$, this quantum state time develops as

$|\psi(t)\rangle={\alpha}e^{iJ(t-T)}|+\rangle+{\beta}e^{-iJ(t-T)}|-\rangle,$
(3)

where we have set the Plank constant as $\hbar=1$ for simplicity. For this
state we measure the observable $Z$. The expectation value of $Z$ time
develops as

$\langle\psi(t)|Z|\psi(t)\rangle=2|\alpha\beta|\cos(2J(t-T)+\theta),$ (4)

where the angle $\theta$ is defined by
$\alpha\beta^{*}=|\alpha\beta^{*}|e^{i\theta}$. Thus at the time $t=T$ we get
the expectation value $2|\alpha\beta|\cos{\theta}$, which may be a good
approximation of the desired value $\langle+|Z|+\rangle=0$ by chance. The
deviation, however, reaches up to $2|\alpha\beta|$ in the time development. We
can obtain the precise value by time averaging
$\langle\psi(t)|Z|\psi(t)\rangle$ over a period. Fig.1(a) shows one of our
quantum simulation results. From the peak to peak value $4|\alpha\beta|$ we
compute the varance $2|\alpha\beta|^{2}$ as $0.000730$, and we get
$2|\beta|^{2}=0.000730$. By another simulation shots we observe $-X$ that
commutes with the Hamiltonian. After the time $t=T$ the expectation value of
$-X$ is almost constant(Fig.1(b)). Its average is $-0.999320$ and its variance
is $1.35\times 10^{-9}$ for one of our simulation result with $10^{6}$ shots.
The value $-0.999320$ almost agrees with the previous value
$-1+2|\beta|^{2}=-0.999270$.

Second, we examine another simple one-qubit model. We take the initial
Hamiltonian as ${\hat{H}}_{0}=-JZ,J>0$, and we take the target Hamiltonian as
${\hat{H}}_{T}=-JH$, where $H$ is the Hadamard gate. We again observe the
physical quantity $Z$. We represent the eigenstates of the target Hamiltonian
${\hat{H}}_{T}=-JH$ as $|h\pm\rangle$, where they satisfy
$H|h\pm\rangle=\pm|h\pm\rangle$. The explicit expressions of $|h\pm\rangle$
are

$|h+\rangle={1\over\sqrt{4-2\sqrt{2}}}(|0\rangle+(\sqrt{2}-1)|1\rangle),$ (5)
$|h-\rangle={1\over\sqrt{4+2\sqrt{2}}}(|0\rangle-(\sqrt{2}+1)|1\rangle),$ (6)

After the adiabatic state preparation process, the observable $Z$ time
develops as

$e^{i{\hat{H}}_{T}t}Ze^{-i{\hat{H}}_{T}t}=e^{-iJ{H}t}Ze^{iJ{H}t}={1\over\sqrt{2}}H-{1\over\sqrt{2}}Y\sin{2Jt}+{1\over
2}(Z-X)\cos{2Jt}.$ (7)

At the time $t=T$ if we have a state
$|\psi(t=T)\rangle=\alpha|h+\rangle+\beta|h-\rangle,|\alpha|^{2}+|\beta|^{2}=1$,
instead of the desired state $|h+\rangle$, we have at a time $t(\geq T)$

$\langle\psi(t)|Z|\psi(t)\rangle={1\over\sqrt{2}}(1-2|\beta|^{2})+\sqrt{2}|\alpha\beta|\cos(2J(t-T)+\theta),$
(8)

where we have again set $\alpha\beta^{*}=|\alpha\beta^{*}|e^{i\theta}$. Since
the physical quantity $Z$ does not anti-commute with the target Hamiltonian
${\hat{H}}_{T}=-JH$, the expectation value $\langle\psi(t)|Z|\psi(t)\rangle$
oscillate in time around the value ${1\over\sqrt{2}}(1-2|\beta|^{2})$ that
slightly less than the desired value ${1\over\sqrt{2}}=0.707107$.

Fig.2 shows one of our simulation results. We have used the second order
Suzuki-Trotter formula[8, 9]. In the result, a time average of
$\langle\psi(t)|Z|\psi(t)\rangle$ is 0.706690, which we have computed from the
average of the maximum value and the minimum value. We can find the value
$|\beta|^{2}$ from the variance of the values of
$\langle\psi(t)|Z|\psi(t)\rangle$ in the range $t\geq T$. In the result, the
variance $|\alpha|^{2}|\beta|^{2}$ is $0.0003222$ and we find
$|\beta|^{2}=0.0003223$. Thus the expectation value of $Z$ is slightly
improved to $0.707145$. Thus the systematic error for the expectation value of
$Z$ that is obtained from the time average has been diminished.

We have studied the quantum state preparation by the adiabatic quantum
computation. We have examined two simple 1-qubit models. For the first case,
the prepared quantum state is supposed to be a superposition of the true
vacuum and the excited state. The expectation value oscillates in time around
the expectation value of the true vacuum. This is rather special case that the
physical quantity anti-commute with the target Hamiltonian. The second model
will represent rather general case. We observe the physical quantity that does
not anti-commute with the target Hamiltonian. In this case the time average of
the expectation value differs from the expectation value for the true vacuum.
We can diminish this difference from the time behavior of the expectation
value. Although our models may be simple, our analysis would be useful to
grasp properties of adiabatically prepared quantum states for more complicated
systems, such as $(1+1)-$dimensional Schwinger model[4].

## References

  * [1] E.A.Martinez, C.A. Muschik, P.Schindler, D.Nigg, A.Erhard, M.Heyl, P.Hauke, M.Dalmonte, T.Monz, P.Zoller and R.Blatt, 534, 516 (2016).
  * [2] T.Kadowaki and H.Nishimori, Phys.Rev.E58, 5355(1998).
  * [3] E.Farhi, J.Goldstone, S.Gutmann, J.Lapan, A.Lundgren and D.Preda, Science292, 472(2001).
  * [4] B.Chakraborty, M.Honda, T.Izubuti, Y.Kikuchi and A.Tomiya, arXiv:2001.00485(hep-lat).
  * [5] M.Honda, E.Itou, Y.Kikuchi, L.Nagano and T.Okuda, arXiv:2105.03276(hep-lat).
  * [6] M. Born and V. A. Fock, Zeit. fur Physik A51, 165(1928),
  * [7] T. Kato, J. Phys. Soc Japan. 5, 435(1950).
  * [8] H.H.Trotter, Proc. Amer. Math. Soc. 10 ,545(1959).
  * [9] M.Suzuki, Comm. Math. Phys. 51, 183 (1976).

Figure Captions

Fig.1(a)
Simulation result of the adiabatic state preparation for the Hamiltonian
${\hat{H}}_{T}=-JX$ by IBM qasm-simulator. We have started from the ground
state of ${\hat{H}}_{0}=-JZ$ and we have observed $Z$. We have set $J=1$, the
adiabatic time period $T=36$, and one time-step width $\delta{t}={1\over 8}$.
The number of shots is $10^{6}$. The orange line represent the theoretical
value. After the time $T$, a time average over a period precisly leads to 0.

Fig.1(b) An expectation value of $-X$. We have used another $10^{6}$ shots.

Fig.2
Simulation result of the adiabatic state preparation for the Hamiltonian
${\hat{H}}_{T}=-JH$ by IBM qasm-simulator. We have started from the ground
state of ${\hat{H}}_{0}=-JZ$ and we have observed $Z$. We have set
$J={\pi\over 4}$, the adiabatic time period $T=36$, and one time-step width
$\delta{t}={1\over 24}$. The number of shots is $10^{6}$. The orange line
represent the theoretical value. After the time $T$, a time average over a
period slightly less than the theoretical value ${1\over\sqrt{2}}$.

Fig.1(a)

Fig.1(b)

Fig.2


11institutetext: Insight SFI Centre for Data Analytics, Ireland
22institutetext: Dublin City University, Ireland

# Synthetic data for unsupervised polyp segmentation ††thanks: This project
has received funding from the European Union’s Horizon 2020 research and
innovation programme under the Marie Skłodowska-Curie grant agreement No
765140. This publication has emanated from research supported by Science
Foundation Ireland (SFI) under Grant Number SFI/12/RC/2289_P2, co-funded by
the European Regional Development Fund.

Enric Moreu 1122 0000-0003-1336-6477 Kevin McGuinness 1122 0000-0003-1336-6477
Noel E. O’Connor 1122 0000-0002-4033-9135

###### Abstract

Deep learning has shown excellent performance in analysing medical images.
However, datasets are difficult to obtain due privacy issues, standardization
problems, and lack of annotations. We address these problems by producing
realistic synthetic images using a combination of 3D technologies and
generative adversarial networks. We use zero annotations from medical
professionals in our pipeline. Our fully unsupervised method achieves
promising results on five real polyp segmentation datasets. As a part of this
study we release Synth-Colon, an entirely synthetic dataset that includes 20
000 realistic colon images and additional details about depth and 3D geometry:
https://enric1994.github.io/synth-colon

###### Keywords:

Computer Vision Synthetic Data Polyp Segmentation Unsupervised Learning

##  1 Introduction

Colorectal cancer is one of the most commonly diagnosed cancer types. It can
be treated with an early intervention, which consists of detecting and
removing polyps in the colon. The accuracy of the procedure strongly depends
on the medical professionals experience and hand-eye coordination during the
procedure, which can last up to 60 minutes. Computer vision can provide real-
time support for doctors to ensure a reliable examination by double-checking
all the tissues during the colonoscopy.

Figure 1: Synth-Colon dataset samples include: synthetic image, annotation,
realistic image, depth map, and 3D mesh (from left to right).

The data obtained during a colonoscopy is accompanied by a set of issues that
prevent creating datasets for computer vision applications. First, there are
privacy issues because it is considered personal data that can not be used
without the consent of the patients. Second, there are a wide range of cameras
and lights used to perform colonoscopies. Every device has its own focal
length, aperture, and resolution. There are no large datasets with
standardized parameters. Finally, polyp segmentation datasets are expensive
because they depend on the annotations of qualified professionals with limited
available time.

We propose an unsupervised method to detect polyps that does not require
annotations by combining 3D rendering and a CycleGAN [23]. First, we produce
artificial colons and polyps based on a set of parameters. Annotations of the
location of the polyps are automatically generated by the 3D engine. Second,
the synthetic images are used alongside real images to train a CycleGAN. The
CycleGAN is used to make the synthetic images appear more realistic. Finally,
we train a HarDNeT-based model [3], a state-of-the-art polyp segmentation
architecture, with the realistic synthetic data and our self-generated
synthetic labels.

The contributions of this paper are as follows:

  * •

To the best of our knowledge, we are the first to train a polyp segmentation
model with zero annotations from the real world.

  * •

We propose a pipeline that preserves the self-generated annotations when
shifting the domain from synthetic to real.

  * •

We release Synth-Colon (see Figure 1), the largest synthetic dataset for polyp
segmentation including additional data such as depth and 3D mesh.

The remainder of the paper is structured as follows: Section 2 reviews
relevant work, Section 3 explains our method, Section 4 presents the Synth-
Colon dataset, Section 5 describes our experiments, and Section 6 concludes
the paper.”

##  2 Related work

Here we briefly review some relevant works related to polyp segmentation and
synthetic data.

###  2.1 Polyp segmentation

Figure 2: Real samples from CVC-ColonDB with the corresponding annotation made
by a medical professionals indicating the location of cancerous polyps.

Early polyp segmentation was based in the texture and shape of the polyps. For
example, Hwang et al. [8] used ellipse fitting techniques based on shape.
However, some corectal polyps can be small (5mm) and are not detected by these
techniques. In addition, the texture is easily confused with other tissues in
the colon as can be seen in Figure 2.

With the rise of convolutional neural networks (CNNs) [10] the problem of the
texture and shape of the polyps was solved and the accuracy was substantially
increased. Several authors have applied deep convolutional networks to the
polyp segmentation problem. Brandao et al. [2] proposed to use a fully
convolutional neural network based on the VGG [16] architecture to identify
and segment polyps. Unfortunately, the small datasets available and the large
number of parameters make these large networks prone to overfitting. Zhou et
al. [22] used an encoder-decoder network with dense skip pathways between
layers that prevented the vanishing gradient problem of VGG networks. They
also significantly reduced the number of parameters, reducing the amount of
overfitting. More recently, Chao et al. [3] reduced the number of shortcut
connections in the network to speed-up inference time, a critical issue when
performing real-time colonoscopies in high-resolution. They focused on
reducing the memory traffic to access intermediate features, reducing the
latency. Finally, Huang et al. [7] improved the performance and inference time
by combining HarDNet [3] with a cascaded partial decoder [21] that discards
larger resolution features of shallower layers to reduce latency.

###  2.2 Synthetic data

The limitation of using large neural networks is that they often require large
amounts of annotated data. This problem is particularly acute in medical
imaging due to problems in privacy, standardization, and the lack of
professional annotators. Table 1 shows the limited size and resolution of the
datasets used to train and evaluate existing polyp segmentation models. The
lack of large datasets for polyp segmentation can be addressed by generating
synthetic data.

Thambawita et al. [18] used a generative adversarial network (GAN) to produce
new colonoscopy images and annotations. They added a fourth channel to SinGAN
[14] to generate annotations that are consistent with the colon image. They
then used style transfer to improve the realism of the textures. Their results
are excellent considering the small quantity of real images and professional
annotations that are used. Gao et al. [6] used a CycleGAN to translate
colonoscopy images to polyp masks. In their work, the generator learns how to
segment polyps by trying to fool a discriminator.

Table 1: Real polyp segmentation datasets size and resolution. Dataset | #Images | Resolution
---|---|---
CVC-T [19] | 912 | 574 x 500
CVC-ClinicDB [1] | 612 | 384 x 288
CVC-ColonDB [17] | 380 | 574 x 500
ETIS-LaribPolypDB [15] | 196 | 1225 x 966
Kvasir [9] | 1000 | Variable

Synthetic images combined with generative networks have also been widely used
in the depth prediction task [11, 12]. This task helps doctors to verify that
all the surfaces in the colon have been analyzed. Synthetic data is essential
for this task because of the difficulties to obtain depth information in a
real colonoscopy.

Unlike previous works, our method is entirely unsupervised and does not
require any human annotations. We automatically generate the annotations by
defining the structure of the colon and polyps and transferring the location
of the polyps to a 2D mask. The key difference between our approach and other
state-of-the-art is that we combine 3D rendering and generative networks.
First, the 3D engine defines the structure of the image and generates the
annotations. Second, the adversarial network makes the images realistic.

Similar unsupervised methods have also been successfully applied in other
domains like crowd counting. For example, Wang et al. [20] render crowd images
from a video game and then use a CycleGAN to increase the realism.

##  3 Method

Our approach is composed of three steps: first, we procedurally generate colon
images and annotations using a 3D engine; second, we feed a CycleGAN with
images from real colonoscopies and our synthetic images; finally, we use the
realistic images created by CycleGAN to train an image segmentation model.

###  3.1 3D colon generation

Figure 3: The structure of the colon is composed by 7 segments to simulate the
curvature of the intestinal tract.

The 3D colon and polyps are procedurally generated using Blender, a 3D engine
that can be automated via scripting.

Our 3D colons structure is a cone composed by 2454 faces. Vertices are
randomly displaced following a normal distribution in order to simulate the
tissues in the colon. Additionally, the colon structure is modified by
displacing 7 segments as in Figure 3. For the textures we used a base color
[0.80, 0.13, 0.18] (RGB). For each sample we shift the color to other tones of
brown, orange and pink. One single polyp is used on every image, which is
placed inside the colon. It can be either in the colon’s walls or in the
middle. Polyps are distorted spheres with 16384 faces. Samples with polyps
occupying less than 20,000 pixels are removed.

Lighting is composed by a white ambient light, two white dynamic lights that
project glare into the walls, and three negative lights that project black
light at the end of the colon. We found that having a dark area at the end
helps CycleGAN to understand the structure of the colon. The 3D scene must be
similar to real colon images because otherwise, the CycleGAN will not
translate properly the images to the real-world domain. Figure 4 illustrates
the images and ground truth generated by the 3D engine.

Figure 4: Synthetic colons with corresponding annotations rendered using a 3D
engine.

###  3.2 CycleGAN

A standard CycleGAN composed by two generators and two discriminators is
trained using real images from colonoscopies and synthetic images generated
using the 3D engine as depicted in Figure 6. We train a CycleGAN for 200
epochs and then we infer real images in the “Generator Synth to Real” model,
producing realistic colon images.

Figure 5 displays synthetic images before and after the CycleGAN domain
adaptation. Note that the position of the polyps is not altered. Hence, the
ground truth information generated by the 3D engine is preserved.

Figure 5: Synthetic images (first row) and realistic images generated by our
CycleGAN (second row). Figure 6: Our CycleGAN architecture. We train two
generator models that try to fool two discriminator models by changing the
domain of the images.

###  3.3 Polyp segmentation

After creating a synthetic dataset that has been adapted to the real colon
textures, we train an image segmentation model. We used the HarDNeT-MSEG [7]
model architecture because of its real-time performance and high accuracy. We
use the same hyperparameter configuration as in the original paper.

##  4 Synth-Colon

We publicly release Synth-Colon, a synthetic dataset for polyp segmentation.
It is the first dataset generated using zero annotations from medical
professionals. The dataset is composed of 20 000 images with a resolution of
500$\times$500\. Synth-Colon additionally includes realistic colon images
generated with our CycleGAN and the Kvasir training set images. Synth-Colon
can also be used for the colon depth estimation task [12] because we provide
depth and 3D information for each image. Figure 1 shows some examples from the
dataset. In summary, Synth-Colon includes:

  * •

Synthetic images of the colon and one polyp.

  * •

Masks indicating the location of the polyp.

  * •

Realistic images of the colon and polyps. Generated using CycleGAN and the
Kvasir dataset.

  * •

Depth images of the colon and polyp.

  * •

3D meshes of the colon and polyp in OBJ format.

##  5 Experiments

###  5.1 Metrics

We use two common metrics for evaluation. The mean Dice score, given by:

$\mathrm{mDice}=\frac{2\times tp}{2\times tp+fp+fn},$ (1)

and the mean intersection over union (IoU):

$\mathrm{mIoU}=\frac{tp}{tp+fp+fn},$ (2)

where in both forumlae, $tp$ is the number of true positives, $fp$ the number
of false positives, and $fn$ the number of false negatives.

###  5.2 Evaluation on real polyp segmentation datasets

We evaluate our approach on five real polyp segmentation datasets. Table 2
shows the results obtained when training HarDNeT-MSEG [7] using our synthetic
data. Note that our method is not using any annotations. Results are
satisfactory considering the fact that labels have been generated
automatically. We found that training the CycleGAN with only the images from
the target dataset performs better than training the CycleGAN with all the
datasets combined, indicating a domain gap among the real-world datasets.

Table 2: Evaluation of our synthetic approach on real-world datasets. The
metrics used are mean Dice similarity index (mDice) and mean Intersection over
Union (mIoU).

| CVC-T | ColonDB | ClinicDB | ETIS | Kvasir
---|---|---|---|---|---
| mDice | mIoU | mDice | mIoU | mDice | mIoU | mDice | mIoU | mDice | mIoU
U-Net [13] | 0.710 | 0.627 | 0.512 | 0.444 | 0.823 | 0.755 | 0.398 | 0.335 | 0.818 | 0.746
SFA [5] | 0.467 | 0.329 | 0.469 | 0.347 | 0.700 | 0.607 | 0.297 | 0.217 | 0.723 | 0.611
PraNet [4] | 0.871 | 0.797 | 0.709 | 0.640 | 0.899 | 0.849 | 0.628 | 0.567 | 0.898 | 0.840
HarDNet-MSEG [7] | 0.887 | 0.821 | 0.731 | 0.660 | 0.932 | 0.882 | 0.677 | 0.613 | 0.912 | 0.857
Synth-Colon (ours) | 0.703 | 0.635 | 0.521 | 0.452 | 0.551 | 0.475 | 0.257 | 0.214 | 0.759 | 0.527

###  5.3 Study with limited real data

In this section we evaluate how our approach based on synthetic imagery and
domain adaptation compares with the fully supervised state-of-the-art HarDNeT-
MSEG network when there are fewer training examples available. We train the
CycleGAN used in the proposed approach, without ground truth segmentation
labels, on progressively larger sets of imagery, and compare this with the
supervised method trained on the same amount of labelled imagery. Table 3
shows the results of the experiment, which demonstrates that synthetic data is
extremely useful for domains where annotations are very scarce. While our
CycleGAN can produce realistic images with a small sample of only five real
images, supervised methods require many images and annotations to achieve good
performance. Table 3 shows that our unsupervised approach is useful when there
are less than 50 real images and annotations. Note that zero images here means
there is no domain adaptation via the CycleGAN.

Table 3: Evaluation of the proposed approach on the Kvasir dataset when few real images are available. The performance is measured using the mean Dice metric. | Synth-Colon (ours) | HarDNeT-MSEG [7]
---|---|---
0 images | 0.356 | -
5 images | 0.642 | 0.361
10 images | 0.681 | 0.512
25 images | 0.721 | 0.718
50 images | 0.735 | 0.781
900 (all) images | 0.759 | 0.912

##  6 Conclusions

We successfully trained a polyp segmentation model without annotations from
doctors. We used 3D rendering to generate the structure of the colon and
generative adversarial networks to make the images realistic, and demonstrated
that it can perform quite reasonably in several datasets, even outperforming
some fully supervised methods in some cases. We hope this study can help
aligning synthetic data and medical imaging in future. As future work, we will
explore how to include our synthetic annotations in the CycleGAN.

## References

  * [1] Bernal, J., Sánchez, F.J., Fernández-Esparrach, G., Gil, D., Rodríguez, C., Vilariño, F.: Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized Medical Imaging and Graphics 43, 99–111 (2015)
  * [2] Brandao, P., Mazomenos, E., Ciuti, G., Caliò, R., Bianchi, F., Menciassi, A., Dario, P., Koulaouzidis, A., Arezzo, A., Stoyanov, D.: Fully convolutional neural networks for polyp segmentation in colonoscopy. In: Medical Imaging 2017: Computer-Aided Diagnosis. vol. 10134, p. 101340F. International Society for Optics and Photonics (2017)
  * [3] Chao, P., Kao, C.Y., Ruan, Y.S., Huang, C.H., Lin, Y.L.: Hardnet: A low memory traffic network. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3552–3561 (2019)
  * [4] Fan, D.P., Ji, G.P., Zhou, T., Chen, G., Fu, H., Shen, J., Shao, L.: Pranet: Parallel reverse attention network for polyp segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 263–273. Springer (2020)
  * [5] Fang, Y., Chen, C., Yuan, Y., Tong, K.y.: Selective feature aggregation network with area-boundary constraints for polyp segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 302–310. Springer (2019)
  * [6] Gao, H., Ogawara, K.: Adaptive data generation and bidirectional mapping for polyp images. In: 2020 IEEE Applied Imagery Pattern Recognition Workshop (AIPR). pp. 1–6. IEEE (2020)
  * [7] Huang, C.H., Wu, H.Y., Lin, Y.L.: Hardnet-mseg: A simple encoder-decoder polyp segmentation neural network that achieves over 0.9 mean dice and 86 fps (2021)
  * [8] Hwang, S., Oh, J., Tavanapong, W., Wong, J., De Groen, P.C.: Polyp detection in colonoscopy video using elliptical shape feature. In: 2007 IEEE International Conference on Image Processing. vol. 2, pp. II–465. IEEE (2007)
  * [9] Jha, D., Smedsrud, P.H., Riegler, M.A., Halvorsen, P., de Lange, T., Johansen, D., Johansen, H.D.: Kvasir-seg: A segmented polyp dataset. In: International Conference on Multimedia Modeling. pp. 451–462. Springer (2020)
  * [10] LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. nature 521(7553), 436–444 (2015)
  * [11] Mahmood, F., Chen, R., Durr, N.J.: Unsupervised reverse domain adaptation for synthetic medical images via adversarial training. IEEE transactions on medical imaging 37(12), 2572–2581 (2018)
  * [12] Rau, A., Edwards, P.E., Ahmad, O.F., Riordan, P., Janatka, M., Lovat, L.B., Stoyanov, D.: Implicit domain adaptation with conditional generative adversarial networks for depth prediction in endoscopy. International journal of computer assisted radiology and surgery 14(7), 1167–1176 (2019)
  * [13] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp. 234–241. Springer (2015)
  * [14] Rott Shaham, T., Dekel, T., Michaeli, T.: Singan: Learning a generative model from a single natural image. In: Computer Vision (ICCV), IEEE International Conference on (2019)
  * [15] Silva, J., Histace, A., Romain, O., Dray, X., Granado, B.: Toward embedded detection of polyps in wce images for early diagnosis of colorectal cancer. International journal of computer assisted radiology and surgery 9(2), 283–293 (2014)
  * [16] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  * [17] Tajbakhsh, N., Gurudu, S.R., Liang, J.: Automated polyp detection in colonoscopy videos using shape and context information. IEEE transactions on medical imaging 35(2), 630–644 (2015)
  * [18] Thambawita, V., Salehi, P., Sheshkal, S.A., Hicks, S.A., Hammer, H.L., Parasa, S., de Lange, T., Halvorsen, P., Riegler, M.A.: Singan-seg: Synthetic training data generation for medical image segmentation. arXiv preprint arXiv:2107.00471 (2021)
  * [19] Vázquez, D., Bernal, J., Sánchez, F.J., Fernández-Esparrach, G., López, A.M., Romero, A., Drozdzal, M., Courville, A.: A benchmark for endoluminal scene segmentation of colonoscopy images. Journal of healthcare engineering 2017 (2017)
  * [20] Wang, Q., Gao, J., Lin, W., Yuan, Y.: Learning from synthetic data for crowd counting in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8198–8207 (2019)
  * [21] Wu, Z., Su, L., Huang, Q.: Cascaded partial decoder for fast and accurate salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3907–3916 (2019)
  * [22] Zhou, Z., Siddiquee, M.M.R., Tajbakhsh, N., Liang, J.: Unet++: A nested u-net architecture for medical image segmentation. In: Deep learning in medical image analysis and multimodal learning for clinical decision support, pp. 3–11. Springer (2018)
  * [23] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Computer Vision (ICCV), 2017 IEEE International Conference on (2017)


§ METHOD

§.§ Wind Data

Real wind forecasts are gathered from the  ECMWF's ERA5 global reanalysis dataset <cit.>.  As mentioned by Bellemare et al., simplex noise <cit.> can be used to augment the wind data, thus emulating forecasting errors <cit.>.  Specifically, wind forecasts between $1$st November $2022$ and $31$st January $2023$ are chosen as seasonal changes can have a big impact on the wind fields.  Furthermore, wind fields within the tropics, at longitude $-113^{\circ}$ latitude $1^{\circ}$, are chosen as they are shown to contain a range of diverse winds <cit.>.

The wind-field contains wind vectors over a grid of points in the parameter space $\mathcal{X} \times \mathcal{Y} \times \mathcal{P} \times  \mathcal{T}\times  \mathcal{V}$, where the longitude $\mathcal{X}$ and and latitudinal $\mathcal{Y}$ positions are sampled with a resolution of $0.4^{\circ}$ at various pressure points $\mathcal{P}$ ranging from $2000$ Pa to $17500$ Pa. Furthermore, forecasts are collected every $6$ hours for the three day episode and each wind-vector has two components in the longitude and latitude direction $(v_{wx}, v_{wy})$.

§.§ High-level Soft Actor-Critic

The Soft Actor-Critic algorithm is selected since the maximization of both the value function and the policy entropy is expected to assist in exploring the wind field, preventing the controller from getting trapped in local minima.

The MDP state space ($\mathcal{S}$) consists of $50$ wind and $27$ ambient features.  The wind features are collected at 25 equally-spaced points between the vertical pressure limits $[5000, 14000]$ Pa.  For each point, the magnitude of the wind speed $|v|$ and bearing error $\theta$ with respect to the target is calculated, as in <cit.>.  These form two individual vectors which then are concatenated $[|v_0|,...,|v_{24}|,\theta_0,...,\theta_{24}]\in\mathbb{R}^{50}$.

The ambient features consist of simulated onboard measurements.  These measurements include the altitude $h_t$, ascent-rate $\dot{h}_t$, wind velocity $|v_h|$ and bearing error at the current altitude $\theta_h$, the balloon envelope drag area $A$ and volume $V$, total system mass $m_T$, distance $|x|$ and heading to the target $[\sin(\theta_x), \cos(\theta_x)]$.  The past three altitudes $[h_{t-1}, h_{t-2}, h_{t-3}]$, ascent rates $[\dot{h}_{t-1}, \dot{h}_{t-2}, \dot{h}_{t-3}]$, and float actions $[a_{2, t-1}, a_{2, t-2}, a_{2, t-3}]$.  Then finally the sand mass $m_s$, and helium mols $n$ are also included.  Both the wind and ambient features are then concatenated together to form a feature set of 74 variables.

We utilise the distance-based reward function proposed by Bellemare et al.  <cit.>, shown in Eq <ref>.

\begin{equation}
\label{eq:reward_function}
\begin{cases}
1.0, & \text{if }|x| < 50 \\
c2^{-(|x|-\rho)/\tau)}, & \text{otherwise},
\end{cases}
\end{equation}

where the cliff constant $c=0.4$ ensures there is a distinct difference in the value of the state inside the target region compared to outside.

The MDP action space ($\mathcal{A}$) consists of three actions $a\in[a_0, a_1, a_2]$.  Representing the action space in this way decouples the ascent rate into desired altitude $a_0\in[14000, 21000]$ km and time-factor $a_1\in[1, 5]$, as a component of time-steps, to reach that altitude.  A larger time-factor $a_1$ represents a slower ascent rate, as the agent decides lower urgency to reach the desired altitude. Decoupling the ascent rate augments the secondary function of resource conservation into the learning process.  From this perspective, altitude $a_0$ relates to wind speeds propelling the balloon to the target direction.  And the time-factor $a_1$ relates to the amount of resources used to reach that altitude.

Another reason for decoupling the action space in this way is to incorporate transparency into the agent's decision making.  Compared to previous works on station-keeping, trajectories generated in the form of desired altitudes and time-factors are more transparent than low-level control inputs <cit.>.  Further work is needed to discern policy explanations from a generated trajectory in relation to the wind-field <cit.>.

The third action $a_2\in[-1, 1]$ is the binary option to float vertically and takes precedence over the other two actions.  To discretise the action, the controller chooses to float if the action is positive, $a_2>0$.  In the event that action $a_2$ is selected, actions $a_0$ and $a_1$ become unnecessary and are subsequently disregarded.

An episode runs for a total of 3 days, where the balloon's initial conditions consists of the relative longitude, latitude and altitude from the target, date, time, and wind noise which are sampled uniformly from a random seed.  Every six hours, wind forecasts are obtained from ECMWF's ERA5 dataset <cit.>. To determine the wind speed at a specific time and position, multidimensional interpolation is applied to gather data from the wind field.  An episode ends when the agent reaches a flight time of 3 days or runs out of resources.

Both the actor and critic networks are parameterised as neural networks, with two hidden layers of size $256$.  The actor network takes input $77$ which is the combination of the wind and ambient features and outputs a mean and covariance for each of the three actions, $\mu\in\mathbb{R}^{3}$, $\log\sigma\in\mathbb{R}^3$.  The critic network instead takes the concatenated vector of both the state and action and calculates the state-action value $Q(s_t, a_t)$.

(52, 76)(0,0)Actor $\theta$
(54, 64)(0,0)$256$
(8, 36)(0,0)s
(32, 58)(0,0)$77$

(75.5, 62)(0,0)$3$
(75.5, 34)(0,0)$3$

(100, 20)(0,0)$\log \sigma$
(127, 50)(0,0)$\log \pi$

(95, 50)(0,0)$\mu$
(127, 21)(0,0)$\tilde{a}$

(190, 85)(0,0)Critic $\phi$

(156, 25)(0,0)$3$
(156, 72)(0,0)$77$
(185, 62)(0,0)$80$
(207, 67)(0,0)$256$
(229, 45)(0,0)$1$
(245, 38)(0,0)Q
(225, 9)(0,0)x$4$

Soft Actor-Critic <cit.> network architecture.  The actor network is modelled as a Feed-forward parameterised Gaussian policy where the actions are represented as the hyperbolic tangent $\tanh$ applied to z values sampled from the mean $\mu$ and covariance given by the neural network.  Whereas the critic is modeled as a soft Q-function.

§.§ Altitude Controller

The altitude controller takes the desired altitude $a_0$, time-factor $a_1$, and float condition $a_2$ from the SAC controller, and either drops or vents helium depending on the calculated desired ascent rate.  Where the current altitude is denoted $h_t$ and the time between each action, stride time $\Delta t$, is set to 20 minutes.

\begin{equation}
\dot{h}_{\text{des}} = \frac{a_0 - h_t}{a_1 \Delta t}
\end{equation}

To prevent the agent from prioritizing receiving rewards by exploiting the environment <cit.>, a minimum limit for ballasting $m_{\text{s\_min}}$ and venting $n_{\text{min}}$ is set.  If ascent rates are chosen that calculate masses less than these values then no action is taken.  There is a loophole in this approach that could allow the agent to fly above or bellow the set altitude limits.  To prevent this behaviour, the controller will float above or bellow the altitude limits unless an action is chosen that propels the balloon between the altitude limits is chosen.  The full controller is shown in Eq <ref>.

\begin{equation}
\label{eq:fullcontroller}
\text{action} =
\begin{cases}\text{Float,} & \text{if } a_2 > 0 \\
\text{Do Nothing,} & \text{if } n_{\text{calc}} < n_{\text{min}} \text{ or } m_{\text{s\_calc}} < m_{\text{s\_min}}\\
\text{Ballast,} & \dot{h}_{\text{des}} > \dot{h}_t \\
\text{Vent,} & \dot{h}_{\text{des}} < \dot{h}_t \\
\end{cases}
\end{equation}

Where $n_{\text{calc}}$ and $m_{\text{s\_calc}}$ are the number of mols and sand mass calculated to vent or ballast, respectively, for the given action.  The equations to calculate these are described below.

§.§ Resources used to float, vent and ballast

To calculate the desired mass to vent or ballast, we assume the balloon has reached steady state and equate Eq <ref> to 0.  To vent, we solve Eq <ref> for the number of mols $n_{\text{calc}}$ of helium given the desired ascent rate $\dot{h}_{\text{des}}$ from the SAC controller.  This results in a polynomial function which we can then solve for the real roots.

\begin{align}
\begin{split}
\rho g \left(\frac{RT}{P} - M\right)n_{\text{calc}}
- \frac{1}{2}\rho\left|\dot{h}\right|\dot{h}&C_d\pi\left(\frac{3RT}{4\pi P}\right)^{\frac{2}{3}}n_{\text{calc}}^\frac{2}{3} \\
& -\left(m_p + m_s\right)g=0
\end{split}
\end{align}

Similarly, to ballast we can solve for the mass of sand $m_{\text{s\_calc}}$ by rearranging Eq <ref>.

\begin{equation}
m_{\text{s\_calc}} = \rho V - \frac{1}{2g} C_d A \left|\dot{h}\right|\dot{h} - m_p - m_h
\end{equation}

When floating, the balloon has no vertical velocity, and hence drag is not acting on the balloon envelope.  This simplifies Eq <ref> by setting the drag force $F_d$ equal to $0$.  Furthermore, we expand the mass term $m$ to include all the components.  This includes the mass of the payload (and envelope) $m_p$, sand $m_s$ and helium $nM_h$.  Where $n$ and $M_h$ are the number of moles and molar mass of helium respectively.

\begin{equation}
0 = \rho V g - (m_p + m_s + nM_h)g
\end{equation}

Subsequently, we select the appropriate method to maintain equilibrium between the buoyancy force and weight of the balloon.  If the buoyancy force exceeds the weight of the balloon, we opt to vent helium $n_{\text{calc}}$.  Conversely, if the weight of the balloon is more dominant, we opt to ballast $m_{\text{s\_calc}}$:

\begin{equation}
\begin{cases}
n_{\text{calc}} = \frac{m_s + m_p}{\frac{\rho R T}{P}-M_h}, & \text{if } \rho gV > mg \\
m_{\text{s\_calc}} = \rho V - m_p - m_h, & \text{otherwise}\\
\end{cases}
\end{equation}