# RootPainter3D: Interactive-machine-learning enables rapid and accurate
contouring for radiotherapy

Abraham George Smith  Department of Computer Science, University of Copenhagen
Department of Oncology, Rigshospitalet, University of Copenhagen<EMAIL_ADDRESS>Jens Petersen  Department of Computer Science, University of Copenhagen
Department of Oncology, Rigshospitalet, University of Copenhagen  Cynthia
Terrones-Campos  Department of Infectious Diseases, Rigshospitalet, University
of Copenhagen  Anne Kiil Berthelsen  Department of Oncology, Rigshospitalet,
University of Copenhagen  Department of Clinical Physiology, Rigshospitalet,
University of Copenhagen  Nora Jarrett Forbes  Department of Computer Science,
University of Copenhagen Department of Oncology, Rigshospitalet, University of
Copenhagen  Sune Darkner  Department of Computer Science, University of
Copenhagen Lena Specht  Department of Oncology, Rigshospitalet, University of
Copenhagen  Ivan Richter Vogelius  Department of Oncology, Rigshospitalet,
University of Copenhagen

###### Abstract

Organ-at-risk contouring is still a bottleneck in radiotherapy, with many deep
learning methods falling short of promised results when evaluated on clinical
data. We investigate the accuracy and time-savings resulting from the use of
an interactive-machine-learning method for an organ-at-risk contouring task.
We compare the method to the Eclipse contouring software and find strong
agreement with manual delineations, with a dice score of 0.95. The annotations
created using corrective-annotation also take less time to create as more
images are annotated, resulting in substantial time savings compared to manual
methods, with hearts that take 2 minutes and 2 seconds to delineate on
average, after 923 images have been delineated, compared to 7 minutes and 1
seconds when delineating manually. Our experiment demonstrates that
interactive-machine-learning with corrective-annotation provides a fast and
accessible way for non computer-scientists to train deep-learning models to
segment their own structures of interest as part of routine clinical
workflows.

Source code is available at this HTTPS URL.

## Introduction

Half of all cancer patients receive radiotherapy [13], which is associated
with a range of dose dependent side effects [4]. Effective mitigation of these
side effects requires accurate delineation of organs-at-risk, such as the
heart and oesophagus [17, 43]. Manual delineation is still widely used but
time-consuming in comparison to automated methods [55] and subject to large
inter-observer variation [37, 63]. A review of auto-segmentation methods for
radiotherapy is presented by [7], indicating deep-learning methods and
convolutional neural networks (CNN) in particular as representing the state-
of-the-art. A survey of deep-learning for radiotherapy is presented by [44],
including explanations of machine learning, deep learning, and CNNs.

Figure 1: Interactive machine learning with corrective annotation puts the
human-in-the-loop during the model training process.

Although CNNs exhibit impressive performance when the training and testing
data are drawn from the same distribution [46, 34, 55, 19, 57], variations
specific to on-site clinical data may result in decreased performance [21, 20,
51]. For example [18] found that organ deformations due to an abdominal
compression technique impaired the performance of an externally trained CNN
model.

Training models on-site is a potential solution but can be challenging as
training neural networks involves time-consuming trial and error [44] and
hiring the appropriate experts is associated with high costs. A lack of large
and high-quality publicly available datasets compounds the problem [59] as
annotating large enough datasets for training deep-learning models with purely
on-site data may be infeasible.

Corrective-annotation is an annotation sparsification strategy that results in
a sub-region of each image being labelled. [65] demonstrate, using a 3D heart
segmentation dataset, that annotation-sparsification allows deep-learning
models to be trained using just 20% of the labelled data whilst obtaining
similar performance. As opposed to [65] who rely on automatic methods to infer
which regions of an image are most useful for labelling, corrective-annotation
utilises human feedback (Figure 1) to identify problematic regions of the
image for the model [22, 53, 29, 39]. Methods such as corrective-annotation
which utilise a human-in-the-loop to improve machine learning implementations
are known as interactive machine learning (IML) [31].

In prior work, corrective-annotation IML methods have been used to train deep-
learning segmentation models for a diverse array of modalities, including 2D
plant photographs [53, 25, 14], laser ablation tomography [16], histopathology
[30] and 3D X-ray CT images of methane in sand [2].

To the best of our knowledge, no prior work has evaluated an IML approach
applied to medical X-ray CT data. Therefore we implement an IML system for
X-ray CT data and investigate its effectiveness for organ-at-risk
segmentation.

We hypothesise that (1) semi-automatic contouring via an IML corrective-
annotation method will provide similar accuracy to existing manual methods;
And (2) that it will offer continuous improvements in contouring time as more
images are annotated, eventually leading to significant time savings.

## Method

We use a corrective-annotation approach to contour a large dataset of hearts
by having a physician correct all generated contours during training (Figure
1). The process is semi-automatic, with the assisting model continuously
learning as more images are annotated.

### Dataset

We used X-ray CT scans from a cohort of patients that had been collated for a
study on the association between mean heart dose and radiation-induced
lymphopenia [56]. The CT scans were from patients who had started to receive
radiotherapy to the chest region between 2009 and 2016 at Rigshospitalet in
Denmark. The cohort was restricted to patients who were at least 18 years old,
had dosimetric data available, a solid malignant tumour (excluding lymphoma)
and a blood count both before and after receiving radiotherapy. This resulted
in 933 X-ray CT scans from 923 patients, of which 308 had breast cancer, 291
had esophageal cancer, 56 had small-cell-lung-carcinoma and 268 had non-small-
cell lung carcinoma. The 933 images had varying slice thicknesses. 209 images
had a slice thickness of 3mm, 721 had a slice thickess of 2mm , and 3 had a
slice thickness of 1mm. The number of voxels in each dimension varied. The
total number of slices (depth) varied from 96 to 489 with a mean of 189. The
width and height of the axial plane ranged from 512 to 658, with a mean of 537
voxels. No pre-processing was performed to normalize the images in any way.
All files were saved in the gzip compressed NIfTI file format with extentsion
.nii.gz.

### Software Implementation

We implemented RootPainter3D, which is based on RootPainter, an open-source
corrective-annotation software application that utilises a client-server
architecture and makes the necessary operations to train and use a deep-
learning model for image segmentation available via a cross-platform graphical
user interface [53].

The original RootPainter uses a variant of U-Net [50] modified to utilise
group norm [61] and residual connections [26]. Compared to the 2D version we
modify the software in several ways. The 2D convolutional layers were
converted to 3D, resulting in a variant of the 3D U-Net architecture which is
known to perform well for organ-at-risk segmentation [19] and sparse data [9].
The interface was modified to allow contrast settings to be adjusted,
navigation and annotation of 3D images, with viewing enabled in both sagittal
and axial views simultaneously. The data augmentation was removed. Although it
is claimed data augmentation is critical to achieving favourable
generalisation performance [52], results indicate the advantages of data-
augmentation can be inconsistent and dataset specific [36].

RootPainter3D allows a user to inspect model predictions on a dataset that
they work through sequentially - one image at a time. For each image, they
initially define a bounding box in order to obtain predictions for a region
containing the organ of interest. The user is able to assign corrections to
the model predictions given for the defined region. When the user clicks ’save
and next’ in the interface the current annotation is saved to disk in a folder
of annotations which is shared with a remote server with a more powerful
graphics processing unit (GPU). The server component of the software
continuously trains a 3D U-Net on the available annotations using stochastic
gradient descent. For RootPainter3D, batches of 4 annotations, with their
associated image regions are sampled from the annotation folder without
replacement. The regions of the images used for training are only those where
user-annotations are assigned. Supervised training is used but the annotations
are sparse. Only the defined regions (what the user has specified as either
foreground or background) are used for computing the loss which is used for
updating the model weights.

(a) Screenshot of RootPainter3D software showing segmentation in blue and
annotation in red for corrections to false negative regions and green for
corrections to false positive regions. The area in light red is outside of the
bounding box and is not part of the predicted or corrected region. The dashed
red line shows the position of the axial slice in the sagittal view. This
dashed line was added after the experiment in this study was completed based
on user-feedback.

(b) Screenshot of RootPainter3D software showing the outline view with
segmentation and annotation hidden.

Figure 2: Screenshots of the RootPainter3D client software. Showing different
views available for heart contouring. The axial view is shown on the right and
sagittal view on the left. The user is able to change what data is shown via
keyboard shortcuts or with the checkboxes shown in each viewer.

The contrast settings (see Contouring procedure for more details). can be
changed to a preset option in the view menu, such as mediastinal, which was
used for our experiments.

### Contouring procedure

To enable comparison to a manual method widely used in the clinic, 20 hearts
were also delineated using the Eclipse treatment planning software from Varian
Medical Systems, Inc.

From the 933 heart CT scans, the annotator delineated the first 10 in Eclipse
and then all 933 in RootPainter3D and then the last 10 in Eclipse. This was
done to allow the dice score to be computed between the contours done in
RootPainter3D and Eclipse, and also to compare delineation time between
RootPainter3D and Eclipse at the start and end of the RootPainter3D model
training process.

Contouring in both Eclipse and RootPainter3D was done using the mediastinal
hounsfield unit (HU) window which ranges from -125 HU to 250 HU. For both
software applications, annotation can be performed in both the sagittal and
axial views but for our experiment all annotation was assigned in the axial
view. In RootPainter3D the user is able to view the model’s initial
segmentation in blue (Figure 2(a)). They can annotate foreground (heart)
regions in red and background (not heart) regions in green (Figure 2(a)). A 3D
segmentation consists of a model prediction for each voxel in an image. If
annotating correctively, the red foreground annotation should correspond to
model predictions that were false negatives, and the green background regions
should be targeted towards false positive model predictions. Taking
corrections into account, the corrected contour can be viewed using the
outline view (Figure 2(b)).

When contouring in Eclipse, every third slice was contoured and then
interpolation was used to join the slices, as this is a standard clinical
procedure used to save time compared to contouring all slices. The same
trained physician delineated all hearts in both RootPainter3D and Eclipse
using the Danish guidelines for whole heart contouring [47].

In order to ensure the physician’s familiarity with both software applications
prior to the experiment, 20 hearts that were not used as part of the
experiment were contoured in both applications.

### Training

The training process for neural networks is a procedure where the network
weights and biases are iteratively updated by a gradient descent algorithm,
which compares the network outputs with the desired output [40].

When a user completes a delineation in the RootPainter3D client software, the
annotation is saved to disk and training is started automatically by sending
an instruction to the server.

The training routine, once started, first creates a model using Kaiming
initialisation to assign random parameters [27], and then trains that model by
continuously iterating over all saved annotations.

The training procedure treats newly saved images similarly to to previously
saved images as all images are equally likely to be used in the next batch of
the training procedure. It loads the annotations from disk and uses these,
combined with the image data, to train a model to predict the annotated
regions correctly. For the RootPainter3D trainer, regions that are not
annotated are set to 0 in both the network predictions and annotation. This
means only the regions of the image explicitly annotated as either foreground
or background by the annotator are used to compute the loss. The loss is then
used to update the network weights. The loss function used in RootPainter3D is
a combination of cross-entropy and dice loss, taken from [54].

A GPU is required for achieving optimal performance when training large
parameter models such as deep neural networks. As the entire dataset cannot
fit in the memory of the GPU, stochastic gradient descent is used to make
network updates based on a subset (known as a batch) of the full available
dataset. In our case the network is trained with a batch size of 4. We used
two NVIDIA Titan RTX GPUs, with 24GB of memory each, using a data-parallel
[11] approach. Due to GPU memory constraints, instead of using the full
images, a sub-region (patch) is sampled from a random location that contains
annotation, from each of the randomly sampled images in the training batch.
The patch dimensions are 228, 228 and 52 voxels, corresponding to width,
height and depth respectively.

For the RootPainter3D trainer application, once training has started, the
process is continuous, even when the client application is closed. The trainer
application keeps track of a counter named epochs_without_progress. This
counter is used to measure the length of time that the network has been
trained without obtaining a new high score on the validation data.

### Validation

In a machine learning context, the purpose of validation is to estimate model
performance on unseen data. Networks with large capacity can overfit arbitrary
datasets [64]. Fortunately, they have a tendency to learn smooth functions and
overfitting can be conveniently mitigated with early stopping [8]. Early
stopping involves checking model performance on a portion of the data not used
for training. This allows generalisation performance to be estimated and a
snapshot of the model weights to be taken before overfitting has substantially
degraded performance.

Similarly to 2D RootPainter [53], a portion of the annotated images are
assigned to the validation set instead of the training set. The first image is
assigned to the training dataset, then the second image is assigned to the
validation dataset. From then on images are only assigned to the validation
dataset if the training dataset is at least 5 times as large as the validation
dataset.

The validation dataset is used for model selection, in a way equivalent to
early stopping. An epoch typically refers to a full iteration over all
available training data. In this context we define epoch length in terms of
the number of examples sampled from the training data and have this
automatically adjusted in relation to the validation set size, which increases
as more images are annotated. If no validation images have been saved then the
epoch length is 128\. Otherwise the length is $max(64,2v)$ where $v$ is the
number of patches containing annotation in the validation set. Setting the
length of the training period based on the validation set size ensures
validation time does not overwhelm the training procedure, despite continuous
addition of new cases.

At the end of each epoch the predictions for the model being trained are
computed on the validation set and a dice score is computed using the
available validation set annotations. If the model-in-training’s dice score is
higher than the most recently saved model so far then it will be saved to disk
and used for generating the segmentation presented to the user in the client
user-interface.

epochs_without_progress will get set to 0 in two cases. Firstly, when new data
is added to the training or validation datasets. Secondly, when a new model is
found which obtains a new high score on the validation set.

Network training procedures are stochastic and dependent on initial weights
[3]. Therefore as the first initial weights are unlikely to be optimal,
repeating the training procedure with different initial weights is a simple
way to obtain better results. For this reason we introduced a restart
procedure. The restart procedure was not functioning for the first 478 hearts,
during which the network would stop after 60 epochs without progress.

The restarting behaviour, used for the latter portion of the experiment to
further boost network accuracy, would start training the network from scratch
after 60 epochs_without_progress were reached. If the newly trained model beat
the best model on the validation set so far then it would be saved (see
Validation for more details). That means if the system is left unattended for
long enough, it will likely find a new best model, and keep trying
relentlessly until it does.

### Dice evaluation

In order to evaluate the accuracy of the RootPainter3D contours before and
after user correction, we compared the dice of both the initial predicted
contour and the version after corrections with contours created manually in
Eclipse for the same 20 images.

For each model generated heart contour, we also compute the dice score between
the predicted heart contour and corrected version for 933 images. This
approach is similar to previous studies that use manually corrected model
predictions as ground-truth contours for evaluation [19, 24].

### Interaction logging and annotation duration

To evaluate the extent to which the trained model assisted in reducing
annotation time, we asked the annotator to log the specific time periods when
they started and stopped annotating. We also automatically logged interaction
events with the user-interface including when the user saved an image, opened
a new image, moved to a different slice in the axial or sagittal views,
changed zoom settings and mouse down and mouse release events.

To compute the amount of time the annotator spent on each image, we filter to
interaction events inside their manually logged annotation period. For each
file, we start accumulating interaction time from when the user opens that
file until they open the next file. We also exclude periods of inactivity from
our duration computation, which we define as no interaction events for at
least 20 seconds.

### Impact on radiation dose

We also compare the computed mean heart dose (Gy) when using the initial
segmentation compared to the corrected contour.

To compare dose absorption we first obtain a dose matrix for each scan from
the clinical dose plan. The dose matrix provides a cumulative level of
radiation dose absorbed by each voxel in the planning CT scan. We compute the
mean heart dose using the dose matrix combined with both the initial
segmentation and corrected structure independently. We then take the absolute
difference between the two mean heart dose measurements to allow us to plot
dose deviation over time as more hearts are annotated.

### Results

#### Duration

In order to evaluate the annotation speed of RootPainter3D, we report
delineation duration as a function of the number of images annotated (Figure
3). The first 10 and last 10 hearts are shown separately to highlight both the
differences with Eclipse and RootPainter3D both before and after a period of
extensive interactive annotation (Figure 4). RootPainter3D is initially slower
and then becomes significantly faster after it has learned from more user
corrections (Figure 4).

After around 110 images, delineation duration in RootPainter3D is less than
the mean delineation of the 20 hearts contoured in Eclipse (Figure 3). With
the exception of temporary fluctuations, the RootPainter3D per-heart
delineation duration continues to decrease as more hearts are contoured
(Figure 3) becoming substantially faster than the comparison manual method
with the last 10 hearts taking an average of 2 minutes and 2 seconds to
delineate compared to 7 minutes and 1 seconds when using Eclipse (Figure 4).
The RootPainter3D delineations also maintain strong agreement with manual
delineations of the same images (Table 1).

Figure 3: Delineation duration for each of the 933 images shown in order of
delineation with the mean delineation duration of the 20 images contoured in
Eclipse. RootPainter3D per-image delineation duration reduces over time.
Figure 4: RootPainter3D and Eclipse delineation duration for the first 10 and
last 10 images out of the 933. RootPainter3D is initially slower than Eclipse
but then becomes significantly faster. The last 10 hearts take an average of 2
minutes and 2 seconds to delineate in RootPainter3D compared to 7 minutes and
1 seconds when using Eclipse. Delineation in RootPainter3D is over three times
faster than delineation in Eclipse.

#### Accuracy

The dice score between the initial predicted heart and the corrected version
are shown with running mean and standard deviation which were computed using a
running average from 60 images (Figure 5). The dice scores of all hearts,
excluding the first (which had a dice score of 0.2) are shown in figure 5.
From the 300th heart onward, the vast majority of hearts have a dice score
above 0.98 (Figure 7) with a few outliers having dice scores between 0.9 and
0.95 and just two extreme outliers having dice scores below 0.7 (Figure 5).

Axial slices from the outliers a and b labelled in figure 5 are illustrated in
figure 6(a) and figure 6(b) respectively. These two hearts were the only ones
out of the last 600 annotated which had a dice score lower than 0.9. In both
cases the model appeared to be confused by a large tumour adjacent to the
heart.

To show the model progression more clearly, in figure 7 the y-axis minimum is
raised to 0.9, which includes most of the values. Although there are
fluctuations, the mean dice score shows a trend of continuous improvement as
more images are correctively delineated (Figure 7).

We found strong agreement between the hearts contoured in RootPainter3D and
Eclipse (Table 1). All of the last 10 hearts in RootPainter3D had higher
agreement with the manual Eclipse delineations after the corrections were
assigned.

Figure 8(a) shows the error measured in mean dose for automatically predicted
heart structure drops as more images are correctively delineated. As shown in
8(b) only 4 of the model predicted hearts in the last 300 result in an error
in dose above 1 Gray.

Table 1: Dice scores between hearts delineated using RootPainter3D and Eclipse, showing strong agreement between methods. However, the RootPainter3D predicted and corrected contours are more similar to each other than either one is to the Eclipse contours. pred refers to RootPainter3D predicted contours and cor refers to the RootPainter3D corrected contours. | Eclipse vs pred | Eclipse vs cor | pred vs cor
---|---|---|---
mean | 0.945 | 0.952 | 0.991
std | 0.008 | 0.007 | 0.005
Figure 5: The dice score for 933 images shown in delineation order. After the
initial training period, the vast majority of dice scores are above 0.9, with
only a few outliers dropping below. Outliers are labelled a and b.

(a) Outlier a: Dice score 0.67

(b) Outlier b: Dice score 0.68.

Figure 6: Outliers a and b. For each heart two axial slices are shown with
contours overlaid. On the left the model prediction is shown in green with the
user corrected heart in yellow. On the right only the corrected heart is
shown. For both the outliers with low dice, the tumour was located adjacent to
the heart. Figure 7: Dice score computed using the initial predicted image and
the corrected version. Y-axis restricted to 0.9 to 1.0 to show improvement in
mean over time. The mean dice score has fluctuations, but shows a general
trend of improvement throughout the experiment.

(a) Running mean using a Gaussian kernel. The mean difference drops from 0.3
to 0.12 Gray as more images are correctively annotated.

(b) Percentage of hearts with absolute differences from 0.25 to 1 Gray and
over 1 Gray for images 1-311, 312-622 and 623 to 923.

Figure 8: Absolute dose difference between predicted and corrected contours.
The dose differences decrease as more images are annotated correctively.

### Discussion

The strong agreement with the manual contours shown in table 1 supports our
first hypothesis that RootPainter3D will result in accurate contours. Prior
heart auto-contouring studies have observed a mean dice score of 0.925 between
model predictions and manual delineation [19]. [62] obtained a dice score of
0.931 when measuring the disagreement between multiple annotators, indicating
the difference between RootPainter3D delineations and our manually created
delineations may be less than the difference between two annotators, even
before corrections are assigned.

The decrease in per-image contouring duration as more hearts are annotated
(figure 3) and significantly faster contouring for the last 10 hearts (Figure
4 support our second hypothesis that RootPainter3D will offer substantial time
savings.

In Eclipse, only one in every three slices was manually delineated, with the
others created using interpolation. Even though in RootPainter3D all slices
were manually corrected, the contouring time was still more than three times
faster than Eclipse (Figure 4). This discrepancy between the number of slices
delineated vs interpolated also explains why RootPainter3D was slower than
eclipse during the initial period of annotation (Figure 4).

The dice scores between the RootPainter3D corrected contours and Eclipse
contours are lower than the dice scores between the RootPainter3D predicted
contours and corrected contours (Table 1).

As the predicted and corrected contours are not done independently, it is
expected that they will have higher agreement than two independently created
contours. For quantifying errors in an existing delineation, there are some
cases where this may be a strength. When delineating from scratch in noisy
images and areas with little contrast, a drop in dice may be caused by
differences in delineation along ambiguous border regions. With corrective-
annotation, the annotator focuses on clear errors and the measured dice is
therefore an indication of how often the system makes such clear errors and
these may be more relevant to quantify. For creating datasets for machine
learning, corrective-annotation could therefore reduce label noise consisting
of natural-perturbations. Although deep-learning is relatively robust to label
noise [49], perturbations along boundaries are particularly problematic for
U-Net model training [28].

In prior studies, correcting contours, as opposed to contouring from scratch
has been shown to increase consistency and reduce inter-observer variation
[12].

The two outliers with low dice score were in patients with anatomical
abnormalities that will also likely be correlated to high dose to the heart
(Figure 6). This is a potential limitation of using fully-automatic contouring
for dose-response modelling and shows the importance of carefully reviewing
the results of such models in clinical applications.

A typical limitation of auto-contouring studies is the small number of scans
used for evaluation, with less than 100 being typical [51]. By evaluating our
contouring software on over 900 hearts we were able to identify outliers that
would have otherwise been missed (Figure 6). It is clear from figure 5 that
smaller datasets may exclude such outliers by chance, resulting in over-
optimistic characterisations of performance on unseen data.

Despite editing auto-contours being identified as a barrier to the adoption of
automatic methods [62], our results, in alignment with previous studies [42,
58, 38], demonstrate that correcting auto-contours saves time compared to a
standard clinical workflow. In prior work, time-savings have been
demonstrated, even when the predicted structures are of particularly low
quality, with a dice score as low as 0.46 [23].

The benefits of corrective-annotation to routine clinical contouring are clear
as correcting inaccurate contours is already essential to optimise treatment
planning [17, 43]. In addition to being usable by non-experts [15] and
providing performance improvements in comparison to fully-automated systems
[32], IML can provide trust and quality control benefits [45] which are of
particular interest in a radiotherapy context.

The IML process (Figure 1) provides feedback to the annotator on how much data
is necessary to train a model to a given level of accuracy and due to their
extended exposure to model behaviour, provides physicians with control of
model training and insight into the strengths and limitations of an AI system,
attributes needed to ensure clinical adoption [6].

[48] characterise IML systems as being used for task-completion or model-
building. As opposed to [53], who evaluated IML with corrective-annotation for
model-building, in this study we also demonstrate the potential for assisting
in task-completion.

We used mean heart dose computation accuracy to measure model quality.
Although it brings our results closer to a value familiar to the working
radiation oncologist, the utility of mean heart dose for understanding cardiac
toxicity has been called into question by recent studies [33]. With faster
contouring capabilities, IML systems such as RootPainter3D provide
capabilities to contour more structures in less time, making it possible to
delineate organ sub-structures in a larger cohort of patients.

We motivated the omission of data-augmentation based on results that indicate
the benefits may be dataset specific [36]. More recently, the same authors
conducted a more thorough analysis, providing stronger indications that a
sensibly designed data-augmentation procedure would provide benefits across
datasets and organs [35]. Thus we expect the addition of data-augmentation
would further improve our results.

Similarly to [5], we observed that many time consuming contour adjustments
were minor changes along the boundary which are in many cases unlikely to have
a large impact on dose. We agree with [5] that there is a need for methods
that guide delineators in determining which corrections are meaningful for
dose-planning.

The combination of electronic health records, stored dose matrices and CT
scans from record and verify systems has great potential to increase our
understanding of radiation dose effects for normal tissues [59]. [1]
investigated the relationship between radiation exposure and lymphopenia for a
cohort of 901 patients by registering all scans to a single reference patient.
Our proposed method would compliment such studies by enabling efficient
segmentation of structures of interest for each individual patient CT scan,
potentially mitigating errors that may result from the registration process.

Our software is made available open source and we emphasise that with a
restarted training process, the same application can learn to segment other
organs of interest in a research setting. As we started training from random
weights and use a fairly generic U-Net architecture, we expected the
performance to be similar.

In conclusion, our results demonstrate the benefits of continual-learning with
correction-annotation, by showing how contouring time can be continually
reduced whilst maintaining accuracy. It may be some time before such systems
can be made widely available to clinics, as medical devices utilising
continual-learning are yet to be approved by the FDA [41, 10].

We also only used a single physician for our experiments and the contours were
collected for research purposes, rather than as part of a routine procedure.
The risks when working with new data and multiple clinicians will need to be
carefully considered before such systems can be adopted in the clinic, and new
tests may need to be designed to ensure robustness and quality [60].

### Acknowledgements

We thank Thomas Carlslund and Kurt Nielsen for IT infrastructure support and
Agata Wlaszczyk for proofreading. We would also like to thank Katrin Elisabet
Håkansson, Mirjana Josipovic and Emmanouil Terzidi for feedback on early
versions of the software and Deborah Anne Schut for feedback on the heart
contouring procedure. We also thank Mikkel Skaarup for feedback on
experimental design and gratefully acknowledge our financial support from
Varian Medical Systems and the Danish Cancer Society (grant no R231-A13976).

## References

  * [1] A. Abravan, C. Faivre-Finn, J. Kennedy, A. McWilliam, and M. van Herk.  Radiotherapy-Related Lymphopenia Affects Overall Survival in Patients With Lung Cancer.  Journal of Thoracic Oncology: Official Publication of the International Association for the Study of Lung Cancer, 15(10):1624–1635, Oct. 2020.
  * [2] F. J. Alvarez-Borges, O. N. F. King, B. Madhusudhan, T. Connolley, M. Basham, and S. I. Ahmed.  U-Net Segmentation Methods for Variable-Contrast XCT Images of Methane-Bearing Sand.  Preprint, Soil Science, Apr. 2021.
  * [3] A. Atiya and C. Ji.  How initial conditions affect generalization performance in large networks.  IEEE Transactions on Neural Networks, 8(2):448–451, Mar. 1997.
  * [4] L. Barazzuol, R. P. Coppes, and P. van Luijk.  Prevention and treatment of radiotherapy-induced side effects.  Molecular Oncology, 14(7):1538–1554, 2020.
  * [5] C. L. Brouwer, D. Boukerroui, J. Oliveira, P. Looney, R. J. H. M. Steenbakkers, J. A. Langendijk, S. Both, and M. J. Gooding.  Assessment of manual adjustment performed in clinical practice following deep learning contouring for head and neck organs at risk in radiotherapy.  Physics and Imaging in Radiation Oncology, 16:54–60, Oct. 2020\.
  * [6] C. J. Cai, S. Winter, D. Steiner, L. Wilcox, and M. Terry.  ”Hello AI”: Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making.  Proceedings of the ACM on Human-Computer Interaction, 3(CSCW):104:1–104:24, Nov. 2019.
  * [7] C. E. Cardenas, J. Yang, B. M. Anderson, L. E. Court, and K. B. Brock.  Advances in Auto-Segmentation.  Seminars in Radiation Oncology, 29(3):185–197, July 2019.
  * [8] R. Caruana, S. Lawrence, and C. L. Giles.  Overfitting in Neural Nets: Backpropagation, Conjugate Gradient, and Early Stopping.  page 7.
  * [9] Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger.  3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation.  In S. Ourselin, L. Joskowicz, M. R. Sabuncu, G. Unal, and W. Wells, editors, Medical Image Computing and Computer-Assisted Intervention – MICCAI 2016, Lecture Notes in Computer Science, pages 424–432, Cham, 2016. Springer International Publishing.
  * [10] S. Cruz Rivera, X. Liu, A.-W. Chan, A. K. Denniston, M. J. Calvert, SPIRIT-AI and CONSORT-AI Working Group, SPIRIT-AI and CONSORT-AI Steering Group, and SPIRIT-AI and CONSORT-AI Consensus Group.  Guidelines for clinical trial protocols for interventions involving artificial intelligence: The SPIRIT-AI extension.  Nature Medicine, 26(9):1351–1363, Sept. 2020.
  * [11] J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, Q. V. Le, and A. Y. Ng.  Large Scale Distributed Deep Networks.  page 9.
  * [12] M. A. Deeley, A. Chen, R. D. Datteri, J. Noble, A. Cmelak, E. Donnelly, A. Malcolm, L. Moretti, J. Jaboin, K. Niermann, E. S. Yang, D. S. Yu, and B. M. Dawant.  Segmentation editing improves efficiency while reducing inter-expert variation and maintaining accuracy for normal brain tissues in the presence of space-occupying lesions.  Physics in Medicine and Biology, 58(12):4071–4097, June 2013.
  * [13] G. Delaney, S. Jacob, C. Featherstone, and M. Barton.  The role of radiotherapy in cancer treatment.  Cancer, 104(6):1129–1137, 2005.
  * [14] R. F. Denison.  Legume-imposed selection for more-efficient symbiotic rhizobia.  Proceedings of the National Academy of Sciences, 118(22):e2107033118, June 2021.
  * [15] J. J. Dudley and P. O. Kristensson.  A Review of User Interface Design for Interactive Machine Learning.  ACM Transactions on Interactive Intelligent Systems, 8(2):8:1–8:37, June 2018.
  * [16] B. Elias, A. Lanba, J. C. Netto, F. Fritschi, and A. Smith.  Interactive Machine Learning Methods for the Quantification of Vascular Features in Soybean Images Obtained via Laser Ablation Tomography (LATscan).  Thinking Matters Symposium, Apr. 2021.
  * [17] G. A. Ezzell, J. M. Galvin, D. Low, J. R. Palta, I. Rosen, M. B. Sharpe, P. Xia, Y. Xiao, L. Xing, C. X. Yu, IMRT subcommitte, and AAPM Radiation Therapy committee.  Guidance document on delivery, treatment planning, and clinical implementation of IMRT: Report of the IMRT Subcommittee of the AAPM Radiation Therapy Committee.  Medical Physics, 30(8):2089–2115, Aug. 2003.
  * [18] X. Feng, M. E. Bernard, T. Hunter, and Q. Chen.  Improving accuracy and robustness of deep convolutional neural network based thoracic OAR segmentation.  Physics in Medicine & Biology, 65(7):07NT01, Mar. 2020.
  * [19] X. Feng, K. Qing, N. J. Tustison, C. H. Meyer, and Q. Chen.  Deep convolutional neural network for segmentation of thoracic organs-at-risk using cropped 3D images.  Medical Physics, 46(5):2169–2180, 2019.
  * [20] M. Ghafoorian, A. Mehrtash, T. Kapur, N. Karssemeijer, E. Marchiori, M. Pesteie, C. R. G. Guttmann, F.-E. de Leeuw, C. M. Tempany, B. van Ginneken, A. Fedorov, P. Abolmaesumi, B. Platel, and W. M. Wells III.  Transfer Learning for Domain Adaptation in MRI: Application in Brain Lesion Segmentation.  arXiv:1702.07841 [cs], 10435:516–524, 2017.
  * [21] E. Gibson, Y. Hu, N. Ghavami, H. U. Ahmed, C. Moore, M. Emberton, H. J. Huisman, and D. C. Barratt.  Inter-site Variability in Prostate Segmentation Accuracy Using Deep Learning.  In A. F. Frangi, J. A. Schnabel, C. Davatzikos, C. Alberola-López, and G. Fichtinger, editors, Medical Image Computing and Computer Assisted Intervention – MICCAI 2018, Lecture Notes in Computer Science, pages 506–514, Cham, 2018\. Springer International Publishing.
  * [22] F. Gonda, V. Kaynig, T. R. Jones, D. Haehn, J. W. Lichtman, T. Parag, and H. Pfister.  ICON: An interactive approach to train deep neural networks for segmentation of neuronal structures.  In 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017), pages 327–331, Apr. 2017.
  * [23] M. J. Gooding, A. J. Smith, M. Tariq, P. Aljabar, D. Peressutti, J. van der Stoep, B. Reymen, D. Emans, D. Hattu, J. van Loon, M. de Rooy, R. Wanders, S. Peeters, T. Lustberg, J. van Soest, A. Dekker, and W. van Elmpt.  Comparative evaluation of autocontouring in clinical practice: A practical method using the Turing test.  Medical Physics, 45(11):5105–5115, 2018.
  * [24] C. Gros, B. De Leener, A. Badji, J. Maranzano, D. Eden, S. M. Dupont, J. Talbott, R. Zhuoquiong, Y. Liu, T. Granberg, R. Ouellette, Y. Tachibana, M. Hori, K. Kamiya, L. Chougar, L. Stawiarz, J. Hillert, E. Bannier, A. Kerbrat, G. Edan, P. Labauge, V. Callot, J. Pelletier, B. Audoin, H. Rasoanandrianina, J.-C. Brisset, P. Valsasina, M. A. Rocca, M. Filippi, R. Bakshi, S. Tauhid, F. Prados, M. Yiannakas, H. Kearney, O. Ciccarelli, S. Smith, C. A. Treaba, C. Mainero, J. Lefeuvre, D. S. Reich, G. Nair, V. Auclair, D. G. McLaren, A. R. Martin, M. G. Fehlings, S. Vahdat, A. Khatibi, J. Doyon, T. Shepherd, E. Charlson, S. Narayanan, and J. Cohen-Adad.  Automatic segmentation of the spinal cord and intramedullary multiple sclerosis lesions with convolutional neural networks.  NeuroImage, 184:901–915, Jan. 2019.
  * [25] E. Han, A. G. Smith, R. Kemper, R. White, J. Kirkegaard, K. Thorup-Kristensen, and M. Athmann.  Digging roots is easier with AI.  bioRxiv, page 2020.12.01.397034, Dec. 2020.
  * [26] K. He, X. Zhang, S. Ren, and J. Sun.  Deep Residual Learning for Image Recognition.  arXiv:1512.03385 [cs], Dec. 2015.
  * [27] K. He, X. Zhang, S. Ren, and J. Sun.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification.  arXiv:1502.01852 [cs], Feb. 2015.
  * [28] N. Heller, J. Dean, and N. Papanikolopoulos.  Imperfect Segmentation Labels: How Much Do They Matter?  arXiv:1806.04618 [cs], June 2018.
  * [29] D. J. Ho, N. P. Agaram, P. J. Schüffler, C. M. Vanderbilt, M.-H. Jean, M. R. Hameed, and T. J. Fuchs.  Deep Interactive Learning: An Efficient Labeling Approach for Deep Learning-Based Osteosarcoma Treatment Response Assessment.  In A. L. Martel, P. Abolmaesumi, D. Stoyanov, D. Mateus, M. A. Zuluaga, S. K. Zhou, D. Racoceanu, and L. Joskowicz, editors, Medical Image Computing and Computer Assisted Intervention – MICCAI 2020, Lecture Notes in Computer Science, pages 540–549, Cham, 2020. Springer International Publishing.
  * [30] D. J. Ho, N. P. Agaram, P. J. Schüffler, C. M. Vanderbilt, M.-H. Jean, M. R. Hameed, and T. J. Fuchs.  Deep Interactive Learning: An Efficient Labeling Approach for Deep Learning-Based Osteosarcoma Treatment Response Assessment.  In A. L. Martel, P. Abolmaesumi, D. Stoyanov, D. Mateus, M. A. Zuluaga, S. K. Zhou, D. Racoceanu, and L. Joskowicz, editors, Medical Image Computing and Computer Assisted Intervention – MICCAI 2020, Lecture Notes in Computer Science, pages 540–549, Cham, 2020. Springer International Publishing.
  * [31] A. Holzinger.  Interactive machine learning for health informatics: When do we need the human-in-the-loop?  Brain Informatics, 3(2):119–131, June 2016.
  * [32] A. Holzinger, M. Plass, M. Kickmeier-Rust, K. Holzinger, G. C. Crişan, C.-M. Pintea, and V. Palade.  Interactive machine learning: Experimental evidence for the human in the algorithmic loop.  Applied Intelligence, 49(7):2401–2414, July 2019.
  * [33] B. S. Hoppe, J. E. Bates, N. P. Mendenhall, C. G. Morris, D. Louis, M. W. Ho, R. T. Hoppe, M. Shaikh, Z. Li, and S. Flampouri.  The Meaningless Meaning of Mean Heart Dose in Mediastinal Lymphoma in the Modern Radiation Therapy Era.  Practical Radiation Oncology, 10(3):e147–e154, May 2020.
  * [34] B. Ibragimov and L. Xing.  Segmentation of organs-at-risks in head and neck CT images using convolutional neural networks.  Medical Physics, 44(2):547–557, 2017.
  * [35] F. Isensee, P. F. Jaeger, S. A. A. Kohl, J. Petersen, and K. H. Maier-Hein.  nnU-Net: A self-configuring method for deep learning-based biomedical image segmentation.  Nature Methods, 18(2):203–211, Feb. 2021.
  * [36] F. Isensee, J. Petersen, S. A. A. Kohl, P. F. Jager, and K. H. Maier-Hein.  nnU-Net: Breaking the Spell on Successful Medical Image Segmentation.  page 9.
  * [37] L. Joskowicz, D. Cohen, N. Caplan, and J. Sosna.  Inter-observer variability of manual contour delineation of structures in CT.  European Radiology, 29(3):1391–1399, Mar. 2019.
  * [38] K. Kiser, A. Barman, S. Stieb, C. D. Fuller, and L. Giancardo.  Novel Autosegmentation Spatial Similarity Metrics Capture the Time Required to Correct Segmentations Better than Traditional Metrics in a Thoracic Cavity Segmentation Workflow.  Preprint, Radiology and Imaging, May 2020.
  * [39] T. Kontogianni, M. Gygli, J. Uijlings, and V. Ferrari.  Continuous Adaptation for Interactive Object Segmentation by Learning from Corrections.  In A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, editors, Computer Vision – ECCV 2020, Lecture Notes in Computer Science, pages 579–596, Cham, 2020. Springer International Publishing.
  * [40] Y. LeCun, Y. Bengio, and G. Hinton.  Deep learning.  Nature, 521(7553):436–444, May 2015.
  * [41] C. S. Lee and A. Y. Lee.  Clinical applications of continual learning machine learning.  The Lancet Digital Health, 2(6):e279–e281, June 2020.
  * [42] T. Lustberg, J. van Soest, M. Gooding, D. Peressutti, P. Aljabar, J. van der Stoep, W. van Elmpt, and A. Dekker.  Clinical evaluation of atlas and deep learning based automatic contouring for lung cancer.  Radiotherapy and Oncology, 126(2):312–317, Feb. 2018.
  * [43] T. R. Mackie, J. Kapatoes, K. Ruchala, W. Lu, C. Wu, G. Olivera, L. Forrest, W. Tome, J. Welsh, R. Jeraj, P. Harari, P. Reckwerdt, B. Paliwal, M. Ritter, H. Keller, J. Fowler, and M. Mehta.  Image guidance for precise conformal radiotherapy.  International Journal of Radiation Oncology, Biology, Physics, 56(1):89–105, May 2003.
  * [44] P. Meyer, V. Noblet, C. Mazzara, and A. Lallement.  Survey on deep learning for radiotherapy.  Computers in Biology and Medicine, 98:126–146, July 2018.
  * [45] C. J. Michael, D. Acklin, and J. Scheuerman.  On Interactive Machine Learning and the Potential of Cognitive Feedback.  arXiv:2003.10365 [cs], Mar. 2020.
  * [46] F. Milletari, N. Navab, and S.-A. Ahmadi.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation.  arXiv:1606.04797 [cs], June 2016.
  * [47] M. L. H. Milo, B. V. Offersen, T. Bechmann, A. C. P. Diederichsen, C. R. Hansen, E. Holtved, M. Josipovic, T. Lörincz, M. V. Maraldo, M. H. Nielsen, M. Nordsmark, P. W. Nyström, M. Pøhl, H. K. Rose, T. Schytte, E. S. Yates, and E. L. Lorenzen.  Delineation of whole heart and substructures in thoracic radiation therapy: National guidelines and contouring atlas by the Danish Multidisciplinary Cancer Groups.  Radiotherapy and Oncology, 150:121–127, Sept. 2020.
  * [48] G. Ramos, C. Meek, P. Simard, J. Suh, and S. Ghorashi.  Interactive machine teaching: A human-centered approach to building machine-learned models.  Human–Computer Interaction, 35(5-6):413–451, Nov. 2020\.
  * [49] D. Rolnick, A. Veit, S. Belongie, and N. Shavit.  Deep Learning is Robust to Massive Label Noise.  arXiv:1705.10694 [cs], Feb. 2018.
  * [50] O. Ronneberger, P. Fischer, and T. Brox.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  In N. Navab, J. Hornegger, W. M. Wells, and A. F. Frangi, editors, Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, volume 9351, pages 234–241. Springer International Publishing, Cham, 2015.
  * [51] J. Schreier, F. Attanasi, and H. Laaksonen.  Generalization vs. Specificity: In Which Cases Should a Clinic Train its Own Segmentation Models?  Frontiers in Oncology, 10, 2020.
  * [52] P. Schwöbel, F. Warburg, M. Jørgensen, K. H. Madsen, and S. Hauberg.  Probabilistic Spatial Transformers for Bayesian Data Augmentation.  arXiv:2004.03637 [cs, stat], Apr. 2020.
  * [53] A. G. Smith, E. Han, J. Petersen, N. A. F. Olsen, C. Giese, M. Athmann, D. B. Dresbøll, and K. Thorup-Kristensen.  RootPainter: Deep Learning Segmentation of Biological Images with Corrective Annotation.  bioRxiv, page 2020.04.16.044461, May 2020.
  * [54] A. G. Smith, J. Petersen, R. Selvan, and C. R. Rasmussen.  Segmentation of roots in soil with U-Net.  Plant Methods, 16(1):13, Feb. 2020.
  * [55] H. Tang, X. Chen, Y. Liu, Z. Lu, J. You, M. Yang, S. Yao, G. Zhao, Y. Xu, T. Chen, Y. Liu, and X. Xie.  Clinically applicable deep learning framework for organs at risk delineation in CT images.  Nature Machine Intelligence, 1(10):480–491, Oct. 2019.
  * [56] C. Terrones-Campos, B. Ledergerber, N. Forbes, A. G. Smith, M. Helleberg, J. Lundgren, L. Specht, and I. R. Vogelius.  Prediction of radiation-induced lymphopenia following exposure of the thoracic region and associated risk of infections and mortality.  submitted.
  * [57] H. Um, J. Jiang, M. Thor, A. Rimner, L. Luo, J. O. Deasy, and H. Veeraraghavan.  Multiple resolution residual network for automatic thoracic organs-at-risk segmentation from CT.  arXiv:2005.13690 [cs, eess], May 2020.
  * [58] F. Vaassen, C. Hazelaar, A. Vaniqui, M. Gooding, B. van der Heyden, R. Canters, and W. van Elmpt.  Evaluation of measures for assessing time-saving of automatic organ-at-risk segmentation in radiotherapy.  Physics and Imaging in Radiation Oncology, 13:1–6, Jan. 2020.
  * [59] I. R. Vogelius, J. Petersen, and S. M. Bentzen.  Harnessing data science to advance radiation oncology.  Molecular Oncology, 14(7):1514–1528, 2020.
  * [60] K. N. Vokinger, S. Feuerriegel, and A. S. Kesselheim.  Continual learning in medical devices: FDA’s action plan and beyond.  The Lancet Digital Health, 0(0), Apr. 2021.
  * [61] Y. Wu and K. He.  Group Normalization.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 3–19, 2018.
  * [62] J. Yang, H. Veeraraghavan, S. G. Armato, K. Farahani, J. S. Kirby, J. Kalpathy-Kramer, W. van Elmpt, A. Dekker, X. Han, X. Feng, P. Aljabar, B. Oliveira, B. van der Heyden, L. Zamdborg, D. Lam, M. Gooding, and G. C. Sharp.  Autosegmentation for thoracic radiation treatment planning: A grand challenge at AAPM 2017.  Medical Physics, 45(10):4568–4581, Oct. 2018.
  * [63] J. Yang, C. Wei, L. Zhang, Y. Zhang, R. S. Blum, and L. Dong.  A statistical modeling approach for evaluating auto-segmentation methods for image-guided radiotherapy.  Computerized Medical Imaging and Graphics, 36(6):492–500, Sept. 2012.
  * [64] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals.  Understanding deep learning requires rethinking generalization.  arXiv:1611.03530 [cs], Feb. 2017.
  * [65] H. Zheng, Y. Zhang, L. Yang, C. Wang, and D. Z. Chen.  An Annotation Sparsification Strategy for 3D Medical Image Segmentation via Representative Selection and Self-Training.  Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):6925–6932, Apr. 2020.