# OpenKBP-Opt: An international and reproducible evaluation of 76 knowledge-
based planning pipelines

Aaron Babier  Rafid Mahmood  Binghao Zhang  Victor G. L. Alves  Ana Maria
Barragán-Montero  Joel Beaudry  Carlos E. Cardenas  Yankui Chang  Zijie Chen
Jaehee Chun  Kelly Diaz  Harold David Eraso  Erik Faustmann  Sibaji Gaj
Skylar Gay  Mary Gronberg  Bingqi Guo  Junjun He  Gerd Heilemann  Sanchit Hira
Yuliang Huang  Fuxin Ji  Dashan Jiang  Jean Carlo Jimenez Giraldo  Hoyeon Lee
Jun Lian  Shuolin Liu  Keng-Chi Liu  José Marrugo  Kentaro Miki  Kunio
Nakamura  Tucker Netherton  Dan Nguyen  Hamidreza Nourzadeh  Alexander F. I.
Osman  Zhao Peng  José Darío Quinto Muñoz  Christian Ramsl  Dong Joo Rhee
Juan David Rodriguez  Hongming Shan  Jeffrey V. Siebers  Mumtaz H. Soomro  Kay
Sun  Andrés Usuga Hoyos  Carlos Valderrama  Rob Verbeek  Enpei Wang  Siri
Willems  Qi Wu  Xuanang Xu  Sen Yang  Lulin Yuan  Simeng Zhu  Lukas Zimmermann
Kevin L. Moore  Thomas G. Purdie  Andrea L. McNiven  Timothy C. Y. Chan Due to
space constraints, funding information and author affiliations for this work
appear in the acknowledgement section.

###### Abstract

We establish an open framework for developing plan optimization models for
knowledge-based planning (KBP) in radiotherapy. Our framework includes
reference plans for 100 patients with head-and-neck cancer and high-quality
dose predictions from 19 KBP models that were developed by different research
groups during the OpenKBP Grand Challenge. The dose predictions were input to
four optimization models to form 76 unique KBP pipelines that generated 7600
plans. The predictions and plans were compared to the reference plans via:
dose score, which is the average mean absolute voxel-by-voxel difference in
dose a model achieved; the deviation in dose-volume histogram (DVH) criterion;
and the frequency of clinical planning criteria satisfaction. We also
performed a theoretical investigation to justify our dose mimicking models.
The range in rank order correlation of the dose score between predictions and
their KBP pipelines was 0.50 to 0.62, which indicates that the quality of the
predictions is generally positively correlated with the quality of the plans.
Additionally, compared to the input predictions, the KBP-generated plans
performed significantly better (P$<$0.05; one-sided Wilcoxon test) on 18 of 23
DVH criteria. Similarly, each optimization model generated plans that
satisfied a higher percentage of criteria than the reference plans. Lastly,
our theoretical investigation demonstrated that the dose mimicking models
generated plans that are also optimal for a conventional planning model. This
was the largest international effort to date for evaluating the combination of
KBP prediction and optimization models. In the interest of reproducibility,
our data and code is freely available at https://github.com/ababier/open-kbp-
opt.

Optimization, open data, inverse problem, knowledge-based planning,
radiotherapy

##  1 Introduction

Automated radiotherapy planning is transforming clinical practice and
personalized cancer treatment [1]. The most common type of automated planning
is knowledge-based planning, which leverages knowledge derived from historical
clinical treatment plans to generate new treatment plans without human
intervention [2, 3, 4]. Most common KBP methods can be thought of as a two-
stage pipeline that first predicts the dose that should be delivered to a
patient, and then converts that prediction into a treatment plan via
optimization (Figure 1). Both stages of this pipeline, which are active areas
of research, can significantly affect the quality of generated treatment plans
[5]. The contributions of this paper are twofold: 1) to provide data that
supports KBP optimization research at scale and 2) to establish a connection
between dose mimicking (a type of KBP optimization) and conventional planning
methods. We expand on the impact of these contributions throughout this paper.

Figure 1: Overview of a complete knowledge-based planning pipeline.

Comparing the quality of competing KBP models from the research community is
difficult because the vast majority of research is conducted with large
private datasets, as noted in several reviews [6, 7, 8, 9]. To help address
this issue, the Open Knowledge-Based Planning (OpenKBP) Grand Challenge was
organized to facilitate the largest international effort to date for
developing and comparing dose prediction models on a single open dataset [10].
The OpenKBP dataset, which includes data for 340 head-and-neck patients
undergoing intensity modulated radiotherapy (IMRT), is limited to dose
prediction research (i.e., it is incompatible with KBP optimization research).
Although there are still no open datasets for KBP optimization research, there
are two open datasets that support research in other areas of plan
optimization [11, 12]. However, it is challenging to use these datasets in KBP
plan optimization research for two reasons. First, neither dataset includes
dose predictions, which are the input to KBP plan optimization models. Second,
they are smaller (123 patients across both datasets), span multiple sites
(prostate, liver, head-and-neck), and multiple modalities (CyberKnife,
volumetric modulated arc therapy, proton therapy, IMRT). While such a
diversity in cases is important to demonstrate the robustness and
generalizability of optimization algorithms across sites and modalities, this
same diversity is a disadvantage when it comes to training dose prediction
models, since there is insufficient data for any one site-modality pair [13].

Most KBP pipelines are developed as fully-automated pipelines that can replace
human treatment planners in the planning process [14, 15, 16, 17]. These
approaches have demonstrated promising results in prospective research studies
where a sizeable portion of KBP-generated plans were considered inferior to
human-generated plans, which suggests that there is an opportunity for
improvement [2, 4]. In those cases, making manual adjustments to the KBP-
generated plan is non-trivial because they are generated by _fully-automated_
pipelines that rely on the quality of the data. In contrast to fully automated
pipelines, _semi-automated_ pipelines rely on both the quality of data and
human expertise, which puts less reliance on the data. For example, a semi-
automated KBP pipeline could enable human planners to improve upon a KBP-
generated plan via an intuitive process (e.g., inverse planning) and thereby
provide a pipeline that leverages human expertise, models, and data. In the
KBP literature, however, there are relatively few papers that describe tools
that humans can intuitively interact with in semi-automated KBP pipeline [18,
19, 20, 21].

In this paper, we extend the results from the OpenKBP Grand Challenge, which
we call OpenKBP, with an international validation of 76 KBP pipelines. We made
this extension, which we call OpenKBP-Opt, open to provide a benchmark for KBP
optimization research and to lower the barriers for contributing to this
research area. We also demonstrate how KBP plan optimization models can be
used to initialize the conventional planning process (i.e., inverse planning)
with good patient-specific parameters (i.e., objective weights) and provide
the means for a semi-automated KBP pipeline. Identifying this relationship
provides a mechanism for transforming existing KBP optimization models, which
are generally fully-automated pipelines that impede manual intervention, into
semi-automated pipelines that promote human planners to improve upon a KBP-
generated plan via inverse planning (i.e., a familiar and intuitive process).
The data and code to reproduce this paper is publicly available at
https://github.com/ababier/open-kbp-opt.

##  2 Methods and materials

Figure 2 summarizes the overall methodological approach into five components.
The first three components (i.e., processing data, developing dose prediction
models, and generating KBP dose predictions) are based on the results of the
OpenKBP Grand Challenge. The final two components (i.e., developing plan
optimization models and generating KBP treatment plans) are an extension of
the OpenKBP Grand Challenge and the focus of this paper. Below, we describe
all five components and our analysis.

Figure 2: An overview of our methods. A full description of each component is
provided in the corresponding subsection.

###  2.1 Processing patient data

We obtained data for 340 patients ($n=340$) with head-and-neck cancer from the
OpenKBP Grand Challenge. The data consisted of a training set ($n=200$), a
validation set ($n=40$), and a testing set ($n=100$). The plans were delivered
via 6 MV step-and-shoot IMRT from nine equidistant coplanar beams at angles
0∘, 40∘, …, 320∘. Those beams were divided into a set of beamlets
$\mathcal{B}$, which make up a fluence map. The relationship between the
intensity $w_{b}$ of beamlet $b$ and dose $d_{v}$ deposited to voxel $v$ was
determined using the influence matrix $D_{v,b}$ generated by the IMRTP library
from A Computational Environment for Radiotherapy Research [22] using MATLAB,
and it is given by

$d_{v}=\sum\limits_{b\in\mathcal{B}}D_{v,b}w_{b}.$ (1)

###  2.2 Developing dose prediction models

All dose prediction models used in this paper were developed in the OpenKBP
Grand Challenge [10]. During the challenge, teams developed dose prediction
models using identical training and validation datasets with access only to
ground truth data (i.e., dose) for the training set. Every dose prediction
model used a neural network architecture that was based on either a U-Net
[23], V-Net [24], or Pix2Pix [25] architecture. Many of the best performing
models also used other generalizable techniques like ensembles [26], one-cycle
learning [27], radiotherapy-specific loss functions [28], and deep supervision
[29].

All teams competed to develop models that minimize one of two pre-defined
error metrics that quantified the difference between the reference dose and a
KBP-generated dose (i.e., KBP prediction or plan dose). The metrics were: 1)
dose error, which was the mean absolute voxel-by-voxel difference between two
dose distributions, and 2) dose-volume histogram (DVH) error, which was the
absolute difference between a DVH point from two dose distributions. The DVH
error was evaluated on two and three DVH points for each organ-at-risk (OAR)
and target, respectively. The OAR DVH points were the $\text{D}_{\text{mean}}$
and $\text{D}_{\text{0.1cc}}$, which was the mean dose delivered to the OAR
and the maximum dose delivered to 0.1cc of the OAR, respectively. The target
DVH points were the $\text{D}_{\text{1}}$, $\text{D}_{\text{95}}$, and
$\text{D}_{\text{99}}$, which was the dose delivered to 1%
(99${}^{\textrm{th}}$ percentile), 95% (5${}^{\textrm{th}}$ percentile), and
99% (1${}^{\textrm{st}}$ percentile) of voxels in the target, respectively.
The models were ranked according to: 1) dose score, which was the average dose
error of a model, and 2) DVH score, which was the average DVH error of a
model.

###  2.3 Generating KBP dose predictions

In this paper, the OpenKBP organizers collaborated with teams that competed in
the OpenKBP Grand Challenge. The 28 teams that completed the ﬁnal phase of the
OpenKBP Grand Challenge were invited to participate in the OpenKBP-Opt
project, and 21 of those teams agreed to participate. We obtained the dose
predictions from all teams for each patient in the test set to create a set of
2100 dose predictions (21 different predictions for each of the 100 patients).
We observed that two models produced dose scores that were over two standard
deviations (6.3 Gy) above the mean (4.0 Gy), whereas the rest were within half
a standard deviation (1.6 Gy) of the mean. Thus, we omitted those two outlier
models and proceeded with only 19 KBP models ($n=1900$ predictions).

###  2.4 Developing plan optimization models

Next, we formulated four dose mimicking models, which are a type of KBP
optimization model. Each model used the same set of structures and objective
functions that we described in Section 2.4.1 and Section 2.4.2, respectively.
However, they differ in how they mimic (i.e., penalize differences) a specific
dose distribution. In particular, they each have a different cost function,
outlined in Section 2.4.3. Note that in this paper the terms “objective
function” and “cost function” refer to distinct concepts, and the cost
functions in this paper are functions of objective functions.

####  2.4.1 Structures

All of our optimization models used the same set of regions-of-interest (ROIs)
$\mathcal{R}_{p}$ for each patient $p\in\mathcal{P}$ in our test set. The set
$\mathcal{R}_{p}$ contains OARs $\mathcal{I}_{p}$, targets $\mathcal{T}_{p}$,
and optimization structures $\mathcal{O}_{p}$. The OARs contained in
$\mathcal{I}_{p}$ were the brainstem, spinal cord, right parotid, left
parotid, larynx, esophagus, and mandible. Each target $t\in\mathcal{T}_{p}$
was a planning target volume (PTV) with a dose level $\theta_{t}$, and those
targets were the PTV56, PTV63, and PTV70. The optimization structures
contained in $\mathcal{O}_{p}$ were the limPostNeck, which was used to limit
dose to the posterior neck, and six PTV ring structures (a $3~{}\text{mm}$
ring and a $6~{}\text{mm}$ ring for each target). These were the same
structures used to generate the plans in the original OpenKBP dataset [10].
Every ROI $r\in\mathcal{R}_{p}$ was also divided into a set of voxels
$\mathcal{V}_{r}$.

####  2.4.2 Objective functions

Our models used the objective functions in Table 1. Each objective function
quantified a different measure of the dose delivered to a single ROI
$r\in\mathcal{R}_{p}$ in a patient $p\in\mathcal{P}$, which we call an
objective value. Specifically, the average and maximum objective values
quantified the average dose and maximum dose delivered to an ROI $r$,
respectively. The high and low conditional value at risk (CVaR) objective
values quantified the average dose in ROI $r$ that was higher and lower,
respectively than the dose threshold $f$.

Table 1: The formulations for objective functions in our models. Name | Objective function
---|---
Average dose | $\underset{v\in\mathcal{V}_{r}}{\text{mean}}\left\\{d_{v}\right\\}$
Maximum dose | $\underset{v\in\mathcal{V}_{r}}{\text{max}}\left\\{d_{v}\right\\}$
High CVaR dose | $\underset{v\in\mathcal{V}_{r}}{\text{mean}}\left\\{\text{max}\left\\{0,\ d_{v}-f\right\\}\right\\}$
Low CVaR dose | $\underset{v\in\mathcal{V}_{r}}{\text{mean}}\left\\{\text{max}\left\\{0,\ f-d_{v}\right\\}\right\\}$

In total, we considered 107 objectives functions: seven per OAR, three per
target, and seven per optimization structure. The objective functions for each
OAR were the mean dose; maximum dose; and high CVaR dose with thresholds $f$
equal to 0.25, 0.50, 0.75, 0.90, and 0.975 of the maximum predicted dose to
that structure. The objective functions for each target were the maximum dose,
low CVaR dose with a threshold equal to the dose level of the target (i.e.,
$f=\theta_{t}$), and a high CVaR dose with a threshold $f$ equal to 1.05 of
the dose level of the target (i.e., $f=1.05\theta_{t}$). The objective
functions for each optimization structure were the same as the OAR objective
functions. Not all patients had all ROIs, so the models associated with those
patients had fewer than 107 objective functions.

####  2.4.3 Model formulations

Our KBP optimization models performed dose mimicking to generate plans with
optimized objective values that closely matched the input objective values
from a dose prediction. To streamline our model formulation, let each
$m\in\mathcal{M}_{p}$ denote one of the 107 objective functions (as outlined
in Section 2.4.2). Let $g_{m}$ and $\hat{g}_{m}$ be objective values of their
corresponding objective functions evaluated over the optimized plan and
predicted dose, respectively. In all models, the cost functions were
formulated such that lower values of $g_{m}$ were favored over higher values.

Table 2 presents the cost functions of our dose mimicking models. Each model
minimized either the mean or max difference between all corresponding pairs
$\left(g_{m},\hat{g}_{m}\right)$ of the objective values, which were
quantified via an absolute (e.g., $g_{m}-\hat{g}_{m}$) or relative (e.g.,
$(g_{m}-\hat{g}_{m})/\hat{g}_{m}$) difference measure, resulting in four dose
mimicking models. In the mean difference models, we chose to prioritize the
positive differences (i.e., where the optimized plan objective value was
higher than the predicted dose objective value) more than the negative
differences, which we assigned a small positive weight $\epsilon$
($\epsilon=0.0001$ in our experiments). This was done to incentivize the model
to do at least as well as the dose prediction before striving to outperform
the dose prediction on other objective functions. In contrast, the max
difference models used only a single term because the max difference naturally
incentivizes the model to outperform the prediction only once the plan
outperforms the prediction across all objective values (i.e., when
$g_{m}\leq\hat{g}_{m},\forall m\in\mathcal{M}_{p}$).

Table 2: The cost functions for each model that minimize mean absolute (MeanAbs), max absolute (MaxAbs), mean relative (MeanRel), and max relative (MaxRel) differences between the optimized and predicted objective values $\left(g_{m},\hat{g}_{m}\right)$. | Optimization model cost function
---|---
MeanAbs | $\underset{m\in\mathcal{M}_{p}}{\text{mean}}\left(g_{m}-\hat{g}_{m}\right)^{+}\ +\ \epsilon\ \underset{m\in\mathcal{M}_{p}}{\text{mean}}\left(g_{m}-\hat{g}_{m}\right)^{-}$
MaxAbs | $\underset{m\in\mathcal{M}_{p}}{\max}\left(g_{m}-\hat{g}_{m}\right)$
MeanRel | $\underset{m\in\mathcal{M}_{p}}{\text{mean}}\left(\frac{g_{m}-\hat{g}_{m}}{\hat{g}_{m}}\right)^{+}\ +\ \epsilon\ \underset{m\in\mathcal{M}_{p}}{\text{mean}}\left(\frac{g_{m}-\hat{g}_{m}}{\hat{g}_{m}}\right)^{-}$
MaxRel | $\underset{m\in\mathcal{M}_{p}}{\max}\left(\frac{g_{m}-\hat{g}_{m}}{\hat{g}_{m}}\right)$

The main constraint in all four models was a constraint to limit plan
complexity. In particular, the sum-of-positive gradients (SPG) [30] of all
plans generated by the models was constrained to be less than or equal to 65,
which was a constraint in the reference plans [10]. The remaining constraints
were simply auxiliary constraints (including auxiliary variables) used to
linearize both the objective and cost functions (i.e., the formulations in
Table 1 and Table 2). The optimization models were all formulated in Python
3.7 using OR-Tools 8.2 and solved using Gurobi 9.1 (Gurobi Optimization, TX,
US) on a single computer with an Intel i7-8700K (6-Core $3.7$ GHz) CPU and
$16$ GB of random access memory. Default parameters were used with the Gurobi
solver except for _Crossover_ set to $0$, _Method_ set to $2$, and
_BarConvTol_ set to $0.0001$, which were selected based on past experience to
improve solve time without compromising solution quality.

###  2.5 Generating KBP treatment plans

Next, we assembled 76 KBP pipelines by combining the 19 dose prediction models
with each of the four dose mimicking models. Each pipeline was applied to the
100 patients in the testing set, resulting in 7600 KBP plans (see Figure 3).
We used these plans in our analysis to measure the quality of the respective
KBP models. We refer to the four plans generated from each dose prediction as
the MeanAbs, MaxAbs, MeanRel, and MaxRel plans.

Figure 3: An overview of our process. First, dose prediction models were
trained on out-of-sample data. Those models were used to predict dose for
input to dose mimicking optimization models to generate KBP plans.

Altogether, after completing the process in Figure 3, we had dose
distributions for a set of reference plans ($n=100$), predictions ($n=1900$),
and KBP plans generated by four dose mimicking models ($n=4\times 1900$). The
reference plans are the plans that were released as part of the OpenKBP Grand
Challenge, and the predictions are dose distributions that were submitted by
19 teams in the final testing phase of OpenKBP. In general, there will be
differences between the reference plan, prediction, and KBP plan dose
distributions. Differences between a dose prediction and its corresponding KBP
plan are due to factors including prediction noise and deliverability of the
dose prediction. Differences between a KBP plan and its corresponding
reference plan reflect different trade-offs in the cost function used to
generate these plans.

###  2.6 Analysis

We conducted three analyses to measure model performance in terms of dose
error, DVH point differences, and clinical criteria satisfaction. We also
investigated the theoretical connection between our dose mimicking models and
inverse planning. Finally, we summarized empirical optimization metadata.

####  2.6.1 Dose score and error

We evaluated the KBP models using the dose score and dose error as defined in
Section 2.2. We calculated the Spearman rank order correlation of the dose
score between the prediction models and corresponding KBP pipelines. The
distribution of dose error was visualized using a box plot. A one-sided
Wilcoxon signed-rank test was used to determine whether the dose error of the
optimization models was the same (null hypothesis) or lower (alternative
hypothesis) than the dose predictions models. For all hypothesis tests in this
paper, $P<0.05$ was considered significant.

####  2.6.2 DVH point differences

To measure the relative quality of dose distributions from a clinical
perspective, we examined the distribution of DVH point differences between the
reference and KBP-generated dose. The differences were evaluated over the DVH
points listed in Section 2.2 and visualized using boxplots. We used the one-
sided Wilcoxon signed-rank test to determine whether the dose generated by all
optimization models performed the same (null hypothesis) or better
(alternative hypothesis) than the dose predictions. This test was chosen to
evaluate the aggregate performance of all optimization models relative to the
predictions. Lower values were better for $\text{D}_{\text{mean}}$,
$\text{D}_{\text{0.1cc}}$, and $\text{D}_{\text{1}}$; higher values were
better for $\text{D}_{\text{95}}$ and $\text{D}_{\text{99}}$.

####  2.6.3 Expected criteria satisfaction

As another measure of plan quality, we examined the proportion of clinical
criteria that were satisfied by the reference plans and KBP-generated dose.
One criterion was evaluated for each ROI (see Table 3). We tabulated the
proportion of criteria that were satisfied by the reference plans, dose
predictions, MeanAbs plans, MaxAbs plans, MeanRel plans, MaxRel plans, and the
plans from the KBP pipeline that satisfied the most clinical criteria overall.
We also plotted the proportion of OAR, target, and all ROI clinical criteria
that each of the 76 KBP pipelines achieved.

Table 3: The clinical criteria that we used to evaluate dose distributions. Before evaluating these criteria, we reinstated any overlap between targets that was removed. Structures | Criteria
---|---
OARs |
| Brainstem | $\text{D}_{\text{0.1cc}}\leq 50.0\text{ Gy}$
| Spinal cord | $\text{D}_{\text{0.1cc}}\leq 45.0\text{ Gy}$
| Right parotid | $\text{D}_{\text{mean}}\leq 26.0\text{ Gy}$
| Left parotid | $\text{D}_{\text{mean}}\leq 26.0\text{ Gy}$
| Esophagus | $\text{D}_{\text{mean}}\leq 45.0\text{ Gy}$
| Larynx | $\text{D}_{\text{mean}}\leq 45.0\text{ Gy}$
| Mandible | $\text{D}_{\text{0.1cc}}\leq 73.5\text{ Gy}$
Targets |
| PTV56 | $\text{D}_{\text{99}}\geq 53.2\text{ Gy}$
| PTV63 | $\text{D}_{\text{99}}\geq 59.9\text{ Gy}$
| PTV70 | $\text{D}_{\text{99}}\geq 66.5\text{ Gy}$

####  2.6.4 Theoretical analysis of dose mimicking models

To justify our choice of dose mimicking models, we conducted a theoretical
analysis into their structure using linear programming duality theory [31,
Chapter 4]. This analysis was based on previous literature that showed a
connection between Benson’s method [32], which identifies efficient solutions
to multi-objective optimization models, and estimating the weights for inverse
planning [33]. We were motivated to conduct a similar analysis as in Chan et
al. [33] because our dose mimicking models are similar to the formulations in
Benson [32]. In particular, we linearized the dose mimicking models, took
their duals, and related the dual variables to objective function weights in a
conventional multi-objective planning problem depicted in model (2).

$\displaystyle\underset{g}{\text{minimize}}$
$\displaystyle\sum\limits_{m\in\mathcal{M}_{p}}\hat{\alpha}_{m}g_{m},$ (2)
subject to $\displaystyle SPG\leq 65,$ Auxiliary constraints to linearize
$\displaystyle\text{functions in Table}~{}\ref{tab:doseObectives}\text{ and
}\ref{tab:models}.$

####  2.6.5 Optimization metadata

Lastly, we summarized the metadata that each optimization model generated. In
particular, we evaluated the average proportion of objective weight that each
model assigned to OAR, target, and optimization structure objective functions.
Additionally, we recorded the average, first quartile, and third quartile
solve time.

##  3 Results

In this section, we summarize the performance of the 19 dose predictions
models, four dose mimicking models, and 76 KBP pipelines.

###  3.1 Dose score and error

Table 4 summarizes the rank order correlation between the dose prediction
models and their corresponding KBP pipelines. We found that the rank of a
prediction model is positively correlated with its corresponding KBP pipeline
rank. However, there was a wide range in correlation from 0.50 to 0.62. This
demonstrates that high quality predictions are correlated with high quality
plans, but this result also indicates that a prediction model that outperforms
a competitor will not always generate better plans. Additionally, the KBP
plans generated by an optimization model that evaluated relative differences
(i.e., MeanRel and MaxRel) achieved higher rank order correlations than their
counterparts that evaluated absolute differences (i.e., MeanAbs and MaxAbs).

Table 4: Each dose mimicking model is compared to the predictions in terms of median rank change and rank order correlation. | MeanAbs | MaxAbs | MeanRel | MaxRel
---|---|---|---|---
Rank order correlation | 0.53 | 0.50 | 0.62 | 0.59
Rank order $P$-value | 0.019 | 0.030 | $0.005$ | 0.008

The dose errors of predictions and KBP plans are shown in Figure 4. Two of the
four sets of KBP plans had a median dose error that was lower than the median
dose error of the predictions (2.79 Gy), implying that it is possible for
optimization models to generate dose distributions that more closely resemble
the reference plan dose, compared to dose predictions. These two models also
achieved a significantly lower error ($P\leq 0.001$) than predictions. The
MaxAbs model achieved the lowest median dose error (2.34 Gy).

Figure 4: The distribution of dose error over all KBP-generated dose ($n=1900$
points in each box). Boxes indicate median and interquartile range (IQR).
Whiskers extend to the minimum of 1.5 times the IQR and the most extreme
outlier.

###  3.2 DVH point differences

Figure 5 shows the DVH point differences between the reference dose and either
the predicted dose or KBP plan dose. In general, dose mimicking tends to
produce a plan dose that is significantly better than the dose it received as
input from a dose prediction model. In particular, the KBP plan dose is
significantly better on 18 of the 23 DVH points than the predicted dose (all
OAR points and four target points). The five DVH points where the plans were
not significantly better are the three $\text{D}_{\text{95}}$ points and two
$\text{D}_{\text{99}}$ points.

(a) OAR $\text{D}_{\text{mean}}$ (b) OAR $\text{D}_{\text{0.1cc}}$

(c) Target $\text{D}_{\text{1}}$ (d) Target $\text{D}_{\text{95}}$ (e) Target
$\text{D}_{\text{99}}$

Figure 5: The distribution of DVH point differences between the reference dose
and each set of KBP-generated dose. Negative differences indicate cases where
the KBP-generated dose had a lower DVH points than the reference dose. Boxes
indicate median and IQR. Whiskers extend to the minimum of 1.5 times the IQR
and the most extreme outlier.

###  3.3 Expected criteria satisfaction

In Table 5, we compare the percentage of criteria that were satisfied by the
reference plans ($n=100$), the predictions ($n=1900$), the plans generated by
each of the four dose mimicking models ($n=4\times 1900$), and the plans
generated by the top performing KBP pipeline ($n=100$). We use the term
baselines to refer to the reference dose and dose predictions collectively.
The top performing KBP pipeline (denoted “Best” in Table 5) was defined as the
single pipeline (i.e., the combination of one dose prediction model and one
dose mimicking model) whose plans satisfied the most clinical criteria. Of all
dose mimicking models, the MaxRel and MeanAbs models generated plans that
satisfied the fewest (69.8%) and most (72.9%) ROI clinical criteria,
respectively. For comparison, predictions only satisfied 66.2% of all clinical
criteria, which was 3.5 percentage points lower than the reference plans
(69.7%). The best KBP pipeline, which used the MeanAbs model and one of the 19
prediction models (discussed later), satisfied 77.0% of all ROI clinical
criteria.

In general, clinical criteria satisfaction varied across each ROI criterion.
The brainstem, spinal cord, esophagus, and mandible criteria were each
satisfied more than 85% of the time across all the baselines and our dose
mimicking models in Table 5. The right parotid, left parotid, and larynx were
satisfied less than 40% of the time for the two baselines. In contrast, each
of our four KBP models generated a higher average criteria satisfaction for
these ROIs compared to the baselines. In fact, some were substantially higher.
For example, the average criteria satisfaction of the MeanAbs model on the
larynx was 71.5%, compared to an average of 36.2% for the baselines. In
aggregate over all 19 prediction models, the performance of the four dose
mimicking model was comparable or slightly worse than the reference dose in
terms of criteria satisfaction in the targets. However, the best KBP pipeline
outperformed the baselines on all criteria.

Table 5: The percentage of clinical criteria satisfied in each set of KBP-generated dose. Note that “Best” is defined as the top performing KBP pipeline that generated plans that satisfied the most ROI clinical criteria. The highest percentage of satisfied criteria is bolded in each row. |  |  | Baselines |  | Dose mimicking models |  |
---|---|---|---|---|---|---|---
|  | Reference | Prediction |  | MeanAbs | MaxAbs | MeanRel | MaxRel |  | Best
OARs |  |  |  |  |  |  |  |  |  |
| Brainstem |  | 96.6 | 97.3 |  | 100.0 | 99.5 | 100.0 | 98.5 |  | 100.0
| Spinal cord |  | 95.5 | 92.7 |  | 99.7 | 97.3 | 100.0 | 95.6 |  | 100.0
| Right parotid |  | 32.3 | 32.7 |  | 46.1 | 38.9 | 45.0 | 38.0 |  | 41.4
| Left parotid |  | 30.6 | 30.1 |  | 43.7 | 35.0 | 41.9 | 35.0 |  | 40.8
| Esophagus |  | 93.0 | 92.7 |  | 100.0 | 95.2 | 100.0 | 97.3 |  | 100.0
| Larynx |  | 37.7 | 34.7 |  | 71.5 | 44.9 | 58.8 | 44.6 |  | 67.9
| Mandible |  | 87.5 | 89.4 |  | 99.6 | 98.7 | 99.2 | 99.0 |  | 93.1
Targets |  |  |  |  |  |  |  |  |  |
| PTV56 |  | 91.2 | 85.8 |  | 83.3 | 91.8 | 84.1 | 84.6 |  | 96.7
| PTV63 |  | 90.5 | 86.2 |  | 82.2 | 89.6 | 84.8 | 84.8 |  | 92.9
| PTV70 |  | 64.0 | 45.7 |  | 37.2 | 51.6 | 40.1 | 47.7 |  | 66.0
All |  |  |  |  |  |  |  |  |  |
| OARs |  | 65.5 | 65.1 |  | 77.1 | 70.6 | 75.3 | 70.2 |  | 74.5
| Targets |  | 79.4 | 68.7 |  | 63.3 | 74.2 | 65.3 | 68.8 |  | 82.8
| ROIs |  | 69.7 | 66.2 |  | 72.9 | 71.7 | 72.3 | 69.8 |  | 77.0

Figure 6 summarizes the clinical criteria that were satisfied by each of the
76 KBP pipelines that we evaluated. The MeanAbs model generated plans that
satisfied more criteria than the other three optimization models for 16 of the
19 dose prediction models (see Figure 6(c)). Additionally, the pipelines that
used better prediction models (i.e., dose score rank closer to 1) generally
produced plans with higher criteria satisfaction. Interestingly, the best
performing KBP pipeline (the last column of Table 5) used the dose prediction
model that ranked 16${}^{\textrm{th}}$ in terms of dose score. The spread in
OAR criteria satisfaction across all 19 models (55.4% to 82.1%) was lower than
that of target criteria satisfaction (24.5% to 89.7%), see Figure 6(a) and
Figure 6(b), respectively. Note that the poor performing KBP pipelines used
the 12${}^{\textrm{th}}$, 13${}^{\textrm{th}}$, 17${}^{\textrm{th}}$,
18${}^{\textrm{th}}$, and 19${}^{\textrm{th}}$ ranked dose prediction models.
Since the columns in Table 5 included all KBP pipelines, these poor performing
models contributed to low performance on the target criteria. In contrast,
many of the KBP pipelines that used the top ranked models prediction models
clearly performed much better on target criteria.

(a) All OAR criteria

(b) All target criteria

(c) All ROI criteria

Figure 6: The percentage of (a) OAR, (b) Target, and (c) all ROI clinical
criteria that were satisfied by each KBP pipeline. The points indicate the
percentage of satisfied criteria for $n=100$ patients. A dashed line indicates
the percentage of criteria satisfied by reference plans.

###  3.4 Theoretical analysis of dose mimicking models

The inverse planning model (2) is shown in model (3) in vector and matrix
notation following Chan et al. [33].

$\displaystyle\underset{\mathbf{x}}{\text{minimize}}$
$\displaystyle\hat{\boldsymbol{\alpha}}^{\prime}\mathbf{C}\mathbf{x}$ (3)
subject to $\displaystyle\mathbf{A}\mathbf{x}=\mathbf{b},$
$\displaystyle\mathbf{x}\geq\mathbf{0}.$

The objective functions are the rows of matrix $\mathbf{C}$ and the objective
function weights are represented by the vector $\hat{\boldsymbol{\alpha}}$.
The decision variables, which include the fluence variables ($w_{b}\ \forall
b\in\mathcal{B}$) and auxiliary variables are represented by vector
$\mathbf{x}$. The SPG and auxiliary constraints are encoded in the matrix
$\mathbf{A}$ and vector $\mathbf{b}$.

Table 6 presents the formulations of the four dose mimicking models and their
respective dual models. The positive and negative differences between
optimized objective values $\mathbf{C}\mathbf{x}$ and predicted objective
values $\mathbf{C}\hat{\mathbf{x}}$ are represented by vectors
$\boldsymbol{\sigma}$ and $\boldsymbol{\delta}$, respectively. The max
difference between the optimized and predicted objective values is expressed
as a scalar $\zeta$. The dual variables of the dose mimicking models are
denoted $\boldsymbol{\alpha}$ and $\mathbf{p}$. The vectors of all $0$ and $1$
are denoted by $\mathbf{0}$ and $\mathbf{e}$, respectively. The symbol $\odot$
denotes element-wise multiplication of two vectors and prime denotes the
transpose operator.

Next, we complete our theoretical analysis. By Proposition 5 from [33], it
follows that an optimal decision vector $\mathbf{x}^{*}$ from each dose
mimicking model is also optimal for the inverse planning model (3)) with an
optimal dual vector $\boldsymbol{\alpha}^{*}$ as objective weights (i.e.,
$\mathbf{x}^{*}$ is an optimal solution for model (3) when
$\hat{\boldsymbol{\alpha}}=\boldsymbol{\alpha}^{*}$). This result means that
the solution to each dose mimicking model is also optimal to the inverse
planning model with a particular set of objective function weights.

Table 6: The dose mimicking models used in this paper are presented in matrix notation with their corresponding dual models. | MeanAbs | MaxAbs | MeanRel | MaxRel
---|---|---|---|---
Dose mimicking model | $\begin{aligned} &\underset{\mathbf{x},\boldsymbol{\sigma},\boldsymbol{\delta}}{\text{min}}&&\mathbf{e}^{\prime}\boldsymbol{\sigma}+\epsilon\mathbf{e}^{\prime}\boldsymbol{\delta}\\\ &\text{s.t.}&&\mathbf{C}\mathbf{x}={\mathbf{C}\hat{\mathbf{x}}+\boldsymbol{\sigma}+\boldsymbol{\delta}}\\\ &&&\mathbf{A}\mathbf{x}=\mathbf{b}\\\ &&&\mathbf{x}\geq\bf{0}\\\ &&&\boldsymbol{\sigma}\geq\bf{0}\\\ &&&\boldsymbol{\delta}\leq\bf{0}\\\ \end{aligned}$ | $\begin{aligned} &\underset{\mathbf{x},\zeta}{\text{min}}&&\zeta\\\ &\text{s.t.}&&\mathbf{C}\mathbf{x}\leq{\mathbf{C}\hat{\mathbf{x}}+\zeta\mathbf{e}}\\\ &&&\mathbf{A}\mathbf{x}=\mathbf{b}\\\ &&&\mathbf{x}\geq\bf{0}\\\ &\\\ &\end{aligned}$ | $\begin{aligned} &\underset{\mathbf{x},\boldsymbol{\sigma},\boldsymbol{\delta}}{\text{min}}&&\mathbf{e}^{\prime}\boldsymbol{\sigma}+\epsilon\mathbf{e}^{\prime}\boldsymbol{\delta}\\\ &\text{s.t.}&&\mathbf{C}\mathbf{x}={\mathbf{C}\hat{\mathbf{x}}\odot(\mathbf{e}+\boldsymbol{\sigma}+\boldsymbol{\delta})}\\\ &&&\mathbf{A}\mathbf{x}=\mathbf{b}\\\ &&&\mathbf{x}\geq\bf{0}\\\ &&&\boldsymbol{\sigma}\geq\bf{0}\\\ &&&\boldsymbol{\delta}\leq\bf{0}\\\ \end{aligned}$ | $\begin{aligned} &\underset{\mathbf{x},\zeta}{\text{min}}&&\zeta\\\ &\text{s.t.}&&\mathbf{C}\mathbf{x}\leq{\mathbf{C}\hat{\mathbf{x}}\odot(\mathbf{e}+\zeta\mathbf{e})}\\\ &&&\mathbf{A}\mathbf{x}=\mathbf{b}\\\ &&&\mathbf{x}\geq\bf{0}\\\ &\\\ &\end{aligned}$
Dual model | $\begin{aligned} &\underset{\boldsymbol{\alpha},\mathbf{p}}{\text{min}}&&{\boldsymbol{\alpha}^{\prime}\mathbf{C}\hat{\mathbf{x}}-\mathbf{b}^{\prime}\mathbf{p}}\\\ &\text{s.t.}&&\mathbf{C}^{\prime}\boldsymbol{\alpha}\geq\mathbf{A^{\prime}p}\\\ &&&{\boldsymbol{\alpha}\leq\mathbf{e}}\\\ &&&{\boldsymbol{\alpha}\geq\epsilon\mathbf{e}}\\\ \end{aligned}$ | $\begin{aligned} &\underset{\boldsymbol{\alpha},\mathbf{p}}{\text{min}}&&{\boldsymbol{\alpha}^{\prime}\mathbf{C}\hat{\mathbf{x}}-\mathbf{b}^{\prime}\mathbf{p}}\\\ &\text{s.t.}&&\mathbf{C}^{\prime}\boldsymbol{\alpha}\geq\mathbf{A^{\prime}p}\\\ &&&{\boldsymbol{\alpha}^{\prime}\mathbf{e}=1}\\\ &&&\boldsymbol{\alpha}\geq\mathbf{0}\end{aligned}$ | $\begin{aligned} &\underset{\boldsymbol{\alpha},\mathbf{p}}{\text{min}}&&{\boldsymbol{\alpha}^{\prime}\mathbf{C}\hat{\mathbf{x}}-\mathbf{b}^{\prime}\mathbf{p}}\\\ &\text{s.t.}&&\mathbf{C}^{\prime}\boldsymbol{\alpha}\geq\mathbf{A^{\prime}p}\\\ &&&{\boldsymbol{\alpha}\odot\mathbf{C}\hat{\mathbf{x}}\leq\mathbf{e}}\\\ &&&{\boldsymbol{\alpha}\odot\mathbf{C}\hat{\mathbf{x}}\geq\epsilon\mathbf{e}}\\\ \end{aligned}$ | $\begin{aligned} &\underset{\boldsymbol{\alpha},\mathbf{p}}{\text{min}}&&{\boldsymbol{\alpha}^{\prime}\mathbf{C}\hat{\mathbf{x}}-\mathbf{b}^{\prime}\mathbf{p}}\\\ &\text{s.t.}&&\mathbf{C}^{\prime}\boldsymbol{\alpha}\geq\mathbf{A^{\prime}p}\\\ &&&{\boldsymbol{\alpha}^{\prime}\mathbf{C}\hat{\mathbf{x}}=1}\\\ &&&\boldsymbol{\alpha}\geq\mathbf{0}\end{aligned}$

###  3.5 Optimization metadata

In Table 7, we present metadata that was generated by each optimization model,
which assigned a different proportion of weight to the objectives for each
group of ROIs (i.e., OARs, targets, optimization structures). The models that
evaluate relative differences (i.e., MeanRel and MaxRel) spread the proportion
of weight relatively evenly between the OAR and target objectives, but the
other two models assigned the majority of the weight to target objectives with
no more than 0.018 weight to OARs. Additionally, the optimization structures
generally received the smallest proportion of weight with the exception of the
MaxAbs model, which assigned more weight to optimization structure objectives
(0.170) than OAR objectives (0.011). There is also a wide range in average
solve time of the models (222 seconds to 393 seconds). On average, the MaxAbs
model was the fastest.

Table 7: A summary of the metadata that each optimization model generated after optimizing 1900 plans. | MeanAbs | MaxAbs | MeanRel | MaxRel
---|---|---|---|---
Objective weight
OARs | 0.018 | 0.011 | 0.554 | 0.417
Targets | 0.976 | 0.819 | 0.418 | 0.569
Optimization | 0.006 | 0.170 | 0.028 | 0.014
Solve time (s)
Average | 389 | 222 | 367 | 393
First quartile | 192 | 107 | 183 | 188
Third quartile | 502 | 261 | 481 | 507

##  4 Discussion

Knowledge-based planning research is flourishing. However, optimization models
for KBP (e.g., dose mimicking) have received much less attention in the
literature than dose prediction models. In this paper, we developed four dose
mimicking models and evaluated their performance with 19 different dose
prediction models, which were inputs to the optimization models. We showed
that both the dose prediction model and optimization model contributed to
considerable variation in the quality of plans generated by the corresponding
KBP pipeline. Additionally, we conducted a theoretical investigation to show
that our dose mimicking models generate plans that are optimal for a multi-
objective inverse planning model with particular weights.

Our data and code is published at https://github.com/ababier/open-kbp-opt. to
enable others to reproduce our results, which meets the gold standard in
reproducibility [34]. Our data includes the first open dataset of predictions
and reference plans to accompany CT images. We hope that this effort produces
a common resource and lowers the barriers for future KBP optimization
research, given that researchers must currently acquire their own private
datasets and develop in-house prediction models before they can start testing
new KBP optimization models.

Our open dataset contains the data for 100 patients who were treated with IMRT
and a sample of high quality dose predictions for those same patients. The
dataset was curated for the purpose of developing new fluence-based KBP
optimization models that use ROI masks, dose influence matrices, and a dose
prediction. The dose predictions were generated by 21 dose prediction models
that were developed by an international group of researchers, which provided a
diverse sample of realistic inputs for a KBP optimization model. Two of those
prediction models (20${}^{\textrm{th}}$ and 21${}^{\textrm{th}}$ ranked model)
were removed from our analysis because their dose scores were low, which we
elaborated on in Section 2.3. For completeness, however, those 200 predictions
are also available as part of our dataset.

We also performed a theoretical analysis to justify our dose mimicking models.
Our key theoretical finding was that dose mimicking and conventional inverse
planning are equivalent under certain specifications of the objective function
weights. This allows us to interpret previous weight estimation techniques
[33] through the more intuitive lens of dose mimicking models. Finally, by
connecting dose mimicking to inverse planning, there is the potential to
convert fully-automated KBP pipelines into semi-automated pipelines.
Specifically, we use dose mimicking to generate a high-quality plan with its
corresponding objective weights, which can be used in an inverse planning
model (i.e., model (3)). This is advantageous because it enables human
planners to improve the quality of plans generated by KBP via a conventional
inverse planning process. By enabling this intuitive human interaction, we
create a semi-automated KBP pipeline that is aligned with a common belief that
AI will augment, rather than replace, the duties of healthcare practitioners
[35].

Evaluating the performance of optimization models using many different dose
predictions helps to identify interaction effects between these two stages of
a KBP pipeline [5]. For example, the 16${}^{\textrm{th}}$ ranked model
generated lower quality predictions (in terms of dose error) than most of its
competitors. However, when used in a KBP pipeline with the right optimization
model, in this case the MeanAbs model, it generated high quality plans that
achieved more clinical criteria than any other KBP pipeline. In other words,
the errors made by the 16${}^{\textrm{th}}$ ranked model that contribute to
its low prediction quality were corrected by the KBP optimization model. Since
these interaction effects contribute to considerable variation in quality, it
is important to evaluate KBP optimization models across a diverse set of dose
prediction models. Additionally, if we can understand what types of prediction
error are most highly correlated with KBP plan quality we could propose better
evaluation metrics to drive KBP prediction research towards making predictions
that consistently translate into higher quality plans.

As in the original OpenKBP challenge, a limitation of this work is that we use
synthetic dose distributions (i.e., the reference dose) as a substitute for
real clinical dose. Although these dose distributions were subject to less
quality assurance than clinical plans, they were previously shown to be of
similar quality [10]. A second limitation of this work is that the dose
prediction models were developed with the goal of optimizing the dose and DVH
scores. There may be other scoring metrics that are better suited for
developing a dose prediction model that excels in a KBP pipeline. This is a
possible direction for future research. Lastly, this work only covers a single
site and treatment modality. There is no guarantee that KBP optimization
models that are developed with this dataset can generalize to other sites or
treatment modalities.

##  5 Conclusion

In this paper, we combined the dose predictions contributed by a large
international team to several KBP optimization models, resulting in 76 KBP
pipelines. This was the largest international effort to date on KBP pipeline
evaluation. We found that the best performing pipeline significantly
outperformed the baseline approaches. In the interest of reproducibility, our
data and code is freely available.

##  6 Acknowledgments

Aaron Babier is with Department of Mechanical and Industrial Engineering,
University of Toronto, Toronto, ON, Canada and Vector Institute, Toronto, ON,
Canada (e-mail<EMAIL_ADDRESS>
Rafid Mahmood and Binghao Zhang are with Department of Mechanical and
Industrial Engineering, University of Toronto, Toronto, ON, Canada (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Victor G. L. Alves, Jeffrey V. Siebers, and Mumtaz H. Soomro are with
Department of Radiation Oncology, University of Virginia Health System,
Charlottesville, VA, United States (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
Ana Maria Barragán-Montero is with Department of Molecular Imaging Radiation
Oncology, UCLouvain, Louvain-la-Neuve, Belgium (e-mail:
<EMAIL_ADDRESS>
Joel Beaudry is with Department of Radiation Oncology, Memorial Sloan
Kettering Cancer Center, New York, NY, United States (e-mail:
<EMAIL_ADDRESS>
Carlos E. Cardenas is with Department of Radiation Oncology, The University of
Alabama at Birmingham, Birmingham, AL, United States (e-mail:
<EMAIL_ADDRESS>
Yankui Chang and Zhao Peng are with Department of Engineering and Applied
Physics, University of Science and Technology of China, Hefei, China (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Zijie Chen and Enpei Wang are with Shenying Medical Technology Co., Ltd.,
Shenzhen, Guangdong, China (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>
Jaehee Chun is with Department of Radiation Oncology, Yonsei University,
Seoul, Korea (e-mail<EMAIL_ADDRESS>
Kelly Diaz, Harold David Eraso, Jean Carlo Jimenez Giraldo, José Marrugo, José
Darío Quinto Muñoz, Juan David Rodriguez, Andrés Usuga Hoyos, and Carlos
Valderrama are with Department of Physics , National University of Colombia,
Medellín, Colombia (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
Erik Faustmann and Christian Ramsl are with Atominstitut, Vienna University of
Technology, Vienna, Austria (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>
Sibaji Gaj, Bingqi Guo, and Kunio Nakamura are with Department of Biomedical
Engineering, Cleveland Clinic, Cleveland, OH, United States (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
Skylar Gay, Mary Gronberg, Tucker Netherton, and Dong Joo Rhee are with
Department of Radiation Physics, The University of Texas MD Anderson Cancer
Center, Houston, TX, United States (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
Junjun He is with Department of Biomedical Engineering, Shanghai Jiao Tong
University, Shanghai, China (e-mail<EMAIL_ADDRESS>
Gerd Heilemann is with Department of Radiation Oncology, Medical University of
Vienna, Vienna, Austria (e-mail<EMAIL_ADDRESS>
Sanchit Hira is with Department of Biomedical Engineering, Johns Hopkins
University, Baltimore, MD, United States (e-mail<EMAIL_ADDRESS>
Yuliang Huang is with Department of Radiation Oncology, Peking University
Cancer Hospital and Institute, Beijing, China (e-mail<EMAIL_ADDRESS>
Fuxin Ji, Dashan Jiang, Shuolin Liu, and Qi Wu are with Department of
Electrical Engineering and Automation, Anhui University, Hefei, China (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
Hoyeon Lee is with Department of Radiation Oncology, Massachusetts General
Hospital, Boston, MA, United States (e-mail<EMAIL_ADDRESS>
Jun Lian is with Department of Radiation Oncology, University of North
Carolina at Chapel Hill, Chapel Hill, NC, United States (e-mail:
<EMAIL_ADDRESS>
Keng-Chi Liu is with Department of Medical Imaging, Taiwan AI Labs, Taipei,
Taiwan (e-mail<EMAIL_ADDRESS>
Kentaro Miki is with Department Of Biomedical and Health Sciences , Hiroshima
University, Hiroshima, Japan (e-mail<EMAIL_ADDRESS>
Dan Nguyen is with Medical Artificial Intelligence and Automation (MAIA)
Laboratory, Department of Radiation Oncology, The University of Texas
Southwestern Medical Center, Dallas, TX, United States (e-mail:
<EMAIL_ADDRESS>
Hamidreza Nourzadeh is with Department of Radiation Oncology, Thomas Jefferson
University, Philadelphia, PA, United States (e-mail:
<EMAIL_ADDRESS>
Alexander F. I. Osman is with Department of Medical Physics, Al-Neelain
University, Khartoum, Sudan (e-mail<EMAIL_ADDRESS>
Hongming Shan is with Institute of Science and Technology for Brain-inspired
Intelligence, Fudan University, Shanghai, China (e-mail<EMAIL_ADDRESS>
Kay Sun is with Studio Vodels, Atlanta, GA, United States (e-mail:
<EMAIL_ADDRESS>
Rob Verbeek is with Department Computer Science, Aalto University, Espoo,
Finland (e-mail<EMAIL_ADDRESS>
Siri Willems is with Department of Electrical Engineering, KULeuven, Leuven,
Belgium (e-mail<EMAIL_ADDRESS>
Xuanang Xu is with Department of Biomedical Engineering, Rensselaer
Polytechnic Institute, Troy, NY, United States (e-mail<EMAIL_ADDRESS>
Sen Yang is with Tencent AI Lab, Shenzhen, Guangdong, China (e-mail:
<EMAIL_ADDRESS>
Lulin Yuan is with Department of Radiation Oncology, Virginia Commonwealth
University Medical Center, Richmond, VA, United States (e-mail:
<EMAIL_ADDRESS>
Simeng Zhu is with Department of Radiation Oncology, Henry Ford Health System,
Detroit, MI, United States (e-mail<EMAIL_ADDRESS>
Lukas Zimmermann is with Faculty of Health, University of Applied Sciences
Wiener Neustadt, Wiener Neustadt, Austria and Competence Center for
Preclinical Imaging and Biomedical Engineering, University of Applied Sciences
Wiener Neustadt, Wiener Neustadt, Austria (e-mail:
<EMAIL_ADDRESS>
Kevin L. Moore is with Department of Radiation Oncology, University of
California, San Diego, La Jolla, CA, United States (e-mail:
<EMAIL_ADDRESS>
Thomas G. Purdie is with Radiation Medicine Program, UHN Princess Margaret
Cancer Centre, Toronto, ON, Canada, Department of Radiation Oncology,
University of Toronto, Toronto, ON, Canada, Techna Institute for the
Advancement of Technology for Health, Toronto, ON, Canada, and Department of
Medical Biophysics, University of Toronto, Toronto, ON, Canada (e-mail:
<EMAIL_ADDRESS>
Andrea L. McNiven is with Radiation Medicine Program, UHN Princess Margaret
Cancer Centre, Toronto, ON, Canada and Department of Radiation Oncology,
University of Toronto, Toronto, ON, Canada (e-mail<EMAIL_ADDRESS>
Timothy C. Y. Chan is with Department of Mechanical and Industrial
Engineering, University of Toronto, Toronto, ON, Canada, Vector Institute,
Toronto, ON, Canada, and Techna Institute for the Advancement of Technology
for Health, Toronto, ON, Canada (e-mail<EMAIL_ADDRESS>
## References

  * [1] K. L. Moore, “Automated radiotherapy treatment planning,” _Semin Radiat Oncol_ , vol. 29, no. 3, pp. 209–218, 2019.
  * [2] M. Cornell, R. Kaderka, S. J. Hild, X. J. Ray, J. D. Murphy, T. F. Atwood, and K. L. Moore, “Noninferiority study of automated knowledge-based planning versus human-driven optimization across multiple disease sites,” _Int J Radiat Oncol Biol Phys_ , vol. 106, no. 2, pp. 430–439, 2020.
  * [3] R. Kaderka, S. J. Hild, V. N. Bry, M. Cornell, X. J. Ray, J. D. Murphy, T. F. Atwood, and K. L. Moore, “Wide-scale clinical implementation of knowledge-based planning: An investigation of workforce efficiency, need for post-automation refinement, and data-driven model maintenance,” _Int J Radiat Oncol Biol Phys_ , vol. 111, no. 3, pp. 705–715, 11 2021.
  * [4] C. McIntosh, L. Conroy, M. C. Tjong, T. Craig, A. Bayley, C. Catton, M. Gospodarowicz, J. Helou, N. Isfahanian, V. Kong, T. Lam, S. Raman, P. Warde, P. Chung, A. Berlin, and T. G. Purdie, “Clinical integration of machine learning for curative-intent radiation treatment of patients with prostate cancer,” _Nat Med_ , vol. 27, no. 6, pp. 999–1005, 2021.
  * [5] A. Babier, R. Mahmood, A. L. McNiven, A. Diamant, and T. C. Y. Chan, “The importance of evaluating the complete automated knowledge-based planning pipeline,” _Phys Med_ , vol. 72, pp. 73–79, 2020.
  * [6] M. Hussein, B. J. M. Heijmen, D. Verellen, and A. Nisbet, “Automation in intensity modulated radiotherapy treatment planning-a review of recent innovations,” _Br J Radiol_ , vol. 91, no. 1092, p. 20180270, 2018.
  * [7] Y. Ge and Q. J. Wu, “Knowledge-based planning for intensity-modulated radiation therapy: A review of data-driven approaches,” _Med Phys_ , vol. 46, no. 6, pp. 2760–2775, 2019.
  * [8] M. Wang, Q. Zhang, S. Lam, J. Cai, and R. Yang, “A review on application of deep learning algorithms in external beam radiotherapy automated treatment planning,” _Front Oncol_ , vol. 10, p. 580919, 2020.
  * [9] S. Momin, Y. Fu, Y. Lei, J. Roper, J. D. Bradley, W. J. Curran, T. Liu, and X. Yang, “Knowledge-based radiation treatment planning: A data-driven method survey,” _J Appl Clin Med Phys_ , vol. 22, no. 8, pp. 16–44, Aug 2021.
  * [10] A. Babier, B. Zhang, R. Mahmood, K. L. Moore, T. G. Purdie, A. L. McNiven, and T. C. Y. Chan, “OpenKBP: The open-access knowledge-based planning grand challenge and dataset,” _Med Phys_ , vol. 48, no. 9, pp. 5549–5561, 2021\.
  * [11] D. Craft, M. Bangert, T. Long, D. Papp, and J. Unkelbach, “Shared data for intensity modulated radiation therapy (IMRT) optimization research: the CORT dataset,” _Gigascience_ , vol. 3, no. 1, p. 37, 2014.
  * [12] S. Breedveld and B. Heijmen, “Data for TROTS - the radiotherapy optimisation test set,” _Data Brief_ , vol. 12, pp. 143–149, 2017.
  * [13] J. J. Boutilier, T. Craig, M. B. Sharpe, and T. C. Y. Chan, “Sample size requirements for knowledge-based treatment planning,” _Med Phys_ , vol. 43, no. 3, pp. 1212–21, 2016.
  * [14] C. McIntosh, M. Welch, A. McNiven, D. A. Jaffray, and T. G. Purdie, “Fully automated treatment planning for head and neck radiotherapy using a voxel-based dose prediction and dose mimicking method,” _Phys Med Biol_ , vol. 62, no. 15, pp. 5926–5944, 2017.
  * [15] J. Fan, J. Wang, Z. Chen, C. Hu, Z. Zhang, and W. Hu, “Automatic treatment planning based on three-dimensional dose distribution predicted from deep learning technique,” _Med Phys_ , vol. 46, no. 1, pp. 370–381, Jan 2019\.
  * [16] P. Bai, X. Weng, K. Quan, J. Chen, Y. Dai, Y. Xu, F. Lin, J. Zhong, T. Wu, and C. Chen, “A knowledge-based intensity-modulated radiation therapy treatment planning technique for locally advanced nasopharyngeal carcinoma radiotherapy,” _Radiat Oncol_ , vol. 15, no. 1, p. 188, 2020.
  * [17] G. Wortel, D. Eekhout, E. Lamers, R. van der Bel, K. Kiers, T. Wiersma, T. Janssen, and E. Damen, “Characterization of automatic treatment planning approaches in radiotherapy,” _Physics and Imaging in Radiation Oncology_ , vol. 19, pp. 60–65, 2021.
  * [18] N. Li, R. Carmona, I. Sirak, L. Kasaova, D. Followill, J. Michalski, W. Bosch, W. Straube, L. K. Mell, and K. L. Moore, “Highly efficient training, refinement, and validation of a knowledge-based planning quality-control system for radiation therapy clinical trials,” _Int J Radiat Oncol Biol Phys_ , vol. 97, no. 1, pp. 164–172, 2017.
  * [19] A. Babier, J. J. Boutilier, M. B. Sharpe, A. L. McNiven, and T. C. Y. Chan, “Inverse optimization of objective function weights for treatment planning using clinical dose-volume histograms,” _Phys Med Biol_ , vol. 63, no. 10, p. 105004, 2018.
  * [20] G. Bohara, A. Sadeghnejad Barkousaraie, S. Jiang, and D. Nguyen, “Using deep learning to predict beam-tunable pareto optimal dose distribution for intensity-modulated radiation therapy,” _Med Phys_ , vol. 47, no. 9, pp. 3898–3912, 2020.
  * [21] T. Zhang, R. Bokrantz, and J. Olsson, “Probabilistic pareto plan generation for semiautomated multicriteria radiation therapy treatment planning,” _arxiv:2110.05410_ , 2021.
  * [22] J. O. Deasy, A. I. Blanco, and V. H. Clark, “CERR: a computational environment for radiotherapy research,” _Med Phys_ , vol. 30, no. 5, pp. 979–85, 2003.
  * [23] O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” _Proceedings of MICCAI_ , vol. 9351, pp. 234–241, 2015.
  * [24] F. Milletari, N. Navab, and S. Ahmadi, “V-net: Fully convolutional neural networks for volumetric medical image segmentation,” in _Fourth International Conference on 3D Vision, 3DV 2016, Stanford, CA, USA, October 25-28, 2016_. IEEE Computer Society, 2016, pp. 565–571.
  * [25] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” _arXiv:1611.07004_ , 2017.
  * [26] D. Nguyen, A. Sadeghnejad Barkousaraie, G. Bohara, A. Balagopal, R. McBeth, M.-H. Lin, and S. Jiang, “A comparison of monte carlo dropout and bootstrap aggregation on the performance and uncertainty estimation in radiation therapy dose prediction with deep learning neural networks,” _Phys Med Biol_ , vol. 66, no. 5, p. 054002, 2021.
  * [27] L. Zimmermann, E. Faustmann, C. Ramsl, D. Georg, and G. Heilemann, “Technical note: Dose prediction for radiation therapy using feature-based losses and one cycle learning,” _Med Phys_ , vol. 48, no. 9, pp. 5562–5566, Sep 2021\.
  * [28] M. P. Gronberg, S. S. Gay, T. J. Netherton, D. J. Rhee, L. E. Court, and C. E. Cardenas, “Technical note: Dose prediction for head and neck radiotherapy using a three-dimensional dense dilated U-Net architecture,” _Med Phys_ , vol. 48, no. 9, pp. 5567–5573, Sep 2021.
  * [29] S. Liu, J. Zhang, T. Li, H. Yan, and J. Liu, “Technical note: A cascade 3D U-Net for dose prediction in radiotherapy,” _Med Phys_ , vol. 48, no. 9, pp. 5574–5582, Sep 2021.
  * [30] D. Craft, P. Suss, and T. Bortfeld, “The tradeoff between treatment plan quality and required number of monitor units in intensity-modulated radiotherapy,” _Int J Radiat Oncol Biol Phys_ , vol. 67, no. 5, pp. 1596–605, 2007.
  * [31] D. Bertsimas and J. N. Tsitsiklis, “Introduction to linear optimization,” _Athena Scientific_ , 1997.
  * [32] H. P. Benson, “Existence of efficient solutions for vector maximization problems,” _Journal of Optimization Theory and Applications_ , vol. 26, no. 4, pp. 569–580, 1978.
  * [33] T. C. Y. Chan, T. Craig, T. Lee, and M. B. Sharpe, “Generalized inverse multiobjective optimization with application to cancer therapy,” _Oper Res_ , vol. 62, no. 3, pp. 680–95, 2014.
  * [34] B. J. Heil, M. M. Hoffman, F. Markowetz, S.-I. Lee, C. S. Greene, and S. C. Hicks, “Reproducibility standards for machine learning in the life sciences,” _Nat Methods_ , vol. 18, no. 10, pp. 1132–1135, 2021.
  * [35] A. S. Ahuja, “The impact of artificial intelligence in medicine on the future role of the physician,” _PeerJ_ , vol. 7, p. e7702, 2019.