# Deep Reinforcement Learning for Beam Angle Optimization of Intensity-
Modulated Radiation Therapy

Peng Bao  Gong Wang  Ruijie Yang  Bin Dong This work was supported by the
National Natural Science Foundation of China under Grant 12090022, 11831002,
71704023, Beijing Natural Science Foundation under Grant Z180001, the National
Key Research and Development Program under Grant 2020YFE0202500, and Beijing
Municipal Commission of Science and Technology Collaborative Innovation
Project under Grant Z221100003522028. (Corresponding authors: Ruijie Yang, Bin
Dong.)Peng Bao is with the Center for Data Science, Peking University, and the
Department of Radiation Oncology, Peking University Third Hospital, Beijing,
China. (e-mail: pengbao7598@gmail.com)Gong Wang, Ruijie Yang is with the
Department of Radiation Oncology, Peking University Third Hospital, Beijing,
China. (e-mail<EMAIL_ADDRESS>ruijyang@yahoo.com)Bin Dong is with
Beijing International Center for Mathematical Research & Center for Machine
Learning Research & National Biomedical Imaging Center, Peking University,
Beijing, China. (e-mail: dongbin@math.pku.edu.cn).

###### Abstract

Objective: Intensity-modulated radiation therapy (IMRT) beam angle
optimization (BAO) is a challenging combinatorial optimization problem that is
NP-hard. In this study, we aim to develop a personalized BAO algorithm for
IMRT that improves the quality of the final treatment. Methods: To improve the
quality of IMRT treatment planning, we propose a deep reinforcement learning
(DRL)-based approach for IMRT BAO. We consider the task as a sequential
decision-making problem and formulate it as a Markov Decision Process. To
facilitate the training process, a 3D-Unet is designed to predict the dose
distribution for the different number of beam angles, ranging from 1 to 9, to
simulate the IMRT environment. By leveraging the simulation model, double
deep-Q network (DDQN) and proximal policy optimization (PPO) are used to train
agents to select the personalized beam angle sequentially within a few
seconds. Results: The treatment plans with beam angles selected by DRL
outperform those with clinically used evenly distributed beam angles. For
DDQN, the overall average improvement of the CIs is 0.027, 0.032, and 0.03 for
5, 7, and 9 beam angles respectively. For PPO, the overall average improvement
of CIs is 0.045, 0.051, and 0.025 for 5, 7, and 9 beam angles respectively.
Conclusion: The proposed DRL-based beam angle selection strategy can generate
personalized beam angles within a few seconds, and the resulting treatment
plan is superior to that obtained using evenly distributed angles.
Significance: A fast and automated personalized beam angle selection approach
is been proposed for IMRT BAO.

Intensity-modulated radiation therapy, beam angle optimization, Markov
decision process, deep reinforcement learning

##  1 Introduction

Cancer is the second leading cause of death worldwide. According to estimates
from World Health Organization (WHO), an estimated 19.3 million new cancer
cases and almost 10 million cancer deaths occurred in 2020 [1]. Therefore, it
is significant to improve cancer treatment. Radiotherapy is a primary way to
treat cancer and is used in more than half of cancer treatments. It uses high
doses of radiation to kill cancerous cells. In this paper, we focus on
Intensity-modulated radiation therapy (IMRT), a primary technique for
radiotherapy.

IMRT treatment planning typically involves three steps: 1) tuning the hyper-
parameters for each patient, including optimization objectives and beam
angles, etc., 2) optimizing the fluence map corresponding to each beam angle,
known as _fluence map optimization_ (FMO), 3) sequencing the multileaf
collimator to generate the deliverable treatment plan. The selection of hyper-
parameters is crucial, as it directly affects the quality of the treatment
plan and ultimately the effectiveness of the treatment [2]. However, finding
the optimal combinations of hyper-parameters for IMRT treatment planning is a
challenging and time-consuming task. In current clinical practice, these
hyper-parameters are usually obtained by a lot of trial-and-error by the
experienced medical physicist, which is time-consuming and makes the quality
of the IMRT treatment plan limited by the expertise of the medical physicist.
Therefore, there is a significant need to develop an automatic approach to
determine the hyper-parameters to generate high-quality treatment plans [3].

Numerous studies have been conducted to address the issue of determining the
optimal hyper-parameters for IMRT treatment planning. For optimization
objectives, one of the most used approaches was to optimize the fluence map
with a predefined set of optimization objectives [4]. Additionally, heuristic
and statistical approaches have also been utilized to find better optimization
objectives [5, 6, 7]. More recently, deep reinforcement learning (DRL) has
been utilized to tune the optimization objectives based on the current state
to mimic the decision-making process of a human planner [8, 9]. Specifically,
the DRL agent takes the dose-to-volume histogram (DVH) of a treatment plan as
the state and determines the action of a specific parameter to improve the
plan quality. However, this approach requires a separate neural network for
each parameter of optimization objectives, which can result in a large number
of networks and is impractical for clinical use. To address this issue, the
hierarchical DRL has been introduced to break down the decision-making process
into three parts, including the structure, the parameter of the selected
structure, and the action to that parameter [10].

In addition to choosing the optimization objectives, beam angle optimization
(BAO) has also been studied extensively [11, 12, 13, 14, 15, 16, 17, 18, 19,
20, 21, 22, 14, 23, 24, 25, 26, 27]. Given the optimization objectives, the
BAO problem can be viewed as a combinatorial optimization problem that aims to
find the optimal set of beam angles of a given size among all possible angles
for the best treatment plan quality. One way to obtain the optimal set of beam
angles is to enumerate all possible beam angle combinations, but it is
impossible to apply this approach in practice because of the large number of
possible combinations. For example, selecting the beam set with 5 beam angles
among 360 candidate beam angles results in more than 4.9$e$10 possible
combinations, and it will take several minutes to solve the FMO problem to
evaluate the plan quality for a given combination of beam angles. As a result,
heuristic approaches such as genetic algorithms [11, 12, 13, 14] and simulated
annealing algorithms [15, 16, 17] have been applied to search for the optimal
combination. However, most of these methods usually take a long time because
of the huge search space and the global optimality of these methods cannot be
guaranteed. Besides, these methods always solve the FMO problem to evaluate
corresponding treatment plans which is extremely time-consuming. Instead of
solving the FMO problem, some approaches use predetermined criteria to rank
the possible beam angles and select the beam angles with the highest scores
[18, 19, 20, 21, 22]. Although these methods are computationally efficient,
the optimality of the selected beam angles cannot be guaranteed since the
predetermined criteria are based on the single beam and do not adequately
consider the interplay between multiple beams. More recently, deep learning
approaches have been applied to address the BAO problem. For instance, some
works use a deep neural network (DNN) as a surrogate model to quickly estimate
the true objective function value of the FMO problem [14, 23]. Moreover, [24]
uses a pre-trained DNN to estimate the beams’ fitness values to guide the
traversal of branches of the Monte Carlo decision tree to select the beam
angle. These methods can reduce the time required to evaluate the treatment
plan quality and provide more accurate estimations.

Inspired by the successful application of the DRL algorithm in solving the
combinatorial optimization problem [28], such as the travelling salesman
problem (TSP) [29, 30], vehicle routing problem (VRP) [31, 32], minimum vertex
cover problem [30], scanning strategy [33, 34], etc., in this work, we propose
to use the DRL algorithms to solve the BAO problem. The proposed method can
select the beam angles sequentially in just a few seconds based on the
patient’s anatomy information, the selected beam angles, and the DVH of the
current treatment plan. Specifically, we first formulate the BAO problem as a
sequential decision problem and design the corresponding Markov Decision
Process (MDP). Then we use DRL algorithms, including double deep-Q network
(DDQN) [35] and proximal policy optimization (PPO) [36], to train the RL agent
to select the beam angles sequentially. Additionally, to obtain the next state
and accelerate training, we design a neural network that can predict the dose
distribution for different beam settings to simulate the IMRT environment.

The structure of this paper is organized as follows: in Section 2, we provide
a brief review on IMRT treatment planning and reinforcement learning. Then, in
Section 3, we elaborate on the MDP for IMRT BAO and IMRT environment
simulation model. Next, we present the experimental setup and representative
results in Section 4. Finally, in Section 5, we include the conclusion.

##  2 Background

###  2.1 IMRT Treatment Planning

IMRT, as an advanced technique of radiation therapy, uses multiple beams with
modulated beam intensity to deliver a more precise radiation dose to the
tumor. Given the optimization objectives, the primary goal of IMRT treatment
planning is to determine the optimal beam angle set and the corresponding
fluence maps to achieve the best possible treatment outcome.

Figure 1: Illustration of fluence map ($10\times 10$). Each square denotes a
beamlet.

In IMRT, the total accumulated dose of the patient’s $i$-th voxel can be
calculated as follows:

$d_{i}=\sum_{b\in\mathcal{B}}\sum_{j_{b}}D_{ij_{b}}x_{j_{b}},$ (1)

where $i=\\{1,2,\dots,I\\}$ denotes the index of the voxels, $b\in\mathcal{B}$
represents the beam angle and $\mathcal{B}$ is the set of all candidate beams,
$j_{b}=\\{1,2,\dots,J_{b}\\}$ denotes the index of the beamlet at beam angle
$b$. $D$ is the dose influence matrix that takes hours to compute, and
$D_{ij_{b}}$ denotes the dose absorbed by $i$-th voxel of the patient when per
unit intensity of radiation is delivered through $j_{b}$-th beamlet at angle
$b$. $x$ is the intensity map as illustrated in Figure 1 and $x_{j_{b}}$ is
the intensity of $j_{b}$-th beamlet at angle $b$. The IMRT optimization
problem can be formulated as follows:

$\displaystyle\underset{x,\theta}{\min}$ $\displaystyle f(d)$ (2)
$\displaystyle{\rm s.t.}$ $\displaystyle
d_{i}=\sum_{b\in\mathcal{B}}\sum_{j_{b}}\theta_{b}D_{ij_{b}}x_{j_{b}}\quad\quad\forall
i=1,2,\cdots,I,$ $\displaystyle x\geq 0,\ \ \theta_{b}\in\\{0,1\\},\ \
\sum_{b\in\mathcal{B}}\theta_{b}\leq B,$

where $\theta_{b}\in\\{0,1\\}$ indicates whether the $b$-th angle is selected
or not, $B$ is the max number of beam angles to be selected. $f(\cdot)$
denotes a function that quantifies the quality of the treatment plan. There
are lots of ways to define $f(\cdot)$, such as linear model[37], mixed integer
model [38], and multi-objective model [39]. In this paper, we consider the
following nonlinear function:

$f(d)=\sum_{o\in\mathcal{O}}\left(\alpha_{o}\left\|d_{i\in\mathcal{V}_{o}}-\lambda_{o}p\right\|_{+}^{2}+\beta_{o}\left\|d_{i\in\mathcal{V}_{o}}-\lambda_{o}p\right\|_{-}^{2}\right),$
(3)

where $\mathcal{O}$ denotes the set of planning target volume (PTV) and
organs-at-risk (OARs), $\alpha_{o},\beta_{o},\lambda_{o}$ denotes the
optimization objective of the organ $o$ which are decided manually by the
medical physicist. $\mathcal{V}_{o}$ denotes the set of the voxels of the
organ $o$, and $p$ is the prescription dose. $\left\|\cdot\right\|_{+}$ and
$\left\|\cdot\right\|_{-}$ are $l_{2}$-norms that only compute the positive
and negative elements respectively.

In current clinical practice, the beam angles are given by the human planner.
Given a set of beam angles, the FMO problem can be formulated as follows:

$\displaystyle\underset{x}{\min}$ $\displaystyle f(d)$ (4) $\displaystyle{\rm
s.t.}$ $\displaystyle d_{i}=\sum_{b\in\mathcal{B}}\sum_{j}D_{ij}^{b}x_{j}^{b}\
\ {\rm for\ all\ }i=1,2,\cdots,I,$ $\displaystyle x\geq 0.$

Since it is time-consuming to solve (4), in this paper, we propose to use a
neural network to directly approximate the corresponding dose distribution
instead of solving the FMO problem. As for beam angle optimization, the
challenging combinatorial problem, we consider using the DRL algorithm to
solve it based on the patient’s anatomy information and the current treatment
plan.

###  2.2 Reinforcement Learning

A sequential decision problem can be described as a Markov Decision Process
(MDP), which is a tuple $\left(\mathcal{S,A},r,\mathbb{P},\gamma\right)$ [40].
Here, $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space,
$r:\mathcal{S\times A}\to\mathcal{R}$ is the reward function,
$\mathbb{P}:\mathcal{S\times A\times S}\to[0,1]$ is the transition probability
of the environment, and $\gamma$ is the discount factor. A policy
$\pi:\mathcal{S\times A}\to[0,1]$ is a probability distribution of actions
$\mathcal{A}$ over states $\mathcal{S}$. Define a trajectory
$\tau=(s_{0},a_{0},r_{0},s_{1},a_{1},r_{1},\dots)$ is the sequence of states,
actions, and rewards, the discounted accumulated reward of a trajectory $\tau$
is defined as $G(\tau)=\sum_{t=0}^{\infty}\gamma^{t}r_{t}$.

Given a policy $\pi$, the state-value function $V^{\pi}(s)$ is defined as the
expected return starting from state $s$ and following policy $\pi$:

$V^{\pi}(s)=\mathbb{E}_{\tau}[G(\tau)|s,\pi].$ (5)

Similar to the state-value function $V^{\pi}(s)$, the state-action-value
function $Q^{\pi}(s,a)$ is defined as the expected return starting from state
$s$, taking action $a$, and then following policy $\pi$:

$Q^{\pi}(s,a)=\mathbb{E}_{\tau}[G(\tau)|s,a,\pi].$ (6)

Given an MDP, the RL agent tries to find the optimal policy $\pi^{*}$ that
maximizes the discounted accumulated reward. The algorithms to find the
optimal policy $\pi^{*}$ can be divided into two categories: value-based
methods and policy gradient methods.

Value-based methods, such as Deep Q-Network (DQN) [41] use a parameterized
neural network $Q(s,a;\psi)$ to approximate the optimal state-action-value
function $Q^{*}(s,a)$ and optimize the neural network by minimizing the
following loss function:

$\mathcal{L}(\psi)=\mathbb{E}_{s,a,r,s^{\prime}}\left[\left(Q(s,a;\psi)-(r+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime};\psi^{\prime}))\right)^{2}\right],$
(7)

where $\psi^{\prime}$ are the parameters used to compute the target and are
copied periodically from $\psi$. However, DQN suffers from an overestimation
problem. Double DQN (DDQN) [35] combines double q-learning [42] and DQN to
alleviate this problem, and has the following loss function:

$\displaystyle\mathcal{L}(\psi)=$
$\displaystyle\mathbb{E}_{s,a,r,s^{\prime}}[(Q(s,a;\psi)$ (8)
$\displaystyle-(r+\gamma\max_{a^{\prime}}Q(s^{\prime},\underset{a^{\prime}}{\rm
arg\ max}\ Q(s^{\prime},a^{\prime};\psi);\psi^{\prime})))^{2}].$

With the approximated optimal Q function, the policy can be obtained by
$\pi(s)=\underset{a}{\rm arg\ max}\ Q(s,a;\psi)$.

In contrast to value-based methods, policy gradient methods directly optimize
a parameterized policy $\pi(a|s;\psi)$ by gradient ascent on
$\mathbb{E}_{\tau}[G(\tau)]$. We use ${\pi_{\psi}}$ to denotes $\pi(a|s;\psi)$
for simplicity. Proximal policy optimization (PPO) [36] is one of the state-
of-the-art policy gradient methods that updates the policy in a proximal
region to address the high variance problem in the vanilla policy gradient.
Specifically, defining the advantage function as
$A^{\pi_{\psi}}(s,a)=Q^{\pi_{\psi}}(s,a)-V^{\pi_{\psi}}(s)$. Given an old
policy $\pi_{\psi_{old}}$, the policy ratio between the new policy and old
policy is $q(\psi)=\frac{\pi_{\psi}}{\pi_{\psi_{old}}}$. PPO optimizes
$\pi_{\psi}$ by maximizing the following objective:

$\displaystyle\mathcal{L}(\psi)=$ (9)
$\displaystyle\mathbb{E}_{s,a\sim\pi_{\psi_{old}}}\left[\min(q(\psi)A^{\pi_{\psi_{old}}},{\rm
clip}(q(\psi),1-\epsilon,1+\epsilon)A^{\pi_{\psi_{old}}})\right],$

where ${\rm clip}(q(\psi),1-\epsilon,1+\epsilon)$ restricts the policy ratio
$q(\psi)$ to $[1-\epsilon,1+\epsilon]$.

##  3 Method

To improve the quality of IMRT treatment planning, we propose a DRL-based
approach to BAO, where the task is regarded as a sequential decision-making
problem and is formulated as an MDP. To facilitate the training process and
obtain the next state, we design a 3D-Unet to simulate the IMRT environment.
The overall pipeline of our approach is illustrated in Figure 2. By leveraging
the simulation model, we train agents using DDQN and PPO to select the beam
angle sequentially, given the patient’s anatomic structure and the current
treatment plan.

Figure 2: The overall pipeline of the proposed method.

###  3.1 MDP Formulation of IMRT BAO

IMRT BAO problem is a combinatorial optimization problem that can be
formulated as a sequential decision-making problem and effectively solved
using DRL. In this regard, we need to select one beam angle based on the
current state at each time step. Given the patient’s segmentation masks $m$,
the total number of all possible angles $N$, the MDP formulation of the BAO
problem consists of the following components:

  * •

The state is a tuple $s_{t}=(\theta_{t},dvh_{t},l_{t})$. $\theta_{t}$ records
the beam angles that are already selected at time step $t$. It is a binary
vector with the size of $N$. The value at each entry equals 1 if the
corresponding beam angle is selected. In this paper, we set $N$ to 180, which
indicates the total number of candidate beam angles at 2-degree resolution.
$dvh_{t}$ denotes the DVH of the current treatment plan based on the selected
beam angles. $l_{t}$ encodes the latent features of the patient’s segmentation
mask $m$ and the selected beam angles $\theta_{t-1}$.

  * •

The action $a_{t}$ is a one-hot vector with the size of $N$, which represents
the beam angle that we choose at time step $t$.

  * •

The reward is calculated as $r_{t}=CI_{t}-CI_{t-1}$, where $CI$ denotes the
conformity index that can evaluate the quality of the current treatment plan
to a certain degree [43]. Specifically, $CI$ can be calculated as follows:

$CI=\frac{V_{PTV,ref}^{2}}{V_{PTV}V_{ref}},$ (10)

where $V_{PTV}$ denotes the number of voxels of PTV, $V_{ref}$ denotes the
number of voxels that received reference dose, and $V_{PTV,ref}$ is the number
of voxels of PTV that received reference dose. $\frac{V_{PTV,ref}}{V_{PTV}}$
and $\frac{V_{PTV,ref}}{V_{ref}}$ can describe the PTV coverage rate and the
OAR sparing rate respectively. $CI\in[0,1]$, the closer $CI$ is to 1, the more
conformal the treatment plan is.

  * •

The transition model $\mathbb{P}$ is detailed as follows: given the state
$s_{t}$ and $a_{t}$, each component of next state $s_{t+1}$ can be calculated
as follows:

    * –

The new beam angles set $\theta_{t+1}=\theta_{t}+a_{t}$. The MDP terminates
once the number of selected beam angles reaches max number $B$.

    * –

The new DVH $dvh_{t+1}$ is calculated according to the dose distribution of
the current treatment plan. However, to obtain the dose distribution, the FMO
problem needs to be solved first, which is time-consuming. Therefore, we
design a 3D-Unet to predict the dose distribution for different beam settings,
which is detailed in Section 3.2.

    * –

The new feature $l_{t+1}$ is the latent space of the 3D-Unet that tasks the
patient’s segmentation mask $m$ and beam angle set $\theta_{t}$ as inputs.

(a) Beam extractor. (b) DVH extractor. (c) Latent extractor.

(d) Q-Network. (e) PPO network. (f) Convolutional block. (g) Fully connected
block.

Figure 3: The components of policy networks.

###  3.2 3D-Unet-based IMRT Environment Simulation

As mentioned in Section 3.1, to obtain the next state, we need to design a
network that can predict the dose distribution for the different number of
beam angles. In [44], the authors propose to use ray-tracing dose
distribution, which is time-consuming to generate, as the input of the network
to represent the beam setup information. In contrast, we propose a 3D-Unet
[45] that directly takes the beam set $\theta$ and segmentation masks $m$ as
inputs, and outputs the predicted dose distribution.

As we use cervical cancer as an example in this paper, thus the segmentation
mask contains 1 PTV channel, 6 OAR channels, and 1 body channel. It is worth
noting that we use the positional encoding technique [46] to encode the beam
angle, allowing the network to fully consider the beam setting. Additionally,
we set the max number of beam angles to be selected as 9, which means we have
9 channels for beam information. We then encode each selected beam angle as a
tensor with the size of $128\times 128\times 64$. Therefore, the input of the
3D-Unet is a 4D tensor with the size of $17\times 128\times 128\times 64$,
consisting of 8 segmentation mask channels and 9 beam channels, for a total of
17 channels. The output of the 3D-Unet is the predicted dose distribution with
the size of $128\times 128\times 64$. Based on the above settings, the
proposed model can predict dose distribution for beam numbers ranging from 1
to 9. The network architecture used in this paper is similar to the one used
in [47], but with two key differences. Firstly, we add the SEblocks [48]
before the max-pooling and the transpose-convolutional layers. Secondly, we
use instance normalization [49] instead of batch normalization since we have a
small batch size. We train our 3D-Unet using the mean squared error loss
function.

###  3.3 Policy Networks

As described in Section 3.1, our proposed MDP for the IMRT BAO problem
involves three components in the state, including the selected beam vector,
the current DVH, and the latent feature of the masks and beam encoding. To
extract the features from these components, we design a feature extractor with
three separate branches for the policy network, as illustrated in Figure 3.
Specifically, for the selected beam vector $\theta_{t}$, we use a three-layer
multi-layer perceptron (MLP) to extract its features, as illustrated in Figure
3a. Similarly, for the patient’s anatomy information $l_{t}$, we also use a
three-layer MLP to extract its features, as shown in Figure 3c. For the
current DVH $dvh_{t}$, we use several 1D convolutional layers and MLPs to
extract its features, as described in Figure 3b. The outputs of these three
branches are then concatenated to form a single feature vector. Finally, for
DDQN, the combined feature is sent to a three-layer MLP to predict the Q
function, as illustrated in Figure 3d. For PPO, the combined feature is first
sent to a shared two-layer MLP and then the output is sent to the two separate
three-layer MLPs to predict the value function and the policy, respectively,
as illustrated in Figure 3e.

##  4 Experiments

To demonstrate the performance of the proposed pipeline for IMRT BAO, we
conduct two kinds of experiments, including IMRT environment simulation and
IMRT BAO. Sixty-seven cervical cancer patients originally treated with
radiotherapy in our institute are included. Fifty cases are randomly selected
as the training data, and the remaining 17 cases are as the testing data. The
anatomy information consists of 8 masks, including PTV, bladder, rectum, left
and right femoral heads, colon, small intestine, and body. The prescription
dose is 50.4 Gy. The patient mask and the dose distribution are down-sampling
to $128\times 128\times 64$. We use several commonly used metrics in the
clinic to evaluate the treatment plan, including CI, the dose to 95% of the
target (D95), the volume receiving a dose at least 45 Gy (V45).

###  4.1 IMRT Environment Simulation

(a) One beam angle.

(b) Two beam angles.

(c) Three beam angles.

(d) Four beam angles.

(e) Five beam angles.

(f) Six beam angles.

(g) Seven beam angles.

(h) Eight beam angles.

(i) Nine beam angles.

Figure 4: Results of IMRT environment simulation. In each sub-figure, the Label denotes the ground truth and the Prediction denotes the output of the 3D-Unet. | Mean absolute error in percentages (%)
---|---
Num. Angles | PTV D95 | CI | Rectum V45 | Bladder V45 | Colon V45 | Small Intestine V45
1 | 4.41$\pm$2.66 | 0.71$\pm$0.56 | 7.56$\pm$8.78 | 5.45$\pm$6.81 | 1.20$\pm$1.15 | 0.88$\pm$1.10
2 | 2.94$\pm$2.27 | 2.05$\pm$1.67 | 9.38$\pm$7.41 | 9.25$\pm$8.32 | 1.94$\pm$2.14 | 2.50$\pm$2.80
3 | 1.59$\pm$1.38 | 2.71$\pm$2.37 | 6.67$\pm$5.23 | 6.13$\pm$5.96 | 2.62$\pm$2.66 | 2.97$\pm$2.74
4 | 0.85$\pm$0.91 | 4.65$\pm$3.62 | 5.05$\pm$4.04 | 4.27$\pm$4.24 | 2.28$\pm$2.76 | 3.36$\pm$3.60
5 | 0.61$\pm$0.58 | 5.09$\pm$3.66 | 4.51$\pm$3.78 | 3.49$\pm$3.21 | 2.10$\pm$2.81 | 2.98$\pm$3.56
6 | 0.47$\pm$0.37 | 5.74$\pm$3.83 | 4.17$\pm$3.41 | 3.11$\pm$2.75 | 1.74$\pm$2.08 | 2.42$\pm$2.95
7 | 0.49$\pm$0.37 | 6.30$\pm$5.97 | 3.96$\pm$2.98 | 3.06$\pm$2.53 | 1.75$\pm$3.05 | 3.04$\pm$5.46
8 | 0.54$\pm$0.42 | 6.00$\pm$6.28 | 3.56$\pm$2.73 | 2.81$\pm$2.71 | 1.70$\pm$3.36 | 2.90$\pm$5.32
9 | 0.53$\pm$0.41 | 5.44$\pm$6.64 | 3.43$\pm$2.81 | 2.78$\pm$2.54 | 1.74$\pm$3.37 | 2.67$\pm$5.48
Overall | 1.38$\pm$1.87 | 4.30$\pm$4.72 | 5.37$\pm$5.37 | 4.48$\pm$5.20 | 1.90$\pm$2.71 | 2.64$\pm$3.99
Table 1: The mean absolute error and its standard deviation (mean$\pm$std) for
clinically relevant metrics on the testing set (average results for 5 folds).
All results are expressed in percentages. Both overall results and the results
of the different number of beam angles are reported.

####  4.1.1 Experiment Setup

In this section, we conduct an experiment to demonstrate that the proposed
IMRT simulation model can predict dose distribution for different beam
settings. Specifically, we use an open-source toolkit MatRad [50] to generate
datasets for training, validation, and testing. For each case, we randomly
generate 80 plans for each number of the beam setting, ranging from 1-9, which
means 720 (80$\times$9) plans per case, resulting in a total of 48240
(720$\times$67) plans. The possible angles are all integers in
$[0^{\circ},360^{\circ})$ with the resolution of $2^{\circ}$, resulting in 180
possible angles. Five-fold cross-validation is used to evaluate the model
performance. The proposed 3D-Unet is trained with six graphics processing
units NVIDIA TITAN V. And We use Adam [51] to optimize the parameters of the
3D-Unet with an initial learning rate of 0.0001. The learning rate is
scheduled according to the cosine annealing method. After training, we test
the learned prediction models on 12240 plans of 17 testing cases.

####  4.1.2 Results

In Figure 4, we visualize some prediction results. We can find that the
prediction results are similar to the label data, which demonstrates that the
proposed 3D-Unet can capture the beam angle information for the model input
and predict the dose distribution for different beam settings. Furthermore,
Table 1 reports the mean absolute error (MAE) of some commonly used metrics in
the clinic to evaluate the model prediction. The overall MAEs are less than 5%
for most metrics, except Rectum, which is slightly bigger than 5%. In
addition, we can notice that the error is relatively large only when the
number of angles is small. With this prediction model, we can predict the dose
distribution for the different number of beam angles less than 1s. Besides, we
can make predictions in parallel which significantly reduces the time to train
the RL agent.

###  4.2 IMRT Beam Angle Optimization

| Num. Angles
---|---
Method | 5 | 7 | 9
DDQN | 0.7702$\pm$0.0225 | 0.7956$\pm$0.0269 | 0.8221$\pm$0.0218
PPO | 0.7829$\pm$0.0410 | 0.8039$\pm$0.0353 | 0.8191$\pm$0.0373
Even | 0.7449$\pm$0.0259 | 0.7780$\pm$0.0247 | 0.7979$\pm$0.0239
Table 2: The mean CI and its standard deviation (mean$\pm$std). The results are computed based on the dose distribution predicted by the environment model the prescription dose is 50.4 Gy. The best results for different beam numbers are shown in bold numbers.  | CI | PTV D95
---|---|---
Method | 5 | 7 | 9 | 5 | 7 | 9
DDQN | 0.6754$\pm$0.0431 | 0.7138$\pm$0.0586 | 0.7520$\pm$0.0465 | 50.0927$\pm$0.5532 | 50.5369$\pm$ 0.3381 | 50.6762$\pm$0.2249
PPO | 0.6932$\pm$0.0383 | 0.7337$\pm$0.0459 | 0.7479$\pm$0.0740 | 50.1135$\pm$0.4577 | 50.4772$\pm$0.3391 | 50.6318$\pm$0.3045
Even | 0.6482$\pm$0.0705 | 0.6823$\pm$0.0761 | 0.7220$\pm$0.0691 | 50.2062$\pm$0.3944 | 50.5985$\pm$0.2947 | 50.7103$\pm$0.2393
| Rectum V45 | Bladder V45
Method | 5 | 7 | 9 | 5 | 7 | 9
DDQN | 0.3233$\pm$0.0433 | 0.3273$\pm$0.0427 | 0.3069 $\pm$0.0536 | 0.3505$\pm$0.0281 | 0.3442$\pm$0.0331 | 0.3433$\pm$0.0358
PPO | 0.3299$\pm$0.0414 | 0.3287$\pm$0.0436 | 0.3143$\pm$0.0452 | 0.3494$\pm$0.0310 | 0.3490$\pm$0.0311 | 0.3457$\pm$0.0372
Even | 0.3405$\pm$0.0443 | 0.3303$\pm$0.0453 | 0.3117$\pm$0.0492 | 0.3538$\pm$0.0.0228 | 0.3442$\pm$0.0322 | 0.3416$\pm$0.0371
| Colon V45 | Small Intestine V45
Method | 5 | 7 | 9 | 5 | 7 | 9
DDQN | 0.3019$\pm$0.1545 | 0.2912$\pm$0.1569 | 0.2796$\pm$0.1515 | 0.1904$\pm$0.1081 | 0.1885$\pm$0.1170 | 0.1702$\pm$0.0944
PPO | 0.3086$\pm$0.1593 | 0.2924$\pm$0.1559 | 0.2778$\pm$0.1532 | 0.1971$\pm$0.0904 | 0.1721$\pm$0.0803 | 0.1693$\pm$0.0797
Even | 0.3095$\pm$0.1619 | 0.3012$\pm$0.1690 | 0.2879$\pm$0.1535 | 0.2139$\pm$0.1061 | 0.2086$\pm$0.1297 | 0.1768$\pm$0.0878
Table 3: The mean and standard deviation of metrics (mean$\pm$std). The
results are computed based on the dose distribution optimized in MatRad and
the prescription dose is 50.4 Gy. The best results for different beam numbers
are shown in bold numbers.  Figure 5: The DVH of a representative case.

(a) DDQN. CI=0.7627.

(b) PPO. CI=0.7556.

(c) Even. CI=0.6728.

Figure 6: The angle selection of IMRT BAO and its dose distribution in Figure
5.

####  4.2.1 Experiment Setup

Based on the pre-trained 3D-Unet as the IMRT environment model, in this
section, we use DDQN and PPO to train the agents to determine the beam angles
with one graphics processing unit NVIDIA TITAN V. Specifically, we set the
number of beam angles to be selected to 5, 7, 9. We test the learned policies
on 17 testing cases and compare the dose metrics of DRL-based results with
those of the evenly distributed beam angles, including CI, PTV D95, Rectum
V45, Bladder V45, Colon V45 and Small Intestine V45. The evenly distributed
beam angles start from $180^{\circ}$, resulting in [$36^{\circ}$,
$108^{\circ}$, $180^{\circ}$, $252^{\circ}$, $324^{\circ}$], [$28^{\circ}$,
$80^{\circ}$, $132^{\circ}$, $180^{\circ}$, $232^{\circ}$, $284^{\circ}$,
$336^{\circ}$], [$20^{\circ}$, $60^{\circ}$, $100^{\circ}$, $140^{\circ}$,
$180^{\circ}$, $220^{\circ}$, $260^{\circ}$, $300^{\circ}$, $340^{\circ}$] for
5, 7, and 9 beam angles respectively.

####  4.2.2 Results

As we use the CI as the reward and train agents based on the simulated
environment, we first present the overall mean and standard deviation of CI
computed based on the dose distribution predicted by the 3D-Unet in Table 2.
We can notice that both DDQN and PPO results have improvement on CI for three
different beam number settings compared with results of evenly distributed
beam angles. To further demonstrate the feasibility of the proposed pipeline,
we present the clinically relevant metrics based on the dose distribution
optimized by MatRad, which we consider as the real environment in this paper,
in Table 3. The results demonstrate that the treatment plans with beam angles
selected by DRL perform well in the real environment with regard to CI, and
the other important metrics are also comparable. This shows the potential of
the proposed DRL-based beam angle selection method in improving the quality of
IMRT treatment planning for clinical use.

To better visualize the performance of the proposed pipeline, we present the
DVH of a representative case with 7 beam angles in Figure 5. We can notice
that DRL-based plans and the evenly distributed-based plan have similar PTV
coverage. However, DRL-based plans have smaller high-dose regions for most
OARs than the evenly distributed-based plan. This suggests that the DRL-based
beam angle selection method can generate plans with improved dose sparing for
OARs while maintaining PTV coverage, which is an important factor in reducing
treatment-related side effects and improving treatment outcomes. We further
present the selected angles and corresponding dose distribution of the case
shown in Figure 6.

##  5 Conclusion

IMRT BAO is a combinatorial optimization problem and is NP-hard which is
difficult to solve. In recent years, the DRL has been proven useful in solving
many combinatorial optimization problems, such as TSP and VRP. In this paper,
we proposed to use DRL algorithms to learn a personalized beam angle selection
strategy for IMRT based on the current treatment plan and the patient’s
anatomy information. Specifically, we formulated the IMRT BAO problem as an
MDP and solved it using DDQN and PPO. To obtain the next state and accelerate
the training process, we also designed a 3D-Unet to simulate the IMRT
environment. This 3D-Unet can predict the dose distribution for the different
number of beam angles within 1 s. With the IMRT environment prediction model,
the time spent to train the RL agent is significantly reduced. After training,
the agent can determine the personalized beam angles within a few seconds.
Experimental results on a dataset of cervical cancer patients showed that the
proposed DRL approach can improve the plan quality compared to the strategy
used in clinical practice. Specifically, the proposed approach can generate
plans with improved dose-sparing for OARs while maintaining PTV coverage.
Furthermore, although this paper takes cervical cancer as an example, the
proposed method can be also used for other sites. In our future work, we will
explore more complex neural network models, such as vision transformer [52,
53], to further improve the accuracy of the environment model prediction.
Furthermore, we may also consider model-based RL algorithms.

## References

  * [1] H. Sung, J. Ferlay, R. L. Siegel, M. Laversanne, I. Soerjomataram, A. Jemal, and F. Bray, “Global cancer statistics 2020: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries,” _CA: A Cancer Journal for Clinicians_ , vol. 71, no. 3, pp. 209–249, 2021.
  * [2] A. Pugachev, J. G. Li, A. L. Boyer, S. L. Hancock, Q.-T. Le, S. S. Donaldson, and L. Xing, “Role of beam orientation optimization in intensity-modulated radiation therapy,” _International Journal of Radiation Oncology, Biology, Physics_ , vol. 50, no. 2, pp. 551–560, 2001.
  * [3] M. Wang, Q. Zhang, S. Lam, J. Cai, and R. Yang, “A review on application of deep learning algorithms in external beam radiotherapy automated treatment planning,” _Frontiers in Oncology_ , vol. 10, p. 580919, 2020.
  * [4] H. Wang, P. Dong, H. Liu, and L. Xing, “Development of an autonomous treatment planning strategy for radiation therapy with effective use of population-based prior data,” _Medical Physics_ , vol. 44, no. 2, pp. 389–396, 2017.
  * [5] T. Lee, M. Hammad, T. C. Chan, T. Craig, and M. B. Sharpe, “Predicting objective function weights from patient anatomy in prostate imrt treatment planning,” _Medical Physics_ , vol. 40, no. 12, p. 121706, 2013.
  * [6] J. J. Boutilier, T. Lee, T. Craig, M. B. Sharpe, and T. C. Chan, “Models for predicting objective function weights in prostate cancer imrt,” _Medical Physics_ , vol. 42, no. 4, pp. 1586–1595, 2015.
  * [7] N. Wahl, M. Bangert, C. P. Kamerling, P. Ziegenhein, G. H. Bol, B. W. Raaymakers, and U. Oelfke, “Physically constrained voxel-based penalty adaptation for ultra-fast imrt planning,” _Journal of Applied Clinical Medical Physics_ , vol. 17, no. 4, pp. 172–189, 2016.
  * [8] C. Shen, Y. Gonzalez, P. Klages, N. Qin, H. Jung, L. Chen, D. Nguyen, S. B. Jiang, and X. Jia, “Intelligent inverse treatment planning via deep reinforcement learning, a proof-of-principle study in high dose-rate brachytherapy for cervical cancer,” _Physics in Medicine & Biology_, vol. 64, no. 11, p. 115013, 2019.
  * [9] C. Shen, D. Nguyen, L. Chen, Y. Gonzalez, R. McBeth, N. Qin, S. B. Jiang, and X. Jia, “Operating a treatment planning system using a deep-reinforcement learning-based virtual treatment planner for prostate cancer intensity-modulated radiation therapy treatment planning,” _Medical Physics_ , vol. 47, no. 6, pp. 2329–2336, 2020.
  * [10] C. Shen, L. Chen, and X. Jia, “A hierarchical deep reinforcement learning framework for intelligent automatic treatment planning of prostate cancer intensity modulated radiation therapy,” _Physics in Medicine & Biology_, vol. 66, no. 13, p. 134002, 2021.
  * [11] Q. Hou, J. Wang, Y. Chen, and J. M. Galvin, “Beam orientation optimization for imrt by a hybrid method of the genetic algorithm and the simulated dynamics,” _Medical Physics_ , vol. 30, no. 9, pp. 2360–2367, 2003.
  * [12] E. Schreibmann, M. Lahanas, L. Xing, and D. Baltas, “Multiobjective evolutionary optimization of the number of beams, their orientations and weights for intensity-modulated radiation therapy,” _Physics in Medicine & Biology_, vol. 49, no. 5, pp. 747–770, 2004.
  * [13] Y. Li, J. Yao, and D. Yao, “Automatic beam angle selection in imrt planning using genetic algorithm,” _Physics in Medicine & Biology_, vol. 49, no. 10, pp. 1915–1932, 2004.
  * [14] J. Dias, H. Rocha, B. Ferreira, and M. d. C. Lopes, “A genetic algorithm with neural network fitness function evaluation for imrt beam angle optimization,” _Central European Journal of Operations Research_ , vol. 22, no. 3, pp. 431–455, 2014.
  * [15] T. Bortfeld and W. Schlegel, “Optimization of beam orientations in radiation therapy: some theoretical considerations,” _Physics in Medicine & Biology_, vol. 38, no. 2, pp. 291–304, 1993.
  * [16] H.-M. Lu, H. M. Kooy, Z. H. Leber, and R. J. Ledoux, “Optimized beam planning for linear accelerator-based stereotactic radiosurgery.” _International Journal of Radiation Oncology, Biology, Physics_ , vol. 39, no. 5, pp. 1183–1189, 1997.
  * [17] D. Djajaputra, Q. Wu, Y. Wu, and R. Mohan, “Algorithm and performance of a clinical imrt beam-angle optimization system,” _Physics in Medicine & Biology_, vol. 48, no. 19, pp. 3191–3212, 2003.
  * [18] E. Woudstra and P. Storchi, “Constrained treatment planning using sequential beam selection,” _Physics in Medicine & Biology_, vol. 45, no. 8, pp. 2133–2149, 2000.
  * [19] S. Gaede, E. Wong, and H. Rasmussen, “An algorithm for systematic selection of beam directions for imrt,” _Medical Physics_ , vol. 31, no. 2, pp. 376–388, 2004.
  * [20] E. Schreibmann and L. Xing, “Dose–volume based ranking of incident beam direction and its utility in facilitating imrt beam placement,” _International Journal of Radiation Oncology, Biology, Physics_ , vol. 63, no. 2, pp. 584–593, 2005.
  * [21] E. Woudstra, B. J. Heijmen, and P. R. Storchi, “Automated selection of beam orientations and segmented intensity-modulated radiotherapy (imrt) for treatment of oesophagus tumors,” _Radiotherapy and Oncology_ , vol. 77, no. 3, pp. 254–261, 2005.
  * [22] R. Vaitheeswaran, V. Sathiya Narayanan, J. R. Bhangle, A. Nirhali, N. Kumar, S. Basu, and V. Maiya, “An algorithm for fast beam angle selection in intensity modulated radiotherapy,” _Medical Physics_ , vol. 37, no. 12, pp. 6443–6452, 2010.
  * [23] A. Sadeghnejad Barkousaraie, O. Ogunmolu, S. Jiang, and D. Nguyen, “A fast deep learning approach for beam orientation optimization for prostate cancer treated with intensity-modulated radiation therapy,” _Medical Physics_ , vol. 47, no. 3, pp. 880–897, 2020.
  * [24] A. Sadeghnejad-Barkousaraie, G. Bohara, S. Jiang, and D. Nguyen, “A reinforcement learning application of a guided monte carlo tree search algorithm for beam orientation selection in radiation therapy,” _Machine Learning: Science and Technology_ , vol. 2, no. 3, p. 035013, 2021\.
  * [25] X. Jia, C. Men, Y. Lou, and S. B. Jiang, “Beam orientation optimization for intensity modulated radiation therapy using adaptive l2, 1-minimization,” _Physics in Medicine & Biology_, vol. 56, no. 19, pp. 6205–6222, 2011.
  * [26] R. Yang, J. Dai, Y. Yang, and Y. Hu, “Beam orientation optimization for intensity-modulated radiation therapy using mixed integer programming,” _Physics in Medicine & Biology_, vol. 51, no. 15, pp. 3653–3666, 2006.
  * [27] G. Wang, H. Wang, H. Zhuang, and R. Yang, “An investigation of non-coplanar volumetric modulated radiation therapy for locally advanced unresectable pancreatic cancer using a trajectory optimization method,” _Frontiers in Oncology_ , vol. 11, p. 717634, 2021.
  * [28] N. Mazyavkina, S. Sviridov, S. Ivanov, and E. Burnaev, “Reinforcement learning for combinatorial optimization: A survey,” _Computers & Operations Research_, vol. 134, p. 105400, 2021.
  * [29] I. Bello, H. Pham, Q. V. Le, M. Norouzi, and S. Bengio, “Neural combinatorial optimization with reinforcement learning,” in _Workshop Proceedings of the 5th International Conference on Learning Representations_ , 2017.
  * [30] E. Khalil, H. Dai, Y. Zhang, B. Dilkina, and L. Song, “Learning combinatorial optimization algorithms over graphs,” in _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , 2017, p. 6351–6361.
  * [31] M. Nazari, A. Oroojlooy, L. Snyder, and M. Takác, “Reinforcement learning for solving the vehicle routing problem,” in _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , 2018, p. 9861–9871.
  * [32] W. Kool, H. Van Hoof, and M. Welling, “Attention, learn to solve routing problems!” in _International Conference on Learning Representations_ , 2019\.
  * [33] L. Pineda, S. Basu, A. Romero, R. Calandra, and M. Drozdzal, “Active mr k-space sampling with reinforcement learning,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2020, pp. 23–33.
  * [34] Z. Shen, Y. Wang, D. Wu, X. Yang, and B. Dong, “Learning to scan: A deep reinforcement learning approach for personalized scanning in ct imaging,” _Inverse Problems and Imaging_ , vol. 16, no. 1, pp. 179–195, 2022.
  * [35] H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning,” in _Proceedings of the 30th AAAI Conference on Artificial Intelligence_ , 2016, p. 2094–2100.
  * [36] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017\.
  * [37] H. E. Romeijn, R. K. Ahuja, J. F. Dempsey, A. Kumar, and J. G. Li, “A novel linear programming approach to fluence map optimization for intensity modulated radiation therapy treatment planning,” _Physics in Medicine & Biology_, vol. 48, no. 21, pp. 3521–3542, 2003.
  * [38] E. K. Lee, T. Fox, and I. Crocker, “Integer programming applied to intensity-modulated radiation therapy treatment planning,” _Annals of Operations Research_ , vol. 119, no. 1, pp. 165–181, 2003.
  * [39] D. L. Craft, T. F. Halabi, H. A. Shih, and T. R. Bortfeld, “Approximating convex pareto surfaces in multiobjective radiotherapy planning,” _Medical Physics_ , vol. 33, no. 9, pp. 3399–3407, 2006.
  * [40] R. S. Sutton and A. G. Barto, _Reinforcement learning: An introduction_. MIT press, 2018.
  * [41] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski _et al._ , “Human-level control through deep reinforcement learning,” _Nature_ , vol. 518, no. 7540, pp. 529–533, 2015.
  * [42] H. Hasselt, “Double q-learning,” in _Proceedings of the 23rd International Conference on Neural Information Processing Systems_ , 2010, p. 2613–2621.
  * [43] A. Van’t Riet, A. C. Mak, M. A. Moerland, L. H. Elders, and W. Van Der Zee, “A conformation number to quantify the degree of conformality in brachytherapy and external beam irradiation: application to the prostate,” _International Journal of Radiation Oncology, Biology, Physics_ , vol. 37, no. 3, pp. 731–736, 1997.
  * [44] A. M. Barragán-Montero, D. Nguyen, W. Lu, M.-H. Lin, R. Norouzi-Kandalan, X. Geets, E. Sterpin, and S. Jiang, “Three-dimensional dose prediction for lung imrt patients with deep neural networks: robust learning from heterogeneous beam configurations,” _Medical Physics_ , vol. 46, no. 8, pp. 3679–3691, 2019.
  * [45] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2015, pp. 234–241.
  * [46] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , 2017, p. 6000–6010.
  * [47] Z. Qilin, B. Peng, Q. Ang, J. Weijuan, J. Ping, Z. Hongqing, D. Bin, and Y. Ruijie, “The feasibility study on the generalization of deep learning dose prediction model for volumetric modulated arc therapy of cervical cancer,” _Journal of Applied Clinical Medical Physics_ , vol. 23, no. 6, p. e13583, 2022.
  * [48] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 7132–7141.
  * [49] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” _arXiv preprint arXiv:1607.08022_ , 2016\.
  * [50] H.-P. Wieser, E. Cisternas, N. Wahl, S. Ulrich, A. Stadler, H. Mescher, L.-R. Müller, T. Klinge, H. Gabrys, L. Burigo _et al._ , “Development of the open-source dose calculation and optimization toolkit matrad,” _Medical Physics_ , vol. 44, no. 6, pp. 2556–2568, 2017.
  * [51] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [52] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in _International Conference on Learning Representations_ , 2021.
  * [53] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 10 012–10 022.