# Robust Path Planning via Learning from Demonstrations for Robotic Catheters
in Deformable Environments

Zhen Li  Graduate Student Member, IEEE  Chiara Lambranzi  Di Wu  Alice Segato
Federico De Marco  Emmanuel Vander Poorten  Jenny Dankelman  and Elena De Momi
Senior Member, IEEE This work was supported by the ATLAS project. This project
has received funding from the European Union’s Horizon 2020 research and
innovation programme under the Marie Sklodowska-Curie grant agreement No
813782.Zhen Li is with the Department of Electronics, Information and
Bioengineering, Politecnico di Milano, 20133 Milan, Italy, and also with the
Department of Biomechanical Engineering, Delft University of Technology, 2628
CD Delft, The Netherlands (e-mail: zhen.li@polimi.it).Chiara Lambranzi is with
the Department of Electronics, Information and Bioengineering, Politecnico di
Milano, 20133 Milan, Italy, and also with Istituto Italiano di Tecnologia,
16163 Genova, Italy (e-mail: chiara.lambranzi@polimi.it).Di Wu is with the
Department of Mechanical Engineering, Katholieke Universiteit Leuven, 3000
Leuven, Belgium, and also with the Faculty of Mechanical, Maritime and
Materials Engineering, Delft University of Technology, 2628 CD Delft, The
Netherlands (e-mail: di.wu@kuleuven.be).Alice Segato was with the Department
of Electronics, Information and Bioengineering, Politecnico di Milano, 20133
Milan, Italy (e-mail: alice.segato@polimi.it).Federico De Marco is with Centro
Cardiologico Monzino, IRCCS, 20138 Milan, Italy (e-mail:
federico.demarco@gmail.com).Emmanuel Vander Poorten is with the Department of
Mechanical Engineering, Katholieke Universiteit Leuven, 3000 Leuven, Belgium
(e-mail: emmanuel.vanderpoorten@kuleuven.be).Jenny Dankelman is with the
Department of Biomechanical Engineering, Delft University of Technology, 2628
CD Delft, The Netherlands (e-mail: j.dankelman@tudelft.nl).Elena De Momi is
with the Department of Electronics, Information and Bioengineering,
Politecnico di Milano, 20133 Milan, Italy (e-mail: elena.demomi@polimi.it).

###### Abstract

Navigation through tortuous and deformable vessels using catheters with
limited steering capability underscores the need for reliable path planning.
State-of-the-art path planners do not fully account for the deformable nature
of the environment. This work proposes a robust path planner via a learning
from demonstrations method, named Curriculum Generative Adversarial Imitation
Learning (C-GAIL). This path planning framework takes into account the
interaction between steerable catheters and vessel walls and the deformable
property of vessels. In-silico comparative experiments show that the proposed
network achieves smaller targeting errors, and a higher success rate, compared
to a state-of-the-art approach based on GAIL. The in-vitro validation
experiments demonstrate that the path generated by the proposed C-GAIL path
planner aligns better with the actual steering capability of the pneumatic
artificial muscle-driven catheter utilized in this study. Therefore, the
proposed approach can provide enhanced support to the user in navigating the
catheter towards the target with greater precision, in contrast to the
conventional centerline-following technique. The targeting and tracking errors
are 1.26$\pm$0.55mm and 5.18$\pm$3.48mm, respectively. The proposed path
planning framework exhibits superior performance in managing uncertainty
associated with vessel deformation, thereby resulting in lower tracking
errors.

Path Planning, Steerable Catheter, Vessel Deformation, Deep Learning,
Endovascular Intervention

PCI

    Percutaneous Coronary Intervention
DFS

    Depth First Search
bi-RRT

    bi-directional Rapidly-exploring Random Tree
ACO

    Ant Colony Optimization
GA

    Genetic Algorithm
LfD

    Learning from Demonstrations
DRL

    Deep Reinforcement Learning
RL

    Reinforcement Learning
OR

    Operating Room
DRU-Net

    Deep Residual U-Net
CTA

    Computed Tomography Angiography
CCM

    Centro Cardiologico Monzino
MDCT

    Multidetector Computed Tomography
ECG

    Electrocardiogram
EM

    Electromagnetic
FoV

    Field-of-View
DSC

    Dice Similarity Coefficient
DDF

    Dense Displacement Field
ROI

    Region of Interest
RRT

    Rapidly-exploring Random Tree
STN

    Spatial Transform Network
CNN

    Convolutional Neural Network
PBD

    Position-based Dynamics
PAMs

    Pneumatic Artificial Muscles
FTL

    Follow-The-Leader
BC

    Behavioral Cloning
GAIL

    Generative Adversarial Imitation Learning
C-GAIL

    Curriculum Generative Adversarial Imitation Learning
GAN

    Generative Adversarial Network
GUI

    Graphical User Interface
BC

    Behavioral Cloning
PPO

    Proximal Policy Optimization
IVUS

    IntraVascular UltraSound
OCT

    Optical Coherence Tomography
FBG

    Fiber Bragg Grating
CTO

    Chronic Total Occlusion
CL

    Curriculum Learning
DoF

    Degree of Freedom

##  1 Introduction

Endovascular procedures are a rapidly emerging field in medicine. The number
of patients treated has constantly increased over the past few decades [1].
These procedures increase patient comfort, reduce risks, and improve outcomes
compared to traditional open surgery. However, navigation through narrow,
fragile, and deformable vessels, using traditional non-steerable catheters and
guidewires, requires considerable skill and experience [2]. Steerable
catheters and navigation guidance could potentially lower the skill that would
be required for percutaneous treatment [3]. Commercial robotic platforms can
attest to the robot-assisted trend, such as CorPath™ GRX (Corindus, Waltham,
USA), Sensei™ X and Magellan (J&J robotics, New Brunswick, USA), Amigo™
(Catheter Robotics Inc. Budd Lake, USA), R-One™ (Robocath, Rouen, France) and
Niobe™ (Stereotaxis, St. Louis, USA). In the last decades, different research
groups have focused their efforts on the development of steerable catheters
[4, 5, 6, 7]. For example, a proof-of-concept medical robotic platform,
composed of a multi-lumen catheter shaft and magnetically actuated
microcatheter, was developed in [7].

In this study, we explore the application of a novel robotic catheterization
system, as detailed in [8, 7], in the context of Percutaneous Coronary
Intervention (PCI) for Chronic Total Occlusion (CTO) treatment. This system
employs a robotic catheter, innovatively designed with one to two internal
lumens. These lumens serve the critical function of carrying microcatheters
with magnetic steering capabilities. A key operational aspect of this system
involves the precise navigation of the robotic catheter to an anchor position
proximal to the aortic root. Upon reaching this location, the magnetic
microcatheters are subsequently deployed into the coronary arteries, setting
the stage for the ensuing treatment procedures. The primary objective of this
research is to develop a safe, accurate, and robust path planner for the
robotic catheter. This path planner is essential for ensuring the robotic
catheter’s efficient navigation to the aforementioned anchor position near the
aortic root. Limited steering capability underscores the need for reliable
path planning [9]. However, the complex interaction between the steerable
catheter and vessel walls and the deformable property of the vessels makes
reliable and real-time path planning a hard problem.

This work presents a robust and accurate path planning framework to improve
risk management. Specifically, this framework can reduce the uncertainty in
vessel deformation, thereby minimizing tracking errors. The main contributions
are:

  * •

proposing a novel path planning approach, named Curriculum Generative
Adversarial Imitation Learning (C-GAIL), which outperforms existing models by
offering optimal path planning while adhering to the constraints of robotic
catheters. Notably, the C-GAIL integrates a Curriculum Learning (CL) module
and a Behavioral Cloning (BC) module, distinguishing it from models like
PPO+GAIL ([10, 11]), by enabling progressive training in complex environments
from demonstrations;

  * •

presenting a path planning framework for a motorized steerable catheter, which
uniquely considers both the deformable nature of the environment and the
dynamic movements of the target, setting it apart from existing path planning
methods;

  * •

validating the proposed path planner for a motorized steerable catheter in an
in-vitro setting using a teleoperation control strategy. These experiments
underscore the algorithm’s feasibility in generating suitable paths that align
with the actual steering capability of the catheter, further demonstrating the
advantages of the C-GAIL model over a traditional centerline-following
approach.

##  2 Related work

Table 1: State-of-the-art path planning methods for endovascular
catheterization (From 2011 to 2022)

Reference |  Method |  Algorithm |  Instrument |  Environment |  Validation
---|---|---|---|---|---
[12] Wang 2011 |  NB |  Centerline-based tree |  Shaped catheter |  Rigid |  in-vitro
[13] Zheng 2018 |  NB |  Centerline-based tree |  \- (Not specified) |  Deformable |  in-vitro
[14] Huang 2011 |  NB |  DFS |  Guidewire |  Rigid |  in-silico
[15] Qian 2019, [16] Cho 2021, [17] Schegg 2022 |  NB |  Dijkstra |  Guidewire |  Rigid |  in-vitro
[18, 19] Ravigopal 2022 |  NB |  Modified hybrid A* |  Steerable guidewire |  Deformable |  ex-vivo
[20] Fagogenis 2019 |  NB |  Wall-following |  Concentric tube robot |  Deformable |  in-vivo
[21], [22], [23] Fauser 2019 |  SB |  bi-RRT |  Catheter / Steerable guidewire |  Rigid |  in-silico
[24] Guo 2021 |  SB |  RRT |  Catheter |  Rigid |  in-silico
[25] Gao 2015 |  OB |  ACO |  Catheter |  Rigid |  in-silico
[26] Qi 2019 |  OB |  Optimal inverse kinematics |  Steerable catheter |  Rigid |  in-vitro
[27] Li 2021 |  OB |  GA |  Steerable catheter |  Rigid |  in-silico
[28], [29] Rafii-Tari 2014 |  LB |  GMM, HMM |  Shaped catheter, pre-loaded guidewire |  Rigid |  in-vitro
[30], [31], [10] Chi 2020 |  LB |  DMPs, GMMs, GAIL |  Shaped catheter, pre-loaded guidewire |  Deformable |  in-vitro
[32] Zhao 2022 |  LB |  GAN |  Guidewire |  Rigid |  in-vitro
[33] Tibebu 2014, [34] You 2019 |  LB |  Q-learning, DQN |  Steerable catheter |  Rigid |  in-vitro
[35] Behr 2019, [36] Karstensen 2020, [37] Kweon 2021 |  LB |  DQN, DDPG |  Shaped guidewire |  Rigid |  in-vitro
[38] Meng 2021 |  LB |  A3C |  Guidewire |  Rigid |  in-silico
[39] Karstensen 2022 |  LB |  DDPG |  Shaped guidewire |  Deformable |  ex-vivo
Proposed |  LB |  C-GAIL |  Steerable catheter |  Deformable |  in-vitro

Over the last decade, several path planning methods for steerable/non-
steerable catheters/guidewires have been proposed to assist clinicians. Table
1 summarizes the state-of-the-art from 2011 to 2022, in terms of path planning
methodology, type of medical instrument used, type of environment (presence of
dynamic changes), and type of validation (in-silico, in-vitro, ex-vivo, in-
vivo). In the following a very brief description of the main types of planners
is given.

###  2.1 Node-Based (NB) Methods

Node-based algorithms use an information structure to represent the
environment map. Studies [12] and [13] extracted the vessel centerline and
built an exploration tree along the centerline. The aim of this method is to
keep the tip of the instrument away from the walls. Nevertheless, path
exploration inside the information structure is not mentioned in those
studies. Graph search strategies such as Depth First Search (DFS) algorithm
[14], Dijkstra algorithm [15, 16, 17] and A* algorithm [18, 19] were employed
to generate a path solution in a tubular environment with multi-branches. For
movement in the cardiac chamber, a wall-following strategy employing haptic
vision was developed in [20] by Fagogenis _et al._ to keep a certain distance
from the heart wall.

###  2.2 Sampling-Based (SB) methods

Sampling-based methods randomly sample in the robot’s configuration space or
workspace to generate new tree vertices. Then collision-free vertices are
connected as tree edges. Fauser _et al._ [21, 23, 22] introduced a bi-
directional Rapidly-exploring Random Tree (bi-RRT) method for instruments that
follow curvature constrained trajectories in vena cava or aorta. The study in
[24] implemented an improved RRT algorithm for cerebrovascular interventions.
The expansion direction of the random tree is a compromise between the new
randomly sampled node and the target. This strategy can improve the
convergence speed of the algorithm. However, their work did not take into
account any kinematic constraints governing catheter movement.

###  2.3 Optimization-Based (OB) Methods

Path planning can be formulated as an optimization problem and solved by
numerical solvers. Gao _et al._ [25] proposed an improved Ant Colony
Optimization (ACO) algorithm to plan an optimal path that also accounted for
factors such as catheter diameter, vascular length, diameter, curvature and
torsion. Nevertheless, the high computational time with an average value of
12.32s makes it infeasible in real-time scenarios. Qi _et al._ [26] formulated
the path planning as an optimization problem under the inverse kinematics
modeling of continuum robots. However, the optimization problem is solved
locally without considering long-term cumulative costs. Li _et al._ [27]
proposed a fast path planning approach via a local Genetic Algorithm (GA)
optimization. The approach is able to account for constraints on the catheter
curvature, but the optimization algorithm is based on vessel centerlines that
are sensitive to deformations of the anatomical model.

###  2.4 Learning-Based (LB) Methods

Learning-based methods use statistical tools and machine learning algorithms
for path planning. Rafii-Tari _et al._ [28, 29] and Chi _et al._ [30, 31, 10]
proposed Learning from Demonstrations (LfD) approaches to optimize
trajectories or learn motion primitives using expert demonstrations. Zhao _et
al._ proposed a Generative Adversarial Network (GAN) framework for real-time
path planning and evaluated it in 2D-DSA images [32]. The work in [34, 35, 36,
33, 37, 38, 39] developed Reinforcement Learning (RL) approaches to predict a
sequence of actions to reach a target. LfD methods based on Generative
Adversarial Imitation Learning (GAIL) were adapted into other medical
scenarios [40, 11] because of their ability to compromise between learning the
distribution and ensuring the generalization of trajectories.

###  2.5 Limitations

Current approaches lack planning capabilities that actually take into account
the deformable nature of the environment, even while those studies were
verified in a soft environment [13, 20, 30, 31, 10, 39, 19]. Moreover, most of
the studies that looked at deformable environments were actually developed for
passive, non-steerable instruments [13, 30, 31, 10, 39]. The wall-following
algorithm [20] was only tested on a short path along the inner heart-wall.
This approach could be considered efficient if there are few feasible routes
to reach the target. However, in scenarios where there are multiple feasible
routes, the solution provided by a wall-following algorithm cannot ensure
optimality and may cause the catheter to enter other branches along the vessel
wall. This algorithm has limited application scenarios. For navigation along
vessels, wall-following is not advisable as it could cause the catheter to
come into excessive contact with fragile tissue, plaque or calcium that should
actually be avoided.

In summary, there is a need for a reliable path planner elaborated in this
work that takes into account the deformable nature of the environment and the
kinematics of steerable catheters. The paper is built up as follows. Section
III introduces the modeling and path planning methods. Section IV presents an
in-silico and in-vitro experimental setup, followed by experimental results in
Sec. V. Conclusions and future directions are summarized in Sec. VI.

##  3 Materials and Methods

###  3.1 Moving Agent

The tip of the catheter is considered as the moving agent. The movement of the
catheter is fully determined by the tip under the assumption of Follow-The-
Leader (FTL) deployment [3]. While this assumption may not seem very realistic
for a catheter with a single bendable segment, it will be shown that it leads
to reasonable results.

A fixed coordinate frame $\mathcal{F}_{A}$ is attached to the tip of the
catheter as shown in Fig. 1. The agent can perform an insertion movement
$\Delta_{l}$ along the $y_{A}$ axis and can bend with angle $\alpha$ about the
$x_{A}$ axis and with angle $\gamma$ about the $z_{A}$ axis, respectively. The
pose of the agent is determined and updated at each time step by the
3-dimensional continuous action space
$\mathcal{A}=[\alpha,\gamma,\Delta_{l}]$. The pose is defined by the tip’s
position $\bm{p_{t}}=[x,y,z]$ and orientation $\bm{r_{t}}=[\alpha,0,\gamma]$
in a global frame $\mathcal{F}_{0}$. Using a transformation matrix, the agent
configuration $\bm{q_{t}}$ can be defined by its pose as below, where the
superscript $T$ means transpose.

$\bm{q_{t}}=\begin{bmatrix}\bm{R}(\alpha,0,\gamma)&[x,y,z]^{T}\\\
\bm{0}^{T}&1\end{bmatrix}$ (1)

The geometric constraints of the catheter, such as its outer diameter and the
length of the distal segment $L$, along with the kinematic constraints, such
as the maximum bending angle $\theta_{max}$ and the maximum insertion speed,
are considered in the agent’s motion. $\theta_{max}^{t}$ is the maximum
bending angle the catheter can bend at time $t$, given an insertion
$\Delta_{l}$. The bending angle is within the range of
$[-\theta_{max}^{t},\theta_{max}^{t}]$. This range further depends on the
insertion speed $v_{t}$ and time interval $\Delta_{t}$ because
$\Delta_{l}=v_{t}\Delta_{t}$.

$\theta_{max}^{t}=\frac{\theta_{max}\Delta_{l}}{L}$ (2)

Figure 1: Parameterization of a robotic catheter agent: the catheter tip has
configuration $\bm{q_{t}}$ at time $t$. The agent can perform an insertion
movement $\Delta_{l}$ along the $y_{A}$ axis and can bend with angle $\alpha$
about the $x_{A}$ axis and with angle $\gamma$ about the $z_{A}$ axis,
respectively, in the tip frame $\mathcal{F}_{A}$. The catheter segments
following the tip adopt the previous configurations sequentially.

###  3.2 Dynamic Environment

An effective planner should address the level of uncertainty that is present
in this problem. Due to the deformable nature of vessels, pre-planned paths
will deviate from the reality. Rigidly following such outdated paths may lead
to intense contact with the fragile anatomy. A realistic and auto-adaptive
simulator to predict vessels’ global deformation induced by the catheter’s
contact and cyclic heartbeat motion was proposed in our previous work [41,
42]. The vessel modeling is based on a Position-based Dynamics (PBD) approach.
It discretizes an object into a particle system composed of particles. Then it
computes the system’s time evolution by directly updating particle positions,
subject to a set of equality and inequality constraints. In the developed
simulator [41], the deformable property was calibrated according to a stress-
strain curve which appropriately depicts the biomechanics properties.
Moreover, the heartbeat motion was calibrated according to the averaged
annulus displacement from 60 patients with aortic stenosis.

In this work, for the construction of the 3D dynamic environment (see Fig. 2),
let us define:

  * •

the “configuration space” $C_{space}$ as the set of all the possible agent
configurations $\bm{q_{t}}$;

  * •

the “obstacle space” $C_{obst}\subset C_{space}$ that is the space occupied by
the vessel wall that limits the area in which the catheter can move;

  * •

the “free space” $C_{free}\subset C_{space}$ that is the set of all possible
agent configurations $\bm{q_{t}}$ within the aorta lumen without collisions
with other objects;

  * •

the “centerline space” $C_{centerline}$ that is the shortest path computed via
the Voronoi Diagram from the descending aorta to the left and right
coronaries;

  * •

the “target space” $C_{target}\subset C_{space}$ that is the volume where the
target configuration $\bm{q_{g}}$ can locate. Once the delivery catheter
reaches the target space, a micro-catheter can be inserted from a channel of
the delivery catheter [7]. $\bm{q_{g}}$ changes randomly at every learning
episode within $C_{target}$. Importantly, the target moves in concert with the
deformation of the vessels, which is induced by the catheter’s contact and the
cyclic motion of the heartbeat. This movement is achieved by linking the
target with one of the particles within the PBD system;

  * •

the agent start configuration $\bm{q_{0}}$ that is located in the descending
aorta.

Figure 2: The environment is represented by an aortic anatomy, the obstacle
space $C_{obst}$, the free space $C_{free}$, the centerline space
$C_{centerline}$, and the target space $C_{target}$. The catheter moves from
the start configuration $\bm{q_{0}}$ and proceeds to move to reach the target
configuration $\bm{q_{g}}$. (A) Top view on the aortic model. (B) Cross-
sectional view of the open lumen of the descending aorta.

###  3.3 Path Planning

The path planning problem can be described as: the agent has to find an
admissible set of configurations $Q_{t}=\\{\bm{q_{0}},...,\bm{q_{g}}\\}$ to
move from a start position $\bm{p_{0}}\in\bm{q_{0}}$ to a target position
$\bm{p_{g}}\in\bm{q_{g}}$. The target position is reached when the distance
between the agent and the target is smaller than a distance threshold
$\epsilon$.

The state of the agent consists of its pose, that can be changed through the
actions of rotation and insertion $\mathcal{A}=[\alpha,\gamma,\Delta_{l}]$. In
training, the agent learns to maximize the reward function by taking actions
according to its policy $\tau$, expressed in the paragraph Reward Function and
by observing its interaction with the environment, described in the subsequent
paragraph Observations.

####  3.3.1 Reward Function

The reward function $R(\tau)=r_{t}$ associated with each time step $t$ is
designed to optimize the path according to a combination of multiple criteria:
the number of steps, the number of collisions, reaching the target position,
passing through the centerline waypoints, bending angle. The reward $r_{t}$
consists of two main parts: $r_{end}$, a reward added at the end of a learning
episode; $r_{in}$, a relatively small reward added at each step during a
learning episode. The reward $r_{t}$ is expressed in (3)-(5).

$r_{t}=r_{end}+r_{in}\,$ (3)
$r_{end}=\begin{cases}r_{obst}\quad\textrm{if}\quad\bm{q_{t}}\in C_{obst}\\\
r_{exit}\quad\textrm{if}\quad\bm{q_{t}}\notin
C_{free}\,\textrm{and}\,\bm{q_{t}}\notin C_{obst}\\\
r_{target}\quad\textrm{if}\quad||\bm{p_{t}}-\bm{p_{g}}||<\epsilon\end{cases}\,$
(4) $r_{in}=r_{step}+r_{centerline}+r_{bending}$ (5)

  * •

$r_{obst}$ is a negative reward that is given if a collision between the
catheter tip and vessel walls is detected. We only take that collision into
account because it has a higher risk during navigation. An episode terminates
when a non-minor collision occurs;

  * •

$r_{exit}$ is a negative reward when the agent tries to exit from the open
lumen down the descending aorta;

  * •

$r_{target}$ is a positive reward given to the agent when it reaches the
target;

  * •

$r_{step}$ is a negative reward given at each time step. It is set to keep the
total number of steps of the trajectory as small as possible;

  * •

$r_{centerline}$ is a positive reward if the agent reaches a waypoint in
$C_{centerline}$;

  * •

$r_{bending}$ is a positive reward that is given when the bending action is
bigger than a threshold. This reward was introduced to overcome the tendency
of the network to avoid producing actions near the catheter’s maximum bending
angle. To pass tortuous areas, the maximal bending range is often needed to be
able to pass.

The values of the reward function parameters obtained with an empirical method
are summarized in Table 2. All rewards are set within the interval of [-1,1].
The minimum value of “-1” is assigned to prohibited behaviors such as violent
collisions with the vessel walls. The maximum value of “1” is assigned to
reaching the target, which is the agent’s task. The other rewards are chosen
based on their frequency. For instance, since $r_{step}$ occurs very often, if
it is not small enough, it can lead to a large cumulative reward. Similarly,
the bending reward also has the potential to lead to a large cumulative
reward. In contrast, the number of centerline points is relatively small, with
only around 100 points, and not all of them are reachable if the catheter
constraints are met. Hence, this positive reward is set slightly larger.

Table 2: Values of the reward function parameters Reward |  Value |  Reward |  Value
---|---|---|---
$r_{obst}$ |  -1 |  $r_{step}$ |  -1e-5
$r_{exit}$ |  -1 |  $r_{centerline}$ |  +0.05
$r_{target}$ |  +1 |  $r_{bending}$ |  +1e-5

####  3.3.2 Observations

At every step, the agent collects observations $o_{t}$, which are composed of:

  * •

the agent configuration $\bm{q_{t}}$;

  * •

the normalized distance from the agent to the target
$u=\frac{||\bm{p_{g}}-\bm{p_{t}}||}{d_{max}}$, where $d_{max}$ is the maximum
distance between the agent and the target when $\bm{q_{t}}\in C_{free}$;

  * •

the direction from the agent to the target $\bm{v}=\bm{p_{g}}-\bm{p_{t}}$;

  * •

a set of raycast observations $o_{ray}$. Each raycast detects the presence of
the aortic wall along its direction within the ray length.

####  3.3.3 C-GAIL network

Figure 3: The proposed C-GAIL network architecture. The extrinsic reward
signal considers the reward given by interacting with the environment, such as
curriculum and PPO modules. The intrinsic reward has a policy that considers
other factors, and it is defined inside the learning algorithm: for GAIL about
the similarity of the path with respect to the expert demonstrations, for
curiosity about the difference between the predicted and the actual path.

The new network proposed in this paper, named C-GAIL network, is built around
the combination of the principles of LfD and RL (see Fig. 3). The LfD
component is realized through BC and GAIL networks. The Proximal Policy
Optimization (PPO) network has two parts: the actor network provides an action
given the observation; the critic network that evaluates the actor network
using the extrinsic reward and suggests modification according to a gradient
ascent policy. The actor network updates its actions according to three
policies: the BC policy, PPO policy, and the GAIL generator policy. The
curiosity module [43] acts as an intrinsic reward signal that enables the
agent to explore its environment in novel states to help escape local minima
of the policy function. The curiosity module contains an inverse and forward
network. The inverse network predicts the action between observations, while
the forward network predicts the next encoded observation. The difference
between the predicted and actual encoded observations is defined as the loss
of the forward network. Therefore, through curiosity-driven exploration, the
agent can predict the outcome of its actions and acquire skills that may be
valuable in the future. The curriculum learning module acts on the environment
by progressively adding complexity during the training [44]. The curriculum
learning module optimizes the bending angle while respecting the reachable
bending range of the catheter. Specifically, in curriculum learning, the
learning progress is measured through the reward function and once the agent
performance improves, $\theta_{max}$ is decreased for the next level of
learning. Finally, the agent is able to try actions in different reachable
bending ranges and obtain globally optimal paths.

The proposed network can be represented as a set of modules with weights, that
define the contribution of each module to the loss function. These modules
interact with the environment and with each other. The goal is to come up with
an optimal combination of the strengths of the different modules such that a
performance is achieved that exceeds those of the offered demonstrations. The
training is based on the linear combination of the respective RL and LfD
losses:

$\mathcal{L}=\kappa(1-\mu){\mathcal{L}}_{PPO}+\lambda{\mathcal{L}}_{GAIL}+\kappa\mu{\mathcal{L}}_{BC}+\nu{\mathcal{L}}_{curiosity}$
(6)

where $\mathcal{L}_{PPO}$ is the PPO critic network loss (see its definition
in [45]), $\mathcal{L}_{GAIL}$ is the GAIL loss [46], $\mathcal{L}_{BC}$ is
the BC loss [47], $\mathcal{L}_{curiosity}$ is the curiosity loss [43]. The
weight $\mu$ indicates the degree to which we prioritize the influence of BC
over the policy relative to PPO, with a higher weight indicating a higher
learning rate of imitation from demonstrations and a lower weight indicating
more operations attempting to maximize reward rather than imitating. The
following weights were found empirically to work well: $\kappa=0.2$,
$\lambda=0.8$, $\mu=0.7$, $\nu=0.02$. The GAIL is given the highest weight
because the method combines two paradigms and reward types, intrinsic and
extrinsic.

Figure 4: Workflow for simulation and in-vitro user study. First, a 3D model
is reconstructed from CTA images for a specific patient, and a deformable
environment is built. Next, based on expert demonstrations, the C-GAIL network
is trained to provide an optimal path. This path is then rendered through a
GUI for the in-vitro experiments and serves as path guidance for users.

###  3.4 Clinical Workflow for the Proposed Path Planner

Regarding the clinical translation of this work, the anticipated clinical
workflow is designed as follows.

  * •

Firstly, a 3D mesh model is reconstructed from the pre-operative Computed
Tomography Angiography (CTA) images of a specific patient using the AW server
(GE Healthcare). Based on this 3D model, a patient-specific deformable
environment is built using the PBD simulator [41].

  * •

Then, the proposed C-GAIL network is trained using patient-specific
demonstrations provided by experts, which may encompass approximately 70
instances. This network model can be trained from scratch for this specific
patient, or retrained from a previous network model with a slightly different
anatomy. This training step may last around six hours in the PBD simulator.

  * •

Given the desired target configuration, the proposed C-GAIL path planner gives
a set of configurations to move from a starting position to the target and
outputs an optimal path for the catheter.

  * •

(optional) A patient-specific phantom is manufactured, and clinicians can
teleoperate the robotic catheter to perform in-vitro experiments. The C-GAIL
path is rendered through a Graphical User Interface (GUI), serving as path
guidance for the clinicians. This enables clinicians to acquire experience
prior to the actual interventions, thereby reducing the likelihood of
encountering unforeseen situations.

  * •

For in-vivo, ex-vivo, or clinical practice, the C-GAIL path is rendered
through a GUI and serves as path guidance for the clinicians. The clinicians
can teleoperate the robotic catheter to reach the clinical target site by
following the C-GAIL path.

##  4 Experimental Setup

The proposed workflow for the simulation and in-vitro user study is
illustrated in Fig. 4. Firstly, a 3D mesh model is reconstructed from the CTA
images of a specific patient, and a deformable environment is built using the
PBD approach [41]. Then, the proposed C-GAIL network is trained using
demonstrations provided by experts, which outputs an optimal path for the
catheter. The C-GAIL network utilizes expert demonstrations to learn and
transfer expert experience, which can provide path references for non-expert
users and even autonomous catheters.

In addition to evaluating the path planning algorithm based on C-GAIL, this
paper also includes control experiments (path following experiments) to verify
the feasibility of executing the planned path. The in-vitro experiments used
human-in-the-loop teleoperation control to guide the catheter along the
planned path. It is worth noting that while control strategies are not the
focus of this paper, they were merely included in the experiments to evaluate
the feasibility and performance of the path planner. Teleoperation is more
commonly used and easier to implement in practice, making it the method of
choice for the in-vitro experiments. It is important to note that any control
algorithm could theoretically be used to execute the path planned in this
paper.

###  4.1 In-silico Path Planning Setup

####  4.1.1 Hardware Specification

Experiments are carried out on a computer equipped with an Intel(R) Core(TM)
i9-9900KF CPU @3.60GHz 3.60 GHz processor and 32.0 GB RAM, with an NVIDIA
GeForce RTX2080Ti GPU card.

####  4.1.2 Experimental Protocol

The purpose of the in-silico experiments is to validate that, given the
curvature constraints of the robotic catheter, the paths obtained by the
proposed C-GAIL network have better performance than the state-of-the-art GAIL
approach in a static environment and are also capable to operate in a complex
dynamic environment.

Training and testing were conducted in a single aortic model. For the
training, 70 demonstrations were recorded using a joystick by one expert user,
who performed catheter navigation through vessels in the aortic model for a
PCI. The number of experiments for comparison was set to 100. Different start
configurations in the descending aorta and possible target positions were
chosen to make the training more generally valid. There are three possible
start configurations distributed within 5.3cm along the descending aorta.
There are five target positions distributed in a spherical space with a radius
of 3cm near the coronaries and the vessel walls. Although having diverse
configurations in the training data might hinder convergence, it could
actually enhance the robustness and generalizability of the network model
[48].

The C-GAIL network training was carried out in two phases: pre-training in a
rigid, computationally less heavy environment, where the tuning of the network
parameter settings is performed; re-training in a deformable environment,
starting with the previously tuned network settings. Pre-training helps reduce
the time required to obtain network settings with empirical parameters and
compare the differences in agent behavior under different settings. The
training parameters for the C-GAIL is presented in Table 3. The work from Chi
_et al._ [10] is used as a reference to compare with. To keep consistency in
comparison, we set the same training parameters for the state-of-the-art GAIL.
The GAIL network architecture of [10] is adopted with minor modifications:
each layer has 64 units and a Swish activation function [49].

Table 3: Training parameters for C-GAIL and GAIL. Parameter |  Value |  Parameter |  Value
---|---|---|---
PPO beta |  5.0e-4 |  PPO gamma |  0.99
max steps |  5.0e5 |  GAIL gamma |  0.99
batch size |  1024 |  buffer size |  10240
All parameters are defined in the Unity ML-Agents Toolkit [50].

###  4.2 In-vitro Validation Setup

####  4.2.1 Hardware Specification

The in-vitro experimental setup, including the following devices, is depicted
in Fig. 5.

##### phantom

experiments were performed in a transparent, deformable silicone aortic
phantom (T-S-N-002, Elastrat Sarl, Geneva, Switzerland).

##### robotic catheter

the custom-made robotic catheter is fabricated out of Nitinol using metal
laser cutting technology and is actuated by four integrated Pneumatic
Artificial Muscles (PAMs) [51, 52]. The outer diameter of the catheter is 7mm,
and the length is 900mm. The distal Nitinol segment is 75mm long and includes
a 50mm long steerable distal segment. This steerable distal segment has 2-
Degree of Freedom (DoF) bending motion by concurrently controlling two
antagonistic pairs of the PAMs. The maximum bending angle is $90^{\circ}$.

##### electromagnetic field generator

an Electromagnetic (EM) field generator (Northern Digital Inc., Waterloo,
Canada) is placed beneath the phantom. A 6-DoF EM sensor (Northern Digital
Inc., Waterloo, Canada) is embedded at the tip of the catheter to track its
pose.

##### catheter driver

the robotic catheter driver system is described in [53]. The sleeve-based
catheter driver has two pneumatically actuated grippers that grasp the
catheter alternately and insert the catheter in a relay fashion. The maximum
insertion (and retraction) speed is set as 5mm/s.

##### joystick

the teleoperated catheter insertion and bending are realized through velocity
control of the catheter driver and pressure control of the PAMs, respectively,
using a wireless controller (Yues, Dublin, Ireland).

##### GUI

the path planning is visualized by a GUI, which includes an external projected
view showing the aorta, path, and pose of the catheter tip, and an internal
view showing the next waypoint and suggested bend direction.

Figure 5: In-vitro experimental setup to validate the proposed path planning
approach: (1) silicone aortic phantom; (2) robotic catheter; (3) Aurora EM
field generator; (4) sleeve-based catheter driver; (5) wireless controller;
(6) the GUI as visual feedback.

####  4.2.2 Experimental Protocol

The objective of the in-vitro experiments is to validate whether the path
obtained from the proposed C-GAIL path planner is more compatible with the
actual steering capability of the catheter. Therefore, it can also be verified
whether this path can better help the user steer the catheter accurately to
the target.

For the in-vitro comparison with C-GAIL, we chose the traditional centerline-
following approach. The centerline path is widely used in interventions [54,
55], whereas GAIL approach (which was served as a comparison in the in-silico
experiments) has not yet been validated in clinical practice. Moreover, both
GAIL and C-GAIL are RL approaches, which have limited interpretability.
Therefore, it may be suggested to use traditional approaches in parallel with
learning-based approaches in order to provide a more transparent and
explainable analysis. By comparing C-GAIL with the traditional centerline
approach, a more interpretable comparison can be achieved.

User-involved control experiments were carried out. In each trial, the user
teleoperates with the robotic catheter, tries to pass through each waypoint,
and finally reaches the target. During experiments, the user is instructed not
to directly observe the transparent phantom. The user is instead required to
rely on visual feedback provided by the GUI displayed on a conventional 2D
monitor for his (her) teleoperation actions. The GUI provides critical
feedback to users. It displays the catheter’s tip pose, captured through EM
tracking. By visualizing this alongside the rendered phantom, users can
effectively gauge their proximity to the vessel walls. This feature is
instrumental in preventing collisions or applying excessive force against the
vessels. The maximum operation time is set to 3 minutes. If the maximum
operation time is exceeded and the target is still not reached, the experiment
will be forcibly stopped and be regarded as a failure. Setting this stopping
criterion can avoid the following two cases: 1) The user rushes to finish the
experiment without considering the performance; 2) The user tries excessively
to get better performance while the procedure is very time-consuming.

The user’s performance was compared in two scenarios: with the C-GAIL
assistance and centerline assistance. The waypoints from $C_{centerline}$ were
utilized in the centerline-following approach. The C-GAIL network was trained
using different targets and obtained a path to reach an unseen target (that
was not used during training). To largely eliminate the impact of the user’s
learning curve, the user is required to be pre-trained. Only when the learning
curve converges to a plateau can we assume that the user has stable catheter
manipulation abilities. The user pre-training experiments are repeated ten
times. After that, control experiments with the same C-GAIL path are conducted
ten times. Similarly, the pre-training experiments with the centerline and the
control experiments with the same centerline are repeated ten times each. The
experiments with the C-GAIL path are first performed to eliminate experience
learnt from the other scenario.

###  4.3 Performance Metrics

The following criteria were chosen to quantify the performance of in-silico
and in-vitro experimental results. The statistically significant difference
between the proposed method and others is evaluated via the Kruskal-Wallis
test in this work, with a significance level of 0.05.

####  4.3.1 Success Rate ($\delta$)

Success rate represents the percentage of successes among the total number of
attempts.

$\delta=n_{s}/n$ (7)

where $n_{s}$ is the number of successes to reach the target within a given
time, and $n$ is the number of attempts.

####  4.3.2 Timesteps ($T_{s}$)

For in-silico experiments, at each time step, the agent receives observations,
takes an action according to its policy and receives rewards. $T_{s}$ is
defined as the number of time steps moving from the start configuration to the
target.

$T_{s}=N_{g}-N_{0}+1$ (8)

where $N_{0}$ and $N_{g}$ are the first and last time step, respectively.

####  4.3.3 Duration ($T$)

Duration is the length of time a single experiment lasts from start to stop.
An experiment stops when the target or the maximum duration has been reached.

$T=t_{g}-t_{0}$ (9)

where $t_{0}$ and $t_{g}$ are the first and last timestamp, respectively.

####  4.3.4 Tracking Error ($T_{r}$)

Tracking error is the distance between the in-vitro trajectory obtained from
the EM sensor and the desired path. The desired path is obtained through path
planning, followed by B-spline curve fitting and resampling. This criterion is
used to evaluate the path following capability.

$T_{r}(t)=||\bm{p^{d}_{t}}-\bm{p_{t}}||$ (10)

where $\bm{p^{d}_{t}}$ is the desired position and $\bm{p_{t}}$ is the actual
position.

####  4.3.5 Targeting Error ($T_{a}$)

Targeting error is the minimum distance to the target along the trajectory.
This criterion is used to evaluate the accuracy when reaching the target.

$T_{a}=\min||\bm{q_{g}}-\bm{p_{t}}||$ (11)

##  5 Results and Discussion

Figure 6: In-silico performance comparison between the proposed C-GAIL network
(purple) and the state-of-the-art GAIL network [10] (red), with respect to (A)
example of trajectories, (B) targeting error $T_{a}$, and (C) timesteps
$T_{s}$. An ablation study of the C-GAIL is presented when $\mu=0$,
$\lambda=0$ or $\nu=0$. ($*,p<0.05$ using Kruskal-Wallis test) Figure 7: In-
silico performance of the proposed C-GAIL network in static and dynamic
environments: (A) The intra-operative adaptation of the pre-operative path
$q_{pre}$ when the environment is deformable and the target moves; (B)
Targeting error $T_{a}$ and (C) Timesteps $T_{s}$ in static and dynamic
environments. ($*,p<0.05$ using Kruskal-Wallis test)

###  5.1 In-silico Path Planning

The performance comparison of the proposed C-GAIL and the state-of-the-art
GAIL [10] in a static environment is presented in Fig. 6. The number of
experiments for comparison is set to 100, and results are reported regardless
of whether the target was successfully reached. Fig. 6A shows an example of
trajectories obtained by these two approaches. The C-GAIL approach generates
paths with smaller targeting errors (see Fig. 6B). An ablation study of the
C-GAIL is also presented, investigating its performance when $\mu=0$,
$\lambda=0$, or $\nu=0$. If we set the threshold $\epsilon=10$, the success
rate of the C-GAIL network, originally at 80%, decreases to 11%, 58%, and 48%
for the scenarios when $\mu=0$, $\lambda=0$, and $\nu=0$ respectively. The
results, as shown in Fig. 6B, indicate that BC, GAIL and curiosity modules are
all critical for accurate target reaching. Furthermore, the robustness of the
proposed network is verified by the success rate of the C-GAIL network, that
is 80%, compared to that of the GAIL network [10] which is only 42%, when
$\epsilon=10$. However, compared to GAIL, C-GAIL requires more timesteps to
reach the target (see Fig. 6C). This indicates that either the path is longer
or there are more samples taken along the path to achieve the same level of
path length. The duration is affected by a large number of failures that often
cause early termination of the experiment. A common failure is that the path
planning tends to stop due to non-minor collision with the aortic arch and are
therefore unsuccessful attempts. Minor collisions that cause slight
deformation within reasonable stress ranges were considered harmless [56].

The agent was then trained using pre-trained weights from the static
environment in a dynamic environment, where deformations are caused by contact
with the catheter and the heartbeat motion. Fig. 7A demonstrates the ability
of real-time planning in a complex dynamic environment with deformable vessels
and a moving target. The top zoom-in view shows that the paths of pre-trained
${\bm{q}}_{pre}$ and intraoperatively trained ${\bm{q}}_{intra}$ are initially
close and then separate in the aortic arch, where the degree of deformation is
greater. The bottom zoom-in view shows the change from the initial target
position ${\bm{q_{g}}}_{pre}$ to the intraoperative target position
${\bm{q_{g}}}_{intra}$. Fig. 7B-C present performance comparisons in static
and dynamic environments. The timesteps are much more in the deformable
environment. This is due to the fact that there are more possibilities when
interacting with a deformable environment than with a static one. Due to the
greater complexity of navigation and low success rate, targeting error is
larger and more widely distributed than in the static environment.
Nevertheless, when considering only the successful trials, the targeting error
is observed to be 4.76$\pm$0.57mm in the static environment and
7.57$\pm$0.65mm in the dynamic environment, respectively, demonstrating
comparable results. The success rate of 17% is relatively low but indicates
that it is possible to find a feasible path in a constrained environment,
compared to the success rate of 0% for the GAIL network [10] in the dynamic
environment. A common failure is the inability to reach the moving targets
with a distance threshold of 10mm. Including this distance threshold to reach
a target in the curriculum module would improve network performance, such as
success rate. This would be investigated at a later stage. The limited bending
capability of robotic catheters and the presence of moving targets in
deformable environments led to a reduced success rate within a limited number
of timesteps. Despite the relatively low success rate of C-GAIL, it has
outperformed the state-of-the-art approach.

One limitation of our proposed method, particularly in medical applications,
is the exclusion of blood fluid flow dynamics in our developed simulator. To
address this, future iterations should incorporate fluid dynamics, potentially
using subject-specific geometries as suggested in [57]. Integrating this
aspect could yield more precise predictions of vessel deformation and catheter
movement. Such enhancements are likely to elevate the realism of the simulated
scenarios, thereby improving the accuracy of the path planning to better
reflect real-world conditions.

###  5.2 In-vitro Validation

To essentially eliminate the impact of the user’s learning curve, the user is
required to be pre-trained. For example, the learning curve of the targeting
error by following the C-GAIL path decreased from 3.6 to $\sim$2mm in ten
trials. The user can achieve a targeting error of 0.23mm in the later stage.
Since the learning curve converges to a plateau, it can be concluded that the
user has stable catheter manipulation ability. A video is made available
showing example experiments of following the C-GAIL path and the centerline
111https://youtu.be/GhBi_xHTMFw.

The performance comparison between the C-GAIL path and the centerline is shown
in Fig. 8. The C-GAIL path leads to a smaller tracking error of
5.18$\pm$3.48mm. It confirms that C-GAIL leads to a path that can be followed
better by the robotic catheter. Furthermore, the C-GAIL path also shows a
smaller targeting error of 1.26$\pm$0.55mm (Fig. 8B). Following the C-GAIL
path takes a longer time than following the centerline. This is interpretable
given that since the user learned that the centerline waypoints could not be
reached anyway when passing the aortic arch, the user kept inserting the
catheter at the maximum bending angle and spent less effort following the
centerline. While in the other scenario, the user realized that (s)he could
follow the C-GAIL path waypoints. Therefore, it takes a longer time than
following the centerline. Two of the ten experiments with the C-GAIL path took
longer because the user made a retraction motion in order to re-reach the
target more accurately. Both the C-GAIL path and centerline following
experiments had a 100% success rate.

Figure 9 presents a representative graphical comparison of actual trajectories
and path guidance in two scenarios: following the C-GAIL path and following
the centerline. The trajectories of the catheter tip following the C-GAIL path
are more stable and smoother, with a curvature of $3.1\pm 26.6$mm-1, compared
to a curvature of $3.5\pm 38.9$mm-1 when following the centerline. In the
other scenario, the catheter tip can follow the centerline well at the
beginning. However, the trajectory then moves away from the centerline as the
catheter passes through the aortic arch due to the limited bending capability.

Figure 8: In-vitro performance comparison between C-GAIL path and centerline,
with respect to (A) tracking error $T_{r}$, (B) targeting error $T_{a}$, and
(C) duration $T$. ($*,p<0.05$ using Kruskal-Wallis test) Figure 9:
Representative in-vitro trajectories (blue) and path guidance (black) using:
(A) C-GAIL path, and (B) centerline.

###  5.3 Transferability

To further generalize the proposed path planning framework, in-silico
experiments were conducted in a different aortic anatomy (Materialise NV,
Leuven, Belgium). 70 demonstrations were recorded, and 100 testing trials were
performed. The success rate of the proposed C-GAIL network was 79% when
$\epsilon=10$. Other performance values with respect to targeting error and
timesteps are summarized in Table 4. Representative paths obtained by the
C-GAIL approach are shown in Fig. 10. These results indicate the robustness
and feasibility of applying the proposed path planning framework in anatomies
with different geometries.

This paper employs an EM sensor to measure the tip pose, which has been
validated in a phantom study. EM sensor has demonstrated its efficacy also in
in-vivo environments [58, 59] considering its small size ($<$1mm in diameter
and $<$10mm in length) and bio-compatibility. Furthermore, other sensors, such
as Fiber Bragg Grating (FBG) sensor, as well as imaging modalities, can also
be utilized to obtain catheter tip pose. In certain specific cases (e.g., free
space), the catheter tip pose can also be obtained with a sensorless approach,
e.g., an accurate forward kinematic model if control variables are provided
[52]. One limitation of this study is that there was no intra-operative
environment reconstruction for the in-vitro experiments. Instead, a simulated
deformable model was utilized to predict intra-operative deformations. In
clinical settings, real-time vessel reconstruction from IntraVascular
UltraSound (IVUS) or Optical Coherence Tomography (OCT) images would be
suitable for providing raycast observations elaborated in this work.

Table 4: In-silico performance of the proposed C-GAIL network in a different aortic anatomy. $T_{a}$ (mm) |  $T_{s}$ ($\times 10^{3}$) | $\delta$
---|---|---
25th |  median |  75th |  mean |  std |  25th |  median |  75th |  mean |  std | (%)
6.58 |  6.84 |  8.57 |  17.98 |  23.51 |  9.41 |  10.22 |  11.59 |  9.71 |  2.78 |  79
Figure 10: Examples of in-silico C-GAIL paths in a different aortic anatomy
(Materialise NV, Leuven, Belgium).

##  6 Conclusion

This study set out to design a robust path planning approach respecting the
kinematics of robotic catheters and real-time changes in deformable cluttered
environments. The insights gained from this study add to the rapidly expanding
field of autonomous navigation of robotic catheters. The findings of this
study suggest that the proposed path planner can effectively handle the
uncertainty present in vessel deformation. A further clinical study involving
real-time vessel reconstruction will be carried out.

## References

  * [1] A. P. Durko _et al._ , “Annual number of candidates for transcatheter aortic valve implantation per country: current estimates and future projections,” _Eur. Heart J._ , vol. 39, no. 28, pp. 2635–2642, 2018.
  * [2] R. Aggarwal _et al._ , “Virtual reality simulation training can improve inexperienced surgeons’ endovascular skills,” _Eur. J. Vascular and Endovascular Surg._ , vol. 31, no. 6, pp. 588–593, 2006.
  * [3] C. Culmone _et al._ , “Follow-the-leader mechanisms in medical devices: A review on scientific and patent literature,” _IEEE Rev. Biomed. Eng._ , 2021.
  * [4] T. da Veiga _et al._ , “Challenges of continuum robots in clinical context: A review,” _Prog. Biomed. Eng._ , vol. 2, no. 3, p. 032003, 2020.
  * [5] A. Ali _et al._ , “First expert evaluation of a new steerable catheter in an isolated beating heart,” _Cardiovascular Eng. Technol._ , vol. 11, no. 6, pp. 769–782, 2020.
  * [6] N. Nayar _et al._ , “Design and control of 5-dof robotically steerable catheter for the delivery of the mitral valve implant,” in _Proc. IEEE Int. Conf. Robot. Automat._ , 2021, pp. 12 268–12 274.
  * [7] M. H. D. Ansari _et al._ , “Proof-of-concept medical robotic platform for endovascular catheterization,” in _Proc. Conf. New Technol. Comput. Robot. Assist. Surg._ , 2022.
  * [8] G. Borghesan _et al._ , “Atlas: Autonomous intraluminal surgery: System specifications for targeted intraluminal interventions,” 2020. [Online]. Available: https://atlas-itn.eu/d102_main/
  * [9] A. Favaro _et al._ , “An evolutionary-optimized surgical path planner for a programmable bevel-tip needle,” _IEEE Trans. Robot._ , vol. 37, no. 4, pp. 1039–1050, 2021.
  * [10] W. Chi _et al._ , “Collaborative robot-assisted endovascular catheterization with generative adversarial imitation learning,” in _Proc. IEEE Int. Conf. Robot. Automat._ , 2020.
  * [11] A. Segato _et al._ , “Inverse reinforcement learning intra-operative path planning for steerable needle,” _IEEE Trans. Biomed. Eng._ , vol. 69, no. 6, pp. 1995–2005, 2021.
  * [12] J. Wang _et al._ , “Intravascular catheter navigation using path planning and virtual visual feedback for oral cancer treatment,” _Int. J. Med. Robot. Comput. Assist. Surg._ , vol. 7, no. 2, pp. 214–224, 2011.
  * [13] J.-Q. Zheng _et al._ , “Towards 3d path planning from a single 2d fluoroscopic image for robot assisted fenestrated endovascular aortic repair,” in _Proc. IEEE Int. Conf. Robot. Automat._ , 2019, pp. 8747–8753.
  * [14] D. Huang _et al._ , “An interactive 3d preoperative planning and training system for minimally invasive vascular surgery,” in _Pro. IEEE Int. Conf. Comput. Aided Des. Comput. Graph._ , 2011, pp. 443–449.
  * [15] H. Qian _et al._ , “Towards rebuild the interventionist’s intra-operative natural behavior: A fully sensorized endovascular robotic system design,” in _Proc. IEEE Int. Conf. Med. Imag. Phys. Eng._ , 2019.
  * [16] Y. Cho _et al._ , “Image processing based autonomous guidewire navigation in percutaneous coronary intervention,” in _Proc. IEEE Int. Conf. Consum. Electron. Asia_ , 2021.
  * [17] P. Schegg _et al._ , “Automated planning for robotic guidewire navigation in the coronary arteries,” in _Proc. IEEE Int. Conf. Soft Robot._ , 2022, pp. 239–246.
  * [18] S. R. Ravigopal _et al._ , “Automated motion control of the coast robotic guidewire under fluoroscopic guidance,” in _Proc. IEEE Int. Symp. Med. Robot._ , 2021.
  * [19] ——, “Fluoroscopic image-based 3-d environment reconstruction and automated path planning for a robotically steerable guidewire,” _IEEE Robot. Automat. Lett._ , vol. 7, no. 4, pp. 11 918–11 925, 2022.
  * [20] G. Fagogenis _et al._ , “Autonomous robotic intracardiac catheter navigation using haptic vision,” _Sci. Robot._ , vol. 4, no. 29, 2019.
  * [21] J. Fauser _et al._ , “Generalized trajectory planning for nonlinear interventions,” in _Comput. Assist. Robot. Endoscopy_ , 2018, pp. 46–53.
  * [22] ——, “Planning for flexible surgical robots via bézier spline translation,” _IEEE Robot. Automat. Lett._ , vol. 4, no. 4, pp. 3270–3277, 2019.
  * [23] ——, “Optimizing clearance of bézier spline trajectories for minimally-invasive surgery,” in _Proc. Int. Conf. Med. Imag. Comput. and Imag. Assist. Interv._ , 2019, pp. 20–28.
  * [24] J. Guo _et al._ , “A training system for vascular interventional surgeons based on local path planning,” in _Proc. IEEE Int. Conf. Mechatronics Automat._ , 2021, pp. 1328–1333.
  * [25] M. Gao _et al._ , “Three-dimensional path planning and guidance of leg vascular based on improved ant colony algorithm in augmented reality,” _J. Med. Syst._ , vol. 39, no. 11, p. 133, 2015.
  * [26] F. Qi _et al._ , “Kinematic analysis and navigation method of a cable-driven continuum robot used for minimally invasive surgery,” _Int. J. Med. Robot. Comput. Assist. Surg._ , vol. 15, no. 4, p. e2007, 2019.
  * [27] Z. Li _et al._ , “Path planning for endovascular catheterization under curvature constraints via two-phase searching approach,” _Int. J. Comput. Assist. Radiol. Surg._ , vol. 16, no. 4, pp. 619–627, 2021.
  * [28] H. Rafii-Tari _et al._ , “Learning-based modeling of endovascular navigation for collaborative robotic catheterization,” in _Adv. Inf. Syst. Eng._ , 2013, pp. 369–377.
  * [29] ——, “Hierarchical hmm based learning of navigation primitives for cooperative robotic endovascular catheterization,” in _Proc. Int. Conf. Med. Imag. Comput. and Comput. Assist. Interv._ , 2014, pp. 496–503.
  * [30] W. Chi _et al._ , “Trajectory optimization of robot-assisted endovascular catheterization with reinforcement learning,” in _Proc. IEEE Int. Conf. Intell. Robot. Syst._ , 2018.
  * [31] ——, “Learning-based endovascular navigation through the use of non-rigid registration for collaborative robotic catheterization,” _Int. J. Comput. Assist. Radiol. Surg._ , vol. 13, no. 6, pp. 855–864, 2018.
  * [32] Y. Zhao _et al._ , “Surgical gan: Towards real-time path planning for passive flexible tools in endovascular surgeries,” _Neurocomputing_ , vol. 500, pp. 567–580, 2022.
  * [33] A. T. Tibebu _et al._ , “Towards autonomous robotic catheter navigation using reinforcement learning,” in _Joint Workshop on New Technol. Comput./Robot. Assist. Surg._ , 2014, pp. 163–166.
  * [34] H. You _et al._ , “Automatic control of cardiac ablation catheter with deep reinforcement learning method,” _J. Mech. Sci. Technol._ , vol. 33, no. 11, pp. 5415–5423, 2019.
  * [35] T. Behr _et al._ , “Deep reinforcement learning for the navigation of neurovascular catheters,” _Current Directions Biomed. Eng._ , vol. 5, no. 1, pp. 5–8, 2019.
  * [36] L. Karstensen _et al._ , “Autonomous guidewire navigation in a two dimensional vascular phantom,” _Current Directions Biomed. Eng._ , vol. 6, no. 1, 2020.
  * [37] J. Kweon _et al._ , “Deep reinforcement learning for guidewire navigation in coronary artery phantom,” _IEEE Access_ , 2021.
  * [38] F. Meng _et al._ , “Evaluation of a reinforcement learning algorithm for vascular intervention surgery,” in _Proc. IEEE Int. Conf. Mechatronics Automat._ , 2021, pp. 1033–1037.
  * [39] L. Karstensen _et al._ , “Learning-based autonomous vascular guidewire navigation without human demonstration in the venous system of a porcine liver,” _Int. J. Comput. Assist. Radiol. Surg._ , pp. 1–8, 2022.
  * [40] A. Pore _et al._ , “Learning from demonstrations for autonomous soft-tissue retraction,” in _Proc. IEEE Int. Symp. Med. Robot._ , 2021.
  * [41] Z. Li _et al._ , “Position-based dynamics simulator of vessel deformations for path planning in robotic endovascular catheterization,” _Med. Eng. Phys._ , vol. 110, p. 103920, 2022.
  * [42] ——, “Simulation of deformable vasculature for robot-assisted endovascular catheterization,” in _Poster Session at Conf. Int. Soc. Med. Innov. Technol._ , 2021.
  * [43] D. Pathak _et al._ , “Curiosity-driven exploration by self-supervised prediction,” in _Proc. Int. Conf. Mach. Learn._ , 2017, pp. 2778–2787.
  * [44] S. Narvekar _et al._ , “Source task creation for curriculum learning,” in _Pro. Int. Conf. Auton. agents & multiagent Syst._, 2016, pp. 566–574.
  * [45] J. Schulman _et al._ , “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017.
  * [46] J. Ho _et al._ , “Generative adversarial imitation learning,” _Adv. Neural Inf. Process. Syst._ , vol. 29, 2016.
  * [47] F. Torabi _et al._ , “Behavioral cloning from observation,” _arXiv preprint arXiv:1805.01954_ , 2018.
  * [48] W. Xiao _et al._ , “Multigoal visual navigation with collision avoidance via deep reinforcement learning,” _IEEE Trans. Instrum. Meas._ , vol. 71, pp. 1–9, 2022.
  * [49] P. Ramachandran _et al._ , “Searching for activation functions,” _arXiv preprint arXiv:1710.05941_ , 2017.
  * [50] A. Juliani _et al._ , “Unity: A general platform for intelligent agents,” _arXiv preprint arXiv:1809.02627_ , 2018.
  * [51] A. Devreker _et al._ , “Fluidic actuation for intra-operative in situ imaging,” in _2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2015, pp. 1415–1421.
  * [52] D. Wu _et al._ , “Hysteresis modeling of robotic catheters based on long short-term memory network for improved environment reconstruction,” _IEEE Robot. Automat. Lett._ , vol. 6, no. 2, pp. 2106–2113, 2021.
  * [53] O. Al-Ahmad, M. Ourak, J. Vlekken, and E. V. Poorten, “Force control with a novel robotic catheterization system based on braided sleeve grippers,” _IEEE Transactions on Medical Robotics and Bionics_ , pp. 1–1, 2023.
  * [54] J. P. Sobocinski _et al._ , “The benefits of evar planning using a 3d workstation,” _Eur. J. Vascular and Endovascular Surgery_ , vol. 46, no. 4, pp. 418–423, 2013.
  * [55] S. Ramcharitar _et al._ , “Technology insight: magnetic navigation in coronary interventions,” _Nature Clin. Pract. Cardiovascular Medicine_ , vol. 5, no. 3, pp. 148–156, 2008.
  * [56] X. Ye _et al._ , “A fast and stable vascular deformation scheme for interventional surgery training system,” _Biomed. Eng. Online_ , vol. 15, no. 1, pp. 1–14, 2016.
  * [57] E. L. Schwarz, L. Pegolotti, M. R. Pfaller, and A. L. Marsden, “Beyond cfd: Emerging methodologies for predictive simulation in cardiovascular health and disease,” _Biophysics Reviews_ , vol. 4, no. 1, 2023.
  * [58] M. Ourak _et al._ , “Fusion of biplane fluoroscopy with fiber bragg grating for 3d catheter shape reconstruction,” _IEEE Robot. Automat. Lett._ , vol. 6, no. 4, pp. 6505–6512, 2021.
  * [59] X. T. Ha _et al._ , “Robust catheter tracking by fusing electromagnetic tracking, fiber bragg grating and sparse fluoroscopic images,” _IEEE Sensors J._ , vol. 21, no. 20, pp. 23 422–23 434, 2021.

  *[PCI]: Percutaneous Coronary Intervention
  *[CTO]: Chronic Total Occlusion
  *[C-GAIL]: Curriculum Generative Adversarial Imitation Learning
  *[CL]: Curriculum Learning
  *[BC]: Behavioral Cloning
  *[DFS]: Depth First Search
  *[bi-RRT]: bi-directional Rapidly-exploring Random Tree
  *[ACO]: Ant Colony Optimization
  *[GA]: Genetic Algorithm
  *[LfD]: Learning from Demonstrations
  *[GAN]: Generative Adversarial Network
  *[RL]: Reinforcement Learning
  *[GAIL]: Generative Adversarial Imitation Learning
  *[FTL]: Follow-The-Leader
  *[PBD]: Position-based Dynamics
  *[PPO]: Proximal Policy Optimization
  *[CTA]: Computed Tomography Angiography
  *[GUI]: Graphical User Interface
  *[PAMs]: Pneumatic Artificial Muscles
  *[DoF]: Degree of Freedom
  *[EM]: Electromagnetic
  *[FBG]: Fiber Bragg Grating
  *[IVUS]: IntraVascular UltraSound
  *[OCT]: Optical Coherence Tomography