# Path planning of magnetic microswimmers in high-fidelity simulations of
capillaries with deep reinforcement learning

Lucas Amoudruz  School of Engineering and Applied Sciences, Harvard
University, Cambridge, MA 02138, United States.  Sergey Litvinov  School of
Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138,
United States.  Petros Koumoutsakos  School of Engineering and Applied
Sciences, Harvard University, Cambridge, MA 02138, United States.

###### Abstract

Biomedical applications such as targeted drug delivery, microsurgery or
sensing rely on reaching precise areas within the body in a minimally invasive
way. Artificial bacterial flagella (ABFs) have emerged as potential tools for
this task by navigating through the circulatory system. While the control and
swimming characteristics of ABFs is understood in simple scenarios, their
behavior within the bloodstream remains unclear. We conduct simulations of
ABFs evolving in the complex capillary networks found in the human retina. The
ABF is robustly guided to a prescribed target by a reinforcement learning
agent previously trained on a reduced order model.

##  1 Introduction

Targeted drug delivery, micro-surgery and micro-sensing represent important
areas of research aimed at revolutionizing precision medicine. These
challenging tasks require precise access to target areas in the body, which
are often difficult to reach in a non-invasive way. Over the past decade,
artificial microswimmers have emerged as potent candidates for navigating to
specific regions of the human body through the circulatory system or tissues
[1]. A particular form of artificial microswimmers, known as artificial
bacterial flagella, features a corked-screw shaped body propelled by rotating
magnetic fields [2]. ABF have been utilized in various applications, including
the navigation through bovine eye tissues ex-vivo [3], assisting spermatozoa
in reaching ovocytes [4], enhancing nanoparticle transport [5] and carrying
drugs for cancer therapy [6].

The swimming properties of ABF are well understood in the idealized scenario
of unbounded viscous fluids [7, 8]. Venugopalan et al. demonstrated that ABF
can propel in low concentrated blood suspensions [9]. Alapan et al. have
guided micro-rollers in physiologically relevant blood concentrations with a
rotating magnetic field [10]. Qi et al. have conducted numerical simulations
of these rollers through blood in straight pipes [11]. However, the swimming
characteristics of ABF within the bloodstream have not been thoroughly
investigated. The control of microswimmers for path planning have been the
subject of recent advances with the emergence of reinforcement learning (RL).
Muiños-Landin et al. used RL in an experimental setup to control a
microswimmer towards a target, with no background flow [12]. Similarly,
several studies applied RL to guide a single [13, 14, 15, 16, 17] or multiple
[18] swimmers towards a target in the presence of a background flow with the
help of numerical simulations. Yang et al. achieved path-planning of point
particles between obstacles represented by red blood cells [19]. However the
numerical models used in these studies do not reflect the complex flow
patterns present in capillaries, the finite size of the swimmer, and the
presence of deformable blood cells.

In this work, we model and simulate an ABF evolving in a capillary network
typically found in the retina. In particular, we aim to learn a control
strategy that guides the ABF to a prescribed target by varying the external
magnetic field. The geometric representation of the environment is reproduced
from a fundus image of a retinal capillary network [20]. The simulations
include an accurate RBC model that was extensively validated on experimental
data [21], coupled with the dissipative particle dynamics (DPD) method to
resolve the hydrodynamics at the microscale [22]. The control policy to guide
the ABF to its target is learned with RL. To substantially reduce the
computational cost, the RL agent is trained on a reduced order model of the
high-fidelity simulations.

##  2 Methods

The simulations consist of an ABF evolving through a network of capillaries
filled with blood at a hematocrit $\mathrm{Ht}=25\%$. The ABF has a radius of
$2\text{\,}\mathrm{\SIUnitSymbolMicro m}$ and a length of
$18.37\text{\,}\mathrm{\SIUnitSymbolMicro m}$, within the range of the ABF
presented in the literature. Blood is modeled as RBC membranes suspended in
the plasma and enclosing the cytosol, a fluid 5 times more viscous than the
plasma. Both the plasma and the cytosol are viscous Newtonian fluids and are
modeled with the DPD method [22]. RBC membranes evolve according to bending
forces, and shear forces that are zero when the membrane takes the shape of
the stress-free state of the RBC cytoskeleton. In addition, area and volume of
the RBC membranes are constrained through penalization forces. The details of
the model and its parameters are described in Amoudruz et al. , where the
model has been extensively validated against experimental data [21]. The ABF
is represented as a set of frozen particles and a surface moving as a rigid
body. The surrounding solvent particles interact with the ABF particles
through DPD forces and are bounced-back from the surface. The magnetic moment
$\mathbf{m}$ of the ABF remains constant in the frame of reference of the
swimmer, perpendicular to its principal axis. The ABF is immersed in an
external, uniform magnetic field $\mathbf{B}$, and experiences a magnetic
torque $\mathbf{T}=\mathbf{m}\times\mathbf{B}$. This torque, combined with the
cork-screw shape of the ABF’s tail, causes the ABF to propel along its main
axis [7]. The external magnetic field varies over time and will be the
controlled quantity used to stir the ABF towards its target.

The boundaries of the capillaries were generated from a human retinal
vasculature fundus image in Ghassemi et al. [20] (see supplementary material).
The walls are formed by a layer of frozen DPD particles, of width larger than
the interaction cutoff. Furthermore, the DPD particles are bounced-back from
the wall surface. The parameters of the DPD forces and interactions between
every objects of the simulations are detailed in Amoudruz [23]. To drive the
flow, an external body force is applied to the DPD particles, following the
approach described in Yazdani et al. [24]. The body force is derived from the
pressure gradient of the flow previously solved by an external CFD package. We
use Aphros [25] to solve the Stokes equation in the retina geometry, assuming
a Newtonian fluid and a constant velocity at the inlet. In the blood
simulations, the body forces were scaled to match a given Reynolds number at
the inlet of the domain. The blood simulations are performed with Mirheo [26],
a high-performance package for microfluidics simulations on multi-GPU
architectures.

##  3 Results

###  3.1 Control via reinforcement learning

We consider the task of bringing the ABF from the entry of the retinal
capillary network to a given target by varying the external magnetic field. RL
algorithms typically require thousands to millions of episodes and it it thus
intractable to directly interact with the high-fidelity model, which requires
64 P100 GPUs over 24 hours per simulation. Instead, we use a reduced order
model to train a policy that we then apply to the high-fidelity model. The
reduced order model consists of a self-propelling point particle advected by a
background velocity field $\mathbf{u}$. The point particle represents the
center of mass of the ABF and the background velocity is obtained from the
grid-based solver Aphros. In addition, we model the collisions with
surrounding RBCs via a stochastic term, resulting in the stochastic
differential equation (SDE)

$\dot{\mathbf{x}}=\mathbf{u}(\mathbf{x})+U\mathbf{p}+\sqrt{D}\boldsymbol{\xi},$
(1)

where $\mathbf{p}$ and $U$ are the self-propelling velocity direction and
magnitude of the ABF, respectively. $D$ is a diffusion coefficient, and
$\boldsymbol{\xi}$ is a vector of Gaussian white noise. In addition, the
particle is bounced-back from the wall boundaries. The parameters $D$ and $U$
were calibrated from the high-fidelity blood simulation (see supplementary
material).

The system is advanced in time with a piece-wise constant action $\mathbf{p}$,
updated every $\Delta t$ units of time. The direction $\mathbf{p}$ is computed
from the ABF’s position $\mathbf{x}$ through the control policy. Each episode
is ended if either the simulation time exceeds a maximal time $T_{\text{max}}$
or if the ABF reaches the target within a distance $\delta$. The reward at
step $t$ is expressed as

$r_{t}=-C\Delta
t+\|\mathbf{x}_{t-1}-\mathbf{x}_{\text{target}}\|-\|\mathbf{x}_{t}-\mathbf{x}_{\text{target}}\|,$
(2)

where $C>0$ is a constant and $\mathbf{x}_{\text{target}}$ is the target
position. The first term in eq. 2 penalizes long trajectories, while the
second term is a reward shaping that is positive when the ABF progresses
towards its target [27].

We train the policy using V-racer [28], an actor-critic off-policy RL
algorithm that was applied to falling objects [29], magnetic microswimmers
[18] and self-propelling fish [30, 31]. The algorithm is implemented in Korali
[32]. Each episode consists of about 500 experiences, and we train the agent
over 10 000 episodes.

Figure 1:  Left: Trajectories of passive tracers (blue) and controlled
swimmers (red) obtained from 100 random seeds, with the reduced order model
and $D=D_{\mathrm{sim}}$ . The circle represents the target. Right: Success
rate of the control policy against the noise level $D$ relative to that
measured from the high-fidelity simulations $D_{\mathrm{sim}}$.

The trajectories obtained with the reduced order model are shown on fig. 1 and
compared to those obtained with passive tracers. We note that passive tracers
are much more sensitive to the noise than the controlled swimmers.
Furthermore, the passive tracers do not reach the target, as opposed to the
swimmers controlled with RL.

To assess the robustness of the control policy, we vary the noise level $D$
and measure the number of times the particle reached its target out of 1000
trajectories. We define the success rate as the ratio of these two quantities.
Figure 1 demonstrates that the control policy brings the swimmer to its target
more than 98% of the time when $D<10D_{\mathrm{sim}}$. Increasing the noise
level further results in a lower success rate. However, the success rate
remains above 85% when $D\leq 20D_{\mathrm{sim}}$. In comparison, passive
particles never reach the target for any values $D\in[0,20D_{\mathrm{sim}}]$.

Figure 2:  Streamlines of the policy at bifurcations. The direction chosen by
the agent is parallel to these streamlines.

Figure 2 shows the policy at a bifurcation. The streamlines of the directions
obtained from the policy converge to a line that connects the starting point
to the target. This line seems to favor regions where the background velocity
is larger and aligned with the swimming direction, thus decreasing the overall
travel time of the swimmer. When the swimmer is away from this line, the agent
chooses a direction that brings the swimmer back to the line. This mechanism
explains the robustness of the control policy to external perturbations.

###  3.2 Control in high-fidelity blood simulations

We now apply the control policy trained on the reduced order model to the
high-fidelity model. The policy maps the swimmer’s position to a swimming
direction $\mathbf{p}$. Since the swimming direction of ABF is perpendicular
to the plane of rotation of the magnetic field, we set

$\mathbf{B}(t)=BR_{x}(\mathbf{p})\begin{pmatrix}0\\\ \cos\omega t\\\
\sin\omega t\end{pmatrix},$ (3)

where $B$ and $\omega$ are the magnitude and frequency of rotation of the
magnetic field, respectively, and $R_{x}(\mathbf{p})$ is the rotation that
transforms the vector $\mathbf{e}_{x}$ into the swimming direction
$\mathbf{p}$ with axis of rotation $\mathbf{e}_{x}\times\mathbf{p}$. The
direction $\mathbf{p}$ is computed from the policy evaluated at the ABF’s
center of mass, and the magnetic field is adapted at every time step of the
simulation.

Figure 3:  Trajectory of the ABF in the blood simulations with feedback
control via the RL policy. The ABF reaches the target (red circle) and follows
a trajectory (red line) similar to that obtained with the reduced order model.
The zoomed-in views show the ABF swimming in the capillaries.

Figure 3 shows that the ABF reaches its target with the policy learned from
the reduced order model. The ABF follows a trajectory very similar to that
obtained with the reduced order model. Furthermore, the ABF tends to align
with the flow direction and the capillaries’ centerline, indicating that the
ABF swims along the flow to reduce its travel time. Away from bifurcations,
the policy seems to keep the ABF near the center of the capillaries, where the
flow is faster. We observe that the ABF stays closer to the wall boundaries in
the vicinity of bifurcations. This strategy can have two benefits: the ABF is
closer to the correct downstream branch, and the ABF has a larger swimming
speed relative to the flow, allowing a finer control in these critical areas.

We emphasize that the policy was trained on an environment that is different
than the high-fidelity simulations but has a similar qualitative response. In
the reduced order model, we assume that the swimmer reorients instantaneously
to the control output, which is not the case in the high-fidelity blood model.
Furthermore, the velocity field is different than that obtained from the
stokes solver with the Newtonian assumption. Finally, the swimming speed of
the ABF with respect to the fluid depends on the local hematocrit, the
configuration of the surrounding RBCs and the geometry of the capillaries,
which we ignore in the reduced order model. Despite these approximations, the
policy learned by RL is successful in the high-fidelity simulations,
indicating that RL is an effective method to learn policies that are robust to
variations in the environment.

##  4 Summary

We have conducted, for the first time, high-fidelity simulations of ABF
evolving in the bloodstream through complex capillaries. The simulations
employ a high-fidelity blood model coupled with the DPD method to resolve the
flow mechanics. The ABF is propelled via an external magnetic field, which is
controlled by a RL agent with the aim of guiding the ABF towards a prescribed
target. The agent was trained on a reduced order model that was calibrated
from the blood simulations. This approach is far less computationally
expensive than using the blood simulations during training and only requires
the geometric representation of the capillaries. The control policy is robust
to external noise in the reduced order environment. Strikingly, it
successfully guides the swimmer to its target in the high-fidelity blood
simulations.

The presented approach is a candidate of choice for designing control methods
in real conditions, and requires only a moderate computational cost as the RL
agent is trained on reduced order models rather than the costly, high-fidelity
ones. Finally, this approach can be used for personalized control policy as
the geometric representation of the capillary networks was generated from
fundus images. This method can serve for the navigation of microswimmers in
the context of targeted drug delivery or microsurgery at precise locations,
and may constitute a crucial tool for precision medicine.

## Acknowledgments

We acknowledge the computational resources granted by the Swiss National
Supercomputing Center (CSCS) under the project ID “s1160”.

## Conflicts of Interest

The authors have no conflicts to disclose.

## References

  * [1] Ada-Ioana Bunea and Rafael Taboryski.  Recent advances in microswimmers for biomedical applications.  Micromachines, 11(12):1048, 2020.
  * [2] Rémi Dreyfus, Jean Baudry, Marcus L Roper, Marc Fermigier, Howard A Stone, and Jérôme Bibette.  Microscopic artificial swimmers.  Nature, 437(7060):862–865, 2005.
  * [3] Jiangfan Yu, Dongdong Jin, Kai-Fung Chan, Qianqian Wang, Ke Yuan, and Li Zhang.  Active generation and magnetic actuation of microrobotic swarms in bio-fluids.  Nature communications, 10(1):5631, 2019.
  * [4] Mariana Medina-Sánchez, Lukas Schwarz, Anne K Meyer, Franziska Hebenstreit, and Oliver G Schmidt.  Cellular cargo delivery: Toward assisted fertilization by sperm-carrying micromotors.  Nano letters, 16(1):555–561, 2016.
  * [5] Simone Schuerle, Ava P Soleimany, Tiffany Yeh, GM Anand, M Häberli, HE Fleming, Nima Mirkhani, Famin Qiu, Sabine Hauert, X Wang, et al.  Synthetic and living micropropellers for convection-enhanced nanoparticle transport.  Science advances, 5(4):eaav4803, 2019.
  * [6] Xu Wang, Jun Cai, Lili Sun, Shuo Zhang, De Gong, Xinghao Li, Shuhua Yue, Lin Feng, and Deyuan Zhang.  Facile fabrication of magnetic microrobots based on spirulina templates for targeted delivery and synergistic chemo-photothermal therapy.  ACS applied materials & interfaces, 11(5):4745–4756, 2019.
  * [7] Li Zhang, Jake J Abbott, Lixin Dong, Bradley E Kratochvil, Dominik Bell, and Bradley J Nelson.  Artificial bacterial flagella: Fabrication and magnetic control.  Applied Physics Letters, 94(6), 2009.
  * [8] Li Zhang, Jake J Abbott, Lixin Dong, Kathrin E Peyer, Bradley E Kratochvil, Haixin Zhang, Christos Bergeles, and Bradley J Nelson.  Characterizing the swimming properties of artificial bacterial flagella.  Nano letters, 9(10):3663–3667, 2009.
  * [9] Pooyath Lekshmy Venugopalan, Ranajit Sai, Yashoda Chandorkar, Bikramjit Basu, Srinivasrao Shivashankar, and Ambarish Ghosh.  Conformal cytocompatible ferrite coatings facilitate the realization of a nanovoyager in human blood.  Nano letters, 14(4):1968–1975, 2014.
  * [10] Yunus Alapan, Ugur Bozuyuk, Pelin Erkoc, Alp Can Karacakol, and Metin Sitti.  Multifunctional surface microrollers for targeted cargo delivery in physiological blood flow.  Science robotics, 5(42):eaba5726, 2020.
  * [11] Xiaojing Qi, Shuo Wang, Shuhao Ma, Keqin Han, Xin Bian, and Xuejin Li.  Quantitative prediction of rolling dynamics of leukocyte-inspired microroller in blood flow.  Physics of Fluids, 33(12), 2021.
  * [12] Santiago Muiños-Landin, Alexander Fischer, Viktor Holubec, and Frank Cichos.  Reinforcement learning with artificial microswimmers.  Science Robotics, 6(52):eabd9285, 2021.
  * [13] Simona Colabrese, Kristian Gustavsson, Antonio Celani, and Luca Biferale.  Flow navigation by smart microswimmers via reinforcement learning.  Physical review letters, 118(15):158004, 2017.
  * [14] Simona Colabrese, Kristian Gustavsson, Antonio Celani, and Luca Biferale.  Smart inertial particles.  Physical Review Fluids, 3(8):084301, 2018.
  * [15] Luca Biferale, Fabio Bonaccorso, Michele Buzzicotti, Patricio Clark Di Leoni, and Kristian Gustavsson.  Zermelo’s problem: optimal point-to-point navigation in 2D turbulent flows using reinforcement learning.  Chaos: An Interdisciplinary Journal of Nonlinear Science, 29(10), 2019.
  * [16] Jaya Kumar Alageshan, Akhilesh Kumar Verma, Jérémie Bec, and Rahul Pandit.  Machine learning strategies for path-planning microswimmers in turbulent flows.  Physical Review E, 101(4):043110, 2020.
  * [17] Francesco Borra, Luca Biferale, Massimo Cencini, and Antonio Celani.  Reinforcement learning for pursuit and evasion of microswimmers at low reynolds number.  Physical Review Fluids, 7(2):023103, 2022.
  * [18] Lucas Amoudruz and Petros Koumoutsakos.  Independent control and path planning of microswimmers with a uniform magnetic field.  Advanced Intelligent Systems, 4(3):2100183, 2022.
  * [19] Yuguang Yang, Michael A Bevan, and Bo Li.  Hierarchical planning with deep reinforcement learning for 3d navigation of microrobots in blood vessels.  Advanced Intelligent Systems, 4(11):2200168, 2022.
  * [20] Pejhman Ghassemi, Jianting Wang, Anthony J Melchiorri, Jessica C Ramella-Roman, Scott A Mathews, James C Coburn, Brian S Sorg, Yu Chen, and T Joshua Pfefer.  Rapid prototyping of biomimetic vascular phantoms for hyperspectral reflectance imaging.  Journal of biomedical optics, 20(12):121312–121312, 2015.
  * [21] Lucas Amoudruz, Athena Economides, Georgios Arampatzis, and Petros Koumoutsakos.  The stress-free state of human erythrocytes: Data-driven inference of a transferable RBC model.  Biophysical Journal, mar 2023.
  * [22] Robert D Groot and Patrick B Warren.  Dissipative particle dynamics: Bridging the gap between atomistic and mesoscopic simulation.  The Journal of chemical physics, 107(11):4423–4435, 1997.
  * [23] Lucas Amoudruz.  Simulations and Control of Artificial Microswimmers in Blood.  PhD thesis, ETH Zurich, 2022.
  * [24] Alireza Yazdani, Mingge Deng, Bruce Caswell, and George Em Karniadakis.  Flow in complex domains simulated by dissipative particle dynamics driven by geometry-specific body-forces.  Journal of Computational Physics, 305:906–920, 2016.
  * [25] Petr Karnakov, Fabian Wermelinger, Sergey Litvinov, and Petros Koumoutsakos.  Aphros: High performance software for multiphase flows with large scale bubble and drop clusters.  In Proceedings of the Platform for Advanced Scientific Computing Conference, pages 1–10, 2020.
  * [26] Dmitry Alexeev, Lucas Amoudruz, Sergey Litvinov, and Petros Koumoutsakos.  Mirheo: High-performance mesoscale simulations for microfluidics.  Computer Physics Communications, 254:107298, 2020.
  * [27] Andrew Y Ng, Daishi Harada, and Stuart Russell.  Policy invariance under reward transformations: Theory and application to reward shaping.  In ICML, volume 99, pages 278–287. Citeseer, 1999.
  * [28] Guido Novati and Petros Koumoutsakos.  Remember and forget for experience replay.  In Proceedings of the 36th International Conference on Machine Learning, 2019.
  * [29] Guido Novati, Lakshminarayanan Mahadevan, and Petros Koumoutsakos.  Controlled gliding and perching through deep-reinforcement-learning.  Physical Review Fluids, 4(9):093902, 2019.
  * [30] Siddhartha Verma, Guido Novati, and Petros Koumoutsakos.  Efficient collective swimming by harnessing vortices through deep reinforcement learning.  P. Natl. Acad. Sci., page 201800923, may 2018.
  * [31] Ioannis Mandralis, Pascal Weber, Guido Novati, and Petros Koumoutsakos.  Learning swimming escape patterns for larval fish under energy constraints.  Physical Review Fluids, 6(9):093101, 2021.
  * [32] Sergio M Martin, Daniel Wälchli, Georgios Arampatzis, Athena E Economides, Petr Karnakov, and Petros Koumoutsakos.  Korali: Efficient and scalable software framework for bayesian uncertainty quantification and stochastic optimization.  Computer Methods in Applied Mechanics and Engineering, 389:114264, 2022.

  *[ABF]: artificial bacterial flagellum
  *[RL]: reinforcement learning
  *[RBC]: red blood cell
  *[DPD]: dissipative particle dynamics
  *[RBCs]: red blood cell
  *[SDE]: stochastic differential equation