∎

††thanks: 1 Ricardo Bedin Grando are with the Technological Unit,
Techonological University of Uruguay – UTEC, Rivera, Uruguay. E-mail:
<EMAIL_ADDRESS>††thanks: 2Junior Costa de Jesus and Paulo L. J.
Drews-Jr are with the NAUTEC, Centro de Ciências Computacionais, Univ. Fed. do
Rio Grande – FURG, RS, Brazil. E-mail<EMAIL_ADDRESS>††thanks: 3Victor
Augusto Kich and Alisson Henrique Kolling are with Federal University of Santa
Maria, Brazil. E-mail<EMAIL_ADDRESS>11institutetext:  Ricardo Bedin
Grando 22institutetext: Technological University of Uruguay
Rivera, Uruguay
22email<EMAIL_ADDRESS>Junior Costa de Jesus
44institutetext: Paulo Lilles Jorge Drews-Jr 55institutetext: Federal
University of Rio Grande
Rio Grande, RS, Brazil
55email<EMAIL_ADDRESS><EMAIL_ADDRESS>Victor Augusto
Kich 77institutetext: Alisson Henrique Kolling88institutetext: Federal
University of Santa Maria
Santa Maria, RS, Brazil
88email<EMAIL_ADDRESS><EMAIL_ADDRESS>
# Double Critic Deep Reinforcement Learning for Mapless 3D Navigation of
Unmanned Aerial Vehicles

Ricardo Bedin Grando1 Junior Costa de Jesus2 Victor Augusto Kich3 Alisson
Henrique Kolling3 Paulo Lilles Jorge Drews-Jr2

(Received: date / Accepted: date)

###### Abstract

This paper presents a novel deep reinforcement learning-based system for 3D
mapless navigation for Unmanned Aerial Vehicles (UAVs). Instead of using a
image-based sensing approach, we propose a simple learning system that uses
only a few sparse range data from a distance sensor to train a learning agent.
We based our approaches on two state-of-art double critic Deep-RL models: Twin
Delayed Deep Deterministic Policy Gradient (TD3) and Soft Actor-Critic (SAC).
We show that our two approaches manage to outperform an approach based on the
Deep Deterministic Policy Gradient (DDPG) technique and the BUG2 algorithm.
Also, our new Deep-RL structure based on Recurrent Neural Networks (RNNs)
outperforms the current structure used to perform mapless navigation of mobile
robots. Overall, we conclude that Deep-RL approaches based on double critic
with Recurrent Neural Networks (RNNs) are better suited to perform mapless
navigation and obstacle avoidance of UAVs.

###### Keywords:

Twin Delayed Deep Deterministic Policy Gradients Soft Actor-Critic Deep
Reinforcement Learning Navigation for Aerial Vehicles

##  1 Introduction

In recent years, Deep Reinforcement Learning (Deep-RL) has been employed in a
wide range of fields, achieving interesting results in many tasks for control
of discrete systems dulac2015deep  and continuous systems
lillicrap2015continuous . More recently, the learning was applied in tasks in
robotics as well, where it was initially used to handle tasks in stable and
observable environments duan2016benchmarking . For mobile robotics, however,
the complexity increases significantly given the interactions with barriers in
the physical workplace drews2016 ; xie2017towards . In this context, Deep-RL
ended up simplifying the problem by discretizing it zhu2017target . Other
approaches have also been exploring continuous control actions in the
navigation of mobile robots and achieving interesting results tai2017virtual ,
chen2017socially , jesus2019deep . However, the results are yet limited for
autonomous Unmanned Aerial Vehicles (UAVs) sampedro2019fully ;
kang2019generalization

In our previous work, we proposed the adaptation of two Deep-RL techniques for
an UAV in two environments grando2020deep  and explored the concept for mobile
robots de2021soft . Based on the state-of-art structure for terrestrial mobile
robots tai2017virtual , we investigated the performance of Deep-RL for 2D
mapless navigation related tasks of UAVs. However, we propose a new structure
for the agents for 3D navigation and develop a new deterministic approach
based on the Twin Delayed Deep Deterministic Policy Gradient (TD3)
fujimoto2018addressing  extending our previous work. Instead of using the
conventional structure used to perform mapless navigation of terrestrial
mobile robots, we propose a new methodology with Recurrent Neural Networks
(RNNs) with more sophisticated Deep-RL approaches.

The objective of this work is to show how well Deep-RL approaches based on
RNNs can perform in 3D navigation-related tasks. In this work we also include
environmental difficulties, such as simulated wind, to improve the overall
robustness. Our approach was done in such a way that the network used had only
26 inputs and 3 outputs, as shown in Figure 1. As inputs, it was used readings
of a distance sensor coupled to the upper part of the vehicle, the previous
linear velocities, the delta yaw angle and the distance and angle of the UAV
related to the target. The actions were set as the outputs of the network to
send it to the vehicle to arrive at the target. We expect that the intelligent
agent will not just be able to arrive at the target goal but also be able to
avoid any possible collision on the way to get it. We also expect the agents
to perform through the environmental difficulties, showing robustness on the
adaptation. Our improved deterministic approach (3DDC-D) and our bias
stochastic approach (3DDC-S) are set to be compared with our previous Deep-RL
approaches and with a behavior-based approach inspired in the work of
marino2016minimalistic  that can be used to perform mapless navigation of
UAVs. 111 Code available at
https://github.com/ricardoGrando/hydrone_deep_rl_jint. 222 Video available at
https://www.youtube.com/watch?v=Kf5T2SgyiRs.

Figure 1: Our second simulated environment (left) and Deep-RL structure
(right).

This work has the following contributions:

  * •

We present a novel methodology that only uses range-data readings and the
vehicle’s relative localization data to successfully perform goal-oriented
mapless 3D navigation and waypoint 3D mapless navigation for an UAV.

  * •

We show that our new methodology based on RNNs improves the results obtained
from the state-of-the-art Deep-RL for terrestrial robots tai2017virtual  and
in our previous work grando2020deep .

  * •

We present a novel deterministic approach based on the double critic actor-
critic TD3 algorithm that outperforms the single critic DDPG algorithm used in
previous works grando2020deep , bedin2021deep .

  * •

We show that our agents manage to learn environmental difficulties, such as
wind, showing that learning-based approaches are more robust and adapt better
than behavior-based algorithms.

  * •

We show that with our novel methodology the robot improves its ability to
avoid collision with obstacles, outperforming our previous approaches in the
avoidance capability.

The work has the following structure: related works are addressed in the Sec.
2. Our novel approaches can be seen in the the Sec. 3. Then, in the Section 5
the tools and techniques that we used are presented. The results achieved and
the comparison with out previous structures and algorithms are presented in
the Sec. 6. For the last, Sec. 7 shows our our overall analysis of the results
and the perspective for future works.

##  2 Related Works

Deep-RL surveys in robotics tasks for the navigation problem have already been
performed to demonstrate how effectively we can solve this problem with
learning approaches kober2013reinforcement , kormushev2013reinforcement ,
tobin2017domain , bedin2021deep .

Tai _et al._ tai2017virtual  used 10-dimensional range findings and a goal
location as inputs and continuous steering commands as outputs for a 2D
mapless motion planner when applied to mobile robots. They concluded that an
approach based on the DDPG method with a fully-connected ANN of three hidden
layers would effectively train a 2D mapless motion planner. Furthermore, the
process of completing the task of reaching a predetermined goal was completed
successfully.

Zhu _et al._ zhu2017target  used an actor-critic Deep-RL model to tackle the
challenge of driving a mobile robot. It took the current state observation and
an image of the target as inputs, producing the action in a 3D environment as
output. Chen _et al._ chen2017socially  discussed the issue of navigation in a
pedestrian-rich environment. It proposed a Deep-RL approach to control a
mobile robot that could navigate in a speed close to a human walking speed. It
also performed in a scenario with multiple humans simulating a crowd. Ota _et
al._ ota2020efficient  developed an approch based on the SAC technique to
efficiently navigate. It was also done in simulation, with the agent learning
the course.

For UAVs, however, Deep-RL has been addressed fewer times. Rodriguez _et al._
rodriguez2018deep  used DDPG to solve the problem of landing on a moving
platform. On the Gazebo simulator, they used Deep-RL in conjunction with the
RotorS framework furrer2016rotors  for aerial vehicles. Sampedro _et al._
sampedro2019fully  suggested that an approach based on the DDPG technique
could perform a task of Search and Rescue in an indoor environment. Sampedro
_et al._ sampedro2019fully  made the use of a real and simulated UAV to
perform this navigation task. Kang _et al._ kang2019generalization  focused on
tasks realted to avoid collision with obstacles. This work also used visual
information and real and simulated data.

He _et al._ he2020deep  combines imitation learning and TD3 technique and
applied on the challenging 3D UAV navigation problem using depth cameras and
sketched in a variety of simulation environments, obtaining great results. Li
_et al._ li2020uav  combines Deep-RL with meta-learning and proposes a novel
approach, named Meta Twin Delayed Deep Deterministic Policy Gradient (Meta-
TD3), to realize the control of an UAV. Thereafter, Li _et al._ compared your
method with DDPG and TD3, obtaining a substantial improvement in terms of both
convergence value and convergence rate.

Differently, our method addresses 3D mapless navigation and waypoint mapless
navigation tasks by relying solely on laser sensor readings and the vehicle’s
relative localization data. Since the TD3 and SAC techniques are state-of-art
for terrestrial mobile robots, we based our approach on them. We also make use
of a LSTM-based structure instead of a MLP structure, showing that RNN based
structures are better suited for this problem. We also differ by imposing
environmental difficulties such as wind, which makes the overall results worse
but more realistic.

##  3 Methodology

In this work, we propose a system for UAVs that can learn to navigate from a
starting position on space to a target position and through a series of target
points, building its own motion plan on environments with obstacles. It was
designed in such a way that it only needs a couple of range sensing data and
the vehicles’ relative localization data to perform the mapless navigation and
the obstacle avoidance. Its movement equation can be defined as:

$\upsilon_{t}=f(x_{t},p_{t},\upsilon_{t-1}),$ (1)

where $x_{t}$ is the raw information from the sensor readings, $p_{t}$ the
relative position and angle of the target and $\upsilon_{t-1}$ the vehicle’s
velocity in the last step. This model allows obtaining the actions that the
robot is able to make, given its current state $s_{t}$. Embodied in a neural
network, the expected result is an action for the current state. This arrange
for the data was explored in Tai et al tai2017virtual , Jesus et al in
jesus2019deep  and in our previous work grando2020deep  for 3D navigation of
UAV.

####  3.0.1 Networks Structure

We defined a structure that has 26 inputs and 3 outputs. Among the 26 inputs,
20 represent range findings from a simulated Lidar, 3 represent the previous
action and the other 3 represent information regarding the target goal. Our
Lidar provides 1080 samples in a $270\degree$ range, 20 of which sampled
equally spaced in $13.5\degree$. The target information used was the vehicles’
relative distance to the goal and two relative angles to the target, one given
by the plan $x-y$ and the second given by the plan $z-distance$. Both the
relative distance and angles were used to make the agent learns towards their
minimization and achieve the goal. The network’s outputs provide the action
for the given step, which are the linear and altitude velocities and the
variation of the yaw angle. These actions are then applied to the vehicle.
Figure 2 shows the proposed architecture.

Figure 2: Network Inputs and Outputs.

Our newly proposed network architecture has a 32-cell LSTM layer for the
actor-network activated using the ReLU function. The hyperbolic tangent
function was used as the activation function in the actor’s outputs. The
values are then scaled between $0$ and $0.25$ $m/s$ for the linear velocity,
$-0.25$ and $0.25$ $rads$ for the delta yaw and $-0.25$ and $0.25$ $m/s$ for
the altitude velocity. The Q-value for the current state and action performed
by the agent is given in the critic-network. We projected the Deep-RL-based
agents similarly so we compare the more fairly the characteristics of each
approach. For the SAC-based approach, specifically, the value network was set
with the same structure as the critic network. Figure 3 shows our new proposed
network structure while Figure 4 shows the previous architecture proposed by
Tai et al. tai2017virtual , also used in our previous work grando2020deep .

Figure 3: Proposed new model architecture.

####  3.0.2 Reward Function

With the environment and the agents defined, it is feasible to train and
evaluate the agents. Before training, it is necessary to create a reward and
penalty system for the agent. These values given to the agents are numbers of
a function model, based on empirical knowledge, created during the process of
solution of the problem.

In our previous work, we defined a system that there were three conditions. In
this current work, one of the conditions regarding the distance in each step
was not used. As consequence, it caused an increased training time. However,
this simplified reward function presented better results at the end of
training when compared with the previous one that was proposed. The reward
function used in this work was the following:

$r(s_{t},a_{t})=\begin{cases}r_{arrive}\ \textrm{if}\ d_{t}<c_{d}\\\
r_{collide}\ \textrm{if}\ min_{x}<c_{o}\end{cases}$ (2)

Only two rewards were given, one for making the task correctly and the second
one in case of failure. The agent receives a positive $r_{arrive}$ reward of
$100$ when the goal is reached in a margin of $c_{d}$ meters This margin,
specifically, was set as $0.5m$. Also, in case of a collision with an obstacle
or with the limits of the scenario, we give a negative reward $r_{collide}$ of
$-10$. The collisions is verified if the readings from the distance sensor is
lower than a distance $c_{o}$ of $0.5$ meters. This simplified rewarding
system also helps to focus on the Deep-RL approaches themselves, their
similarities and differences, instead of focusing on the environment.

##  4 Comparative Approaches

In one of our latest works bedin2021deep , we proposed two 3D quad-rotor
navigation approaches: a deterministic based on DDPG, and a stochastic based
on SAC, both using MLP3. These algorithms are named 3DNDRL-D and 3DNDRL-S,
respectively. These approaches have the same structure like the ones that we
used in our previous work grando2020deep , only adapted to perform the 3D
mapless navigation of a hybrid vehicle instead of the 2D mapless navigation of
an UAV. The networks of 3DNDRL-D and 3DNDRL-S approaches have 3 hidden layers
each. The structure used was a fully-connected multi-layer perception, where
each layers has 512 neurons each. the ReLU activation was used as well. Fig 4
summarizes this structures.

Figure 4: 3-Layers MLP model architecture.

The other algorithm that we used to compare was inspired in the work of Marino
_et al._ marino2016minimalistic . It uses a BUG-like algorithm to perform a 2D
simulated multi-floor indoor navigation with a quad-rotor robot, equipped with
a salient-cue sensor and a laser-range-finder. As well known, the BUG2 tries
to minimize the distance between the robot and the goal, toward the estimated
position of the goal on the horizontal plane while avoiding obstacles. So, it
is possible to implement a local navigation approach for navigating in the
horizontal plane.

##  5 Experimental Setup

For the analysis of this work, in this section is described the setup, tools
and materials that were used. Thus, making it clear the programming language
and framework utilized for our simulation, as well as, to describe the
environments purposed for the task of navigation of a hybrid UAV concept.

###  5.1 Trainig Setup

In this article, the experiments carried out were programmed using the Python
programming language. The neural networks were developed with PyTorchpytorch ,
which provides tensor computation with acceleration via GPU and an automatic
differentiation system and has a varied range of applications such as in
fields of natural language processing, image processing, object recognition,
and othersrukhovich2021iterdet ; Tao2020HierarchicalMA ; shoeybi2020megatronlm
. For the robot simulation, we used the Robot Operating System(ROS)
quigley2009ros  together with Gazebo. ROS is a framework for developing
applications for robots with a collection of tools and libraries that provides
operating system standard services, such as hardware abstraction, device low-
level control and package management. For this work, it was used the Melodic
version of ROS. The gazebo is an open-source 3D robotics simulator, that
provides an integrated physics simulator, 3D rendering and support for the
integration and simulation of sensors and controllers, has available plugins
for the simulation of aerial autonomous vehicles and is fully integrated with
ROS.

###  5.2 Vehicle’s description

The experiments were conducted utilizing an UAV, which consists of a quad-
rotor initially developed as a hybrid vehicle capable of transit through air
and water, as shown in previous worksdrews2014 ; neto2015 . This hybrid UAV,
improved with new models and propellershorn2019study ; horn_lars2020  and with
the Ardupilot and ROS-based control system concept for our developed HUAUV
grando2019deep , succeeded to win the 2019 Brazil Open Flying Robot League
during the 2019 Latin America Robotics Competitiongrando_competi2020 .

To simulate our vehicle, we created a 3D model and a universal robot
description. The package containing the description and the Deep-RL approaches
are provided in code333
$https://github.com/ricardoGrando/hydrone\\_deep\\_rl\\_jint$ . We used the
RotorS framework to create the description of the vehicle using real
dimensions and constants (such as mass, inertia, motor constants, etc). The
visual meshes are based on the model that was developed in Solidworks
software. And we used a simplified collision structure, approximated by cubes
and cylinders.

From the network, our robot receives a linear velocity, an altitude velocity
and a $\Delta$yaw. To control the robot, we used the RotorS’ internal
geometric tracking controller. The conversion from the linear and angular
velocities and the $\Delta$yaw to Cartesian velocities is needed, so a plugin
was developed to convert from polar to Cartesian coordinates. With this
structure, the agents managed to learn the vehicle’s properties, suppress the
environmental difficulties and perform navigation-related tasks.

###  5.3 Simulated Environments

The development of applications for robots is usually difficult to conceive
and maintain given the high cost of hardware and equipment. Many applications
in robotics are then developed in simulation and then deployed to real robots.
Therefore, a good simulation environment is necessary to make progress in the
field. Gazebo simulator koenig2004design  can be used to fulfill this need,
given its extensive compatibility and realistic simulation.

() First environment.

() First environment from above.

() Second environment

() Second environment from above.

Figure 5: 3D Simulated Environments.

Two environments were used to train and test our agents. The first environment
is essentially a box of 10 meters of side, with free space to perform the
navigation. The second environment is just like the first environment but with
4 fixed obstacles added to it. These obstacles were set to represent drilling
risers of an industrial scenario, where this approach could be applied. Both
environments have $5\times 5m$ of area, as can be seen in Figure 5. The blue
lines are the vehicle’s sensor rays.

We also used simulated wind to increase the difficulty. The wind was simulated
using the Ornstein-Uhlenbeck noise in the three axis, with a velocity set to
range from $-0.175$ to $0.175$ m/s. This is considerably high for an indoor
environment, representing $70$% of the maximum velocity that the vehicle can
achieve. Empirically, much higher values made the scenario almost unnavigable,
while lower values present no much influence on the overall results.

##  6 Results

In this section, we address the results obtained during the evaluation phase.
The evaluation was carried out with two tasks and several statistics were
recorded for each one. In the first task, the navigation task, the vehicle
should navigate from a point in space to another, while multiples points
should be visited in the second task, the waypoint navigation task. The
simulated wind was used in both task during the test and the evaluation. Also,
an agent was trained for each one of the scenarios.

Both tasks were evaluated in both scenarios for a total of 100 trials each.
The total of successful trails represents the percentage of navigation where
the vehicle reached the desired goal in the first task, while in the second
task the success represents the percentage of the trail where the trajectory
was completed, from the first point through the final one. The average time
represents the total time that was needed to complete all the 100 trials
divided by 100. For the second task only, it was also registered the average
percentage of the path navigated. For example, if a percentage of 10%, the
vehicle has navigated on average through 10% of all points. This metric was
used to evaluate the learning approaches as well as the BUG2 algorithm. We
defined an initial position for the vehicle at the point (0.0, 0.0, 2.5) and
placed the front part of it pointed towards the positive $x$-axis.

During the training phase, the data from the rewards was collected. We
collected the data for 1000 episodes in the first scenario and 1500 episodes
in the second scenario. The final episode of training was chosen based on the
stagnation of the average reward received. Figures 6 and 6 show the rewards’
moving average of the previous 300 episodes for the first and second scenario,
respectively.

() First environment.

() Second environment.

Figure 6: Moving average of the reward over 300 episodes of the training.

It is possible to observe in the Figures 6 and 6 that the 3DDC-S and 3DDC-D
approaches received the overall higher rewards, manly our newly proposed
approaches based on an RNN. As the obtained reward is intrinsically attached
to the performance of the agent, we are able to estimate the degree of
learning of the agent in the environment. For that, we show the statistics of
the navigation task (Table 1) and for the second task (Table 2).

Table 1: Statistics of the performed navigation for the first task. Scenario | Approach | Success | Average Time (s)
---|---|---|---
1 | 3DDC-S | 100% | $56.48\pm 25.71$
1 | 3DDC-D | 100% | $36.32\pm 18.67$
1 | 3DNDRL-S | 100% | $27.03\pm 18.67$
1 | 3DNDRL-D | 100% | $\textbf{13.69}\pm\textbf{0.85}$
1 | 3DBUG2 | 100% | $21.92\pm 1.341$
2 | 3DDC-S | 69% | $53.82\pm 25.10$
2 | 3DDC-D | 63% | $\textbf{19.56}\pm\textbf{4.28}$
2 | 3DNDRL-S | 19% | $39.43\pm 22.21$
2 | 3DNDRL-D | 17% | $49.27\pm 38.32$
2 | 3DBUG2 | 32% | $51.93\pm 26.83$
Table 2: Statistics of the performed navigation for the second task. Scenario | Approach | Success | Average Time (s) | Percentage
---|---|---|---|---
1 | 3DDC-S | 94% | $155.82\pm 60.69$ | 95.34%
1 | 3DDC-D | 27% | $200.44\pm 17.51$ | 71.42%
1 | 3DNDRL-S | 10% | $135.22\pm 61.15$ | 47.08%
1 | 3DNDRL-D | 77% | $\textbf{76.99}\pm\textbf{29.23}$ | 86.44%
1 | 3DBUG2 | 100% | $145.94\pm 19.45$ | 100%
2 | 3DDC-S | 26% | $145.16\pm 92.29$ | 37.95%
2 | 3DDC-D | 41% | $\textbf{85.88}\pm\textbf{67.88}$ | 51.632%
2 | 3DNDRL-S | 0% | - | 3.46%
2 | 3DNDRL-D | 0% | - | 2.24%
2 | 3DBUG2 | 3% | $90.95\pm 93.07$ | 11.42%

In the first task, we can see that the Deep-RL approaches and the BUG2
algorithm performed with 100% success in the first scenario, with
distinctiveness for the DDPG-based agent which yields the lowest average time
and standard deviation. In the second environment, however, it becomes more
clear how better are the agents based on the RNN are. Besides the average 60%
plus of success, the Deep-RL-LSTM-based approaches outperformed by almost
double the BUG2 algorithm, with a special highlight for the 3DDC-D approach
that performed the task consistently faster than the other approaches.

For the second task, it becomes even more clear the advantages of a Deep-RL
approach with RNN architecture. We can see especially in the second
environment how better the agents performed, even when compared with the BUG2
algorithm. The 3DDC-D approach managed to complete on average 51.65% of the
trajectory while the 3DDC-S agent navigates almost 40%. Both results are
considerably higher than the DDPG(8.36%) and BUG2(11.43%), outperforming by
almost a factor of five. We can also see in table 1 and 2 that the average
time of our adapted 3DNDRL-D was better in the first environment, while the
3DDC-D presented the fastest results in the second. That can be due to fact
that a simpler deterministic approach tends to perform better in a simple
environment, while a most robust one can be more efficient in a more complex
one. It is also interesting to notice that the stochastic approach was
considerably slower given its characteristics, but showed to be more trusty
than the deterministic one. Overall, it is important to highlight how well
agents managed to understand the vehicle and the difficulties that the
environment provided. We can see based on the results of the BUG2 algorithm,
how challenging was to perform this task in an environment with real simulated
wind and dynamics.

Finally we can observe both task performed in both environments in the Figures
7, 8, 9 and 10. It shows the 100 trials performed for each one of the
approaches. We can see even better the more aleatory characteristic of the
stochastic approach when compared to the deterministic one. The 100 samples
for both tasks show that this first approach presents a more exploratory
trajectory, which affects the overall time to perform it. It is also possible
to verify that even the deterministic approach and the BUG2 algorithm suffered
considerably the influence of the simulated wind, showing trajectories with
consistent changes.

In the second task specifically, we can see in Figure 8 one of the main
contribution of this work. It is possible to outline the ability to learn the
simulated wind and to adapt to perform the navigation correctly. When compared
to the the stochastic approach (Figure 8) or even to the BUG2 algorithm (8) we
can see the more stable trajectories which provided the overall better
results. For the second task in the second environment the same can be said,
while 3DDC-D and 3DDC-S learned the task and the environmental difficulties,
the previous methods and the traditional method presented consistently more
difficulties.

() Trajectory 3DNDRL-S method.

() Upper view trajectory 3DNDRL-S method.

() Trajectory 3DNDRL-D method.

() Upper view trajectory 3DNDRL-D method.

() Trajectory 3DDC-D method.

() Upper view trajectory 3DDC-D method.

() Trajectory 3DDC-S method.

() Upper view trajectory 3DDC-S method.

() Trajectory BUG2 method.

() Upper view trajectory BUG2 method.

Figure 7: Navigation task in the first environment.

() Trajectory 3DNDRL-S method.

() Upper view trajectory 3DNDRL-S method.

() Trajectory 3DNDRL-D method.

() Upper view trajectory 3DNDRL-D method.

() Trajectory 3DDC-D method.

() Upper view trajectory 3DDC-D method.

() Trajectory 3DDC-S method.

() Upper view trajectory 3DDC-S method.

() Trajectory BUG2 method.

() Upper view trajectory BUG2 method.

Figure 8: Navigation task in the second environment.

() Trajectory 3DNDRL-S method.

() Upper view trajectory 3DNDRL-S method.

() Trajectory 3DNDRL-D method.

() Upper view trajectory 3DNDRL-D method.

() Trajectory 3DDC-D method.

() Upper view trajectory 3DDC-D method.

() Trajectory 3DDC-S method.

() Upper view trajectory 3DDC-S method.

() Trajectory BUG2 method.

() Upper view trajectory BUG2 method.

Figure 9: Waypoint navigation task in the first environment.

() Trajectory 3DNDRL-S method.

() Upper view trajectory 3DNDRL-S method.

() Trajectory 3DNDRL-D method.

() Upper view trajectory 3DNDRL-D method.

() Trajectory 3DDC-D method.

() Upper view trajectory 3DDC-D method.

() Trajectory 3DDC-S method.

() Upper view trajectory 3DDC-S method.

() Trajectory BUG2 method.

() Upper view trajectory BUG2 method.

Figure 10: Waypoint navigation task in the second environment.

The trials of the 3DDC-D approach (Figures 10 10) and the 3DDC-S approach
(Figures 10 10) in the second scenario shows how effective these two
approaches became when comparing with the others. Both managed to understand
the task, the environment and to avoid obstacles with great distinctiveness.

##  7 Conclusion

In this work, we proposed a novel methodology based on the state-of-art Deep-
RL techniques to address the mapless 3D navigation and waypoint 3D navigation
tasks for UAVs. We showed that approaches with double critic and RNNs are
better suited than the state-of-art approaches with fully-connected ANNs,
commonly used for terrestrial mobile robots. Our proposed system is able to
perform navigation-related tasks by only using the vehicles’ relative
localization data and some range findings, without making the use of imaged-
based sensing which can demand a costly hardware capability to achieve good
performance. We can conclude that our approaches based on double critic actor-
critic SAC and TD3 algorithms also managed to bypass obstacles and reach the
desired goal.

Specifically, it is also possible to conclude that our novel approaches
outperformed a DDPG-based approach and the BUG2 algorithm. The good results
achieved can be due to a simple sensory-based structure and a simple rewarding
system.

As future works, we are aiming to test in our real vehicle and develop a
similar approach for Unmanned Underwater Vehicles (UUVs) and Hybrid Unmanned
Aerial Underwater Vehicles (HUAUVS).

## Aknowledgement

The authors would like to thank to National Council for Scientific and
Technological Development (CNPq), the Coordination for the Improvement of
Higher Education Personnel (CAPES) - Finance Code 001, PRH-ANP and all
participants of VersusAI.

## Declarations

Ethical Approval The article has the approval of all the authors.

Consent to Participate All the authors gave their consent to participate in
this article.

Consent to Publish The authors gave their authorization for the publishing of
this article.

Authors’ contributions:

  * •

Ricardo Bedin Grando conceived the research, writing the article, designed and
program the experiments, collected and processed the test data.

  * •

Junior Costa de Jesus writing of the article, collected and processed the test
data.

  * •

Victor Augusto Kich write the article, program the experiments, collected and
processed the test data.

  * •

Alisson Henrique Kolling write the article, program the experiments, collected
and processed the test data.

  * •

Paulo Lilles Jorge Drews Jr. conceived the research, writing of the article
and discussion of the main ideas of the article.

Funding National Council for Scientific and Technological Development (CNPq),
Coordination for the Improvement of Higher Education Personnel (CAPES) and
National Agency of Petroleum, Natural Gas and Biofuels (PRH-ANP).

Competing interest There are not conflict of interest or competing interest.

Available of data and material available in GitHub
https://github.com/ricardoGrando/hydrone_deep_rl_jint.

## References

  * (1) G. Dulac-Arnold, R. Evans, H. van Hasselt, P. Sunehag, T. Lillicrap, J. Hunt, T. Mann, T. Weber, T. Degris, and B. Coppin, “Deep reinforcement learning in large discrete action spaces,” _arXiv preprint arXiv:1512.07679_ , 2015.
  * (2) T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” in _ICLR_ , 2015.
  * (3) Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel, “Benchmarking deep reinforcement learning for continuous control,” in _International conference on machine learning_. PMLR, 2016, pp. 1329–1338.
  * (4) P. L. J. Drews-Jr, E. Hernández, A. Elfes, E. R. Nascimento, and M. F. M. Campos, “Real-time monocular obstacle avoidance using underwater dark channel prior,” in _IEEE/RSJ IROS_ , 2016, pp. 4672–4677.
  * (5) L. Xie, S. Wang, A. Markham, and N. Trigoni, “Towards monocular vision based obstacle avoidance through deep reinforcement learning,” _arXiv preprint arXiv:1706.09829_ , 2017.
  * (6) Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi, “Target-driven visual navigation in indoor scenes using deep reinforcement learning,” in _IEEE ICRA_ , 2017, pp. 3357–3364.
  * (7) L. Tai, G. Paolo, and M. Liu, “Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation,” in _IEEE/RSJ IROS_ , 2017, pp. 31–36.
  * (8) Y. F. Chen, M. Everett, M. Liu, and J. P. How, “Socially aware motion planning with deep reinforcement learning,” in _IEEE/RSJ IROS_ , 2017, pp. 1343–1350.
  * (9) J. C. Jesus, J. A. Bottega, M. A. Cuadros, and D. F. Gamarra, “Deep deterministic policy gradient for navigation of mobile robots in simulated environments,” in _ICAR_ , 2019, pp. 362–367.
  * (10) C. Sampedro, A. Rodriguez-Ramos, H. Bavle, A. Carrio, P. de la Puente, and P. Campoy, “A fully-autonomous aerial robot for search and rescue applications in indoor environments using learning-based techniques,” _Journal of Intelligent & Robotic Systems_, vol. 95, no. 2, pp. 601–627, 2019.
  * (11) K. Kang, S. Belkhale, G. Kahn, P. Abbeel, and S. Levine, “Generalization through simulation: Integrating simulated and real data into deep reinforcement learning for vision-based autonomous flight,” in _IEEE ICRA_ , 2019, pp. 6008–6014.
  * (12) R. B. Grando, J. C. de Jesus, and P. L. Drews-Jr, “Deep reinforcement learning for mapless navigation of unmanned aerial vehicles,” in _2020 Latin American Robotics Symposium (LARS), 2020 Brazilian Symposium on Robotics (SBR) and 2020 Workshop on Robotics in Education (WRE)_. IEEE, 2020, pp. 1–6.
  * (13) J. C. de Jesus, V. A. Kich, A. H. Kolling, R. B. Grando, M. A. d. S. L. Cuadros, and D. F. T. Gamarra, “Soft actor-critic for navigation of mobile robots,” _Journal of Intelligent & Robotic Systems_, vol. 102, no. 2, pp. 1–11, 2021.
  * (14) S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 1587–1596.
  * (15) R. Marino, F. Mastrogiovanni, A. Sgorbissa, and R. Zaccaria, “A minimalistic quadrotor navigation strategy for indoor multi-floor scenarios,” in _Intelligent Autonomous Systems 13_. Springer, 2016, pp. 1561–1570.
  * (16) R. Bedin Grando, J. Costa de Jesus, V. A. Kich, A. H. Kolling, N. Pieper Bortoluzzi, P. M. Pinheiro, A. Alves Neto, and P. L. J. Drews Jr, “Deep reinforcement learning for mapless navigation of a hybrid aerial underwater vehicle with medium transition,” _IEEE International Conference on Robotics and Automation (ICRA)_ , pp. arXiv–2103, 2021.
  * (17) J. Kober, J. A. Bagnell, and J. Peters, “Reinforcement learning in robotics: A survey,” _The International Journal of Robotics Research_ , vol. 32, no. 11, pp. 1238–1274, 2013.
  * (18) P. Kormushev, S. Calinon, and D. G. Caldwell, “Reinforcement learning in robotics: Applications and real-world challenges,” _Robotics_ , vol. 2, no. 3, pp. 122–148, 2013.
  * (19) J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in _2017 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2017, pp. 23–30.
  * (20) K. Ota, Y. Sasaki, D. K. Jha, Y. Yoshiyasu, and A. Kanezaki, “Efficient exploration in constrained environments with goal-oriented reference path,” in _IEEE/RSJ IROS_ , 2020.
  * (21) A. Rodriguez-Ramos, C. Sampedro, H. Bavle, I. G. Moreno, and P. Campoy, “A deep reinforcement learning technique for vision-based autonomous multirotor landing on a moving platform,” in _IEEE/RSJ IROS_ , 2018, pp. 1010–1017.
  * (22) F. Furrer, M. Burri, M. Achtelik, and R. Siegwart, “RotorS—a modular gazebo MAV simulator framework,” in _Robot Operating System (ROS)_ , 2016, pp. 595–625.
  * (23) L. He, N. Aouf, J. F. Whidborne, and B. Song, “Deep reinforcement learning based local planner for uav obstacle avoidance using demonstration data,” _arXiv preprint arXiv:2008.02521_ , 2020.
  * (24) B. Li, Z. Gan, D. Chen, and D. Sergey Aleksandrovich, “Uav maneuvering target tracking in uncertain environments based on deep reinforcement learning and meta-learning,” _Remote Sensing_ , vol. 12, no. 22, p. 3789, 2020.
  * (25) A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in _Advances in Neural Information Processing Systems_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, Inc., 2019. [Online]. Available: https://proceedings.neurips.cc/paper/2019/file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf
  * (26) D. Rukhovich, K. Sofiiuk, D. Galeev, O. Barinova, and A. Konushin, “Iterdet: Iterative scheme for object detection in crowded environments,” 2021.
  * (27) A. Tao, K. Sapra, and B. Catanzaro, “Hierarchical multi-scale attention for semantic segmentation,” _ArXiv_ , vol. abs/2005.10821, 2020.
  * (28) M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” 2020.
  * (29) M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, and A. Y. Ng, “ROS: an open-source robot operating system,” in _IEEE ICRA - Workshop on open source software_ , vol. 3, no. 3.2. Kobe, Japan, 2009, p. 5.
  * (30) P. L. J. Drews-Jr, A. A. Neto, and M. F. M. Campos, “Hybrid unmanned aerial underwater vehicle: Modeling and simulation,” in _IEEE/RSJ IROS_ , 2014, pp. 4637–4642.
  * (31) A. A. Neto, L. A. Mozelli, P. L. J. Drews-Jr, and M. F. M. Campos, “Attitude control for an hybrid unmanned aerial underwater vehicle: A robust switched strategy with global stability,” in _IEEE ICRA_ , 2015, pp. 395–400.
  * (32) A. C. Horn, P. M. Pinheiro, C. B. Silva, A. A. Neto, and P. L. J. Drews-Jr, “A study on configuration of propellers for multirotor-like hybrid aerial-aquatic vehicles,” in _ICAR_ , 2019, pp. 173–178.
  * (33) A. C. Horn, P. M. Pinheiro, R. B. Grando, C. B. da Silva, A. A. Neto, and P. L. Drews, “A novel concept for hybrid unmanned aerial underwater vehicles focused on aquatic performance,” pp. 1–6, 2020.
  * (34) R. Bedin Grando, P. L. J. Drews Jr, and A. Alves Neto, “Ardupilot and ros-based control system concept for a hybrid unmanned aerial underwater vehicle,” _II Brazilian Humanoid Robot Workshop and III Brazilian Workshop on Service Robotics_ , pp. 26–29, 2019.
  * (35) R. B. Grando, P. M. Pinheiro, N. P. Bortoluzzi, C. B. da Silva, O. F. Zauk, M. O. Piñeiro, V. M. Aoki, A. L. Kelbouscas, Y. B. Lima, P. L. Drews, and A. A. Neto, “Visual-based autonomous unmanned aerial vehicle for inspection in indoor environments,” pp. 1–6, 2020.
  * (36) N. Koenig and A. Howard, “Design and use paradigms for gazebo, an open-source multi-robot simulator,” in _IEEE/RSJ IROS_ , vol. 3, 2004, pp. 2149–2154.


# On the Challenges and Perspectives of Foundation Models for Medical Image
Analysis

Shaoting Zhang
Shanghai AI Laboratory
Shanghai, China
<EMAIL_ADDRESS>
Dimitris Metaxas
Rutgers University
New Brunswick, NJ, USA
<EMAIL_ADDRESS>

###### Abstract

This article discusses the opportunities, applications and future directions
of large-scale pretrained models, i.e., foundation models, which promise to
significantly improve the analysis of medical images. Medical foundation
models have immense potential in solving a wide range of downstream tasks, as
they can help to accelerate the development of accurate and robust models,
reduce the dependence on large amounts of labeled data, preserve the privacy
and confidentiality of patient data. Specifically, we illustrate the
“spectrum” of medical foundation models, ranging from general imaging models,
modality-specific models, to organ/task-specific models, and highlight their
challenges, opportunities and applications. We also discuss how foundation
models can be leveraged in downstream medical tasks to enhance the accuracy
and efficiency of medical image analysis, leading to more precise diagnosis
and treatment decisions.

##  1 Introduction

Figure 1: The spectrum of foundation models in medical image analysis.

A salient distinction exists between traditional pretrained models and
contemporary foundation models. The former Deng et al. (2009); Vaswani et al.
(2017); Devlin et al. (2018); Radford et al. (2018, 2019); Dosovitskiy et al.
(2021) typically require extensive supervised fine-tuning to address specific
downstream tasks, whereas the latter are capable of employing few-shot
learning, zero-shot learning, or prompt engineering to manage a wide variety
of tasks with a singular set of model weights Brown et al. (2020); Chowdhery
et al. (2022); Ouyang et al. (2022); Touvron et al. (2023a, b); Google (2023).
Consequently, foundation models exhibit considerable generalizability and
adaptability, positioning them as a focal point of recent machine learning
(ML) research.

In the field of medical image analysis, however, task-specific ML models are
still the main methods used, especially for clinical applications such as
computer-aided disease diagnosis. Developing medical foundation models
presents a significant challenge due to the diverse imaging modalities used in
medicine, which differ greatly from natural images and are based on a spectrum
of physics-based properties and energy sources. These modalities are based on
the use of light, electrons, lasers, X-rays, ultrasound, nuclear physics, and
magnetic resonance. The images produced span multiple scales, ranging from
molecules and cells to organ systems and the full body. Therefore, it may be
infeasible to develop a unified multi-scale foundation model trained from a
combination of these multi-modality images.

In the following, we will investigate and present our vision for the
“spectrum” of foundation models and their uses in medical image analysis,
ranging from general vision models, modality-specific models, to organ and
task-specific models (Fig. 1). Fortunately, the growing availability of high-
quality publicly available annotated medical data has led to the gradual
emergence of specialized foundational models with an innate capacity for
generating more generalized representations of medical data. Therefore,
foundation models trained with medical images and/or natural images in a self-
supervised manner may serve as an improved solution basis for important
clinical problems, will result in advances in the field of medical imaging,
and will improve the efficacy and efficiency of disease diagnosis and
treatment.

##  2 The Spectrum of Foundation Models

Vision Foundation Models

A straightforward approach is to employ foundation models trained from natural
images Carion et al. (2020); Dosovitskiy et al. (2021); Zhai et al. (2022);
Radford et al. (2021); He et al. (2022); Wang et al. (2022); Oquab et al.
(2023), and then design sophisticated algorithms to solve downstream medical
tasks.

However, the lack of publicly available quality annotations in medical imaging
has been the bottleneck for training large-scale deep learning models for many
downstream clinical applications. It remains a tedious and time-consuming job
for medical professionals to hand-label image data repeatedly, while providing
a few differentiable sample cases is feasible and complies with the training
process of medical residents. Vision foundation models, often trained on
large-scale visual images of various modalities, could serve as the basis for
building medical applications.

However, while vision foundation models can learn general representations,
medical images have unique characteristics whose features and patterns differ
significantly from those in natural images. Therefore, carefully designed
algorithms are necessary to adapt them to domain-specific medical problems.
Fine-tuning, additional adapters, prompting strategies, and specialized
architectures or modifications are potential solutions to achieve optimal
performance in medical problems.

For instance, the recent Segmentation Anything Model (SAM) Kirillov et al.
(2023), a promptable segmentation system with zero-shot generalization to
unfamiliar objects and images, has demonstrated its impressive performance on
natural images. However, its out-of-the-box performance on complex medical
tasks such as pancreas, spine or cell nuclei segmentation is not satisfactory
He et al. (2023); Roy et al. (2023); Deng et al. (2023); Mazurowski et al.
(2023); Shi et al. (2023). SAM can be further tuned to achieve state-of-the-
art performance by leveraging high-quality downstream data and performing
proper fine-tuning strategies Ma and Wang (2023); Paranjape et al. (2023); Cui
et al. (2023), adding adapters with specially designed architectures Wu et al.
(2023a); Gong et al. (2023); Chen et al. (2023a), or effective prompts Huang
et al. (2023a); Cheng et al. (2023) with manual annotations. Venturing a step
further, we could explore the possibility of combining the output of
localization/detection algorithms with SAM or integrating SAM with image
processing and visualization software like 3D slicer Liu et al. (2023a). This
fusion would pave the way for a robust pipeline tailored for complex medical
applications.

In general, a unified foundation model approach can not achieve state-of-the-
art performance in many medical image analysis tasks due to large variations
present in organs and important structures, texture, shape, size and topology
(e.g., blood vessels), and imaging modalities. Furthermore, it is noteworthy
to mention that there are parameter/data efficient tuning methods to adapt
vision foundation models to address image analysis challenges arising from
long-tail medical data.

Modality-specific Foundation Models

Depending on the pathology, various types of imaging modalities are employed
for diagnostic and therapeutic purposes. They include X-rays, Computed
Tomography (CT), Magnetic Resonance Image (MRI), Ultrasound imaging, and
Positron Emission Tomography (PET). A modality-specific foundation model is
specifically designed for a group of imaging modalities such as radiology
images (including X-ray, CT, MR, and Ultrasound) Ghesu et al. (2022), 3D
images (stacks of 2D CT and MR images) Chen et al. (2019), or a particular
medical imaging modality which includes X-ray Tiu et al. (2022), CT Huang et
al. (2023b); Wang et al. (2023a), endoscopy Wang et al. (2023b), and pathology
Chen et al. (2022, 2023b); Vorontsov et al. (2023) images. It is then used to
learn image-based features that are relevant to the intended use of the
particular modality. For example, a CT-specific model may learn to identify
features related to bone density and tissue contrast, while an MRI-specific
model may learn to identify features related to soft tissue contrast and
motion.

Vision foundation models trained on large-scale natural image datasets, can
provide a strong starting point for a wide range of medical imaging analysis
tasks. Using these vision foundation models, modality-specific foundation
models can leverage the unique characteristics of each imaging modality and
can result in models optimized for specific modalities. While they can then
lead to higher accuracy and efficiency for the analysis tasks specific to that
modality, they may not generalize well to other modalities.

Organ/Task-specific Foundation Models

Figure 2: Different image modalities have large image-level variations, which
may cause difficulties when training a unified foundation model. Similarly
each modality given the imaging formation differences, results in images with
significant variations in organ appearance and related structures which
determine which modality needs to be used given a target organ pathology.
Given modalities, leveraging fine-grained models for learning organ appearance
and pathology, will enable important clinical methods and tools such as robust
computer-aided diagnosis and surgical planning.

More specifically, the foundation models could be tailored to a particular
medical organ Li et al. (2020a); Luo et al. (2022); Zhou et al. (2023) or
diagnostic task, such as segmentation Antonelli et al. (2022); Tang et al.
(2022); Butoi et al. (2023). This use aims to address the challenges posed by
the variability in organ appearance across medical images, as well as the
diverse range of clinical tasks that are based on image analysis (Fig. 2).

Collecting data for training organ/task-specific foundation models can be
challenging due to the need for large amounts of labeled data. Nevertheless, a
well-trained organ/task-specific foundation model can provide better accuracy
and interpretability as well as significantly reduce the amount of labeled
data required for new tasks, as it has already learned relevant features from
the previous training.

General vs. Specialized Foundation Models

By definition, specialists possess an in-depth understanding of a particular
subject matter, whereas generalists have a broader purview, either within a
single field or across multiple disciplines.

In the context of medical image analysis, a general AI system is characterized
as a multitask and multimodal platform capable of performing a diverse range
of tasks on multimodal images of different organs and diseases. They include
classification, detection, segmentation and registration and utilize a single
set of model weights Moor et al. (2023); Tu et al. (2023); Wu et al. (2023b).
This approach and related models are shown in the left part of Fig.1, which
shows the spectrum of foundation models.

Conversely, specialized AI systems are designed to perform discrete clinical
tasks, such as the detection of pulmonary nodules, the reconstruction of
coronary arteries, or the diagnosis of hepatocellular carcinoma. These systems
are generally confined to a particular organ and imaging modality, aligning
more closely with the right side of the foundation model spectrum (see Fig.1).

Within the computer science community, there is an emerging focus on the
development of general AI frameworks. This shift is driven by the technical
innovation inherent in exploring large, multimodal generative models capable
of processing diverse types of medical data. However, research in both
academia, medical institutions, and industry, largely remains focused on the
development of specialized AI systems. This focus is attributable to several
factors. Firstly, most existing state-of-the-art medical image analysis
systems use a single type of imaging modality (unimodal) and are trained on a
single task such as segmentation or classification. Secondly, AI currently
serves primarily as an assistant to medical professionals who require targeted
support, which is consistent with their medical training. Consequently, there
is a pragmatic inclination toward designing specialized systems that excel in
terms of performance and accuracy in particular tasks. Moreover, general type
of AI systems tend to consume significantly greater computational resources
and often lack the required accuracy.

Both specialized and general AI system approaches offer distinct advantages
and are suited for different applications. Accordingly, we advocate for a
comprehensive exploration of the foundation model spectrum to ascertain the
optimal trade-off between developmental effort and practical efficacy.

##  3 Data Requirements for Foundation Models

Data to Pretrain the Foundation Models

Data is the cornerstone for training all foundation models. Preparing data for
medical foundation models has unique challenges and requires domain knowledge
since medical data are expensive to collect, annotate interpret, and their
quality varies significantly across hospitals and clinical studies.

Real-world images are 2D projections of the 3D world and therefore it is
relatively easy to collect a large number of images that cover the variability
of object(s)/scenes in terms of viewpoint, angles, scales, appearance and
locations. To the contrary, medical images are acquired for a particular
clinical purpose, through certain protocols and scanners that require use by
an expert who controls the machine settings, including the view angles and
scales. The image complexity and variability typically come from scanner
differences, scanning protocols, and, most importantly, anatomical and other
variations among individuals, disease appearance, location, and stage of the
disease.

These unique properties of medical images manifest during the development of
public datasets. Classical public datasets were collected for a specific
purpose using certain protocols and scanners such as EyePACS De Vente et al.
(2023), SUN-SEG Ji et al. (2022a), ISIC Cassidy et al. (2022),Chestx-ray8 Wang
et al. (2017) CAMELYON Litjens et al. (2018), and EndoVis Allan et al. (2020),
thus usually limited to a single modality and a specific anatomical area for a
particular task. Recently, general-purpose datasets have been obtained using
multiple protocols and scanners, such as Totalseg Wasserthal et al. (2023),
AMOS Ji et al. (2022b), FLARE Ma et al. (2022, 2023), autoPET Gatidis et al.
(2022), BraTS Menze et al. (2014), ISLES Hernandez Petzsche et al. (2022),
DigestPath Da et al. (2022), and MIMICS Johnson et al. (2019).
Correspondingly, the size of datasets has changed from small scale to large
scale, and data variability is increasing. When preparing datasets to train
medical foundation models, one should select the most representative cases to
train generalizable models and cover corner cases instead of simply collecting
large amounts of data.

In addition, instead of building a unified dataset covering all image
modalities (e.g., an“ImageNet” Deng et al. (2009) type of database for medical
data), a more feasible solution is to begin with modality-specific datasets
and then attempt to merge those which are complementary.

Data Adaptation for Downstream Tasks

The emergence of foundation models signals a notable reversal from the
previously encouraging trend of model openness and accessibility within the
scientific community. Although pretrained instances of certain models like
LLaMA Touvron et al. (2023a, b) and SAM Kirillov et al. (2023) are publicly
accessible, models such as GPT-3 Brown et al. (2020), and GPT-4 OpenAI (2023)
are not publicly available; only API access is possible which is restricted to
a limited number of users. Furthermore, the datasets employed for training
foundation models are not made available to the wider research community. The
computational and engineering resources required to train foundation models
from the ground up are cost prohibitive for academia and most companies. This
creates a barrier that prevents the vast majority of artificial intelligence
researchers from participating in this pivotal ML research and methodology.

Recent advancements in ML have facilitated the efficient adaptation of
foundation models for downstream tasks, requiring only a minimal number of
training samples (e.g., prompt engineering and efficient methods for
retraining). Recent research on the use of ML for medical image analysis has
been establishing benchmarks and releasing datasets. Wang et al. (2023c); Yi
et al. (2023). These efforts hold significant promise for enabling the
effective deployment of large-scale foundation models to tackle an array of
clinical challenges.

##  4 Applications and Benefits of Foundation Models

Leveraging foundation models trained on large datasets to address specific
medical needs is crucial for achieving accurate and reliable image analysis
and disease diagnosis and prognosis, minimizing the need for data collection,
reducing the time and cost associated with data labeling, and upholding
patient data privacy and confidentiality.

Long-tailed Problems

Medical image analysis methods often face the challenging long-tail data
scenario, caused by often heavily imbalanced datasets in which many common
disease cases coexist with relatively few rare disease cases. Consequently,
the scarcity of data for training models to accurately identify these rare
cases can lead to significant performance degradation issues. The few-shot
setting aligns perfectly with the long-tailed scenario, which frequently
arises in medical imaging when only a few rare disease cases/high-quality
annotations are available.

The initial stage involves selecting the appropriate foundation model
(including general, modality-specific, and organ/task-specific models),
depending on the application and available resources. Then, data augmentation
techniques are employed to augment the few annotated samples and make full use
of the available supervised information. These techniques include image
augmentation methods like rotation, cropping, color transformation, noise
injection, and random erasing, as well as image generation techniques, e.g.,
GAN and diffusion-based models Chambon et al. (2022); Pinaya et al. (2022);
Ding et al. (2023).

Utilizing medical foundation models that have been trained on vast amounts of
data can reduce the labeled data required for training, which minimizes the
need for manual annotation by medical professionals. Additionally, these
models can lead to more reliable diagnoses and treatment decisions.

Explainable and Generalizable Models

The lack of explainability in deep learning models can lead to distrust issues
when clinicians are accustomed to making explainable clinical inferences.
Similar to explainability, the generalizability of a model (a model trained on
data from one medical center applied to data from other medical centers with
significant variations or domain shifts) is also necessary due to the
previously mentioned dataset limitations. Therefore, innovative methodologies
are needed to improve the explainability and generalizability of the current
models in order to be used effectively in clinical practice Wang et al.
(2023d).

Foundation models provide a unified framework that can support detection,
segmentation, and classification tasks, which is essential for evidence-based
decision-making. Moreover, these models are typically trained on large-scale
datasets covering a wide range of medical centers, scanners, and protocols,
resulting in promising generalizability of the learned feature
representations.

Privacy Preserving Methods

While the computer vision community has an established history of open-
sourcing large-scale datasets, such as the ImageNet, making publicly available
large amounts of medical data is currently not possible due to regulatory and
privacy issues. Foundation models offer an alternative way for knowledge
sharing, while protecting patient privacy. Transfer learning techniques can be
used to adapt the foundation model using a smaller dataset of interest,
avoiding directly accessing massive raw data.

Furthermore, sharing multi-cohort knowledge in foundation models is also
feasible through the use of federated learning Li et al. (2020b); Kaissis et
al. (2021), which enables training on data distributed across multiple
institutions or devices without the data ever leaving the local machines. This
paradigm ensures data privacy when training the foundation models on large
distributed datasets.

Foundation models can also be used to create synthetic data Ding et al. (2023)
which can ensure data privacy preservation. Using generative models to create
synthetic medical images which are statistically similar to real medical
images, researchers can train models on these synthetic images instead of
using real patient data.

Integration with Large Language Models

The study of vision-language models is gaining prominence due to their
capacity for extracting nuanced information and learning superior
representations. However, the majority of existing research predominantly
concentrates on the analysis of X-ray images in conjunction with their
corresponding reports Zhang et al. (2022); Tiu et al. (2022); Zhou et al.
(2022); Lee et al. (2023). The recent advances in large language models (LLMs)
which are trained on vast amounts of text data have significantly improved
natural language processing capabilities Brown et al. (2020); Ouyang et al.
(2022); Touvron et al. (2023a, b); Chowdhery et al. (2022); Google (2023). The
applications are further expanded beyond textual domains with the integration
of vision models OpenAI (2023); Wu et al. (2023c); Alayrac et al. (2022); Li
et al. (2023a); Driess et al. (2023); Wang et al. (2023e); Liu et al. (2023b).
By combining language and vision data, these large-scale vision-language
models have unlocked exciting possibilities for the future use of medical
foundation models.

Integrating medical image analysis systems with general domain LLMs or medical
domain LLMs Singhal et al. (2023a, b) holds immense potential for healthcare
applications. For instance, these models can be trained to generate
descriptive captions for medical images, facilitating automated radiology
reports or succinct summaries of complex visuals. Furthermore, decision
support systems can also benefit from associating visual features from medical
images with text from patient records, providing accurate disease diagnosis
and prognosis Li et al. (2023b); Zhang et al. (2023); Wang et al. (2023f).
However, these frameworks are still preliminary, as they usually integrate
existing LLMs as a module by prompting, without fine-tuning and/or
consolidating data modalities such as medical images to these models. The
efforts to open-source foundation models and the ability to fine-tune them
will be essential in healthcare.

Figure 3: Models trained on multiple medical data modalities can enable
comprehensive clinical solutions.

##  5 Future Directions of Medical Foundation Models

We have discussed the challenges and opportunities of foundation models for
medical image analysis. These insights can help us design more effective and
generalizable foundation models. Since foundation models have only just begun
to transform the way medical image systems are built and deployed worldwide,
issues and challenges are still difficult to predict. We advocate that
researchers from different institutions and disciplines need to collaborate to
investigate and explore the spectrum of foundation models for medical image
analysis, and contribute to the open-source community by releasing a family of
pretrained models which will work on various imaging modalities.

Future directions include multi-modality foundation models, combining various
data types (text, image, video, database, molecule) and scales (molecule,
gene, cell, tissue, patient, population). Foundation models hold enormous
promise as they can assimilate data from various imaging modalities and can
incorporate non-imaging modalities to provide a more comprehensive
understanding of a patient’s condition and its assessment. By leveraging
multi-modality foundation models, medical professionals can achieve a more
accurate disease diagnosis and develop personalized treatment plans and
disease prognosis. These models can potentially improve the overall quality of
medical care by facilitating data sharing among different institutions,
leading to more efficient and effective patient care and healthcare.

Advances in multi-modality foundation models can contribute to the development
of clinical use cases targeting patients with different background and
different diseases. A straightforward application is to support radiologists
throughout their workflow, such as drafting structured radiology reports
automatically and describing possible abnormalities, disease diagnosis and
prognosis, as well as proposed treatment. Another possible use case is to
assist surgeons. Integrating image, language, and audio modalities, surgeons
can communicate with models to make real-time decisions in the operating room
by detecting and identifying the anatomical location of important target
structures often not clearly visible. The quest for comprehensive solutions to
these and other medical problems is expected to intensify and the near future
and produce the desired results (Fig. 3).

## References

  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Radford et al. [2018] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.
  * Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=YicbFdNTTy.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chowdhery et al. [2022] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ , 2022.
  * Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744, 2022.
  * Touvron et al. [2023a] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023a.
  * Touvron et al. [2023b] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023b.
  * Google [2023] Google.  Palm 2 technical report, 2023.  URL https://ai.google/static/documents/palm2techreport.pdf.
  * Carion et al. [2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In _European conference on computer vision_ , pages 213–229. Springer, 2020.
  * Zhai et al. [2022] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer.  Scaling vision transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12104–12113, 2022.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16000–16009, 2022.
  * Wang et al. [2022] Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al.  Internimage: Exploring large-scale vision foundation models with deformable convolutions.  _arXiv preprint arXiv:2211.05778_ , 2022.
  * Oquab et al. [2023] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  _arXiv preprint arXiv:2304.07193_ , 2023.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick.  Segment anything.  _arXiv:2304.02643_ , 2023.
  * He et al. [2023] Sheng He, Rina Bao, Jingpeng Li, P Ellen Grant, and Yangming Ou.  Accuracy of segment-anything model (sam) in medical image segmentation tasks.  _arXiv preprint arXiv:2304.09324_ , 2023.
  * Roy et al. [2023] Saikat Roy, Tassilo Wald, Gregor Koehler, Maximilian R Rokuss, Nico Disch, Julius Holzschuh, David Zimmerer, and Klaus H Maier-Hein.  Sam. md: Zero-shot medical image segmentation capabilities of the segment anything model.  _arXiv preprint arXiv:2304.05396_ , 2023.
  * Deng et al. [2023] Ruining Deng, Can Cui, Quan Liu, Tianyuan Yao, Lucas W Remedios, Shunxing Bao, Bennett A Landman, Lee E Wheless, Lori A Coburn, Keith T Wilson, et al.  Segment anything model (sam) for digital pathology: Assess zero-shot segmentation on whole slide imaging.  _arXiv preprint arXiv:2304.04155_ , 2023.
  * Mazurowski et al. [2023] Maciej A Mazurowski, Haoyu Dong, Hanxue Gu, Jichen Yang, Nicholas Konz, and Yixin Zhang.  Segment anything model for medical image analysis: an experimental study.  _arXiv preprint arXiv:2304.10517_ , 2023.
  * Shi et al. [2023] Peilun Shi, Jianing Qiu, Sai Mu Dalike Abaxi, Hao Wei, Frank P-W Lo, and Wu Yuan.  Generalist vision foundation models for medical imaging: A case study of segment anything model on zero-shot medical segmentation.  _Diagnostics_ , 13(11):1947, 2023.
  * Ma and Wang [2023] Jun Ma and Bo Wang.  Segment anything in medical images.  _arXiv preprint arXiv:2304.12306_ , 2023.
  * Paranjape et al. [2023] Jay N Paranjape, Nithin Gopalakrishnan Nair, Shameema Sikder, S Swaroop Vedula, and Vishal M Patel.  Adaptivesam: Towards efficient tuning of sam for surgical scene segmentation.  _arXiv preprint arXiv:2308.03726_ , 2023.
  * Cui et al. [2023] Can Cui, Ruining Deng, Quan Liu, Tianyuan Yao, Shunxing Bao, Lucas W Remedios, Yucheng Tang, and Yuankai Huo.  All-in-sam: from weak annotation to pixel-wise nuclei segmentation with prompt-based finetuning.  _arXiv preprint arXiv:2307.00290_ , 2023.
  * Wu et al. [2023a] Junde Wu, Rao Fu, Huihui Fang, Yuanpei Liu, Zhaowei Wang, Yanwu Xu, Yueming Jin, and Tal Arbel.  Medical sam adapter: Adapting segment anything model for medical image segmentation.  _arXiv preprint arXiv:2304.12620_ , 2023a.
  * Gong et al. [2023] Shizhan Gong, Yuan Zhong, Wenao Ma, Jinpeng Li, Zhao Wang, Jingyang Zhang, Pheng-Ann Heng, and Qi Dou.  3dsam-adapter: Holistic adaptation of sam from 2d to 3d for promptable medical image segmentation.  _arXiv preprint arXiv:2306.13465_ , 2023.
  * Chen et al. [2023a] Cheng Chen, Juzheng Miao, Dufan Wu, Zhiling Yan, Sekeun Kim, Jiang Hu, Aoxiao Zhong, Zhengliang Liu, Lichao Sun, Xiang Li, et al.  Ma-sam: Modality-agnostic sam adaptation for 3d medical image segmentation.  _arXiv preprint arXiv:2309.08842_ , 2023a.
  * Huang et al. [2023a] Yuhao Huang, Xin Yang, Lian Liu, Han Zhou, Ao Chang, Xinrui Zhou, Rusi Chen, Junxuan Yu, Jiongquan Chen, Chaoyu Chen, et al.  Segment anything model for medical images?  _arXiv preprint arXiv:2304.14660_ , 2023a.
  * Cheng et al. [2023] Dongjie Cheng, Ziyuan Qin, Zekun Jiang, Shaoting Zhang, Qicheng Lao, and Kang Li.  Sam on medical images: A comprehensive study on three prompt modes.  _arXiv preprint arXiv:2305.00035_ , 2023.
  * Liu et al. [2023a] Yihao Liu, Jiaming Zhang, Zhangcong She, Amir Kheradmand, and Mehran Armand.  Samm (segment any medical model): A 3d slicer integration to sam.  _arXiv preprint arXiv:2304.05622_ , 2023a.
  * Ghesu et al. [2022] Florin C Ghesu, Bogdan Georgescu, Awais Mansoor, Youngjin Yoo, Dominik Neumann, Pragneshkumar Patel, RS Vishwanath, James M Balter, Yue Cao, Sasa Grbic, et al.  Self-supervised learning from 100 million medical images.  _arXiv preprint arXiv:2201.01283_ , 2022.
  * Chen et al. [2019] Sihong Chen, Kai Ma, and Yefeng Zheng.  Med3d: Transfer learning for 3d medical image analysis.  _arXiv preprint arXiv:1904.00625_ , 2019.
  * Tiu et al. [2022] Ekin Tiu, Ellie Talius, Pujan Patel, Curtis P Langlotz, Andrew Y Ng, and Pranav Rajpurkar.  Expert-level detection of pathologies from unannotated chest x-ray images via self-supervised learning.  _Nature Biomedical Engineering_ , pages 1–8, 2022.
  * Huang et al. [2023b] Ziyan Huang, Haoyu Wang, Zhongying Deng, Jin Ye, Yanzhou Su, Hui Sun, Junjun He, Yun Gu, Lixu Gu, Shaoting Zhang, et al.  Stu-net: Scalable and transferable medical image segmentation models empowered by large-scale supervised pre-training.  _arXiv preprint arXiv:2304.06716_ , 2023b.
  * Wang et al. [2023a] Guotai Wang, Jianghao Wu, Xiangde Luo, Xinglong Liu, Kang Li, and Shaoting Zhang.  Mis-fm: 3d medical image segmentation using foundation models pretrained on a large-scale unannotated dataset.  _arXiv preprint arXiv:2306.16925_ , 2023a.
  * Wang et al. [2023b] Zhao Wang, Chang Liu, Shaoting Zhang, and Qi Dou.  Foundation model for endoscopy video analysis via large-scale self-supervised pre-train.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2023b.
  * Chen et al. [2022] Richard J Chen, Chengkuan Chen, Yicong Li, Tiffany Y Chen, Andrew D Trister, Rahul G Krishnan, and Faisal Mahmood.  Scaling vision transformers to gigapixel images via hierarchical self-supervised learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16144–16155, 2022.
  * Chen et al. [2023b] Richard J Chen, Tong Ding, Ming Y Lu, Drew FK Williamson, Guillaume Jaume, Bowen Chen, Andrew Zhang, Daniel Shao, Andrew H Song, Muhammad Shaban, et al.  A general-purpose self-supervised model for computational pathology.  _arXiv preprint arXiv:2308.15474_ , 2023b.
  * Vorontsov et al. [2023] Eugene Vorontsov, Alican Bozkurt, Adam Casson, George Shaikovski, Michal Zelechowski, SiQi Liu, Philippe Mathieu, Alexander van Eck, Donghun Lee, Julian Viret, et al.  Virchow: A million-slide digital pathology foundation model.  _arXiv preprint arXiv:2309.07778_ , 2023.
  * Li et al. [2020a] Xiaomeng Li, Mengyu Jia, Md Tauhidul Islam, Lequan Yu, and Lei Xing.  Self-supervised feature learning via exploiting multi-modal data for retinal disease diagnosis.  _IEEE Transactions on Medical Imaging_ , 39(12):4023–4033, 2020a.
  * Luo et al. [2022] Xiangde Luo, Wenjun Liao, Jianghong Xiao, Jieneng Chen, Tao Song, Xiaofan Zhang, Kang Li, Dimitris N Metaxas, Guotai Wang, and Shaoting Zhang.  Word: A large scale dataset, benchmark and clinical applicable study for abdominal organ segmentation from ct image.  _Medical Image Analysis_ , 82:102642, 2022.
  * Zhou et al. [2023] Yukun Zhou, Mark A Chia, Siegfried K Wagner, Murat S Ayhan, Dominic J Williamson, Robbert R Struyven, Timing Liu, Moucheng Xu, Mateo G Lozano, Peter Woodward-Court, et al.  A foundation model for generalizable disease detection from retinal images.  _Nature_ , pages 1–8, 2023.
  * Antonelli et al. [2022] Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald M Summers, et al.  The medical segmentation decathlon.  _Nature communications_ , 13(1):4128, 2022.
  * Tang et al. [2022] Yucheng Tang, Dong Yang, Wenqi Li, Holger R Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh.  Self-supervised pre-training of swin transformers for 3d medical image analysis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 20730–20740, 2022.
  * Butoi et al. [2023] Victor Ion Butoi, Jose Javier Gonzalez Ortiz, Tianyu Ma, Mert R Sabuncu, John Guttag, and Adrian V Dalca.  Universeg: Universal medical image segmentation.  _arXiv preprint arXiv:2304.06131_ , 2023.
  * Moor et al. [2023] Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar.  Foundation models for generalist medical artificial intelligence.  _Nature_ , 616(7956):259–265, 2023.
  * Tu et al. [2023] Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Chuck Lau, Ryutaro Tanno, Ira Ktena, et al.  Towards generalist biomedical ai.  _arXiv preprint arXiv:2307.14334_ , 2023.
  * Wu et al. [2023b] Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie.  Towards generalist foundation model for radiology.  _arXiv preprint arXiv:2308.02463_ , 2023b.
  * De Vente et al. [2023] Coen De Vente, Koenraad A Vermeer, Nicolas Jaccard, He Wang, Hongyi Sun, Firas Khader, Daniel Truhn, Temirgali Aimyshev, Yerkebulan Zhanibekuly, Tien-Dung Le, et al.  Airogs: Artificial intelligence for robust glaucoma screening challenge.  _IEEE Transactions on Medical Imaging_ , 2023.
  * Ji et al. [2022a] Ge-Peng Ji, Guobao Xiao, Yu-Cheng Chou, Deng-Ping Fan, Kai Zhao, Geng Chen, and Luc Van Gool.  Video polyp segmentation: A deep learning perspective.  _Machine Intelligence Research_ , 19(6):531–549, 2022a.
  * Cassidy et al. [2022] Bill Cassidy, Connah Kendrick, Andrzej Brodzicki, Joanna Jaworek-Korjakowska, and Moi Hoon Yap.  Analysis of the isic image datasets: Usage, benchmarks and recommendations.  _Medical image analysis_ , 75:102305, 2022.
  * Wang et al. [2017] Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers.  Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2097–2106, 2017.
  * Litjens et al. [2018] Geert Litjens, Peter Bandi, Babak Ehteshami Bejnordi, Oscar Geessink, Maschenka Balkenhol, Peter Bult, Altuna Halilovic, Meyke Hermsen, Rob van de Loo, Rob Vogels, et al.  1399 h&e-stained sentinel lymph node sections of breast cancer patients: the camelyon dataset.  _GigaScience_ , 7(6):giy065, 2018.
  * Allan et al. [2020] Max Allan, Satoshi Kondo, Sebastian Bodenstedt, Stefan Leger, Rahim Kadkhodamohammadi, Imanol Luengo, Felix Fuentes, Evangello Flouty, Ahmed Mohammed, Marius Pedersen, et al.  2018 robotic scene segmentation challenge.  _arXiv preprint arXiv:2001.11190_ , 2020.
  * Wasserthal et al. [2023] Jakob Wasserthal, Hanns-Christian Breit, Manfred T. Meyer, Maurice Pradella, Daniel Hinck, Alexander W. Sauter, Tobias Heye, Daniel T. Boll, Joshy Cyriac, Shan Yang, Michael Bach, and Martin Segeroth.  Totalsegmentator: Robust segmentation of 104 anatomic structures in ct images.  _Radiology: Artificial Intelligence_ , 5(5):e230024, 2023.  doi:10.1148/ryai.230024.
  * Ji et al. [2022b] Yuanfeng Ji, Haotian Bai, Jie Yang, Chongjian Ge, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhang, Wanling Ma, Xiang Wan, et al.  Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation.  _arXiv preprint arXiv:2206.08023_ , 2022b.
  * Ma et al. [2022] Jun Ma, Yao Zhang, Song Gu, Xingle An, Zhihe Wang, Cheng Ge, Congcong Wang, Fan Zhang, Yu Wang, Yinan Xu, Shuiping Gou, Franz Thaler, Christian Payer, Darko Štern, Edward G.A. Henderson, Dónal M. McSweeney, Andrew Green, Price Jackson, Lachlan McIntosh, Quoc-Cuong Nguyen, Abdul Qayyum, Pierre-Henri Conze, Ziyan Huang, Ziqi Zhou, Deng-Ping Fan, Huan Xiong, Guoqiang Dong, Qiongjie Zhu, Jian He, and Xiaoping Yang.  Fast and low-gpu-memory abdomen ct organ segmentation: The flare challenge.  _Medical Image Analysis_ , 82:102616, 2022.
  * Ma et al. [2023] Jun Ma, Yao Zhang, Song Gu, Cheng Ge, Shihao Ma, Adamo Young, Cheng Zhu, Kangkang Meng, Xin Yang, Ziyan Huang, Fan Zhang, Wentao Liu, YuanKe Pan, Shoujin Huang, Jiacheng Wang, Mingze Sun, Weixin Xu, Dengqiang Jia, Jae Won Choi, Natália Alves, Bram de Wilde, Gregor Koehler, Yajun Wu, Manuel Wiesenfarth, Qiongjie Zhu, Guoqiang Dong, Jian He, the FLARE Challenge Consortium, and Bo Wang.  Unleashing the strengths of unlabeled data in pan-cancer abdominal organ quantification: the flare22 challenge.  _arXiv preprint arXiv:2308.05862_ , 2023.
  * Gatidis et al. [2022] Sergios Gatidis, Tobias Hepp, Marcel Früh, Christian La Fougère, Konstantin Nikolaou, Christina Pfannenberg, Bernhard Schölkopf, Thomas Küstner, Clemens Cyran, and Daniel Rubin.  A whole-body fdg-pet/ct dataset with manually annotated tumor lesions.  _Scientific Data_ , 9(1):601, 2022.
  * Menze et al. [2014] Bjoern H Menze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, et al.  The multimodal brain tumor image segmentation benchmark (brats).  _IEEE transactions on medical imaging_ , 34(10):1993–2024, 2014.
  * Hernandez Petzsche et al. [2022] Moritz R Hernandez Petzsche, Ezequiel de la Rosa, Uta Hanning, Roland Wiest, Waldo Valenzuela, Mauricio Reyes, Maria Meyer, Sook-Lei Liew, Florian Kofler, Ivan Ezhov, et al.  Isles 2022: A multi-center magnetic resonance imaging stroke lesion segmentation dataset.  _Scientific data_ , 9(1):762, 2022.
  * Da et al. [2022] Qian Da, Xiaodi Huang, Zhongyu Li, Yanfei Zuo, Chenbin Zhang, Jingxin Liu, Wen Chen, Jiahui Li, Dou Xu, Zhiqiang Hu, et al.  Digestpath: A benchmark dataset with challenge review for the pathological detection and segmentation of digestive-system.  _Medical Image Analysis_ , 80:102485, 2022.
  * Johnson et al. [2019] Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng.  Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports.  _Scientific data_ , 6(1):317, 2019.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Wang et al. [2023c] Dequan Wang, Xiaosong Wang, Lilong Wang, Mengzhang Li, Qian Da, Xiaoqiang Liu, Xiangyu Gao, Jun Shen, Junjun He, Tian Shen, Qi Duan, Jie Zhao, Kang Li, Yu Qiao, and Shaoting Zhang.  A real-world dataset and benchmark for foundation model adaptation in medical image classification.  _Nature Scientific Data_ , 2023c.
  * Yi et al. [2023] Huahui Yi, Ziyuan Qin, Qicheng Lao, Wei Xu, Zekun Jiang, Dequan Wang, Shaoting Zhang, and Kang Li.  Towards general purpose medical ai: Continual learning medical foundation model.  _arXiv preprint arXiv:2303.06580_ , 2023.
  * Chambon et al. [2022] Pierre Chambon, Christian Bluethgen, Jean-Benoit Delbrouck, Rogier Van der Sluijs, Małgorzata Połacin, Juan Manuel Zambrano Chaves, Tanishq Mathew Abraham, Shivanshu Purohit, Curtis P Langlotz, and Akshay Chaudhari.  Roentgen: vision-language foundation model for chest x-ray generation.  _arXiv preprint arXiv:2211.12737_ , 2022.
  * Pinaya et al. [2022] Walter HL Pinaya, Petru-Daniel Tudosiu, Jessica Dafflon, Pedro F Da Costa, Virginia Fernandez, Parashkev Nachev, Sebastien Ourselin, and M Jorge Cardoso.  Brain imaging generation with latent diffusion models.  In _MICCAI Workshop on Deep Generative Models_ , pages 117–126. Springer, 2022.
  * Ding et al. [2023] Kexin Ding, Mu Zhou, He Wang, Olivier Gevaert, Dimitris Metaxas, and Shaoting Zhang.  A large-scale synthetic pathological dataset for deep learning-enabled segmentation of breast cancer.  _Scientific Data_ , 10(1):231, 2023.
  * Wang et al. [2023d] Guotai Wang, Shaoting Zhang, Xiaolei Huang, Tom Vercauteren, and Dimitris Metaxas.  Editorial for special issue on explainable and generalizable deep learning methods for medical image computing.  _Medical Image Analysis_ , 84:102727, 2023d.
  * Li et al. [2020b] Xiaoxiao Li, Yufeng Gu, Nicha Dvornek, Lawrence H Staib, Pamela Ventola, and James S Duncan.  Multi-site fmri analysis using privacy-preserving federated learning and domain adaptation: Abide results.  _Medical Image Analysis_ , 65:101765, 2020b.
  * Kaissis et al. [2021] Georgios Kaissis, Alexander Ziller, Jonathan Passerat-Palmbach, Théo Ryffel, Dmitrii Usynin, Andrew Trask, Ionésio Lima Jr, Jason Mancuso, Friederike Jungmann, Marc-Matthias Steinborn, et al.  End-to-end privacy preserving deep learning on multi-institutional medical imaging.  _Nature Machine Intelligence_ , 3(6):473–484, 2021.
  * Zhang et al. [2022] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz.  Contrastive learning of medical visual representations from paired images and text.  In _Machine Learning for Healthcare Conference_ , pages 2–25. PMLR, 2022.
  * Zhou et al. [2022] Hong-Yu Zhou, Xiaoyu Chen, Yinghao Zhang, Ruibang Luo, Liansheng Wang, and Yizhou Yu.  Generalized radiograph representation learning via cross-supervision between images and free-text radiology reports.  _Nature Machine Intelligence_ , 4(1):32–40, 2022\.
  * Lee et al. [2023] Hyungyung Lee, Wonjae Kim, Jin-Hwa Kim, Tackeun Kim, Jihang Kim, Leonard Sunwoo, and Edward Choi.  Unified chest x-ray and radiology report generation model with multi-view chest x-rays.  _arXiv preprint arXiv:2302.12172_ , 2023.
  * Wu et al. [2023c] Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan.  Visual chatgpt: Talking, drawing and editing with visual foundation models.  _arXiv preprint arXiv:2303.04671_ , 2023c.
  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Li et al. [2023a] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023a.
  * Driess et al. [2023] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence.  Palm-e: An embodied multimodal language model.  In _arXiv preprint arXiv:2303.03378_ , 2023.
  * Wang et al. [2023e] Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, and Jifeng Dai.  Visionllm: Large language model is also an open-ended decoder for vision-centric tasks.  https://arxiv.org/abs/2305.11175), 2023e.
  * Liu et al. [2023b] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_ , 2023b.
  * Singhal et al. [2023a] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al.  Large language models encode clinical knowledge.  _Nature_ , pages 1–9, 2023a.
  * Singhal et al. [2023b] Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al.  Towards expert-level medical question answering with large language models.  _arXiv preprint arXiv:2305.09617_ , 2023b.
  * Li et al. [2023b] Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao.  Llava-med: Training a large language-and-vision assistant for biomedicine in one day.  _arXiv preprint arXiv:2306.00890_ , 2023b.
  * Zhang et al. [2023] Sheng Zhang, Yanbo Xu, Naoto Usuyama, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, et al.  Large-scale domain-specific pretraining for biomedical vision-language processing.  _arXiv preprint arXiv:2303.00915_ , 2023.
  * Wang et al. [2023f] Sheng Wang, Zihao Zhao, Xi Ouyang, Qian Wang, and Dinggang Shen.  Chatcad: Interactive computer-aided diagnosis on medical image using large language models.  _arXiv preprint arXiv:2302.07257_ , 2023f.