[]This is my name

# Deep Reinforcement Learning framework for Autonomous Driving

Ahmad El Sallab1, Mohammed Abdou1, Etienne Perot2 and Senthil Yogamani3
1 Valeo Egypt, Cairo
2 Valeo Bobigny, France
3 Valeo Vision Systems, Ireland

###### Abstract

Reinforcement learning is considered to be a strong AI paradigm which can be
used to teach machines through interaction with the environment and learning
from their mistakes. Despite its perceived utility, it has not yet been
successfully applied in automotive applications. Motivated by the successful
demonstrations of learning of Atari games and Go by Google DeepMind, we
propose a framework for autonomous driving using deep reinforcement learning.
This is of particular relevance as it is difficult to pose autonomous driving
as a supervised learning problem due to strong interactions with the
environment including other vehicles, pedestrians and roadworks. As it is a
relatively new area of research for autonomous driving, we provide a short
overview of deep reinforcement learning and then describe our proposed
framework. It incorporates Recurrent Neural Networks for information
integration, enabling the car to handle partially observable scenarios. It
also integrates the recent work on attention models to focus on relevant
information, thereby reducing the computational complexity for deployment on
embedded hardware. The framework was tested in an open source 3D car racing
simulator called TORCS. Our simulation results demonstrate learning of
autonomous maneuvering in a scenario of complex road curvatures and simple
interaction of other vehicles.

## INTRODUCTION

A robot car that drives autonomously is a long-standing goal of Artificial
Intelligence. Driving a vehicle is a task that requires high level of skill,
attention and experience from a human driver. Although computers are more
capable of sustained attention and focus than humans, fully autonomous driving
requires a level of intelligence that surpasses that achieved so far by AI
agents.

The tasks involved in creating an autonomous driving agent can be divided into
3 categories, as shown in Figure 1:

1) Recognition: Identifying components of the surrounding environment.
Examples of this are pedestrian detection, traffic sign recognition, etc.
Although far from trivial, recognition is a relatively easy task nowadays
thanks to advances in Deep Learning (DL) algorithms, which have reached human
level recognition or above at several object detection and classification
problems [8] [2]. Deep learning models are able to learn complex feature
representations from raw input data, omitting the need for hand-crafted
features [15] [2] [7]. In this regard, Convolutional Neural Networks (CNNs)
are probably the most successful deep learning model, and have formed the
basis of every winning entry on the ImageNet challenge since AlexNet [8]. This
success has been replicated in lane & vehicle detection for autonomous driving
[6].

2) Prediction: It is not enough for an autonomous driving agent to recognize
its environment; it must also be able to build internal models that predict
the future states of the environment. Examples of this class of problem
include building a map of the environment or tracking an object. To be able to
predict the future, it is important to integrate past information. As such,
Recurrent Neural Networks (RNNs) are essential to this class of problem. Long-
Short Term Memory (LSTM) networks [5] are one such category of RNN that have
been used in end-to-end scene labeling systems [14]. More recently, RNNs have
also been used to improve object tracking performance in the DeepTracking
model [13].

3) Planning: The generation of an efficient model that incorporates
recognition and prediction to plan the future sequence of driving actions that
will enable the vehicle to navigate successfully. Planning is the hardest task
of the three. The difficulty lies in integrating the ability of the model to
understand the environment (recognition) and its dynamics (prediction) in a
way that enables it to plan the future actions such that it avoids unwanted
situations (penalties) and drives safely to its destination (rewards).

Figure 1: High level autonomous driving tasks

The Reinforcement Learning (RL) framework [17] [20] has been used for a long
time in control tasks. The mixture of RL with DL was pointed out to be one of
the most promising approaches to achieve human-level control in [9]. In [12]
and [11] this human-level control was demonstrated on Atari games using the
Deep Q Networks (DQN) model, in which RL is responsible for the planning part
while DL is responsible for the representation learning part. Later, RNNs were
integrated in the mixture to account for partial observable scenarios [4].

Autonomous driving requires the integration of information from multiple
sensors. Some of them are low dimensional, like LIDAR, while others are high
dimensional, like cameras. It is noteworthy in this particular example,
however, that although raw camera images are high dimensional, the useful
information needed to achieve the autonomous driving task is of much lower
dimension.

For example, the important parts of the scene that affect driving decisions
are limited to the moving vehicle, free space on the road ahead, the position
of kerbs, etc. Even the fine details of vehicles are not important, as only
their spatial location is truly necessary for the problem. Hence the memory
bandwidth for relevant information is much lower. If this relevant information
can be extracted, while the other non-relevant parts are filtered out, it
would improve both the accuracy and efficiency of autonomous driving systems.
Moreover, this would reduce the computation and memory requirements of the
system, which are critical constraints on embedded systems that will contain
the autonomous driving control unit.

Attention models are a natural fit for such an information filtering process.
Recently, these models were successfully deployed for image recognition in
[23] and [10], wherein RL was mixed with RNNs to obtain the parts of the image
to attend to. Such models are easily extended and integrated to the DQN [11]
and Deep Recurrent Q Networks (DRQN) [4] models. This integration was
performed in [16]. The success of attention models drives us to propose them
for the extraction of low level information from the raw sensory information
to perform autonomous driving.

In this paper, we propose a framework for an end-end autonomous driving model
that takes in raw sensor inputs and outputs driving actions. The model is able
to handle partially observable scenarios. Moreover, we propose to integrate
the recent advances in attention models in order to extract only relevant
information from the received sensors data, thereby making it suitable for
real-time embedded systems. The main contributions of this paper: 1)
presenting a survey of the recent advances of deep reinforcement learning and
2) introducing a framework for end-end autonomous driving using deep
reinforcement learning to the automotive community. The rest of the paper is
divided into two parts. The first part provides a survey of deep reinforcement
learning algorithms, starting with the traditional MDP framework and
Q-learning, followed by the the DQN, DRQN and Deep Attention Recurrent Q
Networks (DARQN). The second part of the paper describes the proposed
framework that integrates the recent advances in deep reinforcement learning.
Finally, we conclude and suggest directions for future work.

## REVIEW OF REINFORCEMENT LEARNING

For a comprehensive overview of reinforcement learning, please refer to the
second edition of Rich Sutton’s textbook [18]. We provide a short review of
important topics in this section. The Reinforcement Learning framework was
formulated in [17] as a model to provide the best policy an agent can follow
(best action to take in a given state), such that the total accumulated
rewards are maximized when the agent follows that policy from the current and
until a terminal state is reached.

### Motivation for RL Paradigm

Driving is a multi-agent interaction problem. As a human driver, it is much
easier to keep within a lane without any interaction with other cars than to
change lanes in heavy traffic. The latter is more difficult because of the
inherent uncertainty in behavior of other drivers. The number of interacting
vehicles, their geometric configuration and the behavior of the drivers could
have large variability and it is challenging to design a supervised learning
dataset with exhaustive coverage of all scenarios. Human drivers employ some
sort of online reinforcement learning to understand the behavior of other
drivers such as whether they are defensive or aggressive, experienced or in-
experienced, etc. This is particularly useful in scenarios which need
negotiation, namely entering a roundabout, navigating junctions without
traffic lights, lane changes during heavy traffic, etc. The main challenge in
autonomous driving is to deal with corner cases which are unexpected even for
a human driver, like recovering from being lost in an unknown area without GPS
or dealing with disaster situations like flooding or appearance of a sinkhole
on the ground. The RL paradigm models uncharted territory and learns from its
own experience by taking actions. Additionally, RL may be able to handle non-
differentiable cost functions which can create challenges for supervised
learning problems.

Currently, the standard approach for autonomous driving is to decouple the
system into isolated sub-problems, typically supervised-learning-like object
detection, visual odometry, etc and then having a post processing layer to
combine all the results of the previous steps. There are two main issues with
this approach: Firstly, the sub-problems which are solved may be more
difficult than autonomous driving. For example, one might be solving object
detection by semantic segmentation which is both challenging and unnecessary.
Human drivers don’t detect and classify all visible objects while driving,
only the most relevant ones. Secondly, the isolated sub-problems may not
combine coherently to achieve the goal of driving. In RL, this is explicitly
handled by a reward signal corresponding to good driving which can model the
interplay between driving (taking action) and planning (where to drive). As
the reward is based on stable driving and not crashing typically, it is
challenging to train an RL system with a real car because of the risks
involved. Thus most of the current RL research is done using video game
simulation engines like TORCS or Unity. Figure 2 is a screen-shot of multi-
agent simulation in Unity game engine which illustrates a difficult driving
scenario where the white car tries to navigate in heavy traffic with sharp
turns. This problem is relatively easier to model using RL.

Figure 2: Illustration of multi-agent simulation in Unity

### MDP

The model is developed under the Markov Decision Process (MDP) framework,
which is a tuple of (S, A, $P_{sa}$, $\gamma$, R) where: a) Set of Environment
states (S) b) Set of Actions (A) c) Discount Factor ($\gamma$) d) Reward (R)
e) State Transition Probabilities ($P_{sa}$)

We define a value-function that defines the value of being in a state S and
following the policy $\pi$(s) till the end of the episode. The value-function
is the expected sum of the discounted rewards as follows:

$\displaystyle V^{\pi}(s)=E[R(s_{0})+\gamma
R(s_{1})+\gamma^{2}R(s_{1})+\dots|s_{0}=s,\pi(s)]$

The objective is to find the policy that maximizes the expectation of the
accumulated rewards:

$\displaystyle J(\pi)=\max_{\pi}E[V^{\pi}(s)]$
$\displaystyle\pi^{*}(s)=\arg\max_{\pi}J(\pi)$

The solution to such a problem lies in finding a policy $\pi$(s), which is
formed of a set of actions A that maximizes the total rewards from a source to
goal states S. There are popular algorithms to solve this problem for finite
state-space like Value iteration and Policy iteration [17].

### Q-learning

Q-learning [21] is one of the commonly used algorithms to solve the MDP
problem. The actions $a\in A$ are obtained for every state $s\in S$ based on
an action-value function called $Q:S\times A\rightarrow\mathbb{R}$. The
Q-learning algorithm is based on the Bellman equation:

$\displaystyle Q_{t+1}(s,a)\leftarrow
Q_{t}(s,a)+\alpha[r+\gamma\arg\max_{a^{\prime}}Q_{t}(s^{\prime},a^{\prime})-Q_{t}(s,a)]$

The updates to the Q tables are done recursively by Temporal Difference (TD)
incremental learning [17]. The algorithm starts from an initial state, and
proceeds until the episode ends, i.e. a terminal state is reached. In every
step, the agent is in the current state s, it takes an action following the
policy $\pi$(s), and then observes the next state s’ together with the reward
received from the environment r. The algorithm continues until convergence of
the Q function or a certain number of episodes is reached.

## DEEP REINFORCEMENT LEARNING

Depending on the problem domain, the space of possible actions may be discrete
or continuous, a difference which has a profound effect on the choice of
algorithms to be applied. In this section we will discuss two algorithms: one
which operates on discrete actions (DQN) and one which operates on continuous
actions (DDAC).

### Deep Q Networks (DQN)

When the states are discrete, the Q-function can be easily formulated as a
table. This formulation becomes harder when the number of states increases,
and even impossible when the states are continuous. In such case, the
Q-function is formulated as a parameterized function of the states, actions;
$Q(s,a,w)$. The solution then lies in finding the best setting of the
parameter $w$ . Using this formulation, it is possible to approximate the
Q-function using a Deep Neural Network (DNN). The objective of this DNN shall
be to minimize the Mean Square Error (MSE) of the Q-values as follows:

$\displaystyle
l(w)=E[(r+\gamma\arg\max_{a^{\prime}}Q_{t}(s^{\prime},a^{\prime},w)-Q_{t}(s,a,w))^{2}]$
$\displaystyle J(w)=\max_{w}l(w)$

This objective function is differentiable end-to-end in with respect to its
parameters, i.e. $\frac{\partial l(w)}{\partial w}$ exists. Thus, the
optimization problem can be easily solved using Gradient-based methods
(Stochastic Gradient Descent (SGD), Conjugate Gradients (CG),etc). The
algorithm is called Deep Q-Networks (DQN) [12][11].

### Deep Deterministic Actor Critic (DDAC)

The DQN algorithm is suitable for continuous states cases, but action
selection still requires the action values to be discrete. Several algorithms
were suggested for continuous actions cases, where two functions are learned:
1) the actor; which provides the policy mapping from a state to action, and 2)
the critic; which evaluates (criticizes) the value of the action taken in the
given state. The critic function is the same as the Q-function. The algorithms
to learn both functions follow the policy gradient methods [19]. Under the
framework of deep learning, both functions can be learned through two neural
networks; $Q(s,a,w)$ and $\pi(s,u)$ , since the whole objective is still
differentiable w.r.t. the weights of the Q-function and the policy. Hence, the
gradient of the Q-function (the critic) is obtained as in DQN: $\frac{\partial
l(w)}{\partial w}$ , while the gradient of the policy function (the actor) is
obtained using the chain rule as follows:

$\displaystyle\frac{\partial J}{\partial u}=\frac{\partial Q}{\partial
a}|_{a=\pi(s,u)}\frac{\partial\pi(s,u)}{\partial u}$

### Deep Recurrent Reinforcement Learning

The Q-learning algorithms are based on the Markov assumption of the MDP. In
situations where the full observability of the environment is not available,
this assumption is not valid anymore. Partially observable MDP (POMDP) arises
in different scenarios in autonomous driving, like the occlusion of objects
during tracking, mapping and localization. POMDPs are tackled using
information integration over time, where the true state of the environment is
not directly revealed from single observation, but gradually form over
multiple observations at different time stamps. The recurrent neural networks
(RNN) present themselves as a natural framework to tackle POMDPs. In [13] RNN
was successfully applied for the task of end to end multi-object tracking.
Moreover, LSTMs [5] are integrated to the DQNs to form the Deep Recurrent Q
Networks (DRQN) in [4]. The application of DRQN in [4] to Atari games does not
show the full power of the model, since the MDP assumption is usually enough
for Atari games, hence the authors try a variant of the Pong game namely
Flickering Pong, where they show the advantage of adding recurrence.

### Deep Attention Reinforcement Learning

In the DQN model, the spatial features are extracted via a CNN, which learns
the features from data. These features may not be contributing equally to the
final optimization objective. Similar to the recognition process in human
beings, only a limited amount of information is needed to perform the
recognition tasks, and not all the high dimensional sensory data. Attention
models [23] are trying to follow the same concept, where only part of the CNN
extracted features are used in the classification task. This part is learned
in parallel to the original learning process. In [10], a separate network
called “glimpse network” is trained to deploy the kernel operation at certain
parts of the image. The motion of the kernel is learned via the REINFORCE
algorithm [22] to learn the best sequence of motions of the kernel over the
input image. The result is a motion that resembles the reading process in
human beings in case of feeding images of digits for example. The idea is
exploited in [16] to improve the DQN and DRQN models, by including a filter
after the CNN features extractor, resulting in the DARQN model. The DARQN was
shown to attend and highlight parts of the Atari games that are of special
importance. In the “Sequest” game, the agent was able to focus on the oxygen
level of the submarine, while in “Breakout” game; the agent was able to track
the ball position [22].

Figure 3: Deep Reinforcement Learning framework

### Apprenticeship learning

The Reinforcement learning algorithms described so far follow the concept of
“episodic” learning, or “learning from delayed rewards” [20]. In this setting,
the rewards function is assumed to be known to the algorithm. However, in some
cases, instead of having clear mapping of a state to reward function, we have
a demonstrated expert behavior to the agent. The goal of the agent in this
case is to decipher the intention of the expert, and decode the structure of
the reward function. This is referred to as Inverse Reinforcement learning
[1]. The reward function is encoded as a linear combination of “features”,
functions that map the state according to some features of interest. For
example, for the driving task, one feature function could detect “how far is
the car from the lanes”. Another approach is described in [3], where the
demonstrated expert behavior is utilized in a supervised model, where the
actions taken by the expert together with the states are considered as the
training examples to a CNN.

## PROPOSED DRL FRAMEWORK FOR AUTONOMOUS DRIVING

Following the models described for Deep Reinforcement Learning(DRL), we
propose a pipelined framework for end-end training of a DNN for autonomous
driving as illustrated in Figure 3. The inputs are the states of the
environment and their aggregations over time, and the output is the driving
actions. We discuss each part in detail below with reference to a high-level
block diagram in Figure 4.

### Spatial aggregation

The first step of the pipeline is the spatial aggregation network. This stage
contains two networks: 1) Sensors fusion and 2) Spatial features, which are
described below. The car state includes its position, orientation, velocity
and acceleration. In case of autonomous driving, the surrounding environment
state needs to be encoded as well. The environment state may include the
objects, their location, orientation, motion vectors, dimensions,etc.
Traditional autonomous driving algorithms would make use of these state
vectors to plan the motion path in such an environment. The state of the
surrounding objects is usually not directly observed, but rather deduced by
the algorithm through a set of sensors readings (Camera, LIDAR, etc). Fusing
such sensory information is critical for estimation of a robust state, which
by itself is an active area of research. On the other hand, an end-end deep
learning system would use a different encoding of the state. For example in
[12], the states are just the snapshots of the game, which include by default
a lot of implicit information. Such information is not explicitly given to the
algorithm, but rather deduced through the DNN (more specifically the CNN), in
the form of “features”.

#### Sensor fusion

This way of presenting the raw information to the DNN depends on its ability
to extract relevant features, makes sensors fusion a natural task in the
course of the learning process. In other words, it is possible to group all
the sensors’ information that encode the environment in a raw input vector,
and present this to the DNN, leaving the fusion task to the weights adjustment
process taking place inside the DNN using SGD for example. The gradients will
take care of weighting each sensor features according to how relevant it is to
minimizing the overall cost function discussed in the RL section. A CNN is
best suited to this part of the pipeline.

#### Spatial features

The fused sensors representation enables further processing of the data,
focusing on the important part that is relevant to the driving task. A CNN
would extract deeper representations, followed by an attention filter to
direct the convolution kernels to the parts of the data that are of interest.
The integration of attention models has a high potential for embedded systems
integration. The attention models dramatically reduce the dimensionality of
the data, in addition to the number of computations (convolutions) over the
raw input data. To best utilize the value of attention models, it is
recommended to use the idea of action and glimpse networks [10] rather than
applying an attention filter as in [16]. The attention filter does not reduce
the computations, where the convolution kernel is applied to all the data. On
the other hand, the glimpse and action networks learn to only deploy the
“gaze” of the kernel to certain parts of the data, thereby reducing the
computations dramatically in inference time.

Attention Glimpse NetworkCNNKernel Action SignalSpatial Features Sensor Fusion
Representation  Figure 4: Block diagram of modules in DRL framework

### Recurrent temporal aggregation

The environment state is not directly accessible to the agent or the
autonomous car. Instead, sensor readings provide clues about the true state of
the environment. To decode the true environment state, it is not sufficient to
map single snapshot of sensors readings. However, information integration over
time is mandatory, since the state is revealed as you move. A clear example is
the famous robot localization problem. The most popular algorithm to solve
such a problem is the Bayes filter algorithm, which depends on two probability
density functions (pdf), usually known as the motion model and the sensor
model. The Bayes filter is derived under the MDP assumption. When the state is
discrete we have the Hidden Markov Model (HMM). Finally the Kalman filter is
adding the linear state equations assumption, in addition to the Gaussian
assumption over the two pdfs.

As discussed in the previous section, adding recurrence to the pipeline
enables handling of POMDP situations. Such situations are often encountered in
the driving scenarios. Traditional algorithms for tracking and localization,
like Bayes filters, try to handle such situations through information
integration over time. However, such algorithms are still derived under the
framework of the MDP. Moreover, traditional signal processing approaches like
Kalman filters require manual tuning of its model and parameters.
Alternatively, Recurrent Neural Network (RNN) is able in theory to model long
term dependencies on previous states, in addition to the current observation.
In practice, due to limitations of the back propagation of gradients through
time, RNN is not able to memorize long term dependencies, hence, the LSTM
networks are proposed to solve such an issue [5]. LSTMs are able to control
which information to keep from the previous state, in addition to which
information to forget. Additionally, LSTM controls what to add to the hidden
states from the new inputs.

It is worth noting that RNNs and LSTMs follow the same framework of Bayes
filter, in terms of modeling the motion and observation models, where RNN is
modeling the same pdfs using its hidden state weights and its input weights.
In addition, all Bayes filters (and descendants like Kalman filters), RNN and
LSTM have similar controls over what to include in the next hidden state from
the old hidden state and from the new input. The general equation for Bayesian
filtering and the comparison table of state and state transition functions are
provided below.

$\displaystyle h_{t}=\eta p(x_{t}|h_{t})\int p(h_{t}|h_{t-1})h_{t-1}dh$

Type | $h_{t}$ | $p(x_{t}|h_{t})$ | $p(h_{t}|h_{t-1})$
---|---|---|---
RNN | $tanh[(W_{hh}h_{t-1}+(W_{xh}x_{t})]$ | $W_{xh}$ | $W_{hh}$
Kalman | $A_{t}h_{t-1}+K_{t}x_{t}$ | $K_{t}$ | $A_{t}$
LSTM | $\bar{f_{t}}h_{t-1}+\bar{i_{t}}x_{t}$ | $\bar{i_{t}}$ | $\bar{f_{t}}$
GRNN | $\bar{f_{t}}h_{t-1}+\bar{i_{t}}x_{t}$ | $\bar{i_{t}}$ | $\bar{f_{t}}$

Table 1: Comparison of state functions for temporal filters

### Planning

The final part of the end-end pipeline is the Reinforcement learning planning
part. This network follows the same training procedure of the DQN, with a
Q-network on the top having the same objective function as described in DQN
section. In case of continuous actions, the DDAC network can be used as
discussed in DDAC section. The same error is back propagated through the
network structure to obtain the gradients at every network layer.

## RESULTS AND DISCUSSION

We use lane keeping assist algorithm as a sample application to simulate and
test the proposed framework on The Open-source Racing Car Simulator (Torcs),
screenshot shown in Figure 5. To be able to use simulated sensors, we use the
Simulated Car Racing (SCR) add-on, which gives access to the car controls,
like steering, velocity, acceleration, brakes, and car states, like position,
velocity, acceleration, fuel level, etc. In terms of simulation setup, the
input to the network is the trackPos sensor input, which provides the position
of the track borders, in addition to the car speed in the x-position. The
output are the steering, the gear, the acceleration and the brake values. The
network is trained end-end following the same objective of the DQN.

In order to formulate the problem as a classification problem, the actions
(steer, gear, brake, acceleration) are tiled and discretized. In another
setup, the actions are taken as continuous values following the policy
gradient method of the DDAC. The results show successful lane keeping function
in both cases. However, in case of DQN, removing the replay memory trick
(Q-learning) helps to have faster convergence and better performance. In
addition, the tile coding of actions in case of DQN makes the steering actions
more abrupt. On the other hand, using DDAC continuous policy helps smooth the
actions and provides better performance.

Figure 5: TORCS screen-shot of DRL based lane keeping

Apprenticeship learning with demonstrated behavior was also tested. The
experimental setup is similar to the DQN one described above. The demonstrated
behavior is obtained from a proportional controller that is trying to keep the
lane, while at the same time limiting the speed. The level of supervision is
reduced gradually (the epsilon factor), so when epsilon is 0.5, it means 50%
of the time the actions are coming from the model, and 50% from the
p-controller. At epsilon=0, the model is completely in control. A regression
neural network is used to fit the demo p-controller model. The result is a
successful lane keeping behavior with speed limit. Youtube video links are
provided in the appendix section for sample demonstration of training and lane
keeping.

## CONCLUSION

In this paper, we provided a survey of the recent advances in the field of
Deep Reinforcement Learning. We then followed by a proposal of a framework for
an end-end Deep Reinforcement learning pipeline for autonomous driving which
integrates RNNs to account for POMDP scenarios. We integrate attention models
into the framework, making use of glimpse and action networks to direct the
CNN kernels to the regions of the input that are relevant to the driving
process, thereby improving the computational complexity. The framework was
tested for lane keep assist algorithm and test videos with successful learning
were provided. Our future work includes deploying the proposed framework in a
simulated environment, where the sensors and actuators are artificially
controlled with labeled ground truth. This can lead to potential extension of
the framework to real driving scenarios.

## APPENDIX

Sample DRL training and demo sequences are provided as supplementary material
for the review process. Please visit the following youtube links for DRL
training using DQN, DRL training using DDAC and DRL driving demo using
regression neural network.

Entering URLs explicitly in case the hyperlinks are suppressed. DRL training
using DQN - https://youtu.be/hktC8sGURJQ
DRL training using DDAC - https://youtu.be/OtuKpWew6UI
DRL driving demo using regression neural network -
https://youtu.be/RxIkdKGtzTE

## ACKNOWLEDGMENT

The authors would like to thank their employer for the opportunity to work on
fundamental research. Thanks to B Ravi Kiran (INRIA France) and Catherine
Enright (Valeo) for reviewing the paper and providing feedback. Special thanks
to a colleague Matthew Carrigan for detailed review, proof-reading and
editing.

## References

  * [1] Abbeel, P., & Ng, A. Y. (2004). Apprenticeship learning via inverse reinforcement learning. Proceedings of the twenty-first international conference on Machine learning, (p. 1).
  * [2] Badrinarayanan, V., Kendall, A., & Cipolla, R. (2015). SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation. arXiv preprint arXiv:1511.00561 .
  * [3] Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., et al. (2016). End to End Learning for Self-Driving Cars. arXiv preprint arXiv:1604.07316 .
  * [4] Hausknecht, M., & Stone, P. (2015). Deep recurrent q-learning for partially observable mdps. arXiv preprint arXiv:1507.06527 .
  * [5] Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation , 9 (8), 1735-1780.
  * [6] Huval, B., Wang, T., Tandon, S., Kiske, J., Song, W., Pazhayampallil, J., et al. (2015). An Empirical Evaluation of Deep Learning on Highway Driving. arXiv preprint arXiv:1504.01716 .
  * [7] Kendall, A., Badrinarayanan, V., & Cipolla, R. (2015). Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding. arXiv preprint arXiv:1511.02680.
  * [8] Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, (pp. 1097-1105).
  * [9] LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature , 521 (7553), 436-444.
  * [10] Mnih, V., Heess, N., Graves, A., & others. (2014). Recurrent models of visual attention. Advances in Neural Information Processing Systems, (pp. 2204-2212).
  * [11] Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., et al. (2013). Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 .
  * [12] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., et al. (2015). Human-level control through deep reinforcement learning. Nature , 518 (7540), 529-533.
  * [13] Ondruska, P., & Posner, I. (2016). Deep tracking: Seeing beyond seeing using recurrent neural networks. arXiv preprint arXiv:1602.00991.
  * [14] Pinheiro, P. H., & Collobert, R. (2013). Recurrent convolutional neural networks for scene parsing. arXiv preprint arXiv:1306.2795 .
  * [15] Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., & LeCun, Y. (2013). Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229 .
  * [16] Sorokin, I., Seleznev, A., Pavlov, M., Fedorov, A., & Ignateva, A. (2015). Deep Attention Recurrent Q-Network. arXiv preprint arXiv:1512.01693 .
  * [17] Sutton, R. S. (1988). Learning to predict by the methods of temporal differences. Machine learning , 3 (1), 9-44.
  * [18] Sutton, R. S., & Barto, A. G. (2016). Reinforcement learning: An introduction. Online Draft.
  * [19] Sutton, R. S., McAllester, D. A., Singh, S. P., Mansour, Y., & others. (1999). Policy Gradient Methods for Reinforcement Learning with Function Approximation. NIPS, 99, pp. 1057-1063.
  * [20] Watkins, C. J. (1989). Learning from delayed rewards. Ph.D. dissertation, University of Cambridge England.
  * [21] Watkins, C. J., & Dayan, P. (1992). Q-learning. Machine learning , 8 (3-4), 279-292.
  * [22] Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8 (3-4), 229-256.
  * [23] Xu, K., Ba, J., Kiros, R., Courville, A., Salakhutdinov, R., Zemel, R., et al. (2015). Show, attend and tell: Neural image caption generation with visual attention. arXiv preprint arXiv:1502.03044 .

## Author Biography

Ahmad El Sallab is the Chief Engineer of Deep Learning in the Driving Active
Safety unit, Valeo. Ahmad has 11 years of experience in Machine Learning and
Deep Learning, where he acquired his M.Sc. and Ph.D. on 2009 and 2013 in the
field. He has worked for reputable multi-national organization in the industry
since 2005 like Intel and Valeo. He has 15 publications in Deep Learning in
top IEEE and ACM journals and conferences, with applications in speech, NLP
and Robotics. He is also a designated reviewer in a number of top conferences
like EUSIPCO and ITSC.

Mohammed Abdou is a software engineer and researcher in the Driving Active
Safety unit, systems and functions team at Valeo Cairo. He is pursuing his
master’s degree in control systems with focus on Reinforcement Learning.

Etienne has been a research engineer in computer vision & machine learning at
Valeo Driving Assistance Research for 4 years where he applies Deep Learning
extensively for perception and control. He focuses on embedded development,
keeping in mind constraints from low-power devices. Etienne attended the
French Engineering School of Polytech Orleans; his final specialisation year
was spent at the University of Miami where he learned Machine Learning /
Neural Networks from Miroslav Kubat.

Senthil Yogamani is a technical lead in computer vision department at Valeo
for the past three years. He is currently focused on research and design of
the overall computer vision algorithm architecture for next generation
automated parking systems. He is also a member of the internal autonomous
driving and deep learning working group. He has over 10 years of experience in
computer vision and machine learning including 8 years of experience in
industrial automotive systems. He is an author of 14 peer reviewed
publications and technical reports. He is a recipient of best associate editor
award at IEEE ITSC 2015 and best paper award at IEEE ITST 2012.


11institutetext: Department of Engineering Science, Institute of Biomedical
Engineering, University of Oxford, Oxford, UK 22institutetext: Big Data
Institute, University of Oxford, Li Ka Shing Centre for Health Information and
Discovery, Oxford, UK 33institutetext: NIHR Oxford Biomedical Research Centre,
Oxford, UK 44institutetext: Translational Gastroenterology Unit, Experimental
Medicine Div., John Radcliffe Hospital, University of Oxford, Oxford, UK
44email: {numan.celik, sharib.ali<EMAIL_ADDRESS>
# EndoUDA: A modality independent segmentation approach for endoscopy imaging

Numan Celik 1122 0000-0003-1813-1036 Sharib Ali 112233 0000-0003-1313-3542
Soumya Gupta 1122 0000-0001-5874-5273 Barbara Braden 2233 0000-0002-8534-6873
Jens Rittscher 112233 0000-0002-8528-8298

###### Abstract

Gastrointestinal (GI) cancer precursors require frequent monitoring for risk
stratification of patients. Automated segmentation methods can help to assess
risk areas more accurately, and assist in therapeutic procedures or even
removal. In clinical practice, addition to the conventional white-light
imaging (WLI), complimentary modalities such as narrow-band imaging (NBI) and
fluorescence imaging are used. While, today most segmentation approaches are
supervised and only concentrated on a single modality dataset, this work
exploits to use a target-independent unsupervised domain adaptation (UDA)
technique that is capable to generalize to an unseen target modality. In this
context, we propose a novel UDA-based segmentation method that couples the
variational autoencoder and U-Net with a common EfficientNet-B4 backbone, and
uses a joint loss for latent-space optimization for target samples. We show
that our model can generalize to unseen target NBI (target) modality when
trained using only WLI (source) modality. Our experiments on both upper and
lower GI endoscopy data show the effectiveness of our approach compared to
naive supervised approach and state-of-the-art UDA segmentation methods.

###### Keywords:

Barrett’s esophagus Polyp Endoscopy Unsupervised domain adaptation Variational
autoencoder Segmentation

##  1 Introduction

Most gastrointestinal (GI) cancers are preventable. In 2018, the five major
types of GI cancers, which include those of the stomach, liver, oesophagus,
and colorectum account for 26% of the global cancer incidence and 35% of all
cancer deaths [2]. Endoscopy, a vital tool for screening and disease
surveillance, is however heavily operator dependent and 12% of cancers are
missed [11]. Artificial intelligence-based systems can play a vital role in
improving diagnostic quality. However, it is critical that these tools
integrate directly into the existing clinical workflow and generalize well to
the previously unseen data.

Today, high-definition endoscopes provide a sufficient resolution to allow for
a detailed visualisation of the mucosal surface. In addition to the
conventional white light imaging (WLI) modality, autofluorescence imaging, and
electronic chemoendoscopy techniques such as narrow-band imaging have been
developed to improve the early detection of cancer and its precursors [15]. It
is important that any computer assisted methods should work seamlessly in
these different modalities and avoid the need for any modality-specific
training. Developing such a modality agnostic approach for detection and
segmentation of pre-cancerous changes in the esophagus, and polyps in the
lower GI is the focus of this paper.

Convolutional Neural Network (CNN) based approaches [10, 14, 5], all well
established in biomedical imaging, have been applied to segmentation tasks in
GI endoscopy [1, 8, 19]. Wu et al. [19] proposed a dual neural network based
on U-Net with ResNet backbone for automatic esophageal lesion segmentation.
Guo et al. [8] utilized a FCN-based architecture with atrous kernels to
segment polyps. However, all these supervised approaches are trained on WLI
endoscopy and do not support working in different imaging modalities.

Figure 1: Example images and masks of the two GI datasets for which source and
target domains are separated based on their modality.

To date, the problem of quantifying disease related changes in NBI, which can
produce a very different appearance of the mucosal surface when compared to
WLI (see Fig. 1) has not been addressed. Training methods for different
imaging modalities independently is prohibitive. Firstly, extensive training
datasets that cover a similar range of conditions would need to be curated to
ensure a comparable performance. Secondly, manual annotations by human experts
are required which is not only labour intensive but also expensive. To tackle
these issues domain adaptation methods have been used to adapt trained models
to unseen target data for segmentation [17, 7, 9, 13]. Although these methods
promise to generalise, so far there are only a few examples of applying such
methods to medical data [7, 9]. Additionally, the available methods still
under-perform and have not been adapted for extremely diverse modality cases
such as that in the GI endoscopy.

To develop a modality-agnostic approach, we propose to use an unsupervised
domain adaptation (UDA) technique that can be trained on available single WLI
modality dataset and can be applied on an unseen NBI imagery. For such
settings, where classical supervised models fail to generalize and perform
poorly, we show that our proposed target-independent UDA technique, EndoUDA,
has the potential to generalize and to provide improved segmentation on NBI
modality for both BE and polyp datasets.

Figure 2: Complete architecture of the proposed EndoUDA: a target-independent
unsupervised domain adaptation.

##  2 Method

Our proposed EndoUDA consists of components for training and test time
inference shown in Figure 2. The training network, utilizes a shared
EfficientNet-B4 encoder for our variational autoencoder (VAE) and a U-Net-
based segmentation model. The VAE module allows to learn the source domain
representations while U-Net module leverages the learned encoded features for
semantic segmentation. For inference at test time, learnt latent space
embeddings are further optimized to translate to the unseen modality dataset
by continuously updating via a joint loss minimization scheme. Below, details
of each of these modules are presented.

###  2.1 VAE training on source domain

VAE is a latent space derived generative model that is based on variational
inference distributions. Let $\mathscr{P}_{\theta}(\textbf{z}|\textbf{x})$ be
the latent space distribution for sample x and
$\mathcal{Q}_{\phi}(\textbf{z}|\textbf{x})$ be an identical tractable
distribution so that it approximates the true posterior
$\mathscr{P}_{\theta}(\textbf{z}|\textbf{x})$. Then, the variational lower
bound term of the loss $\mathscr{L}(\theta,\phi)$ between probabilistic
$h_{\phi}$ (encoder, outputs the parameters $\mu_{\textbf{z}}$ and
$\sigma_{\textbf{z}}$ of a distribution) and $h_{\theta}$ (decoder) neural
networks. In Eq. (1), approximated $\mathcal{Q}_{\phi}(\textbf{z}|\textbf{x})$
eliminates the non-negative KL-divergence term imposing a lower bound on the
data log-likelihood only. This is maximized by the VAE via parameterizing
$\mathscr{Q}_{\phi}(\textbf{z}|\textbf{x})$ and
$\mathscr{P}_{\theta}(\textbf{x}|\textbf{z})$ which is similar to the
minimization of the reconstruction loss $\mathscr{L}_{r}(\textbf{x})$ in Eq.
(2).

${}\mathscr{L}(\theta,\phi)=\mathbb{E}_{\mathscr{Q_{\phi}}}[\ln{(\mathscr{P}_{\theta}(\textbf{x}|\textbf{z}))}]-\mathbb{D}_{\textit{KL}}[\mathscr{Q}_{\phi}(\textbf{z}|\textbf{x})\|\mathscr{P}_{\theta}(\textbf{z}|\textbf{x})]$
(1)

Compared to the VAE framework in [13], we propose to couple the encoder layer
of VAE as the segmentation network backbone. Based on our experiments, we have
chosen an EfficientNet which computes the approximate posterior
$\mathcal{Q}_{\phi}(\textbf{z}|\textbf{x})$. Unlike Pandey et al. [13], we
remove the originally used perceptual loss module in the decoder layer used to
compute the image likelihood of the observed data
$\mathscr{P}_{\theta}(\textbf{x}|\textbf{z})$. The decoder network of the VAE
contains 5 convolution layers followed by a LeakyRelu function (Leak=0.2) and
a tanh activation function for the reconstructed output in the final
convolution layer. A Sobel filter generated edge map is concatenated with the
sixth layer of the decoder network of the VAE through a tanh nonlinear
function (see Fig. 2). The reconstruction loss $\mathscr{L}_{r}(\textbf{x})$
is minimized for the VAE training network given by:

${}\mathscr{L}_{r}({\textbf{x}})=\mathscr{L}(\theta,\phi)=\frac{1}{N}\sum_{i=1}^{N}\|\textbf{x}^{i}-h_{\theta}(h_{\phi}(\textbf{x}^{i}))\|^{2},$
(2)

where x is the source domain image, $N$ is total number of samples, $h_{\phi}$
and $h_{\theta}$ are encoder and decoder networks, respectively. Here, the
reconstructed image is given by
$\hat{\textbf{x}}=h_{\theta}(h_{\phi}(\textbf{x}))$. In addition, as shown in
Fig. 2, the edges of the input images are extracted and concatenated into the
decoder network in VAE through a skip connection to better generalize the
source and target domains. Fig. 2 shows a schematic diagram of the complete
VAE architecture used for training on the source domain.

###  2.2 Segmentation module

The segmentation network consists of a U-Net with an EfficientNet-B4 [16]
backbone. The input from the reconstructed image obtained from the decoder of
the VAE is fed to the already tuned weights of EfficientNet-B4. The
segmentation network is adapted by a supervised loss on the source domain
given by a classically used binary cross-entropy loss ($\mathscr{L}_{BCE}$)
and dice coefficient loss ($\mathscr{L}_{DSC}$) [12]. $\mathscr{L}_{DSC}$ is
represented as
$\mathscr{L}_{DSC}=(2\sum\textbf{p}\textbf{g})/(\sum\textbf{p}^{2}+\sum\textbf{g}^{2})$,
where p is the predicted probability map and g is the ground truth mask.

###  2.3 VAE latent search optimization via new joint Loss

The trained VAE encoding aims to find the nearest point from the source
domain, given a sample from the target domain, through an optimization process
and updates of the latent space representation z, as can be seen in Fig. 2.
Once the decoder $h_{\theta}$ of the VAE is trained on the source distribution
$\mathscr{P}_{s}(\textbf{x})$, given an image $\tilde{\textbf{x}}_{t}$ from
the target distribution, the latent search algorithm finds the nearest point
by optimizing latent vector over the iterative process. As shown in the Fig.
1, the WLI images are used in the source domain while the NBI are used in the
target domain. For faster optimization convergence of the latent space
encoding with improved generalization ability of the network, we propose to
use a joint loss function that comprises of the normalized cross correlation
(NCC) $\mathscr{L}_{NCC}$ and the structural similarity index measure (SSIM)
[21] loss $\mathscr{L}_{ssim}$ between the image $\tilde{\textbf{x}}_{t}$ from
target domain and the reconstructed target image $\tilde{\textbf{x}}$ from
$h_{\theta}$. The loss function for computing NCC is defined as:

${}\mathscr{L}_{NCC}(\tilde{\textbf{x}},\tilde{\textbf{x}}_{t})=\frac{1}{2N}\sum(\frac{\tilde{\textbf{x}}-\mu_{s}}{\sqrt{\sigma_{s}^{2}+\epsilon^{2}}}-\frac{\tilde{\textbf{x}}_{t}-\mu_{t}}{\sqrt{\sigma_{t}^{2}+\epsilon^{2}}})^{2},$
(3)

where $\tilde{\textbf{x}}$, $\mu_{s}$ and $\sigma_{s}$ are reconstructed
images from decoder network, mean and standard deviation of the latent encoded
image, respectively. Similarly, $\tilde{\textbf{x}}_{t}$, $\mu_{t}$ and
$\sigma_{t}$ correspond to target image input, mean and standard deviation of
the target image. The SSIM for a pair of images ($\tilde{\textbf{x}}$,
$\tilde{\textbf{x}}_{t}$) and loss function $\mathscr{L}_{ssim}$ is defined
as:

${}\begin{split}SSIM(\tilde{\textbf{x}},\tilde{\textbf{x}}_{t})&=l(\tilde{\textbf{x}},\tilde{\textbf{x}}_{t})^{\alpha}*c(\tilde{\textbf{x}},\tilde{\textbf{x}}_{t})^{\beta}*s(\tilde{\textbf{x}},\tilde{\textbf{x}}_{t})^{\gamma},\\\
\mathscr{L}_{ssim}(\tilde{\textbf{x}},\tilde{\textbf{x}}_{t})&=1-SSIM(\tilde{\textbf{x}},\tilde{\textbf{x}}_{t}),\end{split}$
(4)

here l, c and s denote luminance, contrast and structure similarities of the
given pair of images, respectively with the parameters $\alpha$, $\beta$ and
$\gamma$ $>$ 0\. Also, the similarity loss function $\mathscr{L}_{ssim}$ is
presented. The final joint loss function is given by:

${}\mathscr{L}_{joint}=\lambda\mathscr{L}_{NCC}+(1-\lambda)\mathscr{L}_{ssim},$
(5)

where $\lambda=0.75$ is set empirically. After the convergence of
$\mathscr{L}_{joint}$, the optimal latent space representation
$\hat{\textbf{z}}=\textbf{z}+\eta\mathscr{L}_{joint}$ is used to generate the
closest clone of source domain with a learning rate $\eta$ which is then used
as an input to the segmentation network to predict the target mask.

##  3 Experiments and results

####  3.0.1 Implementation Details.

For the training of our VAE network we used RMSProp with learning rate (lr) of
0.0001 while for the segmentation module Adam optimizer was used with
$\beta_{1}$=0.5, $\beta_{2}$=0.999, and $lr=0.001$. The initial learning rate
was reduced by a factor of 10 for every validation loss that did not improve
in the last 3 epochs and a stopping criteria was set for no improvement in
validation loss upto 10 epochs. We used 100 epochs for training our EndoUDA
training module with batch size of 64. For the test module of EndoUDA, we set
learning rate of $\eta=0.001$ for optimization of the latent space
representation and a fixed 40 iterations with batch size of 32 was set. The
proposed method was implemented using the Tensorflow framework on a machine
deployed with NVIDIA RTX 2080Ti graphics card with 12 GB GPU memory. All input
images were resized to 128 $\times$ 128 pixels.

####  3.0.2 Datasets.

We have used two gastrointestinal (GI) endoscopy datasets for analyzing two
distinct endoscopically found precancerous anomalies, namely, Barrett’s
esophagus (BE) and polyps. To evaluate the efficacy of the proposed EndoUDA
method, we have used clinically acquired white light imaging (WLI) and narrow-
band imaging (NBI) modality data for BE and polyps. BE dataset consists of
1182 endoscopy images acquired from 68 unique patients of which 611 WLI images
are used as source domain data (train set: 90%, validation set: 10%) and 571
NBI images as target domain data are used for testing. Similarly, for the
polyp dataset, we used the publicly available Hyper-Kvasir dataset [4] for
training which consists of 1000 images (train set: 80%, validation set: 20%).
In addition, we used clinically acquired 42 NBI images of polyps acquired from
20 patients at our local hospital and from collaborators.

Table 1: Supervised Segmentation analysis: Benchmarking BE and polyp datasets on standard segmentation architectures on 80-20 train-test split.  Models | BE | Polyp
---|---|---
IoU |  Dice |  IoU |  Dice
U-Net[14] |  0.738 |  0.846 |  0.718 |  0.755
UNet++[20] |  0.742 |  0.859 |  0.721 |  0.778
ResUNet[6] |  0.789 |  0.908 |  0.734 |  0.794
EfficientUNet[3] |  0.841 |  0.931 |  0.803 |  0.896
EfficientUNet++ |  0.828 |  0.927 |  0.767 |  0.865
DeepLabv3+ [5] |  0.802 |  0.893 |  0.789 |  0.869

###  3.1 Benchmark comparison and ablation study

Result of supervised segmentation of BE and polyps in our GI dataset are
presented in Table 1. Frequently used supervised learning methods have been
evaluated with respect to IoU and Dice to establish a baseline. It can be
observed that the UNet with an EfficientNet B4 backbone [3] outperformed other
baseline methods including DeepLabv3+ [5] by nearly 4% and 2% on both IoU and
Dice for BE and polyps, respectively. Supplementary material Table 1-4 show
that our choice of the joint loss function, the Sobel edge operator, the
decoder architecture and the computational complexity yield in improved
performance compared to alternatives that are often used.

Table 2: Empirical results of EndoUDA along with SOTA UDA methods with mean $\mu$ and standard deviation $\sigma$ are provided. All comparisons are provided for source only trained model and tested on target data. Paired t-test between EndoUDA and each SOTA methods are shown in Supplementary Table 5.  Models | BE ($\mu\pm\sigma$) | Polyp ($\mu\pm\sigma$)
---|---|---
IoU | Dice | Precision | Recall | IoU | Dice | Precision | Recall
Naive U-Net [3] |  | 0.626
---
${\pm{0.02}}$
| 0.718
---
${\pm{0.015}}$
| 0.654
---
${\pm{0.018}}$
| 0.667
---
${\pm{0.021}}$
| 0.492
---
${\pm{0.018}}$
| 0.564
---
${\pm{0.012}}$
| 0.487
---
${\pm{0.011}}$
| 0.496
---
${\pm{0.011}}$
AdaptSegnet [17] |  | 0.658
---
${\pm{0.019}}$
| 0.749
---
${\pm{0.016}}$
| 0.721
---
${\pm{0.012}}$
| 0.698
---
${\pm{0.022}}$
| 0.519
---
${\pm{0.017}}$
| 0.577
---
${\pm{0.015}}$
| 0.601
---
${\pm{0.016}}$
| 0.583
---
${\pm{0.012}}$
ADVENT [18] |  | 0.667
---
${\pm{0.012}}$
| 0.768
---
${\pm{0.014}}$
| 0.739
---
${\pm{0.015}}$
| 0.706
---
${\pm{0.012}}$
| 0.524
---
${\pm{0.011}}$
| 0.591
---
${\pm{0.019}}$
| 0.662
---
${\pm{0.021}}$
| 0.638
---
${\pm{0.014}}$
CellSegUDA [9] |  | 0.673
---
${\pm{0.02}}$
| 0.771
---
${\pm{0.016}}$
| 0.733
---
${\pm{0.018}}$
| 0.691
---
${\pm{0.013}}$
| 0.533
---
${\pm{0.012}}$
| 0.629
---
${\pm{0.021}}$
| 0.688
---
${\pm{0.018}}$
| 0.641
---
${\pm{0.017}}$
GLSS [13] |  | 0.704
---
${\pm{0.019}}$
| 0.815
---
${\pm{0.022}}$
| 0.846
---
${\pm{0.019}}$
| 0.735
---
${\pm{0.02}}$
| 0.558
---
${\pm{0.017}}$
| 0.621
---
${\pm{0.016}}$
| 0.675
---
${\pm{0.019}}$
| 0.649
---
${\pm{0.012}}$
EndoUDA |  | 0.733
---
${\pm{0.014}}$
| 0.854
---
${\pm{0.021}}$
| 0.832
---
${\pm{0.017}}$
| 0.784
---
${\pm{0.019}}$
| 0.605
---
${\pm{0.014}}$
| 0.693
---
${\pm{0.02}}$
| 0.722
---
${\pm{0.017}}$
| 0.704
---
${\pm{0.018}}$
Figure 3: Qualitative study: Baseline UDA experiments on BE and polyp images.

###  3.2 Comparison with SOTA method

We compared the proposed EndoUDA approach with four SOTA UDA methods:
AdaptSegnet[17], ADVENT[18], CellSegUDA[9], GLSS[13], along with the naive
supervised baseline model. The quantitative results are reported in Table 2
for four different metrics including intersection-over-union (IoU), Dice,
precision and recall metrics. It can be observed that our proposed framework
EndoUDA outperformed other SOTA UDA methods and naive U-Net method. An
improvement of nearly 4% on BE and 7% on polyp dataset for with respect to the
Dice metric compared to the recent SOTA GLSS method. Similar improvements can
be seen for other metrics for both GI datasets and also much higher
improvements compared to other SOTA UDA methods. Table 5 in the supplementary
material shows the statistical significance of the improvement provided by our
method when compared to SOTA methods presented in Table 2 with p-values
$<$0.01 for almost all metrics and SOTA approaches for both BE and polyp
segmentation.

It can be seen in Fig. 3 that the predicted masks using proposed EndoUDA model
for BE and polyp images are very close to the ground truth annotations
compared to those predicted by other SOTA UDA methods.

###  3.3 Effect of out-of-distribution data in supervised training

We further investigated the performance of EndoUDA compared to the supervised
baseline U-Net model (with EfficientNet backbone) by training the model with
10%, 25% and 50% of target data samples. It can be observed (see Table 3 that
the EndoUDA performs consistently better than the baseline approach for a
different proportion of target data inclusion in the training. For example, on
10% and 25% of target data mixed in the training sample, EndoUDA still
provides nearly a 7-10% improvement on IoU and 8-13% with respect to Dice
scores over the baseline method.

Table 3: IoU and dice comparisons for out-of-distribution target samples (NBI) in the source (WLI) GI endoscopy datasets  Method | BEIoU | PolypIoU | BEdice | Polypdice
---|---|---|---|---
U-Net (source-only) | 0.626 | 0.492 | 0.718 | 0.564
EndoUDA | 0.733 | 0.605 | 0.854 | 0.693
U-Net (source 100% + target 10%) | 0.646 | 0.536 | 0.738 | 0.588
EndoUDA (source 100% + target 10%) | 0.744 | 0.613 | 0.866 | 0.706
U-Net (source 100% + target 25%) | 0.691 | 0.579 | 0.761 | 0.641
EndoUDA (source 100% + target 25%) | 0.756 | 0.624 | 0.875 | 0.721
U-Net (source 100% + target 50%) | 0.738 | 0.631 | 0.827 | 0.734
EndoUDA (source 100% + target 50%) | 0.771 | 0.658 | 0.889 | 0.739
U-Net (target-trained) | 0.841 | 0.802 | 0.931 | 0.896

##  4 Conclusion

We have presented a novel target-independent unsupervised domain adaptation
VAE-based method that comprises of a shared encoder for both reconstruction
and segmentation, and a joint loss function for improved unseen target domain
generalization. We have validated our approach on two GI endoscopy datasets
that often require additional modalities for screening and intervention. The
proposed modality agnostic method can be used in clinical GI endoscopy without
imposing any constraints on what imaging modality is being used. Our
qualitative and quantitative results demonstrate the effectiveness and promise
of the proposed method compared to both the naive baseline supervised model
and state-of-the-art UDA segmentation methods.

#### Acknowledgement

NC is supported by the Emerson Collective Cancer Research Fund, SG is funded
by Boston Scientific, SA and JR are supported by National Institute for Health
Research (NIHR) Oxford Biomedical Research Centre (BRC). The views expressed
are those of the authors and not necessarily those of the NHS, the NIHR or the
Department of Health. The authors declare no competing interests.

## References

  * [1] Ali, S., Dmitrieva, M., Ghatwary, N., Bano, S., et al.: Deep learning for detection and segmentation of artefact and disease instances in gastrointestinal endoscopy. Medical Image Analysis 70, 102002 (2021)
  * [2] Arnold, M., Abnet, C.C., Neale, R.E., Vignat, J., Giovannucci, E.L., McGlynn, K.A., Bray, F.: Global burden of 5 major types of gastrointestinal cancer. Gastroenterology 159(1), 335–349 (2020)
  * [3] Baheti, B., Innani, S., Gajre, S., Talbar, S.: Eff-UNet: A Novel Architecture for Semantic Segmentation in Unstructured Environment. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp. 1473–1481 (2020)
  * [4] Borgli, H., Thambawita, V., Smedsrud, P.H., Hicks, S., et al.: HyperKvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy. Scientific Data 7(1), 1–14 (2020)
  * [5] Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 833–851 (2018)
  * [6] Diakogiannis, F.I., Waldner, F., Caccetta, P., Wu, C.: ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data. ISPRS Journal of Photogrammetry and Remote Sensing 162, 94–114 (2020)
  * [7] Dong, N., Kampffmeyer, M., Liang, X., Wang, Z., Dai, W., Xing, E.: Unsupervised Domain Adaptation for Automatic Estimation of Cardiothoracic Ratio. In: Medical Image Computing and Computer Assisted Intervention (MICCAI). pp. 544–552 (2018)
  * [8] Guo, Y., Bernal, J., J Matuszewski, B.: Polyp Segmentation with Fully Convolutional Deep Neural Networks—Extended Evaluation Study. Journal of Imaging 6(7), 69 (2020)
  * [9] Haq, M.M., Huang, J.: Adversarial Domain Adaptation for Cell Segmentation. In: Proceedings of the Third Conference on Medical Imaging with Deep Learning (MIDL). pp. 277–287 (2020)
  * [10] Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3431–3440 (2015)
  * [11] Menon, S., Trudgill, N.: How commonly is upper gastrointestinal cancer missed at endoscopy? A meta-analysis. Endoscopy international open 2(2), E46 (2014)
  * [12] Milletari, F., Navab, N., Ahmadi, S.: V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation. In: International Conference on 3D Vision (3DV). pp. 565–571 (2016)
  * [13] Pandey, P., Tyagi, A.K., Ambekar, S., Prathosh, A.P.: Unsupervised Domain Adaptation for Semantic Segmentation of NIR Images Through Generative Latent Search. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 413–429 (2020)
  * [14] Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional Networks for Biomedical Image Segmentation. In: Medical Image Computing and Computer-Assisted Intervention (MICCAI). pp. 234–241 (2015)
  * [15] Subramanian, V., Ragunath, K.: Advanced endoscopic imaging: a review of commercially available technologies. Clinical Gastroenterology and Hepatology 12(3), 368–376 (2014)
  * [16] Tan, M., Le, Q.: EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In: Proceedings of International Conference on Machine Learning (ICML). pp. 6105–6114 (2019)
  * [17] Tsai, Y., Hung, W., Schulter, S., Sohn, K., Yang, M., Chandraker, M.: Learning to Adapt Structured Output Space for Semantic Segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 7472–7481 (2018)
  * [18] Vu, T.H., Jain, H., Bucher, M., Cord, M., Pérez, P.: Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2512–2521 (2019)
  * [19] Wu, Z., Ge, R., Wen, M., Liu, G., et al.: ELNet: Automatic classification and segmentation for esophageal lesions using convolutional neural network. Medical Image Analysis 67, 101838 (2021)
  * [20] Zhou, Z., Siddiquee, M.M.R., Tajbakhsh, N., Liang, J.: UNet++: A nested u-net architecture for medical image segmentation. In: Deep learning in medical image analysis and multimodal learning for clinical decision support, pp. 3–11 (2018)
  * [21] Zhou Wang, Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13(4), 600–612 (2004)