# Colonoscopy Navigation using End-to-End Deep Visuomotor Control:
A User Study

Ameya Pore1,2,∗, Martina Finocchiaro 2,3, Diego Dall’Alba1, Albert
Hernansanz2, Gastone Ciuti3,
Alberto Arezzo4, Arianna Menciassi3, Alicia Casals2, Paolo Fiorini1 This
project has received funding from the European Union’s Horizon 2020 research
and innovation programme under the Marie Skłodowska-Curie (grant agreement No.
813782 ”ATLAS”)* Corresponding author: Ameya Pore (email:
ameya.pore@univr.it)1 Department of Computer Science, University of Verona,
Italy2 Department of Enginyeria de Sistemas, Automatica i Informatica
Industrial, Univiversitat Politecnica de Catalunya, Barcelona, Spain 3 The
BioRobotics Institute, Scuola Superiore Sant’Anna, Pisa, Italy 4 Department of
Surgical Sciences, University of Torino, Turin, Italy

###### Abstract

Flexible endoscopes for colonoscopy present several limitations due to their
inherent complexity, resulting in patient discomfort and lack of intuitiveness
for clinicians. Robotic devices together with autonomous control represent a
viable solution to reduce the workload of endoscopists and the training time
while improving the overall procedure outcome. Prior works on autonomous
endoscope control use heuristic policies that limit their generalisation to
the unstructured and highly deformable colon environment and require frequent
human intervention. This work proposes an image-based control of the endoscope
using Deep Reinforcement Learning, called Deep Visuomotor Control (DVC), to
exhibit adaptive behaviour in convoluted sections of the colon tract. DVC
learns a mapping between the endoscopic images and the control signal of the
endoscope. A first user study of 20 expert gastrointestinal endoscopists was
carried out to compare their navigation performance with DVC policies using a
realistic virtual simulator. The results indicate that DVC shows equivalent
performance on several assessment parameters, being more safer. Moreover, a
second user study with 20 novice participants was performed to demonstrate
easier human supervision compared to a state-of-the-art heuristic control
policy. Seamless supervision of colonoscopy procedures would enable
interventionists to focus on the medical decision rather than on the control
problem of the endoscope.

##  I INTRODUCTION

Colonoscopy screening programs remain the gold standard for the diagnosis and
treatment of lower-gastric pathologies such as colorectal cancer (CRC), which
is the third most common malignancy worldwide [1]. During a routine diagnostic
procedure, a Flexible Endoscope (FE) is firstly inserted from the rectum to
the caecum and then retracted slowly to detect possible early-stage CRC
lesions. Early-stage detection of CRC can improve the five-year survival rate
by 90% [2]. FE-based procedures are complex due to non-intuitive mapping
between the endoscope tip and the control steering knobs, which requires a
long and extensive training process to be mastered [3].

Figure 1: Deep Visuomotor Control (DVC) flow diagram. The environment provides
a state observation $S_{t}$. The DVC agent uses the state input to generate an
action $a_{t}$ that is applied to the environment. During the training phase,
DVC learns a task-conditioned policy $\pi_{\phi}$ to perform autonomous
colonoscopy navigation. In the evaluation phase, the clinicians can supervise
and override DVC decisions through action $a_{t^{\prime}}$.

Consequently, these procedures are more susceptible to human errors and
increase the risks of tissue stretching and perforation, which are the prime
cause of patient discomfort, and pain [4]. Moreover, some studies have
reported work-related musculoskeletal injuries among endoscopists due to
awkward neck and body posturing [5]. The shortage of adequately trained
endoscopists with respect to the increasing clinical demand of colonoscopy
procedures can lead to potential loss of human lives [6].

To overcome these limitations, wireless capsule endoscopes have been developed
[7]; however, wireless devices lack the control of endoscopic point of view
and increase the risk of missing some pathological areas [3]. Therefore,
current research efforts are developing navigation systems using robotised FE,
such as the STRAS system [8], or Magnetic actuated FEs [9]. Robotised FE
enables the introduction of automation technologies to enhance human operator
abilities, in particular by adding autonomous navigation, which is the most
time-consuming step of a routine colonoscopy procedure [10]. This will allow
the endoscopists to focus on the clinical aspect of the procedure rather than
the manual control of FE, potentially improving the overall procedure outcome
and reducing the training time [4].

During the navigation phase, the clinician mainly uses visual feedback from
the FE camera to advance through the lumen [11]. A common gesture observed by
endoscopists during a colonoscopy procedure is to centralise the target
direction of the endoscope towards the lumen centre. Prior works on endoscopic
navigation have built rule-based controllers to replicate this gesture by
reducing the distance error between the image centre and the detected lumen
centre [12]. These algorithms fail in situations when the tip of the endoscope
approaches close to the colon wall. Such situations occur due to the highly
deformable nature of the colon and the variable mobility introduced by patient
movements, peristalsis and breathing, which lead to changes in lumen diameter
and haustral folds due to which lumen detection is not trivial. These
situations require human interventions to find the correct motion direction,
or they can be handled by adaptive exploration methods, as proposed in this
work.

Originally postulated rule-based controllers are being progressively replaced
by data-driven approaches such as Deep Reinforcement Learning (DRL), since
they are able to provide some degree of adaptability [13, 14]. However, the
application of DRL in learning surgical task policies has been limited to low-
dimensional physical state features such as robot kinematic data, which are
widely accepted to be sample-efficient and trivial to learn [14, 15]. This
paper proposed an image-based DRL approach for endoscopic control (Fig. 1)
focussing on learning the navigation task by devising an end-to-end policy to
map the raw endoscopic images to the control signal of the endoscope, referred
henceforth as Deep Visuomotor Control (DVC). We primarily evaluate DVC control
through a user study with 20 expert GastroInstestinal (GI) endoscopists who
perform the navigation task in a realistic virtual simulator.

The introduction of autonomous navigation can improve clinical practice,
relieving clinicians from demanding cognitive and physical tasks. However, in
safety-critical areas, such as medical robotics, it is highly desirable to
maintain human supervision to address ethical and legal concerns [16]. Hence,
it is essential to consider human-in-the-loop for DVC deployment in realistic
surgical scenarios. Therefore, we conducted a second user study with 20 novice
participants to demonstrate that non-expert users can easily supervise
autonomous navigation, and DVC reduces the need for human intervention
compared to a state-of-the-art method.

This work presents an initial study towards generating adaptive control for
the colonoscopy procedure by proposing a DVC control policy for autonomous
navigation and providing its performance evaluation with expert GI
endoscopists.

The content of this paper is organised as follows: Sec. II describes the
related works, while implemented methods are explained in Sec. III. In Sec.
IV, we elaborate on the experimental evaluation. Finally, the results and
conclusions are discussed in Sec. V and Sec. VI respectively.

##  II RELATED WORKS

The advantages of autonomous navigation in colonoscopy have prompted several
studies in this field. In [17], a screw-type colonic endoscope is developed,
and motion adjustment is demonstrated using reinforcement learning. This study
uses robot kinematics variables as state inputs; however, navigation through
the straight segments was slow, and navigation through bends proved awkward
due to the robot’s size. Several studies have focused on magnetic guided
endoscopes [9, 10, 4]. In [9], navigation by following simple predefined
trajectories is presented; hence extending this method to complex non-linear
trajectories is challenging. Heuristic path planning algorithms are used in
[4] to generate a feasible path in a colon map. This approach employs force-
based real-time sensing to guide navigation. Force-based sensing is still not
widely available in existing endoscopic devices; moreover, the interpretation
of robotic actions without scene visualisation is challenging, hence not
suitable for human supervision. In [10], a static perception model is
developed, which extracts the centre of the lumen from raw image observation.
The control of endoscope position and orientation is imparted by a
proportional controller that aligns the endoscopic image with the centre of
the lumen. Similar rule-based controllers have been previously developed in
[12]; however, they require significant manual tasking for non-linear
components such as analytically computing image jacobian, and interaction
matrix [18]. Moreover, lumen detection could be unstable and prone to errors
due to the dynamic nature of the colon and its sharp bends. Such scenarios
require the vision-based control system to improve during policy training
which is limited with hand-engineered features for perception [18]. Learning
end-to-end visuomotor representations for direct control using DRL overcomes
these limitations without separately designing perception and control models
and offers the ability to improve model parameters while training [19, 20].

Some studies have proposed frameworks for training DRL policies to automate
surgical tasks [13, 14, 21, 22] such as manipulation of rigid and deformable
objects. These studies use simplified environments designed explicitly for
robot-assisted surgery to learn the instrument control during the procedure.
Recently, [23] proposed a DRL method for optimising the endoscopic camera
viewpoint. These studies use low-dimensional state information for training
DRL algorithms, such as kinematic values of the robot, the position of target
etc. [13, 14, 21, 23]. In a real colonoscopy scenario, it is challenging to
accurately capture the endoscope kinematics due to limits on the sensing
capabilities [4], and intra-operative guidance is solely based on visual
feedback.

Figure 2: Colon models used in the experimental phase. (From left to right)
ranked in increasing complexity order, $C_{0}$, $C_{1}$, $C_{2}$ and $C_{3}$
colon models. The model complexity is characterised by the centerline distance
of the model from rectum to caecum, and the number of acute bending, i.e.
$>$90 degree, which is estimated through visual inspection.

##  III METHODS

Our objective is to develop end-to-end joint training for perception and
control to learn navigation policies that map raw endoscopic image
observations directly to the control signals of the robotised FE (e.g. motor
torques). We develop a realistic colonoscopy simulator with deformable tissue
dynamics, described in Sec. III-A. Further, we explain the implementation
details of DVC in Sec. III-B.

###  III-A Simulation platform

Colon simulation \- A public CT colonography dataset from the Cancer Imaging
Archive [24] is used to derive the colon models. The 3D models of the bowel
are segmented in a semi-automated way [25]. The segmented models are refined,
and volumetrical and superficial meshes are generated. Realistic render
textures are created using the real endoscopy images from KVASIR dataset [26]
and applied to the models. Thus, the reconstructed colon models are loaded
into SOFA (Simulation Open Framework Architecture), where a realistic
mechanical model based on real-time finite element analysis is generated,
setting the simulation parameters in order to mimic the colon tissue behaviour
[27]. Additionally, the collision detection between the endoscope and the
colon is implemented, and physical constraints are included to realistically
restrain the colon deformations. For creating high quality and realistic
visual rendering, Unity3D is adopted, exploiting the High Definition Render
Pipeline (HDRP), which allows for additional visual effects such as reflection
on the organ surface or vignetting the peripheral darkening of the endoscopy
image [26]. Therefore, the SofaAPAPI-Unity3D is used to interface the SOFA
simulation with the Unity visual rendering.

Figure 3: Representation of the local frame at the endoscope tip. The X-Y
plane of the camera is parallel to the image frame, while the z-axis
represents the direction of insertion. Tip bending is carried out on the X-Y
plane while the roll is carried on the z-axis. DVC uses a low-resolution image
as state input. The green region represents the detected lumen centre.

Endoscope simulation \- We assume a scenario close to a magnetically guided FE
where external magnets control the motion of the magnetic tip while the tether
follows the tip passively [9]. Hence, in this preliminary simulator version,
we neglect the effect of the endoscope tether due to multiple collision points
with the colon model that could lead to simulation instability. The endoscope
tip is modelled as a rigid capsule with weight, length and diameter of 20g,
36mm and 14mm, respectively [28]. An angular drag of 4 rad/sec2 is added to
account for the frictional resistance [28]. The endoscope tip embeds a camera
and has four degrees of freedom for the motion as shown in Fig. 3, i.e.
translation (insertion/retraction), roll, bending in two perpendicular
directions (pitch/yaw).

###  III-B Deep Visuomotor control

DRL background \- The colon navigation problem is formalised into a Markov
Decision Process (MDP) represented by a tuple
$(\mathcal{S},\mathcal{A},\mathcal{R},\mathcal{P},\gamma,T)$, where
$\mathcal{S}$ denotes the state space, $\mathcal{A}$ is the action space,
$\mathcal{P}$ is the transition probability distribution, $\mathcal{R}$ is the
reward space, $\gamma\in[0,1]$ is the discount factor and T is the time
horizon per episode. At each timestep $t$, the environment produces a state
observation $s_{t}\in\mathcal{S}$. The agent then generates an action
$a_{t}\in\mathcal{A}$ according to a policy $a_{t}\sim\pi(s_{t})$, and applies
it to the environment to receive a reward $r_{t}\in\mathcal{R}$ [29]. As a
consequence, the agent transitions to a new state $s_{t+1}$ sampled from the
transition function $p(s_{t+1}|s_{t},a_{t})$, $p\in\mathcal{P}$ or terminates
the episode at state $s_{T}$.

Learning algorithm \- The agent’s goal is to learn a stochastic behaviour
policy $\pi$ parametrised by $\phi$,
$\pi_{\phi}:\mathcal{S}\rightarrow\mathcal{P}(\mathcal{A})$ to maximise the
expected future discounted reward $E[\sum_{i=0}^{T-1}\gamma^{i}r_{i}]$. We
chose PPO [30] as a consolidated DRL algorithm over Soft-Actor Critic [31] and
Deep deterministic policy gradient [32] due to overall returns in terms of
wall-clock training time and hyper-parameter tuning. It is out of the scope of
this work to propose a novel DRL method, while the main goal is to perform a
user study to evaluate the performance of image-based DRL in colonoscopy
navigation. PPO consists of a value and a policy network that uses shared
parameters to estimate the state value ($V$) and predict the action vector
($a$). In the training session, the length of each episode is set as 10k
iteration steps, $\gamma$ = 0.99, and the batch size and the learning rate
hyperparameters are 64 and 3e-4, respectively. The PPO clip ratio was 0.2 with
4 mini-batches per epoch and 4 epochs per iteration. A loss term proportional
to negative policy entropy was added, with a coefficient of 0.01. Each
training lasted for 1.5 million iteration steps, which was the time taken for
the reward function to converge (Fig.6).

Action space \- The preliminary manual control of the endoscope revealed that
if the endoscope is directed against the colon wall, especially at the sharp
turns, the lumen is not visible. Hence, it is critical to avoid the
translation of the endoscope in such scenarios. Therefore, we develop an
action strategy, where a translation motion with a constant velocity of
$v_{end}=$ 10mm/sec is carried out only when the lumen is detected. The action
space consists of discrete angular rotation values in the three degrees of
freedom at the endoscope tip, $\delta\theta_{j}=\alpha$,
$\alpha\in\\{0,-1,+1\\}$ in the $j^{th}$ spatial dimension. In the tip local
reference frame, $j\in{x,y,z}$ corresponds to the orientation alignment in the
horizontal and vertical directions in the image plane and the endoscope roll,
respectively (Fig. 3). In cases when the lumen is not visible, the translation
velocity of the endoscope is set to zero, and the agent carries out
orientation changes to detect the lumen.

Observation space and policy \- The sensory input to the DVC agent is composed
of a downscaled endoscopic image. The RGB images rendered by the endoscopic
camera (1024x1024 pixels) is downscaled to 128x128 pixels to reduce the
computational complexity of the training DVC network. The policy $\pi_{\phi}$
is represented by a CNN architecture, consisting of two convolutional layers
(Fig. 1) for encoding visual scene representations. The network details are
publicly available on the project website111https://github.com/Ameyapores/DVC.
The output of the convolutional layers are fed into a combination of fully
connected layers and Long Short-Term Memory (LSTM) layer to represent time-
dependent behaviour, each with 128 rectified units, followed by the linear
connections to the output logits $\pi_{t}$ for each action $a_{t}$ and values
estimate $V_{t}$. A softmax function transforms the logits to action
probabilities. The complete network is trained end-to-end to acquire task-
specific visual features.

Figure 4: Proposed adaptive threshold segmentation pipeline for lumen detection. Each RGB frame captured by the endoscopic camera is passed through the adaptive filter to detect the dark pixels a) original RGB frame b) Image mask for the detected lumen (in green) c) distance vector between the image centre $P_{c}$ and the centroid of the detected darkest regions $P_{L}$. Figure 5: Manual supervision through a joystick while navigating by autonomous control strategies. supervision is printed on the screen, indicating the switch to manual control. When the endoscope is oriented towards the lumen (green point), the user can give back the control to the autonomous agent. A low-resolution (128x128 pixels) image is displayed to facilitate interpretability of machine decisions, however users have the option to change to high resolution (1024x1024 pixel) display. TABLE I: Navigation parameters used for validation with their description |  Navigation metrics |  Description
---|---|---
1 |  Time of insertion (TOI) |  TOI is measured from the time point where the initial movement of the endoscope is detected to the time point when the caecum is reached.
2 |  Perforation |  Perforation refers to the scenario when excessive force is applied on the colon wall (especially at the turning point) that can lead to severe injuries. Studies based on tensile property analysis of human rectal tissue reported the maximum elongation of 62% [27]. The average diameter of the colon models used is 5cm, hence a threshold of $\delta d=3cm$ is decided to classify the deformation as perforation.
3 |  Normalised distance travelled |  Distance travelled is crucial as multiple backward motions, reversing the direction, can lead to suboptimal trajectories. The distance travelled is measured using the position values of the endoscope tip. This distance is normalised by the centerline distance of the colon model in order to compare among different colon models. Normalised distance above 1 indicates a path distance longer than the centerline path, while a normalised distance below 1 indicates a shorter path than the centerline was followed.
4 |  Average LD |  Lumen centralisation is believed to create smooth insertion trajectories hence the lumen distance in the image plane is recorded at each timepoint. This distance is normalised by the size of the image to get a value in [0,1]. Lumen distance value 0 denotes image centre ($P_{c}$) coincides with the detected lumen ($P_{L}$), and value 1 denotes that the detected lumen is at the farthest point.

Reward function \- The goal of navigation is to reach the end of the colon
without any significant complications. Visuomotor control should be able to
track the colon during the whole procedure. Successful tracking requires the
lumen centre $P_{L}$ to be close to the image centre $P_{c}$. Hence, a dense
reward $r_{t}(s_{t},a_{t})$ is designed as follows:

$r_{t}(s_{t},a_{t})=\begin{cases}C(1-(||P_{L}-P_{c}||_{2}/D_{max})),&L=1\\\
-1,&L=0\end{cases}$ (1)

where $D_{max}=1/2*(Imagewidth)=64$, is the normalisation factor which is the
maximum distance possible, $L$ represents the lumen detection flag, (1 denotes
lumen detected, 0 denotes no lumen detected), the hyperparameter $C$ is chosen
as 1. Moreover, the agent is awarded a reward of +10 if the colon end is
reached and -10 if it returns to the original starting point, to encourage the
agent to move unidirectional towards the caecum. To detect the colon lumen in
the endoscope image, we build a threshold segmentation algorithm that runs in
real-time at 30fps based on [33], where the image is first segmented to detect
the darkest and most distinct region, with the presumption that this area
contains the distal lumen with high probability (Fig.4). The segmentation is
performed by converting the RGB image to greyscale and cropping a circular
region centred with the image and a diameter equal to the image width to
remove the vignette effect on the corner.

Figure 6: Learning curve of DVC trained on varying complexity of colon, using
three colon models. Cumulative reward is normalised in the range $[-1,1]$. The
shaded area spans the range of values obtained when training the agent
starting from five different initialisation seeds.

(a) Lumen distance

(b) Perforations

(c) Normalised distance

(d) Time of insertion

Figure 7: Navigation performance comparison plots between DVC and
endoscopists. Several parameters are plotted a) Lumen distance, b)
perforations, c) Normalised distance, d) Time of insertion.

##  IV EXPERIMENTAL EVALUATION

The experimental goal is to compare the navigation performance of the DVC
agents, the baseline method of rule-based control [10] and the endoscopists.
Hence, we create a pipeline where the position and orientation values of the
endoscope, lumen distance in the image space, colon deformations, and camera
image can be recorded within our developed simulator. During the experiments,
these parameters were synchronised and recorded for future processing using
labstreaminglayer software, which is a unified system for the collection of
time-series measurements [34].

Endoscopist data acquisition \- A group of 20 expert GI endoscopists (more
than four years of experience) were asked to make navigation attempts in the
colonoscopy simulation scene developed in Sec. III-A. Due to the time
constraints and COVID regulations at the hospital222Ospedale Le Molinette
(Torino, Italy), four colon models were selected considering the opinion of
domain experts to represent progressively more complex scenarios (Fig. 2).
Each endoscopist was instructed to navigate the colon models from the rectum
to the caecum using a through PlayStation (Sony Interactive Entertainment,
USA) joystick device. The colon model $C_{0}$, which depicts a simplified
colon model that conforms with the shape and size of the average human colon,
was used to familiarise the endoscopists with the controls before initialising
the trials. The trials started with endoscopist attempts on the $C_{1}$ colon,
followed by randomised attempts on $C_{2}$ and $C_{3}$ colon. The randomness
between $C_{2}$ and $C_{3}$ colon was introduced to identify performance bias
based on the colon model.

Training DVC agents \- We conduct three experiments to validate the DVC.
First, our aim is to determine the sample efficiency of training on different
levels of colon complexity. Hence, we train DVC agents separately using the
same models employed during the endoscopist experiment. Second, to establish a
comparative analysis between the DVC and endoscopists, a similar experimental
workflow was followed as in the endoscopist experiments, where the DVC was
only trained on $C_{0}$ ($DVC_{C_{0}}$) and tested on $C_{1}$, $C_{2}$ and
$C_{3}$ colons. Third, the DVC was trained on the $C_{0}$ model followed by
training on the $C_{1}$ ($DVC_{C_{0}+C_{1}}$) to test if training on a complex
colon after a simple one improves performance. To keep the overall iteration
steps for DVC training at 1.5 million, the training on $C_{0}$ was terminated
after 1 million iteration steps and loaded back to train on $C_{1}$ for 500k
iteration steps (Table. LABEL:Tab:abb). table

Supervision \- 20 novice participants (no endoscopy experience) were asked to
supervise the performance of the rule-based controller agent and the DVC
agent. The experimental workflow consisted of three trials; inside each trial,
the participants attempted to navigate $C_{1}$, $C_{2}$ and $C_{3}$ colon
models. Each trial was characterised by one of the following control
strategies.

  1. 1.

Manual control: Participants were instructed to exclusively control the
endoscope using a joystick during the entire duration of the procedure.

  2. 2.

Rule-based baseline [10]: A proportional controller is generated for
orientation control that aligns the image center ($P_{c}$) to the detected
lumen ($P_{L}$) as follows:

$\delta\theta=\beta\begin{bmatrix}P_{L_{x}}-P_{c_{x}}\\\
P_{L_{y}}-P_{c_{y}}\end{bmatrix}$ (2)

We refer to the distance between $P_{L}$ and $P_{c}$ as Lumen Distance (LD).

  3. 3.

DVC: A fully trained $DVC_{C_{0}}$ was deployed.

In control strategy 2, the rule-based controller indicates the requirement of
manual supervision when the lumen centre is not detected (Fig. 5). In control
strategy 3, the agent is given ($\Delta_{t}=50$) iteration steps to search for
lumen when the lumen is not detected. After $\Delta_{t}$ steps, the DVC
notifies the requirement of human supervision, and manual control is
activated. In both the control strategies, the user has an override option to
take control when unsafe behaviour is encountered, e.g. collision with the
colon wall or direction of motion reversed. Once the manual control is active,
the participants can navigate the endoscope safely and give back the control
to the DVC or rule-based controller. During each attempt, the number of
interventions by the participant was recorded. After all the trials, users
were asked to complete a NASA Task Load Index (TLX) questionnaire [35], to
score human-perceived workload.

TABLE II: NASA Task Load Index for novice users. Lower score indicate good user experience |  Manual control |  Rule-based control | DVC
---|---|---|---
Mental demand | 63 | 33 | 18
Physical demand | 65 | 38 | 9
Temporal demand | 30 | 47 | 17
Performance | 25 | 34 | 12
Effort | 57 | 38 | 10
Frustration | 42 | 41 | 12
Mean workload | 47 | 38 | 13

(a) $C_{1}$

(b) $C_{2}$

(c) $C_{3}$

Figure 8: Trajectory plot of DVC, complex and smoothest endoscopist
performance for a) $C_{1}$ b) $C_{2}$ 3) $C_{3}$ models respectively.

Data Analysis \- Four different parameters are used to score the navigation
performance. Time of Insertion (TOI) and the number of colon perforations are
qualitative assessment measures for colonoscopy procedures [36], while average
LD and the normalised distance travelled are the two metrics devised in this
study to measure the accuracy of the trajectories. The details of each
parameter are elaborated in Table. I. When the user or DVC reversed its
direction of motion and returned to the rectum or perforated heavily to
destabilise the colon model, it was considered a failed navigation attempt.

##  V RESULTS AND DISCUSSION

The learning curves when DVC is trained on different levels of colon
complexity are presented in Fig. 6. $C_{0}$ represents a simplistic model;
hence, the DVC agent reaches high reward values in relatively fewer steps than
in other colon models. A high reward indicates that the agent successfully
learns to complete the navigation task. Whereas $C_{2}$ represents high
complexity, the agent requires 1.2 million steps for high-reward convergence.
The $C_{1}$ training curve lies between $C_{0}$ and $C_{2}$. This suggests
that the training time is related to colon complexity. However, note that
$DVC_{C_{0}}$ can navigate other complex colon models, i.e. it acquires task-
specific features that can generalise to other colon models (Table.
LABEL:Tab:abb).

Comparative analysis \- The performance data of 20 endoscopists was acquired
while 10 different DVC agents were trained on the $C_{0}$ starting with a
different random seed. The realism of the simulation was confirmed and
validated by expert clinicians. Additionally, all the users positively
evaluated the joystick used to navigate the endoscope inside the colon as
intuitive, user-friendly and easy to learn. Fig. 7 shows the comparison of the
average LD, the number of perforations, the completion time and the normalised
distance travelled. There is a significant difference in the average LD and
the number of perforations between the endoscopists and the DVC. DVC shows
precise tip centralisation and less number of perforations compared to
endoscopists. One of the reasons for this difference is that clinicians tend
to push the colon wall at acute bends of colon junctions (See supplementary
video). This is a gesture sometimes clinicians follow due to the rigid
constraints of the clinically available FEs. Whereas DVC is trained on reward
feedback to minimise LD, it stays centralised to avoid contact with the wall.
For the normalised distance and TOI, a substantial difference is not noted.
There is more variance observed in the performance of the endoscopists. Some
endoscopists followed a convoluted trajectory that increased the normalised
distance and time of insertion, while others followed smoother trajectories
that resulted in the lower normalised distance and TOI. Fig. 8 shows the most
complex and smoothest trajectories demonstrated by the endoscopists and the
trajectory executed by $DVC_{C_{0}}$ for the $C_{1}$, $C_{2}$ and $C_{3}$
colons. The smoothness of a trajectory is estimated using a jerk index J
($cm/sec^{3}$) which characterises the average rate of change of acceleration
in a movement [37]. Human operators tend to show wide variance in performing
optimal trajectories, while DVC performance stays in the average range.

The result of splitting the training into two colon models $DVC_{C_{0}+C_{1}}$
and evaluating on other colon models are shown in Table. LABEL:Tab:abb. There
is an improvement in the lumen detection performance for $DVC_{C_{0}+C_{1}}$
in comparison to $DVC_{C_{0}}$. $DVC_{C_{0}}$ reaches high rewards at 500k
iteration steps; hence there is no additional feedback to improve the
performance. We speculate that the agent reaches suboptimal local minima,
while, when the DVC trained on $C_{0}$ is loaded to train on $C_{1}$, it
encounters acute bends, which offers the potential to maximise the cumulative
reward. There is no considerable improvement on other navigation parameters,
i.e. perforation, TOI and normalised distance.

Supervision \- The human interventions are divided into two parts. First,
where the user overrides the control due to unsafe behaviour and second, where
the system demands human supervision. The average human intervention required
for rule-based baseline was $5\pm 1.8$ for human override and $2.5\pm 1.5$
when the system demanded human control, while for DVC, the average number of
human interventions are $0.1\pm 0.5$ for human override and $0.05\pm 0.2$ when
the system demanded human control. This difference is attributed to DVC’s
adaptability to search for new insertion directions when the lumen is not
easily detected, whereas the rule-based controller lacks this ability. The
NASA-TLX for each control strategy is shown in Table. II. Regarding ease of
use, participants found manual control and rule-based controller more
demanding in all task load categories, while a substantial workload reduction
is observed for DVC.

##  VI CONCLUSIONS

Prior works on autonomous colonoscopy navigation use heuristic control
policies that fail to adapt to situations where detecting lumen is not
straightforward and requires frequent human intervention. This article
proposes an end-to-end DVC that learns a mapping between the endoscopic images
and the endoscope’s control signal, such as tip orientation. DRL has been
applied in the surgical domain; however, these works use low-dimensional
physical robotic state features that are challenging to obtain using FEs. To
evaluate the navigation performance of DVC, motion data from 20 GI
endoscopists was acquired and compared to DVC control. Our experimental
validation shows an equivalent performance in terms of the time of insertion
and the distance travelled. However, DVC reduces the number of perforations
and shows efficient lumen tracking, improving safety. Furthermore, we
conducted a second novice user study to demonstrate that supervision of DVC
control significantly reduces the user workload with overall performance
comparable to expert GI endoscopists.

On the contrary, there are some limitations of this work. First, it is not
straightforward to know the direction of motion of the endoscope. Hence, our
newer version of the virtual scene will simulate the endoscope body dynamics,
providing the insertion length. Second, if the robot needs to learn from raw
image observations, it also needs to evaluate the reward function from raw
image observations, which itself requires a hand-designed perception system.
This can be mitigated by using online user interaction through human-in-the-
loop reinforcement learning [38]. The results obtained in Table. LABEL:Tab:abb
presents an opportunity to study curriculum learning-based setup [39], where
colon navigation can be trained in increasing levels of colon complexity. Our
future work will demonstrate the formal validation of the realism of the
proposed virtual simulator.

## References

  * [1] F. Bray, J. Ferlay, I. Soerjomataram, R. L. Siegel, L. A. Torre, and A. Jemal, “Global cancer statistics 2018: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries,” _CA: a cancer journal for clinicians_ , vol. 68, no. 6, pp. 394–424, 2018.
  * [2] C. Chen, M. Hoffmeister, and H. Brenner, “The toll of not screening for colorectal cancer,” _Expert review of gastroenterology & hepatology_, vol. 11, no. 1, pp. 1–3, 2017.
  * [3] G. Ciuti, K. Skonieczna-Żydecka, W. Marlicz, V. Iacovacci, H. Liu, D. Stoyanov, A. Arezzo, M. Chiurazzi, E. Toth, H. Thorlacius _et al._ , “Frontiers of robotic colonoscopy: a comprehensive review of robotic colonoscopes and technologies,” _Journal of Clinical Medicine_ , vol. 9, no. 6, p. 1648, 2020.
  * [4] H.-E. Huang, S.-Y. Yen, C.-F. Chu, F.-M. Suk, G.-S. Lien, and C.-W. Liu, “Autonomous navigation of a magnetic colonoscope using force sensing and a heuristic search algorithm,” _Scientific reports_ , vol. 11, no. 1, pp. 1–15, 2021.
  * [5] A. K. Shergill, K. R. McQuaid, and D. Rempel, “Ergonomics and gi endoscopy,” _Gastrointestinal endoscopy_ , vol. 70, no. 1, pp. 145–153, 2009.
  * [6] S.-H. Lee, Y.-K. Park, D.-J. Lee, and K.-M. Kim, “Colonoscopy procedural skills and training for new beginners,” _World Journal of Gastroenterology: WJG_ , vol. 20, no. 45, p. 16984, 2014.
  * [7] G. Iddan, G. Meron, A. Glukhovsky, and P. Swain, “Wireless capsule endoscopy,” _Nature_ , vol. 405, no. 6785, pp. 417–417, 2000.
  * [8] F. Nageotte, L. Zorn, P. Zanne, and M. De Mathelin, “Stras: A modular and flexible telemanipulated robotic device for intraluminal surgery,” in _Handbook of Robotic and Image-Guided Surgery_. Elsevier, 2020, pp. 123–146.
  * [9] A. Z. Taddese, P. R. Slawinski, K. L. Obstein, and P. Valdastri, “Nonholonomic closed-loop velocity control of a soft-tethered magnetic capsule endoscope,” in _2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2016, pp. 1139–1144.
  * [10] J. W. Martin, B. Scaglioni, J. C. Norton, V. Subramanian, A. Arezzo, K. L. Obstein, and P. Valdastri, “Enabling the future of colonoscopy with intelligent and autonomous magnetic manipulation,” _Nature machine intelligence_ , vol. 2, no. 10, pp. 595–606, 2020.
  * [11] D. Kragic, H. I. Christensen _et al._ , “Survey on visual servoing for manipulation,” _Computational Vision and Active Perception Laboratory, Fiskartorpsv_ , vol. 15, p. 2002, 2002.
  * [12] N. v. d. Stap, C. H. Slump, I. A. Broeders, and F. v. d. Heijden, “Image-based navigation for a robotized flexible endoscope,” in _International Workshop on Computer-Assisted and Robotic Endoscopy_. Springer, 2014, pp. 77–87.
  * [13] F. Richter, R. K. Orosco, and M. C. Yip, “Open-sourced reinforcement learning environments for surgical robotics,” _arXiv preprint arXiv:1903.02090_ , 2019\.
  * [14] E. Tagliabue, A. Pore, D. Dall’Alba, E. Magnabosco, M. Piccinelli, and P. Fiorini, “Soft tissue simulation environment to learn manipulation tasks in autonomous robotic surgery,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 3261–3266.
  * [15] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq _et al._ , “Deepmind control suite,” _arXiv preprint arXiv:1801.00690_ , 2018.
  * [16] E. Parliament and C. of the European Union, “Artificial intelligence act: Regulation laying down harmonised rules on artificial intelligence and amending certain union legislative acts,” _Proposal for Regulation COM/2021/206 final, Brussels, Belgium_ , 2021.
  * [17] G. Trovato, M. Shikanai, G. Ukawa, J. Kinoshita, N. Murai, J. Lee, H. Ishii, A. Takanishi, K. Tanoue, S. Ieiri _et al._ , “Development of a colon endoscope robot that adjusts its locomotion through the use of reinforcement learning,” _International journal of computer assisted radiology and surgery_ , vol. 5, no. 4, pp. 317–325, 2010.
  * [18] A. Saxena, H. Pandya, G. Kumar, A. Gaud, and K. M. Krishna, “Exploring convolutional networks for end-to-end visual servoing,” in _2017 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2017, pp. 3817–3823.
  * [19] J. Ibarz, J. Tan, C. Finn, M. Kalakrishnan, P. Pastor, and S. Levine, “How to train your robot with deep reinforcement learning: lessons we have learned,” _The International Journal of Robotics Research_ , vol. 40, no. 4-5, pp. 698–721, 2021.
  * [20] S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training of deep visuomotor policies,” _The Journal of Machine Learning Research_ , vol. 17, no. 1, pp. 1334–1373, 2016.
  * [21] J. Xu, B. Li, B. Lu, Y.-H. Liu, Q. Dou, and P.-A. Heng, “Surrol: An open-source reinforcement learning centered and dvrk compatible platform for surgical robot learning,” in _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2021, pp. 1821–1828.
  * [22] P. M. Scheikl, B. Gyenes, T. Davitashvili, R. Younis, A. Schulze, B. P. Müller-Stich, G. Neumann, M. Wagner, and F. Mathis-Ullrich, “Cooperative assistance in robotic surgery through multi-agent reinforcement learning,” in _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2021, pp. 1859–1864.
  * [23] Y.-H. Su, K. Huang, and B. Hannaford, “Multicamera 3d viewpoint adjustment for robotic surgery via deep reinforcement learning,” _Journal of Medical Robotics Research_ , vol. 6, no. 01n02, p. 2140003, 2021.
  * [24] K. Clark, B. Vendt, K. Smith, J. Freymann, J. Kirby, P. Koppel, S. Moore, S. Phillips, D. Maffitt, M. Pringle _et al._ , “The cancer imaging archive (tcia): maintaining and operating a public information repository,” _Journal of digital imaging_ , vol. 26, no. 6, pp. 1045–1057, 2013.
  * [25] B. H. Jeong, H. K. Kim, and Y. D. Son, “Depth estimation of endoscopy using sim-to-real transfer,” _arXiv preprint arXiv:2112.13595_ , 2021.
  * [26] K. İncetan, I. O. Celik, A. Obeid, G. I. Gokceler, K. B. Ozyoruk, Y. Almalioglu, R. J. Chen, F. Mahmood, H. Gilbert, N. J. Durr _et al._ , “Vr-caps: A virtual environment for capsule endoscopy,” _Medical image analysis_ , vol. 70, p. 101990, 2021.
  * [27] M. B. Christensen, K. Oberg, and J. C. Wolchok, “Tensile properties of the rectal and sigmoid colon: a comparative analysis of human and porcine tissue,” _Springerplus_ , vol. 4, no. 1, pp. 1–10, 2015.
  * [28] Y. Wang, Y. He, X. Deng, Z. Lei, Y. Chen, and M. Li, “Learning friction model for tethered capsule robot,” in _2021 6th International Conference on Robotics and Automation Engineering (ICRAE)_. IEEE, 2021, pp. 280–285.
  * [29] R. S. Sutton and A. G. Barto, _Reinforcement learning: An introduction_. MIT press, 2018.
  * [30] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017\.
  * [31] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel _et al._ , “Soft actor-critic algorithms and applications,” _arXiv preprint arXiv:1812.05905_ , 2018.
  * [32] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” _arXiv preprint arXiv:1509.02971_ , 2015.
  * [33] D. Wang, X. Xie, G. Li, Z. Yin, and Z. Wang, “A lumen detection-based intestinal direction vector acquisition method for wireless endoscopy systems,” _IEEE Transactions on Biomedical Engineering_ , vol. 62, no. 3, pp. 807–819, 2014.
  * [34] M. Merino-Monge, A. J. Molina-Cantero, J. A. Castro-García, and I. M. Gómez-González, “An easy-to-use multi-source recording and synchronization software for experimental trials,” _IEEE Access_ , vol. 8, pp. 200 618–200 634, 2020.
  * [35] S. G. Hart and L. E. Staveland, “Development of nasa-tlx (task load index): Results of empirical and theoretical research,” in _Advances in psychology_. Elsevier, 1988, vol. 52, pp. 139–183.
  * [36] M. F. Kaminski, S. Thomas-Gibson, M. Bugajski, M. Bretthauer, C. J. Rees, E. Dekker, G. Hoff, R. Jover, S. Suchanek, M. Ferlitsch _et al._ , “Performance measures for lower gastrointestinal endoscopy: a european society of gastrointestinal endoscopy (esge) quality improvement initiative,” _Endoscopy_ , vol. 49, no. 04, pp. 378–397, 2017.
  * [37] R. Shadmehr and S. Wise, “Supplementary documents for “computational neurobiology of reaching and pointing”,” 2005.
  * [38] S. Chen, J. Gao, S. Reddy, G. Berseth, A. D. Dragan, and S. Levine, “Asha: Assistive teleoperation via human-in-the-loop reinforcement learning,” _arXiv preprint arXiv:2202.02465_ , 2022.
  * [39] Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in _Proceedings of the 26th annual international conference on machine learning_ , 2009, pp. 41–48.


###### Abstract

Time is a parameter playing a central role in our most fundamental modelling
of natural laws. Relativity theory shows that the comparison of times measured
by different clocks depends on their relative motion and on the strength of
the gravitational field in which they are embedded. In standard cosmology, the
time parameter is the one measured by fundamental clocks (i.e., clocks at rest
with respect to the expanding space). This proper time is assumed to flow at a
constant rate throughout the whole history of the universe. We make the
alternative hypothesis that the rate at which the cosmological time flows
depends on the dynamical state of the universe. In thermodynamics, the arrow
of time is strongly related to the second law, which states that the entropy
of an isolated system will always increase with time or, at best, stay
constant. Hence, we assume that the time measured by fundamental clocks is
proportional to the entropy of the region of the universe that is causally
connected to them. Under that simple assumption, we find it possible to build
toy cosmological models that present an acceleration of their expansion
without any need for dark energy while being spatially closed and finite,
avoiding the need to deal with infinite values.

###### keywords:

cosmology: theory; entropy; arrow of time; dark energy

x 10.3390/—— xx Received: 29 May 2017; Accepted: 12 July 2017; Published: 14
July 2017 Cosmological Time, Entropy and Infinity Clémentine Hauret *,†,
Pierre Magain † and Judith Biernaux Clémentine Hauret, Pierre Magain and
Judith Biernaux Correspondence<EMAIL_ADDRESS>Tel.:
+32-4-366-9678 These authors contributed equally to this work.

##  1 Introduction

Since its introduction nearly a century ago (Einstein, 1916), general
relativity (GR) has been brilliantly confirmed by a number of observations,
most notably the perihelion precession of Mercury, the gravitational redshift,
and the deflection of light by massive bodies. GR has also been used in
cosmology to describe the evolution of the universe as a whole, and the model
that currently gives the best description of its large-scale structure and
evolution (namely, the lambda cold dark matter ($\Lambda$CDM) model) is based
on the equations of GR. However, in order to provide an accurate description
of large-scale (galaxy-scale and higher) and long-term phenomena, a number of
ingredients had to be added to the theory. The motion of stars in the
outskirts of spiral galaxies and the velocity dispersions of galaxies in
clusters required the addition of dark matter. The acceleration of the
expansion of the universe called for the introduction of a repulsive component
called dark energy.

One may note that all the successful tests of GR are dealing with small scales
and are quasi-instantaneous in terms of cosmological time. Conversely,
additional ingredients (dark matter and dark energy) are necessary to bring
the models into agreement with observations dealing with very large scales and
very long-term phenomena. We are thus faced with the following alternative:
either these dark components are actual constituents of the universe (in which
case they should sooner or later be identified) or there is something wrong
with the application of GR on very large scales, and especially on the
universe as a whole.

The vast majority of specialists are in favour of the existence of dark matter
and dark energy. However, despite considerable effort, these dark components
have not yet been identified. Some alternative theories have been proposed,
the most popular being the modified Newtonian dynamics (MOND), Milgrom (1983).

In this paper, we want to explore another hypothesis—that the problems
encountered when dealing with large scales are related to our fundamental
understanding of time. Special relativity has shown that the time measured by
different clocks depends on their relative motion. GR has shown that the time
they measure also depends on the strength of the gravitational field in which
they are embedded. We wish to go one step further and postulate that time
depends on the dynamical state of the universe in which it is set.

The arrow of time is often put in relation with the second law of
thermodynamics, which states that the entropy of an isolated system can only
increase (or at best stay constant) as time flows. In this context, we propose
that time might be regarded as an emergent property in the universe,
proportional to the entropy of the causally connected volume, considered as
the ultimate isolated system. Note that other theories dealing with emergent
time Page (1994) have been presented in the past. More recently, emergent
gravity Verlinde (2016) was also introduced in the context of string theories.

Starting from this assumption, we develop a very simple, semi-empirical, and
mathematically basic toy model of our universe. Our goal in this paper is to
design a straightforward implementation of our hypothesis and to determine its
potential consequences on the (dark matter and dark energy) content and
evolution of our universe. We thus emphasize that this study is only
preliminary. However, as the results are promising with an acceleration of the
expansion of the universe without recourse to dark energy, this model is worth
exploring further in forthcoming papers—particularly since other theories
investigating the link between entropy and the accelerated expansion of our
universe have recently been proposed Pavón and Radicella (2014)Pandey (2017)

For the purpose of this paper, we must make a distinction between two concepts
of time: the physical one (also called cosmological time in this paper) and
the coordinate one. We will define and characterise them in further detail in
Section 2. After that, in Section 3, we will specify our assumption on the
relation between the dynamical state of the universe (here, the entropy) and
the (cosmological) time concept. We stress that we only present in this
article one example among others of practical implementation of this link.
Obviously, other mathematical dependences between time and entropy or between
time and a parameter characterising the dynamical state of the universe (such
as its curvature or its density parameters, for example) might be proposed
or—even better—derived from theoretical considerations. Nevertheless, in
Section 4 we build a semi-empirical toy model and show that it can produce a
spatially closed universe with an accelerating expansion without dark energy.

##  2 Cosmological Time and Coordinate Time

When applying GR to the whole universe in order to derive the $\Lambda$CDM
model, one assumes that it is homogeneous on large scales and that the small-
scale inhomogeneities have no impact on its evolution as a whole. One also
assumes that the four-dimensional space-time can be sliced into three-
dimensional space-like hypersurfaces labeled with a time parameter $t$. This
time parameter is usually taken as the time measured by fundamental comoving
clocks Hobson et al. (2006). The time $t$ is then identified with the proper
time of fundamental observers. It is generally taken for granted that if the
universe is homogeneous, this proper time flows at the same rate all along the
history of the universe. When Einstein derived the basic theory of GR, the
universe was believed to be essentially static and there was no reason to
question that hypothesis. However, in the framework of an evolving universe,
one might ask whether the rate at which time flows could depend on its
dynamical state.

We thus propose to distinguish between two different times: (1) the
(conventional) coordinate time parameter $t$, which is the one measured by
fundamental clocks at the Big Bang and is assumed to flow at a constant rate
along the evolution of our universe; and (2) the cosmological time $\tau$,
which is the one measured by fundamental clocks at any times and is assumed to
depend on the state of the universe and to control all physical processes.

To sum up, this $\tau$ time is the time we live in. Every measurement is thus
made in this time. However, the unit of $\tau$ varies with the evolution of
the universe (i.e., a unit of $\tau$ at time $\tau_{1}$ is not equal to a unit
of $\tau$ at time $\tau_{2}\neq\tau_{1}$), as can be seen at the bottom of
Figure 1. On the other hand, the coordinate time $t$ is chosen so that its
unit does not vary with time. To set the unit of $t$, we decide to make it
equal to the unit of $\tau$ at the Big Bang, as shown in this same Figure 1.

Figure 1:  Illustration of the difference between the cosmological time
$\tau$, of which the units are of varying length, and the coordinate time $t$,
which flows at a constant rate. The unit of $t$ is chosen to be equal to the
unit of $\tau$ at the Big Bang.

Cosmological models with varying fundamental constants such as the
gravitational constant $G$, the fine-structure constant $\alpha$, or the speed
of light $c$ have already existed for a long time. In 1957, Dicke published
one of the first theories of gravitation with variable constants Dicke (1957)
and a review of the numerous models developed since then can be found in Chiba
(2011). In this paper, we go one step further with a varying time flow.

##  3 Cosmological Time, Ultimate Isolated System, and Entropy of the Universe

In the following and as already stated, we make a specific hypothesis on the
relation between the cosmological time $\tau$ and the physical properties of
the universe to develop a mathematically rudimentary cosmological model.

It is often argued that the direction of time flow—the arrow of time—is
dictated by the second law of thermodynamics: the direction of time flow is
the one for which the entropy of an isolated system increases. Let us consider
fundamental observers in the universe. The largest system possibly interacting
with such observers (i.e., the ultimate isolated system from their point of
view) is the region of the universe that is causally connected to them. That
region is bounded by the particle horizon, defined by the distance a light
signal could travel from the beginning of time to the observers. From their
very point of view, that system can be considered isolated since no
interaction can happen with objects located further than the particle horizon.
We thus make the simple assumption that the cosmological time $\tau$ measured
by such observers is proportional to the entropy of the region of the universe
that is causally connected to them.

Conventional estimates of the entropy budget (e.g., Egan and Lineweaver
(2010)) result in black holes dominating the entropy of the universe by far.
Indeed, if one admits the Bekenstein–Hawking estimate for the black hole
entropy Bekenstein (1973)Hawking (1976), the entropy budget of the stellar
black holes would exceed that of the (much more numerous) stars by 17 orders
of magnitude and that of the cosmic microwave background (CMB) by seven to
eight orders of magnitude Egan and Lineweaver (2010). Even more dramatically,
a handful of supermassive black holes like those found at the centres of
galaxies would completely dominate the entropy budget of the universe.

We do not consider these possible sources of entropy in our calculations for
the following reason. Let us consider the typical situation in which a stellar
black hole can be detected. This is a binary stellar system in which one of
the stars is a red giant losing matter to its compact companion. An accretion
disk forms around the compact object and emits high energy radiation as its
temperature reaches high values close to the central object. Whether the
compact object is a neutron star or a black hole can only be known by
estimating its mass and verifying whether or not it exceeds the maximum mass
of neutron stars ($\sim 3M_{\odot}$). The gravitational effect of a black hole
just above that mass limit on the accretion disk (and even more, on the
causally connected region) hardly differs from that of a neutron star just
below the mass limit. However, in the Bekenstein–Hawking framework, there
would be a dramatic entropy discrepancy between those two objects, without any
measurable effect on the causally connected region, let alone on the
cosmological time. To avoid that inconsistency, we choose not to consider the
Bekenstein–Hawking entropy. Even if we did, it is reasonable to assume that
the black hole entropy has no effect on the entropy budget outside of its
horizon. Regardless, this hypothesis could always be rewritten such that
cosmological time is proportional to the radiation entropy in the causally
connected region of the universe. For any of these three reasons, we chose not
to consider black holes’ entropy in our calculation.

The entropy of the universe is hence dominated by the CMB Egan and Lineweaver
(2010). Relic neutrinos are predicted to contribute nearly as much as the CMB.
However, as their entropy has the same dependence on temperature and volume as
the CMB photons, including them would only change the proportionality
coefficient between entropy and cosmological time, and not the relation
between cosmological and coordinate times.

##  4 Evolution of Cosmological Time

The CMB is a photon gas very close to thermodynamic equilibrium, whose entropy
$S$ only depends on its volume $V$ and temperature $T$ via Egan and Lineweaver
(2010)

$S=\frac{4\pi^{2}k_{B}^{4}}{45c^{3}\hbar^{3}}VT^{3},$ (1)

where $k_{B}$ is the Boltzmann constant, $c$ the speed of light and $\hbar$
the reduced Planck constant. $V_{\rm horiz}$ being the causally connected
volume, we assume $\tau\propto S(V_{\rm horiz})\propto V_{\rm horiz}T^{3}$. To
compute the causally connected volume, we solve the Einstein equations of GR
in a metric representing our universe.

By construction, GR is a local theory. Applying it to the whole universe is an
attempt to use a local theory to solve a global problem. If the universe is
spatially homogeneous, applying the Einstein equations in the vicinity of any
observer is equivalent to applying them anywhere else, which means this local
theory may be extended to any point in space. However, in our toy model, this
is not true for any instant in time, as the cosmological time $\tau$ will not
flow at a constant rate with the evolution of our universe.

As GR has been successfully verified locally, we make the hypothesis that the
Einstein equations are valid at any location in space-time, when written as a
function of the local space-time coordinates $(t,r,\theta,\phi)$. Then, under
the assumptions of homogeneity and isotropy of the universe, the interval $ds$
is given by the Robertson–Walker metric:

$ds^{2}=c^{2}dt^{2}-R^{2}\left[\frac{dr^{2}}{1-kr^{2}}+r^{2}(d\theta^{2}+\sin^{2}\theta\,d\phi^{2})\right],$

where $(r,\theta,\phi)$ are the comoving spherical coordinates, $R$ is the
scale factor of our universe, and $k$ measures its spatial curvature. The
Robertson–Walker metric and the Einstein equations are thus assumed to be
valid at any point in space and at any time. However, the proper time flow of
fundamental comoving observers will vary with time. After solving the
equations for any local proper time $t$, we must connect these local proper
times together by replacing them with the cosmological proper time $\tau(t)$.
The equations cannot be directly solved as a function of $\tau$, as the latter
depends on the global properties of the universe and is not an independent
variable. Of course, and as already pointed out, such a procedure should later
be tested in a more rigorous mathematical framework, as the aim of the present
paper is only to illustrate the consequences our basic hypothesis might have
on the content and evolution of the universe.

Hence, in this specific implementation of our basic assumption, we have to
solve the Einstein equations in coordinate time $t$, as it is the only time we
can access at this stage of the calculations. Through this, we aim to obtain
the evolution (still in $t$ time) of the entropy in the causally connected
volume $S_{\rm horiz}(t)$ to apply our assumed relation between the
cosmological time $\tau$ and this entropy ($\tau(t)\propto S_{\rm horiz}(t)$).
Thanks to this assumption, we can then associate each coordinate time $t$ (and
thus each value of the scale factor $R$) with the cosmological time $\tau$,
which effectively controls the physical phenomena. The resolution of equations
in coordinate time $t$ (i.e., a resolution of the usual Friedmann equation) is
thus only (in the toy model presented here) a way to determine the physical
time $\tau$.

The Einstein equations in $t$ are simplified to the usual Friedmann equation.
The analytical solutions $t(R)$ can be easily found in the case of a universe
where matter dominates the matter–energy budget (as is the case now) and used
to determine the comoving distance of the horizon $r_{\mathrm{horiz}}$:

$r_{\mathrm{horiz}}=\frac{1}{\sqrt{|k|}}~{}\Theta_{k}\left(\sqrt{|k|}\int_{0}^{R}\frac{c}{R}\frac{dt}{dR}dR\right)$
(2)

where the $\Theta_{k}(x)$ function is defined according to the sign of $k$:

$\begin{array}[]{cc}&\\\ \Theta_{k}(x)&=\\\
&\end{array}\left\\{\begin{array}[]{cccc}\sin(x)&\mathrm{if~{}}k>0\\\
x&\mathrm{if~{}}k=0\\\ \sinh(x)&\mathrm{if~{}}k<0\end{array}\right..$

We can solve Equation (2) to obtain a simpler expression of
$r_{\mathrm{horiz}}$ as a function of the present Hubble constant $H_{0,t}$
and of the present matter-density parameter $\Omega_{0,t}$, both defined in
coordinate time $t$ by

$H_{0,t}=\frac{1}{R_{0}}\frac{dR}{dt}\Big{|}_{t=t_{0}}~{}~{}\mathrm{and}~{}~{}\Omega_{0,t}=\frac{8\pi
G\rho_{0}}{3H_{0,t}^{2}}$ (3)

where $G$ is the gravitational constant and $\rho_{0}$ the present matter
density. We also set the present scale factor value $R_{0}$ to unity.
$r_{\mathrm{horiz}}$ is then given—regardless of the spatial curvature of the
universe—by

$r_{\mathrm{horiz}}=\frac{2c}{H_{0,\mathrm{t}}\Omega_{0,t}}\sqrt{R}\sqrt{\Omega_{0,t}+R(1-\Omega_{0,t})}.$

From that equation, we can compute the expressions of the causally connected
volume $V_{\rm horiz}=4\pi
R^{3}\int_{0}^{r_{\mathrm{horiz}}}x^{2}/\sqrt{1-kx^{2}}dx$, of the entropy
$S_{\rm horiz}$ in this volume thanks to Equation (1), and of the temporal
variation of this entropy $dS_{\rm horiz}/dt$ :

$\frac{dS_{\rm
horiz}}{dt}=\frac{64\pi^{3}k_{\mathrm{B}}^{4}T_{0}^{3}}{45\hbar^{3}}\frac{1}{H_{0,t}^{2}\Omega_{0,t}^{2}}\left(\Omega_{0,t}+R(1-\Omega_{0,t})\right),$

where $T_{0}$ is the present temperature of the CMB and where we have used the
relation $TR=T_{0}R_{0}$. From that, we easily determine the relation between
the two times $t$ and $\tau$, as we assume that $\tau\propto S_{\rm horiz}$
(see Section 3 and the beginning of Section 4):

$\frac{d\tau}{dt}=\frac{dS_{\mathrm{horiz}}/dt}{dS_{\mathrm{horiz}}/dt|_{\mathrm{BB}}}=1+R(\frac{1}{\Omega_{0,t}}-1).$
(4)

The constant of proportionality $dS_{\mathrm{horiz}}/dt|_{\mathrm{BB}}$ stands
for the temporal variation of the entropy in the causally connected volume at
the Big Bang, so that the coordinate time and the cosmological time are set
equal at the Big Bang.

One can directly notice that $d\tau/dt=1$ for a flat universe
($\Omega_{0,t}=1$). Thus, in the geometrically flat case, the cosmological
time flows at a constant rate along the evolution of the universe, $\tau$ and
$t$ become identical, and our usual representation of time does not need to be
modified. In non-flat cases, however, the distinction between these times must
obviously be maintained. Therefore, as $\tau$ is the physical time we live in,
we do not measure the present Hubble and matter-density parameters in $t$
time, but in $\tau$ time. They are defined as in Equation (3), but as
functions of the cosmological time:

$H_{0,\tau}=\frac{1}{R_{0}}\frac{dR}{d\tau}\Big{|}_{\tau=\tau_{0}}~{}~{}\mathrm{and}~{}~{}\Omega_{0,\tau}=\frac{8\pi
G\rho_{0}}{3H_{0,\tau}^{2}}.$

Combining these definitions in our two different times, we obtain:

$\Omega_{0,\tau}=\frac{8\pi
G\rho_{0}}{3\left(\frac{1}{R_{0}}\frac{dR}{d\tau}\Big{|}_{\tau=\tau_{0}}\right)^{2}}=\frac{8\pi
G\rho_{0}}{3\left(\frac{1}{R_{0}}\frac{dR}{dt}\Big{|}_{t=t_{0}}\frac{dt}{d\tau}\Big{|}_{\tau=\tau_{0}}\right)^{2}}=\Omega_{0,t}\left(\frac{d\tau}{dt}\Big{|}_{t=t_{0}}\right)^{2}.$

Using Equation (4) at present time (i.e., with $R_{0}=1$), we compute the
observed matter-density parameter $\Omega_{0,\tau}$ from the $t$ matter-
density parameter $\Omega_{0,t}$:

$\Omega_{0,\tau}=\frac{1}{\Omega_{0,t}}.$

A particularly interesting case concerns models with $\Omega_{0,t}>1$, i.e.
spatially closed cosmological models. Indeed, the spatial curvature $k$ is
given by $k=H_{0,t}^{2}(\Omega_{0,t}-1)/c^{2}$ with negative/null/positive
values of $k$ indicating spatially open/flat/closed universes. We illustrate
this interesting case in Figures 2 and 3 where we plot the evolution of the
scale factor $R$ of our toy universe as a function of our times for two
particular values of $\Omega_{0,t}>1$, hence $\Omega_{0,\tau}<1$.

In Figures 2a and 3a, this evolution as a function of $t$ is displayed and one
can directly notice the typical behaviour of closed models (an expansion phase
with a maximum value of $R$, followed by a contraction phase towards a Big
Crunch). In these same figures is also shown the correspondence (calculated
via Equation 4) between the constantly flowing time $t$ and its associated
true varying time $\tau$ (abscissa axes of Figures 2a and 3a). Indeed, one can
notice that a universe like the one we are currently dealing with (that is to
say, with $\Omega_{0,t}>1$) will have its $\tau(t)$ continuously slowing down
(as displayed in Figures 4 and 5), while the opposite behaviour will obviously
be observed in a universe with $\Omega_{0,t}<1$.

If the $\tau$ time is wrongly considered as having a constant flow (as it is
the case if we try to represent an universe with the usual cosmological
models), the $R(\tau)$ curve presents an apparent acceleration, as is shown in
Figures 2b and 3b. This is a similar phenomenon to the transformation of a
logarithmic axis of a chart to a linear one, a straight line in one
representation of this axis being an exponential (accelerating) curve in the
other one.

a

b

Figure 2: Evolution of the scale factor $R$ of a universe as a function of the
times $t$ and/or $\tau$ for our model (solid blue line; we choose to take
$\Omega_{0,\tau}=0.30$) and for a flat lambda cold dark matter ($\Lambda$CDM)
model (dashed green line; we fix its matter density parameter at
$\Omega_{0}=0.30$ to be consistent with the most recent observations of type
Ia supernovae Betoule et al. (2014) or of the Planck telescope Planck
Collaboration et al. (2016)). (a) When depicted as a function of the
constantly flowing coordinate time $t$ or as a function of the varying
cosmological time $\tau$, our model shows the usual behaviour of a spatially
closed universe; (b) When $\tau$ is wrongly interpreted as a constantly
flowing time, this universe seems to accelerate. One will notice that the
expansion of the universe reaches a maximum value of $R\approx 1.4$,
corresponding to the maximum value of the scale factor of a closed universe
(i.e., the maximum of the blue curve in Figure 2a). It does not continue to
expand indefinitely, as in the case of a $\Lambda$CDM model. With
$\Omega_{0,\tau}=0.30$ and a Hubble constant fixed at $H_{0,\tau}=70$
km/s/Mpc, our model predicts an age of this universe (as measured in
cosmological proper time units) of $12.81$ billion years.

a

b

Figure 3: Same as in Figures 2a and 2b for a model with
$\Omega_{0,\tau}=0.05$. With this value of $\Omega_{0,\tau}$ and a Hubble
constant fixed at $H_{0,\tau}=70$ km/s/Mpc, our model predicts an age of this
universe (as measured in cosmological proper time units) of $26.06$ billion
years. The flat $\Lambda$CDM model has the same matter density parameter as
previously. Figure 4: Variation of the cosmological time flow $d\tau/dt$ as a
function of times $t$ and $\tau$ for a matter-dominated universe with
$\Omega_{0,\tau}=0.30$. The cosmological time $\tau$ continuously slows down
in the present matter-dominated era. The units of $\tau$ and $t$ are set equal
at the beginning of the matter era when the universe was essentially flat, and
$\tau=0$ corresponds to the present epoch. Figure 5:  Same as in Figure 4 for
a model with $\Omega_{0,\tau}=0.05$.

Consequently, the main point of this article is that in our varying vision of
time with the hypothesis that sets cosmological time proportional to the
entropy of the causally connected region, we can produce a universe that seems
to accelerate while being in reality spatially closed. This is an interesting
result, as in usual Friedmann–Lemaître cosmological models, a universe with
such an accelerating behaviour is interpreted as open and thus infinite.
Hence, in our interpretation of time, a universe appearing in acceleration can
in fact be closed and finite, saving ourselves from philosophical problems
related to an infinite universe (i.e., containing an infinite amount of
matter).

Indeed, there is no question about the usefulness of the concept of infinity
in mathematics. Positive integer numbers can be summed up without any limit,
reaching towards infinitely large numbers. Conversely, any positive real
number can be divided by two, for example, without any limit, heading towards
infinitely small numbers. However, this is not the case in the physical world.
Consider some macroscopic quantity of water. You can always divide it in half,
but you cannot do so an arbitrarily large number of times, because at some
point you reach the atomic (or nuclear, or quark) level and you cannot divide
any more.

The only object in the physical world that is postulated to be spatially
infinite is the universe. Such a universe with—on average—a uniform density of
matter, would mean an infinite number of atoms, stars, galaxies, etc. It would
imply that this infinite amount of matter would have appeared during the Big
Bang. In our opinion, this makes little physical sense. Indeed, in a rigorous
mathematical approach, infinity is immeasurable. A variable can tend towards
it, but anything measurable, like a quantity of matter, cannot be equal to it.
Yet, within the scope of our toy model, the apparent openness and infinity of
the universe is simply an artefact emerging from the variable time flow,
whereas its real nature corresponds to a geometrically closed universe.

##  5 Conclusions

As already stated in the introduction, our goal with this paper is only to
design a simple implementation of our hypothesis of variable time in order to
determine its potential consequences on the content and evolution of the
universe.

From the basic assumption that the proper time of fundamental observers is
proportional to the entropy of the region of the universe causally connected
to them, we obtain cosmological toy models that present accelerating expansion
without recourse to dark energy. They also display positive geometric
curvatures, indicating finite and spatially closed universes. We thus avoid
the philosophical problem of an infinite universe, containing an infinite
amount of matter.

We nevertheless want to stress that—as pointed out earlier—other mathematical
dependences between time and entropy or any other parameter characterising the
dynamical state of the universe might be proposed or even better derived.

Obviously, the next step for us is now to compare our model predictions to the
various cosmological observations available. However, to put these tests into
practice, we have to develop a complete and mathematically thorough cosmology.
Our aim is to address this in forthcoming papers.

With a correct metric in $\tau$, we will be able to calculate distances and in
particular luminosity distances in order to test our hypothesis on the
observed Hubble diagram built with standard candles such as type Ia
supernovae. A priori, since our model shows an accelerating evolution, we can
expect a similar behaviour as the one from a usual Friedmann–Lemaître model
with dark energy. Moreover, the effects of the slowing down of proper time on
the rotation curves of spiral galaxies and on the velocity dispersions of
galaxies in clusters have to be investigated. Qualitatively, one would predict
an apparent acceleration—similar to the one observed for the scale factor
$R$—in these systems, which might explain the larger than expected observed
velocities. Of course, very careful calculations will be necessary to confirm
this tendency. Finally, cosmological nucleosynthesis and CMB anisotropies will
provide further tests of such a model.

###### Acknowledgements.

Judith Biernaux acknowledges the support of the F.R.I.A. fund (Fonds pour la
formation à la Recherche dans l’Industrie et dans l’Agriculture) of the F.R.S.
- F.N.R.S. (Fonds de la Recherche Scientifique). Clémentine Hauret and Pierre
Magain contributed equally to this work. Judith Biernaux contributed in
discussions and manuscript improvement. The authors declare no conflict of
interest. The founding sponsors had no role in the design of the study; in the
collection, analyses, or interpretation of data; in the writing of the
manuscript, and in the decision to publish the results.

## References

  * Einstein (1916) Einstein, A.  Die Grundlage der allgemeinen Relativitätstheorie.  Ann. Phys. 1916, 354, 769–822.
  * Milgrom (1983) Milgrom, M.  A modification of the Newtonian dynamics as a possible alternative to the hidden mass hypothesis.  Astrophys. J. 1983, 270, 365–370.
  * Page (1994) Page, D.N.  Clock time and entropy.  In _Physical Origins of Time Asymmetry_ ; Halliwell, J.J., Pérez-Mercader, J., Zurek, W.H., Eds.; Cambridge University Press: Cambridge, UK, 1994; pp. 287–298.
  * Verlinde (2016)

Verlinde, E.P. Emergent Gravity and the Dark Universe. SciPost Phys. 2017,
doi:10.21468/SciPostPhys.2.3.016.

  * Pavón and Radicella (2014) Pavón, D.; Radicella, N.  Why We Need Dark Energy.  In _Accelerated Cosmic Expansion_ ; Moreno González, C., Madriz Aguilar, J.E., Reyes Barrera, L.M., Eds.; Springer: Berlin, Germany, 2014; Astrophysics and Space Science Proceedings Volume 38, p. 143.
  * Pandey (2017) Pandey, B.  Does Information Entropy Play a Role in the Expansion and Acceleration of the Universe?  arXiv 2017, arXiv:1705.08945.
  * Hobson et al. (2006) Hobson, M.P.; Efstathiou, G.P.; Lasenby, A.N.  General Relativity; Cambridge University Press: Cambridge, UK, 2006.
  * Dicke (1957) Dicke, R.H.  Gravitation without a Principle of Equivalence.  Rev. Mod. Phys. 1957, 29, 363–376.
  * Chiba (2011) Chiba, T.  The Constancy of the Constants of Nature: Updates.  Prog. Theor. Phys. 2011, 126, 993–1019.
  * Egan and Lineweaver (2010) Egan, C.A.; Lineweaver, C.H.  A Larger Estimate of the Entropy of the Universe.  Astrophys. J. 2010, 710, 1825–1834.
  * Bekenstein (1973) Bekenstein, J.D.  Black Holes and Entropy.  Phys. Rev. D 1973, 7, 2333–2346.
  * Hawking (1976) Hawking, S.W.  Black holes and thermodynamics.  Phys. Rev. D 1976, 13, 191–197.
  * Betoule et al. (2014) Betoule, M.; Kessler, R.; Guy, J.; Mosher, J.; Hardin, D.; Biswas, R.; Astier, P.; El-Hage, P.; Konig, M.; Kuhlmann, S.; et al.  Improved cosmological constraints from a joint analysis of the SDSS-II and SNLS supernova samples.  Astron. Astrophys. 2014, 568, A22.
  * Planck Collaboration et al. (2016) Planck Collaboration.; Ade, P.A.R.; Aghanim, N.; Arnaud, M.; Ashdown, M.; Aumont, J.; Baccigalupi, C.; Banday, A.J.; Barreiro, R.B.; Bartlett, J.G.; et al. Planck 2015 results. XIII. Cosmological parameters.  Astron. Astrophys. 2016, 594, A13.


# MPC-based Imitation Learning for Safe and Human-like Autonomous Driving

Flavia Sofia Acerbo  Jan Swevers  Tinne Tuytelaars  Tong Duy Son

###### Abstract

To ensure user acceptance of autonomous vehicles (AVs), control systems are
being developed to mimic human drivers from demonstrations of desired driving
behaviors. Imitation learning (IL) algorithms serve this purpose, but struggle
to provide safety guarantees on the resulting closed-loop system trajectories.
On the other hand, Model Predictive Control (MPC) can handle nonlinear systems
with safety constraints, but realizing human-like driving with it requires
extensive domain knowledge. This work suggests the use of a seamless
combination of the two techniques to learn safe AV controllers from
demonstrations of desired driving behaviors, by using MPC as a differentiable
control layer within a hierarchical IL policy. With this strategy, IL is
performed in closed-loop and end-to-end, through parameters in the MPC cost,
model or constraints. Experimental results of this methodology are analyzed
for the design of a lane keeping control system, learned via behavioral
cloning from observations (BCO), given human demonstrations on a fixed-base
driving simulator.

Safe Imitation Learning, Model Predictive Control

##  1 Introduction

Currently, the design of autonomous vehicles (AVs) is primarly focused on
safety. However, to ensure a broad use by the general public, AVs should also
be comfortable and enjoyable. Comfort in AVs is a complex and multifaceted
entity, concerning attributes such as motion sickness, perceived safety,
understandability, trust and many others (Elbanhawi2015). Hence, comfort
optimization covers different areas of AVs research, such as personalized
driving controllers, novel cabin and seat design, active integrated chassis
control or motion sickness minimization. In this work, we consider _human-like
driving_ , a solution to improve AVs comfort which entails an implicit
personalization of AVs behavior inspired by human driving behaviors
(Bellem2016; Kalabic2019). This can be realized in a model-based or in a data-
driven approach; intensive research has been done on human driving models,
with sensory, cognitive and muscular level of detail (Nash2019). Though
accurate, these models are usually very task-specific and their identification
for different tasks is expensive (Kolekar2020). Therefore, only a relatively
small part of research on human-like AVs is implementing human driving models
in the controllers, e.g. (Lazcano2021). For higher level of autonomy, research
is primarly focusing on data-driven approaches, such as imitation learning
(IL), which can obtain policies mimicking driving demonstrations experienced
or labeled by the user as adequate. In this domain, various solutions from
both academia and industry (Hawke2020) show great potential on learning
policies end-to-end, i.e. from sensory data to control actions or states.

Imitation learning solutions for human-like driving usually involve deep
learning (DL) in their architecture, and this limits the verifiability of the
safety and stability properties of the closed-loop system. To provide
guarantees, many works have proposed a hierarchical framework, where the DL
components are combined with safety filters, frequently model-based
controllers. In (Pan2017; Chen2019; Pulver2021; Acerbo2020) such DL components
represent trajectories fed to tracking controllers, but learning is done
without directly considering their effect, hence disrupting the end-to-end
learning. In the majority of these works, model predictive control (MPC) is
used, due to its ability to explicitly handle hard constraints. Moreover, MPC
is a generic controller, which can be designed to have parameters in its cost
function, model or constraints. In this regard, there have been recent
advances on the use of Model Predictive Control (MPC), as a differentiable
policy in the context of IL (Amos2018) and reinforcement learning (RL)
(Gros2020).

This work is a preliminary study to extend existing hierarchical structures of
IL for AVs to differentiable MPC, which allows for end-to-end learning while
guaranteeing safety and stability of the controlled system. Specifically, MPC
is used as a reactive control layer and its parameters are provided by other
DL indirect controllers, which map from sensory data to the parameters. The
paper is structured as follows: first, we provide an introduction on related
works concerning IL for AVs and previous studies on MPC with IL and RL. Then,
we present the generic formulation of the proposed approach, namely MPC-IL,
which leverages the MPC differentiability and the hierarchical decomposition.
After that, we detail the approach under the behavioral cloning from
observations algorithm (MPC-BCO) and finally we provide preliminary
experimental results of MPC-BCO applied to the design of a human-like lane
keeping controller from human demonstrations.

##  2 Background and Related Work

###  2.1 Imitation Learning for Autonomous Driving

Let us consider a fully observable Markov Decision Process characterized by a
set of continuous states $s\in\mathcal{S}\subset\mathbb{R}^{n_{s}}$, which can
be controlled with a set of continuous actions
$a\in\mathcal{A}\subset\mathbb{R}^{n_{a}}$ and with a state transition
probability distribution $s_{t+1}\sim P(\cdot|s_{t},a_{t})$. An agent can
control the system with a policy $\pi(a_{t}|s_{t})$. We denote the _state
occupancy measure_ $\rho_{\pi}(s)$ induced by a policy $\pi$ as the density of
occurrence of states, while following that policy, over an infinite time
horizon, discounted by the factor $\gamma$:
$\rho_{\pi}^{s}=\sum_{t=0}^{\infty}\gamma^{t}P(s|\pi).$ Given an expert policy
$\pi_{E}$ to be imitated, IL tries to find a policy $\pi$ minimizing the
occupancy measure distance formulated as:

$\min_{\pi}\operatorname{\mathbb{E}}_{s\sim\rho_{\pi}^{s}}[\mathcal{L}(\pi_{E},\pi)]=\mathcal{L}(\rho_{\pi_{E}}^{s},\rho_{\pi}^{s})\;.$
(1)

As demonstrated by (Ghasemipour2019), all IL algorithms can be read under this
formulation. However, in the context of AVs, behavioral cloning (BC) is the
algorithm with the most promising history of success (Bansal2018; Scheel2021).
Standard BC provides a straightforward solution to the IL problem, by learning
the policy in a supervised way through direct maximum likelihood, i.e. by
minimizing the distance between the action distributions under the expert
state occupancy measure, as:
$\min_{\pi}\operatorname{\mathbb{E}}_{s\sim\rho_{\pi_{E}}^{s}}[\mathcal{L}(\pi_{E},\pi)]$.
This suffers from the covariate shift problem, since the policy $\pi$
generates a different occupancy measure when run in closed-loop, and then
becomes unpredictable in its own induced distribution of states. Among
different solutions to mitigate this problem, it has been shown by
(Ghasemipour2019), that learning the marginal state distribution of the expert
in addition to the action distribution, can heavily influence the results.
Hence, behavioral cloning from observations (BCO) (Torabi2018) aims at
minimizing the Kullback–Leibler (KL) divergence on the state occupancy
measures as: $\min_{\pi}\text{KL}(\rho_{\pi_{E}}^{s}||\rho_{\pi}^{s}).$
Nevertheless, BCO is more expensive than BC, since it requires to run
simulations in the learning loop and needs solutions to estimate the policy
gradient from the states.

###  2.2 Model Predictive Control and Reinforcement Learning

Model predictive control (MPC) is becoming a popular technique in AV research
and industry. It is an optimal controller with receding horizon denoted in its
most general form as:

$\displaystyle a=\operatorname*{arg\,min}_{x,u}$
$\displaystyle\sum_{k=0}^{N-1}l(\theta,x_{k},u_{k})+l_{N}(\theta,x_{k},u_{k})$
(2) $\displaystyle\operatorname{s.t.}$ $\displaystyle x_{0}=\hat{x}$
$\displaystyle x_{k+1}=f(x_{k},u_{k},\theta)$ $\displaystyle
h(x_{k},u_{k},\theta)\leq 0$ $\displaystyle a=u_{0}\;.$

At each time step, the current state of the system is estimated as $\hat{x}$.
Then, MPC minimizes a cost function across a prediction horizon $N$, where the
evolution of the model is simulated with $f$ and where its states and controls
should not violate the constraints specified by $h$. Usually, when talking
about learning-based MPC, we refer to system identification of the underlying
model $f$, given data from the controlled system. However, it is possible to
parametrize also its cost function and constraints. In this regard, MPC was
recently proposed as a solution for safety and sample efficiency issues in IL
and RL. (Gros2020) showed that using a highly parametric MPC as a
differentiable function approximator for RL results in safe and stable RL
policies and allows MPC to optimize its closed-loop performance, even when its
underlying model $f$ is wrong and system identification is not performed.
(Amos2018) have benchmarked the use of differentiable MPC for classical IL/RL
problems showing the advantages in terms of sample efficiency and learning
flexibility with respect to generic IL and system identification. However,
these studies did not consider learning MPC parameters in the form of goals,
references or other generic complex behavior to be imitated and therefore
considered MPC as the entire policy, without combining it with other learning
components.

##  3 MPC-based Imitation Learning

This section presents the formulation and generic algorithmic framework of our
MPC-IL method. Here, MPC is used as the differentiable final control layer of
a hierarchical policy. This means that MPC parameters are given by preceding
indirect controllers, which are learnt with end-to-end imitation learning.
Moreover, a specific IL algorithm, namely behavioral cloning from observations
(BCO), is detailed as an example of the MPC-IL approach.

###  3.1 Problem Formulation

As defined in Section 2.1 we consider IL as the minimization of the occupancy
measure distance, according to a certain metric $\mathcal{L}$. The system
under study is controlled by the human with an (unknown) stochastic policy
$\pi_{H}$ as $a_{t}\sim\pi_{H}(\cdot|s_{t})$, while the mimicking controller
controls it with a deterministic policy parametrized by $\theta$ as
$a_{t}=\pi_{\theta}(s_{t})$. We now formulate this problem in the context
where the controller part of the policy is represented by a MPC, optimizing
its control action $a$ according to the state $s$ of the system and the
parameters $\theta$ that are part of the objective function, model and
constraints:

$\displaystyle\min_{\theta}$
$\displaystyle\operatorname{\mathbb{E}}_{s\sim\rho_{\pi_{\theta}}^{s}}[\mathcal{L}(\pi_{H},\pi_{\theta})]=\mathcal{L}(\rho_{\pi_{H}}^{s},\rho_{\pi_{\theta}}^{s})$
(3) with $\displaystyle\pi_{\theta}\in\begin{aligned}
\operatorname*{arg\,min}_{x,u}\quad&\sum_{k=0}^{N-1}l(\theta,x_{k},u_{k})+l_{N}(\theta,x_{k},u_{k})\\\
\operatorname{s.t.}\quad&x_{0}=s\\\ &x_{k+1}=f(x_{k},u_{k},\theta)\\\
&h(x_{k},u_{k},\theta)\leq 0\\\ &a=u_{0}\;.\end{aligned}$

Equation (3) describes a _bilevel optimization problem_ , where the upper-
level problem is the imitation of policy $\pi$ with respect to the human
policy $\pi_{H}$ and the lower-level problem in the MPC optimization, given a
state $s$. Among gradient-based solution techniques it is common to replace
the lower level optimization problems with their Karush-Kuhn-Tucker (KKT)
conditions as constraints (Hatz2012). However, directly solving a constrained
optimization problem is not trivial in the context of imitation and
reinforcement learning. Moreover, it requires second order information (i.e.
Hessian matrices) and that does not scale well when neural networks and large
amount of data are involved. Conversely, the adopted solution is based on
first-order information only (Amos2018). The core idea is to compute the
gradient of the parametrized solution of the MPC with respect to the
optimization variables of the upper-level imitiation problem, and then solve
it as an unconstrained one with gradient descent steps of size $\alpha$ as:

$\displaystyle\theta$
$\displaystyle\leftarrow\theta-\alpha\left[\frac{\partial{\mathcal{L}}}{\partial{\theta}}\right]^{T}$
(4)
$\displaystyle\left[\frac{\partial{\mathcal{L}}}{\partial{\theta}}\right]^{T}$
$\displaystyle=\left[\frac{\partial{\pi_{\theta}}}{\partial{\theta}}\right]^{T}\left[\frac{\partial{\mathcal{L}}}{\partial{\rho_{\pi_{\theta}}^{s}}}\frac{\partial{\rho_{\pi_{\theta}}^{s}}}{\partial{\pi_{\theta}}}\right]^{T}.$

The computation of
$\left[\frac{\partial{\mathcal{L}}}{\partial{\theta}}\right]^{T}$ requires:

  * •

$\left[\frac{\partial{\mathcal{L}}}{\partial{\rho_{\pi_{\theta}}^{s}}}\frac{\partial{\rho_{\pi_{\theta}}^{s}}}{\partial{\pi_{\theta}}}\right]^{T}$,
which depends on the adopted IL algorithm, defining the distance $\mathcal{L}$
and how to approximate the occupancy measure $\rho$.

  * •

$\left[\frac{\partial{\pi_{\theta}}}{\partial{\theta}}\right]^{T}$, i.e. the
derivative of the MPC problem at its (locally) optimal solution. In the next
section we describe how to efficiently compute it via automatic
differentiation.

###  3.2 Backpropagation through the MPC Problem

We now detail how to compute the gradient of a policy $\pi_{\theta}$ with a
MPC control layer. Let us introduce the Lagrangian associated to the MPC
problem of Equation (3):

$\displaystyle\mathscr{L}(x,u,\mu,\lambda,\theta)=$
$\displaystyle\sum_{k=0}^{N-1}l(\theta,x_{k},u_{k})+l_{N}(\theta,x_{k},u_{k})$
(5) $\displaystyle+$
$\displaystyle\mu^{T}h(x,u,\theta)+\lambda_{0}^{T}(x_{0}-s)$ $\displaystyle+$
$\displaystyle\sum_{k=0}^{N-1}\lambda^{T}_{k+1}\left[f(x_{k},u_{k},\theta)-x_{k+1}\right],$

where $\mu,\lambda$ are the multipliers associated to the dual solution. For
simplicity of notation, we denote as $z=\left[x,u,\lambda,\mu\right]$ the
vector of the primal and dual variables associated with the optimal solution,
each of them with $N$ components e.g., $x_{0},x_{1},\dots,x_{N}$. At a local
optimum $z^{*}$ and if the problem satisfies the linear independence
constraint qualification (LICQ) and the second-order sufficient conditions
(SOCS) (Gros2020), then the following Karush–Kuhn–Tucker (KKT) conditions
hold:

$\displaystyle\nabla_{x,u}\mathscr{L}(z^{*},\theta)=0$ (6)
$\displaystyle\begin{aligned} &x^{*}_{0}=s,\\\
&x^{*}_{k+1}=f(x^{*}_{k},u^{*}_{k},\theta)\;\text{for}\>k=0,...,N-1,\\\
&h(z^{*},\theta)\leq 0\end{aligned}$ (7) $\displaystyle\mu^{*}\geq 0$ (8)
$\displaystyle\mu^{*T}h(z^{*},\theta)=0.$ (9)

If the active set of the inequalities constraints at $z^{*}$ is known, then we
can write the previous equations as an implicit function
$F(z^{*},\theta)\leftrightarrow z^{*}=\pi_{\theta}$ as:

$F(z^{*},\theta)=\begin{bmatrix}\nabla_{x,u}\mathscr{L}(z^{*},\theta)\\\
I_{h}h(z^{*},\theta)-I_{h}^{0}\mu^{*}\\\
\mathbf{f}(z^{*},\theta)\end{bmatrix}=0\;,$ (10)

where $I_{h}$ and $I_{h}^{0}$ are diagonal matrices denoting active and
inactive inequality constraints, respectively. $\mathbf{f}$ represents the
matrix form of the $N$ equality constraints from Equation (LABEL:primalfeas).
According to the implicit function theorem, we can write the Jacobian of
$\pi_{\theta}$ as:

$\frac{\partial\pi_{\theta}}{\partial\theta}=-\frac{\partial F}{\partial
z}^{-1}\frac{\partial F}{\partial\theta}\;.$ (11)

The full $\frac{\partial\pi_{\theta}}{\partial\theta}$ may be very expensive
to compute. However, we are interested in the Jacobian-times-vector product
$\left[\frac{\partial{\pi_{\theta}}}{\partial{\theta}}\right]^{T}\bar{z}$.
This is also known as adjoint sensitivity and can be computed efficiently
within automatic differentiation (AD) engines at the cost of a linear system
solve and a reverse mode sweep (Andersson2018) as:
$\frac{\partial\pi_{\theta}}{\partial\theta}\bar{z}=-\frac{\partial
F}{\partial\theta}^{T}\frac{\partial F}{\partial z}^{-T}\bar{z}\;.$

######  Remark 3.1.

In order to compute the gradient we must ensure that $\frac{\partial
F}{\partial z}$ is invertible. This means that, alongside the conditions of
LICQ and SOCS, it is required that the optimal solution $z^{*}$ where we
compute the gradient is at least locally unique. This is not guaranteed in the
case of nonlinear MPC and therefore we should impose some sort of
regularization on the primal variables.

###  3.3 Hierarchical Decomposition

In the available literature the parameters $\theta$ refer to physical
properties in the model or to the weight matrices in a quadratic regulator
cost, and are therefore static (Amos2018). Nevertheless, to learn more complex
and strategic behaviors, as it is the case for end-to-end AV control from
demonstrations, we suggest the need to combine MPC control with other
differentiable layers, as graphically represented in Figure 1. Let us consider
some context information about the system at time $t$ in the form of a latent
variable $\chi_{t}$, inferrable from the system current state as
$\chi_{t}=Q(s_{t})$; for example, $\chi_{t}$ may be a high-dimensional input
(e.g. camera images), from which it would be convenient to learn end-to-end.
For this purpose, we suggest to learn a function $\theta=g(\chi_{t})$ that
maps the latent variable $\chi_{t}$ to the MPC parameters $\theta$. For
example, $\theta$ can represent the goal state of the system or a trajectory
to follow, which changes depending on the current situation at time $t$; We
denote this approach as _hierarchical decomposition_ ; the MPC is responsible
for the reactive control behavior, i.e. the mapping $s_{t}\rightarrow a_{t}$,
while the preceding layers learn subtasks that indirectly influence the
closed-loop behavior of the system. Such layers are said to be _indirect
controllers_. Then, the policy is written as:
$a_{t}=\pi_{\theta(\chi_{t})}(s_{t})$.

Figure 1: Hierarchical decomposition of the policy with a MPC control layer:
the MPC provides the mapping $s_{t}\rightarrow a_{t}$, based on a constrained
optimization. The indirect controller outputs the MPC parameters $\theta$
based on the latent variable $\chi_{t}$. The dashed line show the flow of the
gradient through the whole policy, possible thanks to the MPC
differentiability.

###  3.4 MPC-based Behavioral Cloning from Observations (MPC-BCO)

It is now detailed how we use the MPC-IL method to learn a deterministic
hierarchical policy $\pi_{\theta}$ with a MPC control layer using closed-loop
policy learning via behavioral cloning from observations (BCO) and a
differentiable simulator.

#### Algorithm

We consider rollout state trajectories of length $T$, induced by the policy
$\pi_{\theta}$: $\tau_{\pi_{\theta}}=s_{0},s_{1},\dots,s_{T}$, and state
trajectories sampled from the human demonstrations:
$\tau_{\pi_{H}}=s_{0}^{*},s_{1}^{*},\dots,s_{T}^{*}$. We frame the problem as
the minimization of the L2 state error
$L(s_{t},s_{t}^{*})=||s_{t}-s_{t}^{*}||_{2}^{2}$ between the human and policy
on sequences of induced states, when starting from the same initial state
$s_{0}^{*}$. These sequences are acquired through a simulation model $F$,
providing the state transitions $s_{t+1}=F(s_{t},a_{t})$. Let us assume that
the first derivative of $F$ around a certain $s_{t},a_{t}$ can be computed
(this assumption is clarified in the last paragraph of this section).
Therefore, we suggest computing $\nabla_{\theta}L(s_{t})$ with backpropagation
through time (BPTT) as:

$\displaystyle\nabla_{\theta}L_{|s=s_{t}}$
$\displaystyle=\nabla_{s}L_{|s=s_{t}}\left(\right.\nabla_{a}F\nabla_{\theta}a_{|a=a_{t-1}}+$
(12)
$\displaystyle+\nabla_{a}F\nabla_{s}a\nabla_{\theta}s_{|a=a_{t-1},s=s_{t-1}}+$
$\displaystyle+\nabla_{s}F\nabla_{\theta}s_{|s=s_{t-1}}\left.\right)\;.$

Hence, with BPTT future state errors are accounted by the earlier actions
taken by the policy.

We define
$J(\theta)=\operatorname{\mathbb{E}}_{s_{t}\sim\rho_{\pi_{\theta}}^{s}}\operatorname{\mathbb{E}}_{s_{t}^{*}\sim\rho_{\pi_{H}}^{s}}\left[\sum_{t=0}^{T}L(s_{t},s_{t}^{*})\right]$
the sum of the L2 pose errors along a trajectory. Then, to update the policy,
we compute the derivatives of $J$ over $(s_{t},a_{t},s_{t+1})$ transitions
with BPTT, leveraging on the differentiability of the policy and of the
simulator. From (Heess2015), the gradient is computed by the following
recursive formulas:

$\displaystyle\frac{\partial J_{t}}{\partial s_{t}}$
$\displaystyle=\frac{\partial L}{\partial s_{t}}+\frac{\partial
J_{t+1}}{\partial s_{t+1}}\left(\frac{\partial F}{\partial
s_{t}}+\frac{\partial F}{\partial a_{t}}\frac{\partial\pi_{\theta}}{\partial
s_{t}}_{|\chi_{t}}\right)$ (13) $\displaystyle\frac{\partial
J_{t}}{\partial\theta}$ $\displaystyle=\frac{\partial J_{t+1}}{\partial
s_{t+1}}\frac{\partial F}{\partial
a_{t}}\frac{\partial\pi_{\theta}}{\partial\theta}_{|\chi_{t}}+\frac{\partial
J_{t+1}}{\partial\theta}\;.$

The final gradient $J(\theta)$ is computed by applying the equations from (13)
recursively, starting from $t=T$ up to $t=t_{s}$. We suggest to not
backpropagate until $t_{s}=0$ to avoid bias; since the initial state
$s_{0}^{*}$ belongs to the human state occupancy metric, there may be a
transitory part before the policy reaches a steady state corresponding to its
actual occupancy metric. The complete algorithm is described in Algorithm
LABEL:alg:mpcbco.


# FlexEdit: Flexible and Controllable Diffusion-based Object-centric Image
Editing

Trong-Tung Nguyen 11 Duc-Anh Nguyen 11 Anh Tran 11 Cuong Pham 1122