# State Dropout-Based Curriculum Reinforcement Learning for Self-Driving at
Unsignalized Intersections*

Shivesh Khaitan1 and John M. Dolan2 *This work was supported by the CMU Argo
AI Center for Autonomous Vehicle Research.1,2The authors are with the Robotics
Institute, Carnegie Mellon University, Pittsburgh, PA, the USA
<EMAIL_ADDRESS>
###### Abstract

Traversing intersections is a challenging problem for autonomous vehicles,
especially when the intersections do not have traffic control. Recently deep
reinforcement learning has received massive attention due to its success in
dealing with autonomous driving tasks. In this work, we address the problem of
traversing unsignalized intersections using a novel curriculum for deep
reinforcement learning. The proposed curriculum leads to: 1) A faster training
process for the reinforcement learning agent, and 2) Better performance
compared to an agent trained without curriculum. Our main contribution is two-
fold: 1) Presenting a unique curriculum for training deep reinforcement
learning agents, and 2) showing the application of the proposed curriculum for
the unsignalized intersection traversal task. The framework expects processed
observations of the surroundings from the perception system of the autonomous
vehicle. We test our method in the CommonRoad motion planning simulator on
T-intersections and four-way intersections.

##  I INTRODUCTION

Urban intersections pose a difficult driving challenge. Even for human
drivers, traversing an intersection requires undivided attention. The
situation gets much more complicated without traffic signals. Unsignalized
intersections require much more coordination among the incoming vehicles. Such
intersections have seen many fatal accidents. The US Federal Highway
Administration has reported that about 71.4% of intersection-related fatal
crashes occur at unsignalized intersections [1]. Autonomous vehicles can
reduce the chances of accidents by eliminating human error. However,
intersections are difficult for autonomous vehicles as well. This is
especially because predicting the intent of other incoming drivers from
different directions at intersections is in itself a complicated problem for
self-driving.

A popular choice for decision-making and planning at intersections for
autonomous vehicles has been rule-based methods [2]. These were especially
very popular for the DARPA Urban Challenge [3, 4]. Although they make the
system interpretable, tuning the parameters to avoid collision in every
situation is very hard. The rule-based methods also usually try to be overly
cautious, leading to suboptimal behaviors. Moreover, a rule-based system needs
to consider the trajectory generator’s ability to generate feasible
trajectories based on the goal or intent set by the decision-making module,
which might not be possible in all circumstances.

Reinforcement learning (RL) and deep reinforcement learning (DRL) have
recently gained a lot of attention in the literature for autonomous driving.
RL’s ability to deal with complex scenarios has been unmatched by rule-based
methods. It also does not require massive datasets as supervised learning does
and can replace individual software modules or be used as a complete end-to-
end solution for driving. Its abilities and performance have been demonstrated
in many previous works [5, 6]. However, complex tasks like autonomous driving
usually require long training periods to produce acceptable results. This is
because of the high-dimensional state space of the autonomous driving task.

To accelerate the training process, curriculum learning [7] has been proposed.
The basic idea in curriculum learning is to design the training process as a
set of graduated steps with increasing complexity of tasks. It draws
inspiration from the way humans learn by starting with easier tasks and
gradually increasing the complexity to become an expert. The idea of
curriculum learning has also been applied to reinforcement learning. For a
detailed review on the applications of curriculum for reinforcement learning,
see [8]. Apart from accelerating the training process, it has also been
proposed that using a curriculum can help find better local minima as compared
to training without curriculum [7].

Previously, most works have used hand-designed task-based curricula for
training, which rely on segregating the tasks at hand based on their
difficulty and training sequentially with increasing levels of difficulty.
Such manual segregation of tasks can be laborious and time-consuming. In this
work, we propose a unique automated curriculum for training that is easy to
design and is also more effective in dealing with unsignalized intersections.
We test our method using CommonRoad-RL [9] on T-intersections and four-way
intersections without traffic control and compare it with standard PPO and
rule-based TTC method. The main contributions of this work are:

  * •

An automated curriculum for training deep reinforcement learning agents.

  * •

An application of the proposed method to the task of traversing unsignalized
intersections.

The rest of this paper is organized as follows. Section II provides a review
of some important related work. Section III gives an introduction to the
intersection problem addressed in this work. Section IV contains necessary
background on key methodologies. Section V presents the experimental results.
The conclusions and the future work are in Section VI.

##  II RELATED WORK

This section summarizes relevant previous work categorized as follows: a) work
related to planning for autonomous vehicles focusing on reinforcement learning
and b) techniques to accelerating training.

###  II-A Planning for autonomous vehicles

In the existing literature, planning for autonomous vehicles has used several
methods, including rule-based methods, imitation learning, reinforcement
learning, and inverse reinforcement learning.

The most popular rule-based method is the time-to-collision (TTC) method [2].
This method relies on hand-crafted rules and parameters to control a vehicle
and avoid collisions. Trajectory optimization-based methods have also been
extensively used for motion planning. Model Predictive Control (MPC) [10, 11],
which is a receding horizon framework with repeated optimization, is a widely
used method. [12] proposed the use of a prediction framework combined with
constrained iterative LQR, which is a lightweight optimization method as
compared to MPC. However, designing the heuristics for optimization is
extremely laborious and often leads to suboptimal results, as they do not
adapt well to different driving environments.

Recently, learning-based methods have been quite successful for many robotics
and autonomous systems tasks. For self-driving, imitation learning leverages
expert driving data to train an autonomous driver, which can work in an end-
to-end framework by directly mapping sensor input to trajectory or control
commands. Behavior cloning [13] uses supervised learning on collected expert
data. However, it suffers from the problem of distribution shift between
training and test environments. This is tackled by online interactive data
collection methods [14, 15] for imitation learning, which require an online
demonstrator at training time. These generalize much better to varied
scenarios, but the requirement of an expert demonstrator itself makes training
challenging.

Deep reinforcement learning is a promising framework for complex self-driving
planning tasks. Recently, it has been extensively used for motion planning in
complex environments [16, 17]. However, certain complex driving scenarios like
unsignalized intersections still remain a challenge. [18] uses a hierarchical
framework with a high-level RL-based decision-making planner and a low-level
MPC controller. However, it does not consider cases when the ego-vehicle might
have to turn left or right at intersections instead of just going straight.
[19] deals with this uncertainty at intersections, but it focuses on
T-intersections only, which is a simpler scenario as compared to four-way
intersections. [20] used Deep Q-learning Network (DQN) for intersection
handling. However, DQNs can only be used for problems with a discrete action
space and hence cannot be used for continuous control commands. The Deep
Deterministic Policy Gradient (DDPG) method [21] works for continuous control
spaces. However, both DQN and DDPG require long training periods to converge.
In this work, we use Proximal Policy Optimization (PPO) [22], which is a
variant of the actor-critic family and uses an adaptive KL divergence penalty
to control the change of policy at each iteration. To accelerate training and
improve performance, we use the state dropout-based curriculum described in
section IV-B.

###  II-B Techniques for accelerating training periods

A major drawback of RL methods is long training periods and convergence to
sub-optimal policies. This usually happens because the dynamics of the RL
training environments are unknown to the agent, and it takes a sufficient
number of interactions with the environment before the agent can understand
the dynamics to improve its policy. Sparse rewards and partial observability
aggravate the problem. This prohibits the usage of RL in safety-critical
systems, as the systems might be required to enter unsafe states while the
agent trains. Some techniques to alleviate this problem have been proposed in
the literature.

Transfer learning (TL) is a technique which pretrains a network on a different
task before training on the target task. The network benefits by using the
skills acquired during pretraining. For a comprehensive literature review on
transfer learning for RL, see [23]. Apart from just having the advantage of
faster training time, using transfer learning also leads to better
generalization and better performance than an agent trained without using the
pretrained network.

Curriculum learning [24], another approach to expedite training, uses a
specially designed schedule for training. It trains on easier tasks to start
with and gradually increases the difficulty level. This draws inspiration from
how humans learn to do complex tasks by “starting small” [25]. The main idea
is that once the agent has learned to do a simple task well, it will take
fewer iterations to adapt to a harder task having the simpler task as a subset
because of the stability of the training process. Apart from just accelerating
the training process, it has also been shown that certain tasks have been
nearly impossible to train without following a curriculum [26]. [27] used
curriculum reinforcement learning for overtaking in an autonomous racing
scenario. However, hand segregating tasks for designing a curriculum might be
cumbersome, as deciding the difficulty level of a task might not be trivial.
[28] proposes an automatic curriculum generator that tries to alleviate the
problem.

In this work, we propose unique automated curricula for autonomous driving at
unsignalized intersections. The curricula are easier to train and reduce the
training time. For the current work, we assume perception to be solved so that
the framework accurately observes the processed road boundary and obstacle
information. However, the curricula can be extended to end-to-end frameworks
as well.

##  III PROBLEM DEFINITION

###  III-A Problem Statement

The problem at hand is generating control commands for an ego-vehicle to
traverse unsignalized intersections and reach a predefined goal state. We
consider two types of intersection scenarios, as shown in Fig. 1 and Fig. 2.
The ego-vehicle is depicted in green, target vehicles are in blue and the goal
region is in yellow. The dotted lines starting from target vehicles show their
future trajectories, which do not change in response to the ego-vehicle
behavior. This increases the difficulty level of the problem significantly.

####  III-A1 T-intersection

These scenarios are adopted from the CommonRoad benchmarks: ZAM_Tjunction
(Fig. 1).

Figure 1: T-intersection

####  III-A2 Four-way intersection

These are four-way intersection scenarios (Fig. 2) custom-generated using the
CommonRoad scenario designer tool [29], as the official CommonRoad benchmarks
lacked four-way intersections. The goal state for these scenarios can be in
the left, straight-ahead, or right lane. The ego-vehicle has to reach the goal
region within $\pm 11.45^{\circ}$ of the road lane orientation to successfully
complete a scenario. The limits on orientation ensure that the ego-vehicle
does not reach the goal with an arbitrary orientation, which can make it
harder for the vehicle to correct the orientation when moving further in the
goal lane. The scenario has to be completed before a time-out of 150 sec.

Figure 2: Four-way intersection

###  III-B Observation and Action space

The observation space is as defined in Table I. The observations can be
divided into four categories: 1) ego-vehicle state; 2) goal and reference
path-related observations; 3) surrounding vehicles-related observations; and
4) road network-related observations. The dynamics constraints are based on
the kinematic bicycle model (KS1) as described in [30]. The reference path
observation consists of waypoints from the path generated by A∗ search over
the lanelet network in Commonroad. The distance advancements for longitudinal
and lateral directions are calculated along this reference path. For the
surrounding vehicles, we include the current and future states (for 5 time-
steps with a time discretization of 0.1 sec) of each of the five vehicles in
the ego-vehicle frame. The information in future states is as described in
section IV-B.

The action space consists of acceleration and steering for the ego-vehicle
$A=\\{accel,steer\\}$. The controls are continuous.

TABLE I: Observation Space

| Description | Values
---|---|---
$v_{t}$ | Ego-vehicle absolute velocity | $\mathbb{R}$
$a_{t-1}$ | Ego-vehicle previous acceleration | $\mathbb{R}$
$\delta_{t-1}$ | Ego-vehicle previous steering | $\mathbb{R}$
$\theta_{t}$ | Ego-vehicle orientation | $\mathbb{R}$
$r$ | Reference path waypoints ($x,y,\theta$) in ego-vehicle frame | $\mathbb{R}^{15}$
$d\theta_{t}$ | Ego-vehicle orientation deviation from reference path | $\mathbb{R}$
$d_{long}$ | Longitudinal distance advancement towards goal | $\mathbb{R}$
$d_{lat}$ | Latitudinal distance advancement towards goal | $\mathbb{R}$
$t_{out}$ | Time remaining before time-out | $\mathbb{R}$
$c$ | Lane curvature | $\mathbb{R}$
$l_{o}$ | Ego-vehicle offset from lane centerline | $\mathbb{R}$
$r_{l}$ | Ego-vehicle distance from left road boundary | $\mathbb{R}$
$r_{r}$ | Ego-vehicle distance from right road boundary | $\mathbb{R}$
$l_{l}$ | Ego-vehicle distance from left lane boundary | $\mathbb{R}$
$l_{r}$ | Ego-vehicle distance from right lane boundary | $\mathbb{R}$
$o$ | Target vehicles’ current and future states∗($x,y,v,\theta$) | $\mathbb{R}^{100}$

  * •

$*$ The information in future states depends on the curriculum

##  IV METHODOLOGY

This section introduces the reward structure for the PPO agent, and the
proposed curricula for learning.

###  IV-A Reward Structure

The components of the reward function are as follows:

  * •

A positive reward for distance advancement along the reference path as
described in section III-B. The reward is calculated as
$\rho_{1}d_{long}+\rho_{2}d_{lat}$ where $\rho_{1}$ and $\rho_{2}$ are
positive constants.

  * •

A constant negative reward $\sigma_{1}$ for collision with obstacles, going
off-road, and violating dynamics constraints.

  * •

A constant negative reward $\sigma_{2}$ for time-out.

  * •

A constant positive reward $\phi$ for reaching the goal within the allowed
orientation error.

An episode is terminated when any one of the following conditions is met: ego-
vehicle reaches the goal within allowed orientation error, ego-vehicle goes
off-road, ego-vehicle collides with another obstacle, applied controls violate
dynamics constraints or time-out.

###  IV-B State dropout-based curriculum for PPO

Training and getting an acceptable performance with PPO or other existing RL
algorithms in the described intersection problem is tough. Often the RL agents
settle for a suboptimal policy. A prominent reason for this is that once an RL
agent adopts a suboptimal policy, it further keeps exploiting the suboptimal
policy and takes a long time to escape local minima and generate better
actions after that. Moreover, unsignalized intersections also require that a
driver is able to determine the intent of multiple other incoming vehicles.
Training a simple PPO agent in this scenario leads to a suboptimal behavior
even with long training periods, as implicitly learning to predict the
behavior of the vehicles while simultaneously learning to control the car is
not trivial for the PPO agent.

Our state dropout-based curriculum overcomes the challenge by “starting
small”. An easy scenario for an RL agent in intersections is when the
intentions of the surrounding drivers are known in advance. In such scenarios,
the RL agent can learn an optimal policy for traversing. Thus we incorporate
the future state information (for $N$ time-steps) about the surrounding
vehicles in the observation space for PPO. We propose two different curricula
to deal with the additional privileged observation:

####  IV-B1 Curriculum 1

In this method we train the agent in $N+1$ phases starting with phase 0. We
begin training with the future state information for $N$ time-steps. After
phase 0, we drop the $N^{th}$ future state information and continue training.
In the subsequent phases we keep on dropping more future state information and
in the final phase fine-tune the agent without any future state information.
Algorithm 1 describes the $STEP$ function for this method.

Algorithm 1 Step function for curriculum 1

1:procedure step(action, phase)

2: Apply acceleration and steering from action

3: obs = Observe()

4: reward, done = Reward(obs)

5: for $i\leftarrow(N-phase+1)$ to $N$ do

6: Drop $i^{th}$ future prediction from obs

7: end for

8: return obs, reward, done

9:end procedure

####  IV-B2 Curriculum 2

In this method, instead of dropping the future state information sequentially
after each phase, we augment the action space of the agent with an additional
action such that the augmented action space is $A=\\{accel,steer,pred\\}$.
Here $pred$ determines which future state information for the surrounding
vehicles will be dropped from the observation in the next step. The reward
structure as defined in section IV-A is also augmented to add constant
positive rewards $\psi_{i}$ for choosing to drop the future states from the
$i^{th}$ to the $N^{th}$ time-step. The decision is based on parameters
$\kappa_{i}$. Here, $\forall i\in[2,N]$

$\psi_{i}<\psi_{i-1}$ $None$ $\kappa_{i}<\kappa_{i-1}$ $None$

This incentivizes the agent to drop more future states. Algorithm 2 defines
the modified $STEP$ function for this method. Thus we let the network choose
when to drop the future states. At the beginning of training, when the agent
has not learned to predict the behavior of the vehicles, it chooses to use the
future information. As the training progresses and the agent learns to control
the ego-vehicle, $\psi_{i}$ incentivizes the agent to learn to predict the
driving intention as well in order to get higher rewards.

An added advantage of this method is that the $pred$ value can be used at test
time to determine whether the agent understands the behavior of the other
vehicles or not. This can further be used as a confidence score for the agent
commands.

Algorithm 2 Step function for curriculum 2

1:procedure step(action)

2: Apply acceleration and steering from action

3: obs = Observe()

4: reward, done = Reward(obs)

5: for $i\leftarrow 1$ to $N$ do

6: if $action.pred\geq\kappa_{i}$ then

7: Drop $i^{th}$ future prediction from obs

8: reward = reward \+ $\psi_{i}$

9: end if

10: end for

11: return obs, reward, done

12:end procedure

##  V EXPERIMENTS

The curriculum was tested on the CommonRoad simulator for unsignalized
T-intersections and four-way intersections.

###  V-A Experimental Setup

For the T-intersections, we have 544 ZAM_Tjunction scenarios from the
CommonRoad benchmarks, which have scenarios with ego-vehicles and target-
vehicles initialized at different positions with different velocities. 380
scenarios are used for training, and the rest are used for testing.

We generated 2000 four-way intersection scenarios. For each scenario, the ego-
vehicle is spawned at a randomly sampled location in lane 1 (Fig. 2) with
appropriate orientation. The ego-vehicle always starts at rest. The obstacles
are spawned at randomly sampled locations in lanes 2, 3 and 4 with appropriate
orientation and randomly sampled initial velocities $\in[5,10]$ m/s. 1400
scenarios were used for training and the rest were used for testing.

###  V-B Network Architecture

To represent the policies, we used fully-connected (FC) networks with 8 hidden
layers of 64 units each and $tanh$ nonlinearities. The hyperparameter values
for PPO are listed in table II. Empirically best performing hyperparameters
and network architecture were choosen for the experimental results.

###  V-C Training Details

The training was done on a desktop with 3.5 GHz AMD Ryzen 9 5900hs CPU. We
used a customized version of the CommonRoad-RL framework for training and
testing. We trained three different agents:

TABLE II: Hyperparameters

Parameter | Value
---|---
Training Iterations (T-intersections) | 600
Training Iterations (Four-way intersections) | 650
Discount ($\gamma$) | 0.99
GAE parameter ($\lambda$) | 0.95
Clipping parameter | 0.2
Learning-rate | 0.0005
No. of environments | 32
Time horizon | 512
Batch size | 16384*

  * •

* Batch size = Time Horizon $\times$ No. of environments

  1. 1.

A standard PPO agent without any curriculum. The target vehicles’ future state
information is dropped by default in the standard agent.

  2. 2.

A PPO agent using curriculum 1 with $N=4$. The agent is thus trained in $5$
phases.

  3. 3.

A PPO agent using curriculum 2 with $N=4$.

###  V-D Performance Evaluation

To understand the impact of the proposed curricula, we compare the training
curves of the agents trained using the curricula with the training curve of
the standard PPO agent. We compare the success rate and mean reward to
evaluate the performance of our method. The results presented are the best
results obtained by repeating each experiment 3 times with a random seed for
the network. The future states of the surrounding vehicles are not available
during testing for all three methods.

####  V-D1 T-intersection

The training curves for the three methods are shown in Fig. 3. The rewards are
averaged across 10 updates. It can be seen that both the curriculum-based
methods outperform the standard PPO baseline in terms of sample efficiency.
While PPO agents with the curricula start converging around 6000000 steps, the
baseline standard PPO converges around 8250000 steps.

Figure 3: Training curves (T-intersection)

Fig. 4 shows the success rate of the methods on 164 testing scenarios. Both
the curricula outperform the standard PPO baseline with their best performance
of 99.39% as compared to 92.68% success rate for the baseline.

Figure 4: Success rates (T-intersections)

####  V-D2 Four-way intersection

The training curves for the three methods are shown in Fig. 5. The rewards are
averaged across 10 updates. It can be seen that both the curriculum based
methods outperform the standard PPO baseline in terms of sample efficiency.
While PPO with curriculum 1 starts converging around 7250000 steps, the
curriculum 2 approach converges around 8500000 steps. The baseline converges
around 9250000 steps.

Figure 5: Training curves

Fig. 6 shows the performance of the methods on the 600 testing scenarios.
Curriculum 2 shows the best performance with its best model having a success
rate of 93.83%. Curriculum 1 achieves 92.66% and the baseline achieves 81.33%.

Figure 6: Success rates

We also compare the results with the rule-based TTC method in Table III.

TABLE III: Success rates of curriculum methods versus baselines Method | T-intersection (%) | Four-way intersection (%)
---|---|---
TTC | 90.85 | 84.33
Standard PPO | 92.68 | 81.33
Curriculum 1 | 99.39 | 92.66
Curriculum 2 | 99.39 | 93.83

###  V-E Discussion

The training curves and success rates of the curriculum-based agents show that
using the curriculum results in higher sample efficiency, with the agents
converging much faster as compared to the baseline agent. Also, both the
curricula have higher success rates than the baselines in both the scenarios.
This shows that the agents converge to better optima as the curriculum makes
the training stable. Though we used ground truth prediction for the target
vehicles available in the simulation, for real-world training, predictions
from the perception layer can be used. Further avenues of privileged
information e.g. bird’s eye view in the augmented observation space can also
be experimented with.

##  VI CONCLUSIONS

In this work, we propose a novel curriculum for training a deep reinforcement
learning agent where future state predictions can achieve faster training and
avoid convergence to suboptimal policies. We test the performance of the
curriculum on the unsignalized intersection traversal task for autonomous
driving. The curriculum outperforms the standard baseline in sample efficiency
and test time performance. Though in this paper we show the application of the
curriculum to the intersection traversal task only, it has a broader scope and
can be generally applied to other similar tasks as well. In future work, we
aim to extend the curriculum for end-to-end learning and real-world testing.

## References

  * [1] N. Administration, “Fatality analysis reporting system (fars) nhtsa,” 2022. [Online]. Available: https://www.nhtsa.gov/research-data/fatality-analysis-reporting-system-fars
  * [2] D. N. Lee, “A theory of visual control of braking based on information about time-to-collision,” _Perception_ , vol. 5, no. 4, pp. 437–459, 1976.
  * [3] C. Urmson, J. Anhalt, D. Bagnell, C. Baker, R. Bittner, M. Clark, J. Dolan, D. Duggins, T. Galatali, C. Geyer, _et al._ , “Autonomous driving in urban environments: Boss and the urban challenge,” _Journal of Field Robotics_ , vol. 25, no. 8, pp. 425–466, 2008.
  * [4] M. Montemerlo, J. Becker, S. Bhat, H. Dahlkamp, D. Dolgov, S. Ettinger, D. Haehnel, T. Hilden, G. Hoffmann, B. Huhnke, _et al._ , “Junior: The stanford entry in the urban challenge,” _Journal of field Robotics_ , vol. 25, no. 9, pp. 569–597, 2008.
  * [5] C.-J. Hoel, T. Tram, and J. Sjöberg, “Reinforcement learning with uncertainty estimation for tactical decision-making in intersections,” in _2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)_ , 2020, pp. 1–7.
  * [6] D. Isele, R. Rahimi, A. Cosgun, K. Subramanian, and K. Fujimura, “Navigating occluded intersections with autonomous vehicles using deep reinforcement learning,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 2034–2039.
  * [7] Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in _Proceedings of the 26th annual international conference on machine learning_ , 2009, pp. 41–48.
  * [8] S. Narvekar, B. Peng, M. Leonetti, J. Sinapov, M. E. Taylor, and P. Stone, “Curriculum learning for reinforcement learning domains: A framework and survey,” _arXiv preprint arXiv:2003.04960_ , 2020.
  * [9] X. Wang, H. Krasowski, and M. Althoff, “Commonroad-rl: A configurable reinforcement learning environment for motion planning of autonomous vehicles,” in _IEEE International Conference on Intelligent Transportation Systems (ITSC)_ , 2021.
  * [10] J. Ji, A. Khajepour, W. W. Melek, and Y. Huang, “Path planning and tracking for vehicle collision avoidance based on model predictive control with multiconstraints,” _IEEE Transactions on Vehicular Technology_ , vol. 66, no. 2, pp. 952–964, 2016.
  * [11] F. Borrelli, P. Falcone, T. Keviczky, J. Asgari, and D. Hrovat, “MPC-based approach to active steering for autonomous vehicle systems,” _International Journal of Vehicle Autonomous Systems_ , vol. 3, no. 2, 2005\.
  * [12] Y. Pan, Q. Lin, H. Shah, and J. M. Dolan, “Safe planning for self-driving via adaptive constrained ilqr,” in _2020 International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020.
  * [13] F. Codevilla, M. Müller, A. López, V. Koltun, and A. Dosovitskiy, “End-to-end driving via conditional imitation learning,” in _2018 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2018, pp. 4693–4700.
  * [14] S. Ross, G. Gordon, and D. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” in _Proceedings of the fourteenth international conference on artificial intelligence and statistics_. JMLR Workshop and Conference Proceedings, 2011.
  * [15] K. Menda, K. Driggs-Campbell, and M. J. Kochenderfer, “Ensembledagger: A bayesian approach to safe imitation learning,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2019, pp. 5041–5048.
  * [16] L. Tai, G. Paolo, and M. Liu, “Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation,” in _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2017, pp. 31–36.
  * [17] M. Everett, Y. F. Chen, and J. P. How, “Motion planning among dynamic, decision-making agents with deep reinforcement learning,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 3052–3059.
  * [18] T. Tram, I. Batkovic, M. Ali, and J. Sjöberg, “Learning when to drive in intersections by combining reinforcement learning and model predictive control,” in _2019 IEEE Intelligent Transportation Systems Conference (ITSC)_. IEEE, 2019, pp. 3263–3268.
  * [19] V. Sezer, T. Bandyopadhyay, D. Rus, E. Frazzoli, and D. Hsu, “Towards autonomous navigation of unsignalized intersections under uncertainty of human driver intent,” in _2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2015.
  * [20] D. Isele, A. Nakhaei, and K. Fujimura, “Safe reinforcement learning on autonomous vehicles,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2018, pp. 1–6.
  * [21] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” _arXiv preprint arXiv:1509.02971_ , 2015.
  * [22] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017\.
  * [23] Z. Zhu, K. Lin, and J. Zhou, “Transfer learning in deep reinforcement learning: A survey,” _arXiv preprint arXiv:2009.07888_ , 2020.
  * [24] Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in _Proceedings of the 26th Annual International Conference on Machine Learning_ , ser. ICML ’09. Association for Computing Machinery, 2009, p. 41–48.
  * [25] J. L. Elman, “Learning and development in neural networks: The importance of starting small,” _Cognition_ , vol. 48, no. 1, 1993.
  * [26] I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, _et al._ , “Solving rubik’s cube with a robot hand,” _arXiv preprint arXiv:1910.07113_ , 2019.
  * [27] Y. Song, H. Lin, E. Kaufmann, P. Dürr, and D. Scaramuzza, “Autonomous overtaking in gran turismo sport using curriculum reinforcement learning,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 9403–9409.
  * [28] Z. Qiao, K. Muelling, J. M. Dolan, P. Palanisamy, and P. Mudalige, “Automatically generated curriculum based reinforcement learning for autonomous vehicles in urban environment,” in _2018 IEEE Intelligent Vehicles Symposium (IV)_ , 2018, pp. 1233–1238.
  * [29] M. K. Sebastian Maierhofer and M. Althoff, “Commonroad scenario designer: An open-source toolbox for map conversion and scenario creation for autonomous vehicles,” in _Proc. of the IEEE Int. Conf. on Intelligent Transportation Systems_ , 2021, pp. 3176–3182.
  * [30] M. Althoff, M. Koschi, and S. Manzinger, “Commonroad: Composable benchmarks for motion planning on roads,” in _Proc. of the IEEE Intelligent Vehicles Symposium_ , 2017.