# Eco-driving for Electric Connected Vehicles at Signalized Intersections: A
Parameterized Reinforcement Learning approach

Xia Jianga Jian Zhanga and Dan Lib CONTACT Jian Zhang. Email:
<EMAIL_ADDRESS>aSchool of Transportation, Southeast University, Nanjing,
Jiangsu, China; bSchool of Engineering, Tibet University, Tibet, China

###### Abstract

This paper proposes an eco-driving framework for electric connected vehicles
(CVs) based on reinforcement learning (RL) to improve vehicle energy
efficiency at signalized intersections. The vehicle agent is specified by
integrating the model-based car-following policy, lane-changing policy, and
the RL policy, to ensure safe operation of a CV. Subsequently, a Markov
Decision Process (MDP) is formulated, which enables the vehicle to perform
longitudinal control and lateral decisions, jointly optimizing the car-
following and lane-changing behaviors of the CVs in the vicinity of
intersections. Then, the hybrid action space is parameterized as a
hierarchical structure and thereby trains the agents with two-dimensional
motion patterns in a dynamic traffic environment. Finally, our proposed
methods are evaluated in SUMO software from both a single-vehicle-based
perspective and a flow-based perspective. The results show that our strategy
can significantly reduce energy consumption by learning proper action schemes
without any interruption of other human-driven vehicles (HDVs).

###### keywords:

eco-driving, reinforcement learning, connected vehicles, signalized
intersections

††articletype: Original article

##  1 Introduction

Transportation system has been recognised as one of the major sources of
energy consumption and air pollution. The authorities have tried their utmost
to build a sustainable and efficient transportation system by applying
advanced technologies. One promising way to facilitate the progress is to
increase the market penetrating rate (MPR) of electric vehicles with effective
energy management strategies. On one hand, the electrification of vehicles
promote the use of cleaner energy, which is of benefit to achieve low-emission
outcomes (Li et al. 2019). On the other hand, making full use of existing
energy can help reduce energy consumption during the operation period of the
vehicles. As an energy efficient purpose is inseparable from advanced
technologies, the application of vehicle-to-everything (V2X) communication to
traffic control and management have triggered a possible revolution towards an
ecological and efficient transportation system. A plenty of research-based
evidences can be found to support the view that such connected technologies
are promising to enhance traffic performance (Jia and Ngoduy 2016; Talebpour
and Mahmassani 2016; Ard et al. 2021; Shi et al. 2021; Jiang et al. 2022).
Based on the belief that the communication modules will be deployed
simultaneously on both vehicles and traffic infrastructures , the vehicle-to-
infrastructure (V2I) technique can be widely implemented in the near future,
which enables the connected vehicles (CVs) to obtain the information
transmitted from traffic infrastructures such as traffic lights and ramp
meterings, with the intent to help CVs make better decisions to improve energy
efficiency.

One of a successful applications of V2I communication in urban traffic is the
deployment of signal phase and timing (SPaT) message, furnishing CVs with
signal status information in a consistent manner (Mintsis, Vlahogianni, and
Mitsakis 2020). Since traffic lights in urban signalized intersections can
interrupt the operation of vehicles, CVs may avoid the red phases and achieve
energy-efficient driving through the use of real-time SPaT data, while idling
at the traffic signals is one major cause of increased energy consumption and
greenhouse gas emissions(Kim and Kim 2020). Given this, the integration of
SPaT information and advanced control policies initiate eco-driving concept,
which has potential to significantly reduce energy consumption in the
proximity of signalized intersections (Li et al. 2018). Generally, the control
policies take vehicle location, SPaT data, and local traffic state as input
and generate speed profile for the controlled vehicle (i.e., the ego vehicle),
and thus enhance the performance of the vehicle in terms of energy
conservation (Asadi and Vahidi 2011; Mintsis et al. 2021; Dong et al. 2021; Ma
et al. 2021).

Eco-driving strategy at signalized intersections is usually accomplished by
keeping up a smooth driving speed, refraining from aggressive acceleration,
and avoiding long-time idling at the intersections, resulting in higher
performance in energy economy and travel efficiency (Gao et al. 2019;
Typaldos, Papamichail, and Papageorgiou 2020). By drawing upon the V2I
communication, relevant strategies place value on the design of control
algorithms in assorted traffic contexts, aiming at explicitly optimize the
energy-saving speed profiles of vehicles. As such, the existing eco-driving
strategies can be fundamentally classified into three broad categories: rule-
based strategy, optimization-based strategy, and learning-based strategy. The
mathematical formulation of each kind of strategy is different, but energy-
saving performances with varying degrees are reported for all of the above
methods.

Predefined rules are provided for a CV in rule-based strategies when the ego
vehicle approaching the intersection. The basic ideas of rule-based methods
are similar: they all divided the state into several categories according to
the signal status and vehicle status, provided that the SPaT messages are
available from V2I communication (Rakha and Kamalanathsharma 2011; Xia,
Boriboonsomsin, and Barth 2013; Ci et al. 2019). As a result, the trajectory
of the ego vehicle can be simply planned with classical kinematics formulas,
trying to let the vehicle cross the junction within green phase.

An advantage of the rule-based methods is that they are easy to be
implemented. However, the approaches cannot adapt to dynamic traffic, because
it is difficult to take the surrounding vehicles into account. For example, if
the vehicle ahead of the ego CV slow down unexpectedly or the vehicle in
adjacent lane insert to the placement in front of the ego CV, the rule-based
methods may lose their efficacy, whereas some of them only consider the signal
status and the vehicle status. Moreover, the rule-based nature is prone to
lead local optimum as the speed change rate is usually fixed. In this case,
the results can leave much to be desired.

In order to achieve better performance, the optimization-based approaches are
extensively studied. One representative way in optimization-based strategies
is to formulate an optimal control model and generate corresponding speed
profiles, which is usually attained by pontryagin’s minimum principle (Jiang
et al. 2017) or dynamic programming (Kuriyama, Yamamoto, and Miyatake 2010;
Feng, Yu, and Liu 2018). Since it is difficult to deal with the nonlinearness,
time varying and environment uncertainty in such vehicle control problems,
some studies adopt model predictive control (MPC) to obtain near-optimal
control laws (Asadi and Vahidi 2011; Kamal, Taguchi, and Yoshimura 2015; Zhao
et al. 2018; Wang and Lin 2020). Another way is establishing a stage-wise
approximation mathematical programming model and solving the model through
heuristic algorithms, thereby providing an optimal speed control strategy.
Chen et al. (2014) developed an optimization model to determine the optimal
longitudinal trajectory, which is resolved by a genetic algorithm (GA). Li et
al. (2018) applied a hybrid algorithm merging GA and particle swarm
optimization (PSO) to solve the multi-stage optimization model. Whereas the
analytical solving process of a complex mathematical program model is time-
consuming, the MPC approaches and heuristic algorithms try to come to an
equitable compromise among optimality and time cost.

There is no doubt that the optimization-based strategies have shown great
potential to implement eco-driving control. Nevertheless, there still exists
some major issues in this kind of strategies. Firstly, solving the
mathematical models in each time step will impose a heavy burden on the on-
board calculation units. The strategies may have limited applicability if the
algorithm cannot be real-time. Secondly, it is hard to consider multiple
vehicle behaviors (e.g., car following, lane changing, overtaking, etc) by
adding a series of cost function items and constraints to the model, while a
complex model can be too burdensome to be numerically solved. Thus, more
adaptive strategy is supposed to be implemented to meets the requirements of
real-time and dynamic.

Recently, with the rapid development of artificial intelligence, the
reinforcement learning (RL) algorithms have attracted the gaze of researchers
in both optimal control community and transportation community. Through
conducting the learning procedure, an agent can select an optimal action for
each observation to maximize its cumulative expected reward (i.e., usually is
the optimization goal), and this process can be usually achieved in a real-
time fashion (Chow et al. 2021). By leveraging deep neural networks, the
integrating outcome, which is known as deep reinforcement learning (DRL), has
the ability to approximate the optimal policy in most of the control tasks,
even with high-dimension state space . Consequently, the potent DRL-based
methods are introduced to control the vehicle in the vicinity of signalized
intersections.

Shi et al. (2018) used a regular Q-learning algorithm to implement eco-driving
for a CV in free flow condition. They took the total CO2 emission as the
reward signal, aiming at optimize the driving behavior of the ego vehicle by
yielding a discrete acceleration rate in each time step. As one of the value-
based reinforcement learning algorithms, the Q-learning approach cannot
control the vehicle in continuous acceleration space, and thereby causes local
optimum and uneven trajectory in most cases. The framework developed by Mousa
et al. (2020) provided insight into the DRL-based eco-driving system, which
introduced deep Q network (DQN) to improve the fuel performance of the
controlled CV. However, major disadvantage similar to the work of Shi et al.
(2018) was encountered in their study, namely, the losing efficacy in
continuous action space. As the policy-based algorithms are capable of
learning control policies with continuous action sets, researchers tend to
apply typical policy-based DRL algorithms to surmount the barrier of
continuous space. Zhou, Yu, and Qu (2020) proposed a car following model based
on the deep deterministic policy gradient (DDPG) algorithm, which is proved
can improve travel efficiency, fuel consumption and safety at an isolated
signalized intersection. Similar study was also formulated for EV when traffic
oscillations were concerned (Qu et al. 2020). Guo et al. (2021) combined DDPG
and DQN to formulate a hybrid reinforcement learning framework, and the
developed approach can not only control the longitudinal motion, but also
optimize the lateral decision of the ego vehicle. Wegener et al. (2021)
studied the application of twin-delayed deep deterministic policy gradient
(TD3) algorithm, which conformed to the similar basic idea of DDPG, but
introduced a series of tricks to tackle the Q function overestimate problem of
DDPG. Their simulation study proclaimed that the DRL algorithm can adapt to
the eco-driving scenarios with the presence of other surrounding vehicles.

Table 1: The summary of existing RL-based eco-driving frameworks

Study |  Vehicle type |  RL algorithm | car follow | lane change |  action space
---|---|---|---|---|---
Shi et al. (2018) |  GV |  Q-learning | $\surd$ | - |  Discrete
Mousa et al. (2020) |  GV |  DQN | $\surd$ | - |  Discrete
Zhou, Yu, and Qu (2020) |  GV |  DDPG | $\surd$ | - |  Continuous
Qu et al. (2020) |  EV |  DDPG | $\surd$ | - |  Continuous
Wegener et al. (2021) |  EV |  TD3 | $\surd$ | - |  Continuous
Guo et al. (2021) |  GV |  DDPG+DQN | $\surd$ | $\surd$ |  Hybrid
Bai et al. (2022) |  EV |  Dueling DQN | $\surd$ | $\surd$ |  Discrete

  * *

GV - Gasoline Vehicle; EV - Electric Vehicle

Focusing on the energy-efficient driving process at signalized intersections,
the existing RL-based frameworks differ from each other in multiple
dimensions, such as the category of the algorithm, the studied vehicle type,
the driving scenario, the design of the action space, etc. Table 1 presents
the summary of relevant studies. It is found that the lane changing behavior
is rarely taken into account when developing eco-driving strategies. Lane
changing during the driving process may bring about a long-term energy
benefit, especially under the situations that the preceding vehicle or queue
can interfere with the running of a CV. Although Guo et al. (2021) considered
the combination of car following and lane changing as their hybrid RL-based
approach simply formulated two Markov Decision Processes (MDP) for
longitudinal motion and lateral decision of the ego vehicle. Therefore, they
only control the two dimensional motions separately by DQN and DDPG. The
inherent mechanism of the hybrid MDP is not clear, and it may lead to local
optimum because the movements in two dimensions are not optimize jointly. It
should be noted that the work of Bai et al. (2022) also considers lane
changing, but the nature of DQN determines that it cannot guarantee a smooth
trajectory and corresponding global optimum, whereas the running of a vehicle
should be continuous process. In this case, in order to handle the application
of discrete-continuous hybrid action space, we introduce the parametrized
action space and compose an unified MDP for eco-driving on the signalized
urban road. Differing from traditional RL methods, the proposed parametrized
reinforcement learning (PRL) controller is capable of optimizing the
longitudinal motion and the lateral motion jointly and stipulate a better
performance, further exploiting the energy-saving potential of electric and
connected vehicles through DRL-based speed control.

In addition, since the inappropriate driving behaviors usually undermine the
holistic performance of traffic, the operation of surrounding normal human-
driven vehicles (HDVs) may be interrupted by the CVs controlled by eco-driving
controllers (Zhao et al. 2018). We call this kind of strategy ”selfish” eco-
driving, which may caused by frequent lane changing or unexpected
deceleration. Compared with most of the other learning-based strategies, which
usually pay close attention to single-vehicle performance, in this paper, we
will restrict CVs from frequent lane changing and unexpected change of
acceleration when designing the reward function, and test whether the proposed
strategy has detrimental influence on other HDVs.

In summary, the particular contributions and the novelty of this paper lie
within the following points:

  * •

An MDP with discrete-continuous hybrid action space is established for CVs in
signalized urban scenario, further tapping the energy-saving potential of CVs
by searching in the longitudinal acceleration/deceleration space and the
lateral lane changing space.

  * •

Instead of simply blending algorithms or discretize the hybrid action space,
the proposed PRL algorithm can be naturally applied to MDP with hybrid action
space, which provides a paradigm for such microscopic CV control problems.

  * •

The proposed strategy can be harmonious by designing a restricted reward
function so that to eliminate the potential adverse impact of the selfish
behaviors of the ego vehicles.

  * •

The developed methodology is tested in both microscopic (single vehicle-based)
perspective and macroscopic (mixed traffic flow-based) perspective,
illuminating the energy-saving capability of the learning-based approach.

The organization of the remainder paper is as follows: The preliminary of RL
is presented in Section 2. Section 3 introduces the system components
regarding the eco-driving process of CVs based on RL controller. Section 4
describes the establishment of the MDP with the implementation of the PRL
algorithm. Numerical results are presented and discussed in Section 5 by
carrying out simulations in microscopic simulation platform SUMO (Lopez et al.
2018). Finally, some concluding remarks are provided in Section 6.

##  2 Background of Reinforcement Learning

The environment (i.e., traffic environment) of the RL agent (i.e., the ego
vehicle) is modeled by an MDP
$M=\\{\mathcal{S},\mathcal{A},\mathcal{P},r,\gamma\\}$ in reinforcement
learning framework, where $\mathcal{S}$ denotes the state space, $\mathcal{A}$
denotes the action space, $\mathcal{P}$ is the Markov transition probability
distribution, $r$ is the reward function, and $\gamma\in[0,1]$ is the discount
factor that measures the relative value of future rewards and current reward.

In the general setup of RL, an agent interacts with the environment
sequentially as follows. At time step $t$, the agent observes a state
$s_{t}\in\mathcal{S}$ and selects an action $a_{t}\in\mathcal{A}$, and then
the agent receives an immediate reward value calculated by $r(s_{t},a_{t})$.
Subsequently, the next state becomes $s_{t+1}~{}P(s_{t+1}|s_{t},a_{t})$. Let
$R_{t}=\sum_{j\geq t}\gamma^{j-t}r(s_{j},a_{j})$ be the cumulative discounted
reward starting from time step $t$, for any policy $\pi$, we define the state-
action value function as $Q^{\pi}(s,a)=\mathbb{E}[R_{t}|s_{t}=s,a_{t}=a,\pi]$,
where $a_{t+k}\sim\pi(\cdot|s_{t+k})$ for all $a_{t+k}$ and $s_{t+k}$ for
$k\in[t+1,\infty)$. Meanwhile, we define the state value function as
$v^{\pi}(s)=\mathbb{E}[R_{t}|s_{t}=s,\pi]$. According to the Bellman equation,
we have $v^{\pi}(s)=\sum_{a\in\mathbb{A}}\pi(a|s)Q^{\pi}(s,a)$, which
illuminates the relationship between the two function. The goal of the agent
is to learn the optimal policy $\pi^{*}$ that can maximize its expected
cumulative discounted reward from the initial state, which can be represented
as $J(\pi)=\mathbb{E}(R_{0}|\pi^{*})$, while the learning procedure is usually
accomplished by estimating the optimal state-action value function
$Q^{\pi^{*}}$.

Value-based methods and policy-based methods are two categories of RL
algorithms, the former estimates $Q^{\pi^{*}}$ and generates actions in a
greedy way, while the latter directly learns the optimal policy $\pi^{*}$ by
optimizing $J(\pi)$. As one typical representative of value-based methods, the
Q-learning algorithm (Watkins and Dayan 1992) is deduced by Bellman equation:

$Q(s,a)=\underset{r_{t},s_{t+1}}{\mathbb{E}}[r_{t}+\gamma\max_{a^{{}^{\prime}}\in\mathcal{A}}Q(s_{t+1},a^{{}^{\prime}})|s_{t}=s,a_{t}=a]$
(1)

which takes $Q^{\pi^{*}}$ as the unique solution. The learning process
iteratively updates the function $Q(s,a)$ through Monte-Carlo method over a
series of sample transitions. Conventionally, $Q(s,a)$ is stored in tabular
data structure for finite state space configuration. Nevertheless, the problem
of large state space arises along with the increase of the task complexity,
When the state space is too large to be stored in computer memory or $Q$-value
cannot be retrieved in real time, the function approximation approach is
introduced, especially promoted by the development of deep neural networks.
DQN utilizes a deep neural network $Q(s,a;\theta)\approx Q(s,a)$ with
parameter set $\theta$ to approximate $Q^{\pi^{*}}$ (Mnih et al. 2015). The
learning of the agent is fundamentally based on the gradient descent theory,
which takes the following loss function:

$L_{t}(\theta)=\\{Q(s_{t},a_{t};\theta)-[r_{t}+\gamma\max_{a^{{}^{\prime}}\in\mathcal{A}}Q(s_{t+1},a^{{}^{\prime}};\theta_{t})]\\}^{2}$
(2)

Differing from the value-based approaches, the policy-based algorithms with
deep neural network directly model the parametrized policy $\pi_{\theta}$. The
parameter set $\theta$ is updated iteratively via gradient descent to maximize
$J(\pi_{\theta})$. A typical application is REINFORCE algorithm, which takes
the corresponding loss function:

$\Delta_{\theta}J(\pi_{\theta})=\underset{s_{t},a_{t}}{\mathbb{E}}[\Delta_{\theta}log\pi_{\theta}(a_{t}|s_{t})Q^{\pi_{\theta}}(s_{t},a_{t})]$
(3)

##  3 Methodology

###  3.1 Scenario description

In this paper, we consider a fully electric vehicle environment to better
measure the holistic energy performance of the mixed road traffic. which is
composed of HDVs and CVs in urban scenario, while only CVs can receive the
SPaT data and be controlled by eco-driving controller. By incorporating such a
mixed traffic condition, the developed strategy is closer to the urban traffic
in the near future so that it can be more practical.

Figure 1: The driving scenario for controlled vehicle in the proximity of a
signalized intersection.

Figure 1 shows the general eco-driving process of the ego vehicle. The left
half of the figure depicts a typical longitudinal trajectory and speed profile
of a CV with eco-driving system. Compared with HDVs, the CV with eco-driving
controller can generate more smooth speed profile and navigate the vehicle
cross the junction in green phase, in order to minimize its electricity
consumption. Considering a more specific scenario, which is shown in the right
half of Figure 1, the CV on the road can constantly detect the surrounding
traffic state within its perceived range. Inspired by (Wang et al. 2021), we
define the perceived range of the CV as a two-dimensional occupancy grid. The
grid cells reflect the occupancy situation of corresponding road segments
around the CV, so we call the grid local traffic state. Whereas the CVs are
equipped with multiple sensor devices, such as camera, Lidar, and radar, this
process can be certainly achieved. In addition to local traffic state, the CV
could also acquire the SPaT data based on V2I communication, provided that it
enters the V2I communication range. The local traffic state and SPaT-related
features are expected to serve as the state of the CV agent, which is one of
the main components in the MDP model.

According to the position of the vehicle, the operation of the CV is
categorized into two cases: (1) When the position of the CV is not within the
V2I communication range of the first downstream traffic signal, it operates by
leveraging local traffic state and default value of the SPaT information. This
is also true of the circumstance when the ego vehicle is inside the
intersection for a multi-intersection scenario. (2) During the period that the
V2I communication is available for the ego CV, it continuously acquire traffic
signal state, together with local traffic state, to adjust its longitudinal
acceleration and make proper lateral decision under control of the eco-driving
system. For longitudinal dimension, the ego vehicle is supposed to cruise with
smooth speed profile and crossing the intersection in green phase; for lateral
dimension, the ego vehicle can stay at current lane or change to adjacent lane
to mitigate the interruption caused by other HDVs or vehicle queues.

###  3.2 Agent framework

The CVs are deemed as agents, which are supposed to adapt to the dynamic
traffic environment and make proper decisions. An agent is a combination of a
PRL policy and several model-based policies. The motivation of the hybrid
policy is: (1) The DRL algorithms always suffer from credibility due to its
unexplainability, and this may cause issues related to driving safety, while
model-based policies explicitly constraint the behavior of vehicles to ensure
a collision-free driving. (2) The model-based driving policies cannot react to
traffic environment adaptively with pre-defined rules, while DRL policy with
”trial-and-error” mechanism can explore the action space more effectively.
This hybrid form endows the agent with the ability to cope with complex
driving scenarios on the premise of safety.

When the agent enters the V2I communication range, its workflow can be seen in
Figure 2. In each step, the agent can observe a state $s_{t}$ and perform
action $a_{t}$ which is output by the PRL policy, and subsequently receive a
reward value $r_{t}$. We abstract the lateral decision and the longitudinal
control of the CV as discrete process and continuous process, respectively.
The action $a_{t}$ thereby can be defined as a tuple $(k_{t},x_{k_{t}})$,
where $k_{t}$ is an integer value that represents the possible lateral
decision that is shown in Figure 1, and $x_{k_{t}}$ is an float value that
represents longitudinal acceleration.

During the learning process of the agent, rear-end or lateral collision may be
caused by unreasonable actions. Concerning that the original action generated
by the PRL policy $a_{t}=(k_{t},x_{k_{t}})$ may be unsafe, we propose the clip
operation and mask operation to preprocess the original action $a_{t}$, and
then decompose the processed action $\overset{.}{a_{t}}$ to perform reliable
motion control in both longitudinal and lateral dimension. This facilitates
more efficient agent training as apriori knowledge is added to prevent the
vehicle from unexpected collision.

Figure 2: The workflow of the agent when V2I communication is available.

In order to provide full explanation of how the framework works, we need to
specify the formulation of the MDP for the agent:

Firstly, the state space $S$ of the agent comprises two parts, namely, the
local traffic state and the logic state, which are specified as:

  * •

Local traffic state $M_{t}^{i}$ for agent $i$ at time $t$ is defined as a two-
dimensional matrix that reflects the occupancy situation of the perceived
range of the agent, which is illustrated in Figure 1. We assume the perceived
range of the CV is $L_{p}$, and it can be divided into forward range
$L_{head}$ and backward range $L_{tail}$. The forward range $L_{head}$ can
help the CV maintain a reasonable speed, when the vehicle agent can obtain the
relative distance between itself and the front vehicle. The backward range
$L_{tail}$ can prompt the CV to find a reasonable lane-changing interval that
can be safe and harmonious. The shape of the local traffic state matrix is
$(\frac{L_{p}}{L_{c}},5)$ for a 5-lane road with the cell length $L_{c}$. The
elements in the matrix are filled with ”0” if the associated detected cells
are occupied by vehicles, otherwise they are filled with ”1”.

  * •

Logic state $V_{t}^{i}$ for agent $i$ at time $t$ is defined as a vector that
includes the physical state of the ego vehicle and the state of the traffic
signal. The logic state of the agent is denoted as
$V_{t}^{i}=[E_{t}^{i},d_{t}^{i},v_{t}^{i},f_{t}^{i},g_{t}^{i}]^{T}$, where
$E_{t}^{i}$ is the one-hot encoding of the lane index that the ego vehicle $i$
located; $d_{t}^{i}$ describes the distance between agent $i$ and the
downstream traffic signal at time $t$; $v_{t}^{i}$ is the velocity of the
agent at time $t$; $f_{t}^{i}$ is a boolean variable that indicates whether
the next traffic signal is in red phase; $g_{t}^{i}$ denotes the time duration
between time $t$ to the time that the next traffic signal turn to green phase
for the agent. When V2I communication is unavailable, the value of $f_{t}^{i}$
and $g_{t}^{i}$ are all set to default value -1.

As a result, the state in time $t$ for agent $i$ is defined as
$s_{t}^{i}=(M_{t}^{i},V_{t}^{i})$, in which the former element is expected to
be processed by a convolutional neural network block to extract latent
features, and the latter element will concatenate to the extracted local
traffic state features and be input to a multi-layer perceptron block to train
the PRL policy.

Secondly, the action space in this paper is considered as a hybrid form that
support both longitudinal control and lateral decision. As mentioned above,
the basic representation of the action at time $t$ is
$a_{t}=(k_{t},x_{k_{t}})$, where $k_{t}\in\\{-1,0,1\\}$ and
$x_{k_{t}}\in[a_{min},a_{max}]$. The value set $\\{-1,0,1\\}$ of $k_{t}$
represent that the ego vehicle can change to the left lane, stay at the
current lane, and change to the right lane. We assume that the lane changing
process can be finished in a fixed duration $T_{c}$. The space of $x_{k_{t}}$
limits the acceleration of the CV to the range of its maximal deceleration
$a_{min}$ and maximal acceleration $a_{max}$. Instead of combining discrete
speed variation rate and lane changing decision to formulate an overall
discrete action space, we utilize the hybrid discrete-continuous action space
to effectively explore the solution space and obtain better performance. Since
imposing the original $a_{t}$ on the ego vehicle may make it be exposed to
traffic accident, we introduce conventional car-following model and lane
changing model to modify the dangerous driving behavior. In this paper, we
utilize the Krauss car-following model (Krauß 1998) and the ”LC2013” lane-
changing model (Erdmann 2015), which are embedded in SUMO simulation software,
to perform clip and mask operations and obtain the modified action
$\overset{.}{a_{t}}=(\overset{.}{k_{t}},\overset{.}{x_{k_{t}}})$:

  * •

Clip operation aims at clipping the potential risking acceleration to a safe
value to avert rear-end collisions or beyond the road speed limit $V_{max}$.
We assume that the acceleration output by the utilized car-following model is
$x_{t}^{CF}$ at time $t$ according to current driving status, and then the
corrected acceleration is transformed to
$\overset{.}{x_{k_{t}}}=\min(x_{CF}^{t},x_{k_{t}})$. Hence, the velocity of
agent $i$ at time $t$ is clipped as
$v_{t}^{i}=\max(\min(V_{max},v_{t-1}^{i}+\overset{.}{x_{k_{t}}}),0)$, which
confines the velocity of the CV to a reasonable range.

  * •

Mask operation is used to judge whether the lateral decision produced by PRL
policy is safe, in order to prevent lateral collisions or deadlocks. For each
non-zero discrete action $k_{t}$, the model-based lane changing policy output
an bool value $\xi$ to resolve that if the agent could change lane. The
corrected lateral action is denoted as $\overset{.}{k_{t}}=k_{t}\times\xi$.

Thirdly, we discuss different settings of the reward function $r_{t}$, while
different reward signal may trigger various learning directions of the agent.
The general goal of the eco-driving strategy is to make a trade-off between
energy consumption and travel time, as the energy-efficient driving mode may
cause the increase of the delay (Wang et al. 2020). An intuitive way to set
the reward function is simply returning a linear combination of total
electricity consumption $E_{f}$ and travel time $T_{f}$:

$r_{t}^{1}=\begin{cases}-(E_{f}+\varphi T_{f}),\mbox{if }t=t_{f}\\\
0,\mbox{otherwise}\end{cases}$ (4)

where, $\varphi$ is a weighting parameter to measure the relative importance
of energy indicator and mobility indicator; $t_{f}$ is the terminal time step
(i.e., the time that the ego vehicle leaves the junction).

However, the travel time can only be calculated at the end of the travel, and
it is difficult for the vehicle agent to learn effectively from such a delay
reward signal. Given this, Guo et al. (2021) used a step-wise travel distance
$l_{t}^{i}$ as the proxy of the travel time. Incorporating with the step-wise
energy consumption $e_{t}^{i}$, the instantaneous reward function is defined
as:

$r_{t}^{2}=l_{t}^{i}-\varphi e_{t}^{i}$ (5)

Despite signs of an improvement in the energy-saving performance, we find that
it is hard to generate a smooth trajectory for the agent by deploying
$r_{t}^{2}$. Meanwhile, this design can be defective because the agent tends
to maximize its future cumulative travel distance, while the remaining
distance from the location of the CV to the intersection is fixed so that the
overall travel time cannot be well reflected. Thus, we devise a novel reward
function composed of step-wise reward and terminal reward to facilitate the
progress of the eco-driving task in a hybrid action space. It is based on the
following considerations:

  * •

The velocity is one of the most important factors when we place value on the
mobility of the CVs, when continuously low speed will lead to an increase of
travel delay. Thus, we set a penalty item with a constant value $r_{p}^{s}$
when the velocity of the agent is lower than a threshold value $v_{\zeta}$.

  * •

Frequent changes of acceleration can cause traffic oscillation by fluctuating
the speed profile of vehicles, and impair the driving comfort at the same time
(Zhao et al. 2018). The differential form of the acceleration is usually
called jerk, which manifests the variation rate of the acceleration. Given
this, we set another penalty item $r_{p}^{j}$ when the value of jerk is higher
than a threshold value $j_{\zeta}$.

  * •

Since frequent lane changing will interfere with the normal operation of other
CVs or HDVs, we set a harmony coefficient $r_{p}^{c}$ for to penal each lane
changing behavior of the agent so that it can change lane in necessary cases.

The three penalty items instruct the agent maintain a smooth speed profile
with as little stopping and lane changing as possible. Finally, our reward
function is defined in Equation 6. It should be noted that the calculation of
$E_{f}$ and $T_{f}$ is only triggered when the episode reaches the terminal
state.

$r_{t}=r_{p}^{s}+r_{p}^{j}+r_{p}^{c}-(E_{f}+\varphi T_{f})$ (6)

##  4 PRL for eco-driving

###  4.1 Reinforcement Learning in Hybrid Action Space

In addition to normal discrete control and continuous control, which are
widely studied by previous works, some control tasks intrinsically have hybrid
action spaces. Controlling of CV is one representative in this field, as the
vehicle can move longitudinally and laterally on the road. While the majority
of the existing studies are confined to car following motion (Shi et al. 2018;
Mousa et al. 2020; Zhou, Yu, and Qu 2020; Wegener et al. 2021), this paper
develops a parameterized action space to naturally describe the control
problem with hybrid actions and thereby implement joint optimization of car-
following and lane-changing movement.

Figure 3: The illustration of a parameterized action space.

Figure 3 presents an example of parameterized action space. We assume that the
lane changing process of the vehicle agent is controlled in a discrete space
(i.e., changing to the left lane, staying at current lane, changing to the
right lane), which is marked with rectangles in orange. Each discrete action
has a continuous parameter space (i.e., the longitudinal
acceleration/deceleration) marked with rectangles in grey. The hierarchical
structure formulates the parameterized action space in which the agent select
actions.

Considering an MDP with parameterized action space, the agent first selects a
discrete action $k_{t}\in\mathcal{K}$ at time $t$, and then selects an
associated continuous parameter $x_{k_{t}}\in\mathcal{X}_{k}$. In this case,
the Bellman equation can be transformed to:

$Q(s_{t},k_{t},x_{k_{t}})=\underset{r_{t},s_{t+1}}{\mathbb{E}}[r_{t}+\gamma\max_{k\in\mathcal{K}}\sup_{x_{k}\in\mathcal{X}_{k}}Q(s_{t+1},k,x_{k})|s_{t}=s,a_{t}=(k_{t},x_{k_{t}})]$
(7)

Note that searching for an optimal action parameter $x_{k}$ over continuous
space in each time step is computationally intractable. Therefore, by setting
up a function $x_{k}^{Q}:\mathcal{S}\rightarrow\mathcal{X}_{k}$ to represent
$\arg\sup_{x_{k}\in\mathcal{X}_{k}}Q(s,k,x_{k})$, the Bellman equation can be
rewritten as Equation 8 (Xiong et al. 2018).

$Q(s_{t},k_{t},x_{k_{t}})=\underset{r_{t},s_{t+1}}{\mathbb{E}}[r_{t}+\gamma\max_{k\in\mathcal{K}}Q(s_{t+1},k,x_{k}^{Q}(s_{t+1}))|s_{t}=s]$
(8)

Given this, a deep neural network with parameter set $\theta$ can be used to
approximate Q function: $Q(s,k,x_{k};\theta)\rightarrow Q(s,k,x_{k})$. For
such a $Q(s,k,x_{k};\theta)$, another neural network with parameter set
$\omega$ can serve as the a deterministic policy network to approximate
$x^{Q}_{k}(s)$, that is,
$x_{k}(:|\omega):\mathcal{S}\rightarrow\mathcal{X}_{k}$. When $\theta$ is
determined, the purpose of the algorithm is to find $\omega$ such that

$Q(s,k,x_{k}(s;\omega);\theta)\approx\sup_{x_{k}\in\mathcal{X}_{k}}Q(s,k,x_{k};\theta)\mbox{\quad
for each \quad}k\in\mathcal{K}$ (9)

Subsequently, we can estimate $\theta$ by minimizing the mean-squared Bellman
error via typical gradient descent, and this resembles the learning process of
DQN. The formation of the loss function for $\theta$ is similar to Equation 2:

$L_{t}^{Q}(\theta)=\frac{1}{2}[Q(s_{t},k_{t},x_{k_{t}};\theta)-y_{t}]^{2}$
(10)

where, $y_{t}$ denotes the cumulative future reward
$r_{t}+\gamma\max_{a^{{}^{\prime}}\in\mathcal{A}}Q(s_{t+1},a^{{}^{\prime}};\theta_{t})$
in time $t$. When $\theta$ is fixed, we aim to regulate $\omega$ to maximize
$Q(s,k,x_{k}(s;\omega))$, so the loss function for $\omega$ is as following

$L_{t}^{\omega}(\omega)=-\sum^{\mathcal{K}}_{k=1}Q(s,k,x_{k}(s;\omega);\theta_{t})$
(11)

We thereby deduce the reinforcement learning theory in a hybrid action space.
To sum up, the framework utilizes a parameterized action space to integrate
the hybrid action sets in a hierarchical fashion. Then we formulate two neural
networks to learn the optimal discrete action and its associated continuous
action, respectively.

###  4.2 PRL algorithm

The DRL algorithm with parameterized action space should not only accommodates
the eco-driving environment, but also considers the implementation details,
such as the exploration process, the design of replay buffer, the setting of
target networks, to make the training process effective and stable.

Firstly, the architecture of the policy network for the agent should be
specified by several neural networks, which is shown in Figure 4, in order to
formulate the backbone of the agent. It takes an action parameter network with
weights $\omega$ to output the parameter vector $x$, and uses an action
network with weights $\theta$ to approximate the optimal parameterized Q
function. For each neural network, the local traffic state is taken as the
input of a convolutional neural network to extract latent features, and then
the flatten features is concatenated with logical state. For the action
network, the parameter vector $x$, which is produced by action parameter
network, is fitted together in the concatenation operation. After a series of
fully-connected layers, the corresponding output is obtained.

In addition, target network is introduced for both action parameter network
and action network to curd instability during training. A target network is a
copy of the original network that is held fixed to serve as a stable target
for some number of steps. We update the target network in a soft way:

$\theta^{{}^{\prime}}=\tau_{1}\theta+(1-\tau_{1})\theta^{{}^{\prime}}$ (12)

$\omega^{{}^{\prime}}=\tau_{2}\omega+(1-\tau_{2})\omega^{{}^{\prime}}$ (13)

where, $\theta^{{}^{\prime}}$ and $\omega^{{}^{\prime}}$ are the parameter set
of target action network and target action parameter network; $\tau_{1}$ and
$\tau_{2}$ are hyperparameters to determine the averaging extents.

Figure 4: The architecture of the policy network.

As both longitudinal decision and lateral decision should be optimized, it is
necessary to perform exploration for the hybrid action set during training so
that more effective motion patterns can be searched. We deploy two exploration
strategies for the agent, namely, $\epsilon\text{-greedy}$ strategy and
Ornstein–Uhlenbeck process noise. The former strategy is usually utilized in
value-based DRL algorithm, similar to the DQN approach, while the latter
resembles that in DDPG algorithm. Meanwhile, a linear decay method is achieved
to regulate the trade-off between exploration and exploitation over the whole
training process with numerous episodes. In this case, the formulation of the
decaying $\epsilon\text{-greedy}$ strategy is

$k=\begin{cases}\arg\max_{k\in\mathcal{K}}Q(s,k,x_{k}(s)),\mbox{with
probability }(1-\epsilon)\\\ exploration,\mbox{with probability
}\epsilon\end{cases}$ (14)

where, $exploration$ represents a randomly selected discrete action. The value
of $\epsilon$ is determined by

$\epsilon=(\epsilon_{init}-\epsilon_{end})\max(\frac{N-n_{step}}{N},0)+\epsilon_{end}$
(15)

where, $\epsilon_{init}$ and $\epsilon_{end}$ are the initial value and the
terminal value of exploration factor $\epsilon$ during the overall learning
episodes; $N$ denotes the total decaying episodes; $n_{step}$ denotes the
index of current episode.

Ornstein–Uhlenbeck process (Uhlenbeck and Ornstein 1930) is implemented to
generate temporally correlated noise, which embodies the exploration mechanism
for action parameter network. By adding noise sampled from a noise process
$\mathcal{N}$, the continuous action produced by action parameter network
$\mu(s|\omega)$ can be slightly disturbed, that is

$x_{k}=\mu(s|\omega)+\mathcal{N}_{k}$ (16)

The holistic process of the proposed PRL algorithm is presented in Algorithm
1. Before the main loop, the weights of the action network, action parameter
network, and the target networks are initiated. For each episode of training,
line 3 shows the initial operation of the simulation traffic environment. The
initialization of the simulation should consider different introductory states
to enhance generalization ability of the agent, and this will be introduce in
section 5. The updating of the weights of the neural networks follows line 17
and line 18 in Algorithm 1. By iteratively updating the parameters, the ego
vehicle, which is seen as the training agent, is capable of conforming to the
dynamic traffic environment and the changing of downstream traffic signals.

Algorithm 1 The PRL algorithm for eco-driving agent training

Hyperparameters: averaging rate $\tau_{1}$ and $\tau_{2}$, exploration factor
$\epsilon$, initial exploration factor $\epsilon_{init}$, terminal exploration
factor $\epsilon_{end}$, the number of decaying episodes $N$, stepsizes
$\\{\alpha_{t},\beta_{t}\\}_{t\geq 0}$, and minibatch size $B$
Initialize: action network $Q(s,k,x_{k}|\theta)$ and action parameter network
$\mu(s|\omega)$, target action network $Q^{{}^{\prime}}$ with weights
$\theta^{{}^{\prime}}\leftarrow\theta$ and target action parameter network
$\mu^{{}^{\prime}}$ with weights $\omega^{{}^{\prime}}\leftarrow\omega$,
replay buffer $\mathcal{R}$, $n_{step}\leftarrow 0$

1:for each episode do

2: $n_{step}\leftarrow n_{step}+1$

3: loading initial traffic environment and observe current state $s_{t}$

4: if $n_{step}<N$ then

5: update exploration factor $\epsilon$ according to Equation 15

6: end if

7: initialize Ornstein–Uhlenbeck process $\mathcal{N}$ for continuous action
exploration

8: for each step in simulation do

9: compute action parameters $x_{k_{t}}$ according to Equation 16

10: compute action $k_{t}$ according to Equation 14

11: take action $a_{t}=(k_{t},x_{k_{t}})$, observe reward $r_{t}$ and next
state $s_{t+1}$

12: store transition $\\{s_{t},a_{t},r_{t},s_{t+1}\\}$ into $\mathcal{R}$

13: sample $\mathcal{B}$ transitions
$\\{s_{b},a_{b},r_{b},s_{b+1}\\}_{b\in\mathcal{B}}$ randomly from
$\mathcal{R}$

14: decompose $a_{b}$ into $k_{b}$ and $x_{k_{b}}$

15: compute future cumulative reward $y_{b}$ for batch $\mathcal{B}$ based on
$Q^{{}^{\prime}}$ and $u^{{}^{\prime}}$

16: use data $\\{y_{b},s_{b},k_{b}\\}_{b\in\mathcal{B}}$ and
$\\{y_{b},s_{b},x_{k_{b}}\\}_{b\in\mathcal{B}}$ to compute the stochastic
gradient $\nabla_{\theta}L_{t}^{Q}(\theta_{t})$ and
$\nabla_{\omega}L_{t}^{\omega}(\omega_{t})$

17: update $\theta$ and $\omega$ by
$\theta_{t+1}\leftarrow\theta_{t}-\alpha_{t}\nabla_{\theta}L_{t}^{Q}(\theta_{t})$
and
$\omega_{t+1}\leftarrow\omega_{t}-\beta_{t}\nabla_{\omega}L_{t}^{\omega}(\omega_{t})$

18: update the parameters of target neural networks according to Equation 12
and Equation 13

19: end for

20:end for

The hyperparameters of the algorithm are collected in Table 2, which are
resolved after running a number of simulations. Meanwhile, the configuration
of the neural networks is determined as follows: the filter number of layer
”Conv_1_1” and ”Conv_1_2” are 8 and 16, respectively. For each convolutional
layer, the kernel size is set to 3, while for layer ”Conv_1_2” an one-
dimension padding operation is added to support the convolutional calculation.
In addition to the convolutional layers, the kernel sizes of max-pooling
layers are all set to 2 with a 2-step stride, while an one-dimension padding
operation is also attached to the second max-pooling layer. The configuration
of layer ”Conv_2_1” and ”Conv_2_2” is the same as those in action parameter
network. The number of neurons for ”Dense_1_1” and ”Dense_1_2” is set to 128
and 64 , and for ”Dense_2_1” and ”Dense_2_2” is 256 and 64.

Table 2: Hyperparameters of the PRL algorithm Hyperparameter |  Symbol |  Value
---|---|---
Discount factor |  $\gamma$ |  0.99
Averaging rate for $Q$ |  $\tau_{1}$ |  $10^{-2}$
Averaging rate for $\mu$ |  $\tau_{2}$ |  $10^{-3}$
Step size for $Q$ |  $\alpha$ |  $10^{-4}$
Step size for $\mu$ |  $\beta$ |  $10^{-5}$
Initial exploration factor |  $\epsilon_{init}$ |  1
Terminal exploration factor |  $\epsilon_{end}$ |  $10^{-2}$
The number of decaying episodes |  $n_{step}$ |  $10^{3}$
Minibatch size |  $\mathcal{B}$ |  128
Replay buffer size |  $\mathcal{R}$ |  $5\times 10^{5}$

##  5 Simulation and discussion

###  5.1 Simulation configuration

The simulation environment is built in SUMO (Lopez et al. 2018) to test the
performance of both training phase and evaluation phase. As an highly portable
microscopic traffic simulation platform, SUMO offers abundant pre-defined
driver models, and meanwhile the operation of vehicles can be controlled
through Traci interface by Python programs. Therefore, we delimit the HDVs and
CVs by adding PRL policy programmed in Python module to CVs, while the
operation of HDVs is based on the default Krauss car-following model and
LC2013 lane-changing model. The energy model that produces transient
electricity consumption is derived from (Kurczveil, López, and Schnieder
2013), which leverages a braking energy recovery mechanism. It means that the
value of instantaneous energy consumption $e_{t}^{i}$ is negative when the
vehicle perform a braking operation, which presents challenges to searching a
globally optimal eco-driving policy. The specific form and parameters of the
energy model are given in our previous article (Zhang et al. 2022). The
proposed PRL policy do not need to explicitly analysis the energy model, while
it only receive a transient signal that indicates the energy consumption in a
model-agnostic manner.

We create a coordinated signal control scenario and a non-coordinated signal
control scenario in a multi-intersection environment to evaluate the proposed
framework. As illustrated in Figure 5, the corridor contains five consecutive
signalized intersections, for which a V2I communication range is set to
$300m$. The traffic lights with cycle time of $90s$ change synchronously for
non-coordinated scenario, while for coordinated scenario a series of offset is
set up to ensure that a vehicle can encounter a green wave under free flow
condition.

Figure 5: The road network scenario used for simulation.

The parameters related to the simulation follow those in Table 3 if no
otherwise specified. Besides, the values of the penalty items is set as
$r_{p}^{s}=-30$, $r_{p}^{j}=-30$, and $r_{p}^{c}=-50$. Together with the
weighting parameter $\varphi$, the reward-related parameters will be
investigated in sensitivity analyses.

Table 3: Parameter configuration for the simulations Simulation parameter |  Symbol |  Value |  Unit
---|---|---|---
Maximum deceleration |  $a_{min}$ |  -4 |  $m/s^{2}$
Maximum acceleration |  $a_{max}$ |  3 |  $m/s^{2}$
Road speed limit |  $V_{max}$ |  50 |  $km/h$
Cell length |  $L_{c}$ |  1 |  $m$
Forward perceived range |  $L_{head}$ |  50 |  $m$
Backward perceived range |  $L_{tail}$ |  10 |  $m$
Vehicle length |  - |  5 |  $m$
Lane changing duration |  $T_{c}$ |  3 |  $s$
Velocity penalty threshold |  $v_{\zeta}$ |  1.5 |  $m/s$
Jerk penalty threshold |  $j_{\zeta}$ |  4 |  $m/s^{3}$

In addition to the general parameter settings of the simulation platform, we
still need to take measures to provide a more dynamic traffic condition, with
the intent to train the agent efficaciously. On one hand, traffic flow with
$1000veh/h$ is generated in the simulation environment in a random departure
mode, which aims at modelling a random road traffic situation; on the other
hand, the ego vehicle is inserted to the beginning of the road network in a
random fashion during training period. More precisely, the insertion time of
the ego vehicle adheres to an uniform distribution $U(150s,300s)$, making the
agent encounter different traffic signal status and vehicle queue situations,
which is of benefit to enhance the robustness of the algorithm.

###  5.2 Training phase discussion

At first, we discuss about the settings of the reward function. The study
investigate not only the forms of the reward function (i.e., $r_{t}^{1}$,
$r_{t}^{2}$, and $r_{t}$), but also the impact of weighting parameter
$\varphi$, to analyze the optimal reward configuration by empirical
experiments. It should be noted that we also add penalty $r_{p}^{c}$ for
$r_{t}^{1}$ and $r_{t}^{2}$ to prevent repeated lane changing behaviors.

(a)

(b)

Figure 6: The changing trend of corresponding indicators during training
period. The data collected in episode $D_{k}$ is smoothed by $D_{k}\leftarrow
0.98D_{k-1}+0.02D_{k}$. (a) Energy consumption, (b) Travel time.

Figure 6 compares three reward configurations in terms of the energy
consumption and travel time for the ego vehicle throughout the 1000-episode
training period. Different patterns are seen in the variation of energy and
time consumption for $r_{t}^{1}$ and the other reward settings. No matter how
the parameter $\varphi$ changes, the downward trend of the metrics is not
significant for $r_{t}^{1}$, illuminating that the agent cannot learn
effective information with feedback only at the end of the episodes. In
contrast, the converged energy consumption of $r_{t}^{2}$ and our $r_{t}$ are
much lower, but the gaps between the those two rewards are likely to widen. In
addition to the energy consumed by the ego vehicle, the changing trend of time
consumption of both $r_{t}^{2}$ and $r_{t}$ are similar, but the figures of
$r_{t}$ see a relatively stable pattern.

On the other side, the impact of the weighting parameter $\varphi$ is likely
to be insignificant, because the results remain basically unchanged with the
increase of $\varphi$. It is possibly due to that the optimal driving patterns
are identical when we give similar importance to mobility and energy saving.
As a result, we will use $\varphi=1$ in the rest of the simulations since the
numerical magnitudes of the two indicators are close.

(a)

(b)

Figure 7: The heatmap with different value of $r_{p}^{s}$ and $r_{p}^{j}$. (a)
Energy consumption, (b) Travel time.

With regard to the values of $r_{p}^{s}$, $r_{p}^{j}$, and $r_{p}^{c}$, a
series of simulation-based training are carried out to explore the optimal
configuration. We fix $r_{p}^{c}=50$ and vary $r_{p}^{s}$ and $r_{p}^{j}$ from
10 to 50 with $step=10$ on the grounds that the result may be changed by
different relative ratio of the three items. The simulation investigation is
based on the environment with non-coordinated traffic signal setting, since
this scenario can be more challenging. The results are shown in Figure 7,
which utilizes average value among 100 test cases for each setting. According
to the graph, we find that the growth of $r_{p}^{S}$ can significantly reduce
travel time but increase the energy consumption. This is reasonable because
the vehicle may speed up as soon as possible if no sufficient penalty is
imposed on its reward system, which also resembles some human drivers. By
comparison, the variation of $r_{p}^{j}$ seems to have much lower impact on
travel time and electricity consumption, but it helps to smooth the trajectory
of the CV, as negative reward will be imposed if the vehicle perform violent
acceleration or deceleration. As a result, we choose $r_{p}^{s}=40$ and
$r_{p}^{j}=30$ to keep a trade-off between energy efficiency and traffic
mobility.

###  5.3 Overall performance of single vehicle

After obtaining the well-trained models of the agent, we can deploy them in
new environment to test the performance of the proposed method. We firstly
introduce a baseline that embedded in SUMO to verify if our model can
outperform human-driving behaviors. In this case, there is no difference
between CVs and HDVs, whereas the CVs follow the default Krauss car-following
model and the LC2013 lane-changing model to simulate the driving behaviors of
human drivers. Meanwhile, we deploy the well-trained agent to the environments
with the same insertion time of the ego vehicle, and record the data of
trajectory, speed, and cumulative energy consumption.

For both coordinate and non-coordinate signal environment, we generate three
scenarios respectively to test the method and make visually comparisons, and
the results are illustrated in Figure 8 and Figure 9. It can be found that the
agent trained by PRL can adapt to different traffic environment: (1) in
coordinate traffic signal environment, the speed profiles of PRL-based vehicle
are similar to those based on human-driving model, but it learns a strategy of
early deceleration to decrease energy consumption. (2) in non-coordinate
traffic signal environment, the energy-efficiency of the vehicle can be better
reflected. While the vehicle with baseline model is obliged to stop at every
red signal, the CV controlled by PRL policy can improve its energy-efficiency
by early deceleration and braking-reacceleration in small range to recover the
electricity through the braking-recovery mechanism. In addition to the energy
indicator, the increase of travel time is not significant for PRL policy. Due
to the fact that the vehicle with default model always drives in its maximum
acceptable velocity, it is difficult to improve mobility and energy-efficiency
at the same time. Therefore, our learning-based framework is capable of
reducing the consumed energy with similar traffic delay of baseline, which is
much more practical in real situations.

Figure 8: The comparison of baseline and PRL in coordinate signal environment.
Figure 9: The comparison of baseline and PRL in non-coordinate signal
environment.

In addition, 100 test cases are simulated to compare the general performance
of the proposed algorithm. Table 4 presents the overall performance of the PRL
algorithm. Compared with the manual driving behavior, although a fraction of
travel time is sacrificed, the proposed framework can significantly reduce the
electricity consumption, showing a decrease of 4.1% in coordinate signal
environment and 27.13% in non-coordinate signal environment.

Table 4: Average performance of the PRL algorithm among 100 test cases Method | Energy consumption (Wh) | Travel time (s)
---|---|---
3 |  |  |  |
5 |  |  |  |
| coord. | non-coord. | coord. | non-coord.
Baseline | 231.13 | 253.79 | 189.08 | 260.21
PRL | 221.75 | 184.93 | 204.79 | 289.82
Imp. $\uparrow$ | 4.1% | 27.13% | -7.93% | -11.25%
|  |  |  |

In order to show the superiority of our method in a more general way, we make
further comparison studies with the following strategies:

  * •

DDPG+LC2013: the longitudinal motions of the CVs are expected to be controlled
by a DDPG policy with the same MDP configurations of our PRL (the harmony
coefficient $r_{p}^{c}$ is excluded), while the lateral decisions are based on
the classical LC2013 model.

  * •

DQN: the CVs are controlled by a DQN policy in this case. Due to that the
value-based DRL algorithm can only handle with discrete action space, a
predefined longitudinal action set is given as:

$\begin{split}A^{DQN}_{lon}=\\{1.0a_{max},0.75a_{max},0.5a_{max},0.25a_{max},0.0,\\\
0.25a_{min},0.5a_{min},0.75a_{min},1.0a_{min}\\}\end{split}$

Combining with the lateral action set $A^{DQN}_{lat}=\\{-1,0,1\\}$ by
Cartesian product, we get a 27-dimensional vector to represent the action
space.

With the intent to make a fair comparison, the hyperparameters for DDPG and
DQN, such as $\gamma$, $\alpha$, $\mathcal{B}$, and $\mathcal{R}$ are the same
as our PRL, as well as the general architectures of the neural networks. The
exploration mechanism of DDPG and DQN are identical to the OU process noise
and decaying $\epsilon$-greedy strategy in the PRL algorithm, respectively.

(a)

(b)

Figure 10: The performance of the ego vehicle with different control methods.
(a) Energy consumption, (b) Travel time.

To alleviate the impact caused by randomness, the results are collected in 10
independent group of simulations with different value of random seed. Figure
10 shows the collected data in a violin plot form, which depicts summary
statistics and the density of the results. From the figure, we can draw
following conclusions: (1) DQN cannot satisfactorily be applied in the task,
whereas it sees an unacceptable performance in terms of travel time. This can
be caused by the large discrete action space of the DQN agent when we
incorporating longitudinal and lateral decisions. (2) The performance of DDPG
is close to the baseline approach, but much worse in non-coordinate signal
settings due to the increased travel time. (3) In contrast, the proposed PRL
method is more impressive with dynamic traffic flow and signal changing
status. Compared with other DRL-based methods, the distribution of energy
consumption for PRL agent is close to a smaller value, with only a modest
growth of travel time.

###  5.4 Lateral performance of single vehicle

Figure 11: The scenarios to test the lateral decision of the agent.

(a) Scenario A

(b) Scenario B

Figure 12: The performance of the ego vehicle in two synthetic scenarios.

It is difficult to illuminate the benefit of lane changing accurately as this
benefit is not instantaneous. One feasible way to demonstrate the
effectiveness of proper lateral decisions is to observe the behavior of the
agent on some specific occasions. Thus, we take two synthetic scenarios as
examples from a microscopic perspective and demonstrate that the agent is
capable of making adaptive lateral decisions. As shown in Figure 11, scenario
A presents a situation that ego vehicle follows HDV1, which will be imposed a
braking with deceleration of $2m/s^{2}$ in three consecutive time steps;
scenario B depicts the ego vehicle follows a platoon composed of HDVs. As the
queues of HDVs at the intersection is detected by the agent, it is supposed to
change lane and plan its velocity in order to mitigate the interruption of the
queue.

Figure 12 depicts the simulation results in coordinate graphs, the time-
distance graphs, and the time-speed graphs of scenario A and scenario B.
Figure 12(a) shows the lane changing behaviors of the ego vehicle after HDV1
performing deceleration. Due to the existence of HDV2, the agent choose to
decelerate and then change to the second lane. In order to shun the
interruption of HDV2, it immediately change to the third lane and keep a
smooth speed profile to cross the intersections. Figure 12(b) shows that the
agent can select the lane with the least number of vehicles and slow down
early to shun the potential influence of the leading HDV.

###  5.5 Performance of mixed traffic

(a) Average energy consumption (coordinate signal control)

(b) Average speed (coordinate signal control)

(c) Average energy consumption (non-coordinate signal control)

(d) Average speed (non-coordinate signal control)

Figure 13: The performance of traffic with different traffic volume and CV MPR
in different traffic signal environment.

Deploying the proposed strategy in mixed traffic flow is an essential part to
verify the applicability of the study. Based on the belief that the co-
existence of CVs and HDVs will last for many years, it is necessary to explore
the potential impact of the strategy with different MPR of the CV.

Figure 13 illustrates the traffic performance in terms of average electricity
consumption and average speed with different MPR of CV and traffic volume. The
time duration of single episode of simulation is set to $1800s$, and a group
of simulation contains 5 independent experiments to collect data and calculate
average results. It is obvious that marginal increase of MPR of CV will bring
about relatively considerable energy reduction of the traffic. Compared with
pure HDV scenario, the mean electricity consumption of scenario with CV MPR=1
decrease to 240Wh/veh-250Wh/veh, showing a decline of around 13.8%. By
deploying the PRL algorithm to the agent, the average energy consumption and
average speed can maintain a stable value with different level of traffic
demand when the MPR is fixed. Meanwhile, we can observe that the decrease of
mean speed is not significant with the growth of MPR, demonstrating the
sacrifice of vehicle mobility is negligible. Another finding is that the loss
of mean speed in non-coordinate setting is lower than that in coordinate
setting with the increase of MPR. This is due to the HDVs in non-coordinate
signal environment are more likely to encounter red light and wait at the
intersection. As such, the proposed method manages to transform this part of
time to the duration that vehicles running with relatively lower speed to
avert red traffic signals.

(a)

(b)

(c)

Figure 14: Average performance of CVs and HDVs in different MPR and traffic
volume. (a) Average energy consumption, (b) Average travel time, (c) Average
speed.

Since many eco-driving strategies may impede the normal operation of HDVs so
as to exert an adverse impact on traffic flow, we run simulations to observe
that if the CVs with PRL strategy interfere with other vehicles. Figure 14
categorise the vehicles into CVs and HDVs, and compare the metrics in non-
coordinate signal settings. The situations of low traffic volume
(200veh/h/lane) and high traffic volume (700veh/h/lane) are compared. The
figures demonstrate that the operations of HDVs are not influenced by CVs,
while thier metrics keep at a steady level, no matter how the volume and MPR
of CVs changes.

##  6 Conclusions

Eco-driving is one of the potent approaches to improve the energy efficiency
of the electric vehicles. This paper propose a PRL-based framework, which can
cope with the vehicle control task with hybrid action space, to implement eco-
driving in urban connected intersections. Incorporating with model-based car-
following and lane-changing mechanism, the PRL algorithm trains the CV to
learn to make joint decisions in terms of longitudinal dimension and lateral
dimension. By analysing different reward settings, which are the core part of
the DRL algorithm, this paper combines step-wise reward and terminal reward,
and formulates an MDP model in a hybrid action space.

Simulations are carried out in both coordinate signal setting and non-
coordinate signal setting, with microscopic perspective and macroscopic
perspective. The microscopic simulations investigate the performance of single
agent in the dynamic traffic environment and show that the consumed energy can
be reduced by 4.1% with coordinate signal setting and 27.13% with non-
coordinate signal setting. Meanwhile, the PRL-based framework can outperform
DQN and DDPG, which are used in some previous works to control CVs. The
macroscopic simulations study the performance of the methodology when
deploying the algorithm to multi-agent situations. It is found that the
increase of CV MPR is of benefit to the overall performance of the traffic
flow, while the operations of HDVs are not impaired by CVs with the PRL
controller. These findings make the framework more practical when considering
to apply DRL-aided policies in real-world intelligent vehicles.

Further exploration of the learning-based eco-driving strategies is very much
needed, including the ways to consider the social welfare of the application
of connected technologies and learning algorithms, so as to improve the
performance of HDVs. In addition, the interactions between CVs is another area
that deserves more attentions, which can promote cooperative eco-driving of
CVs. By employing cooperative multi-agent DRL methods with specific
communication protocols, the collaborative perception and cooperative
decisions can be achieved to further dig the potential value of advanced eco-
driving systems, especially under low CV MPR circumstances.

## Disclosure statement

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

## Funding

This research was supported by the National Key R&D Program of China (Grant
2021TFB1600500), and the Key R&D Program of Jiangsu Province in China (Grant
No. BE2020013).

## References

  * Ard et al. (2021) Ard, Tyler, Longxiang Guo, Robert Austin Dollar, Alireza Fayazi, Nathan Goulet, Yunyi Jia, Beshah Ayalew, and Ardalan Vahidi. 2021. “Energy and flow effects of optimal automated driving in mixed traffic: Vehicle-in-the-loop experimental results.” _Transportation Research Part C: Emerging Technologies_ 130: 103168.
  * Asadi and Vahidi (2011) Asadi, Behrang, and Ardalan Vahidi. 2011. “Predictive Cruise Control: Utilizing Upcoming Traffic Signal Information for Improving Fuel Economy and Reducing Trip Time.” _IEEE Transactions on Control Systems Technology_ 19 (3): 707–714.
  * Bai et al. (2022) Bai, Zhengwei, Peng Hao, Wei Shangguan, Baigen Cai, and Matthew J. Barth. 2022. “Hybrid Reinforcement Learning-Based Eco-Driving Strategy for Connected and Automated Vehicles at Signalized Intersections.” _IEEE Transactions on Intelligent Transportation Systems_ 1–14.
  * Chen et al. (2014) Chen, Zhi, Yunlong Zhang, Jinpeng Lv, and Yajie Zou. 2014. “Model for Optimization of Ecodriving at Signalized Intersections.” _Transportation Research Record_ 2427 (1): 54–62.
  * Chow et al. (2021) Chow, Andy H.F., Z.C. Su, E.M. Liang, and R.X. Zhong. 2021. “Adaptive signal control for bus service reliability with connected vehicle technology via reinforcement learning.” _Transportation Research Part C: Emerging Technologies_ 129: 103264.
  * Ci et al. (2019) Ci, Yusheng, Lina Wu, Jiafa Zhao, Yichen Sun, and Guohui Zhang. 2019. “V2I-based car-following modeling and simulation of signalized intersection.” _Physica A: Statistical Mechanics and its Applications_ 525: 672–679.
  * Dong et al. (2021) Dong, Haoxuan, Weichao Zhuang, Boli Chen, Guodong Yin, and Yan Wang. 2021. “Enhanced Eco-Approach Control of Connected Electric Vehicles at Signalized Intersection With Queue Discharge Prediction.” _IEEE Transactions on Vehicular Technology_ 70 (6): 5457–5469.
  * Erdmann (2015) Erdmann, Jakob. 2015. “SUMO’s Lane-changing model.” In _2nd SUMO User Conference_ , edited by Michael Behrisch and Melanie Weber, Vol. 13 of _Lecture Notes in Control and Information Sciences_ , 105–123. Springer Verlag.
  * Feng, Yu, and Liu (2018) Feng, Yiheng, Chunhui Yu, and Henry X. Liu. 2018. “Spatiotemporal intersection control in a connected and automated vehicle environment.” _Transportation Research Part C: Emerging Technologies_ 89: 364–383.
  * Gao et al. (2019) Gao, Zhiming, Tim LaClair, Shiqi Ou, Shean Huff, Guoyuan Wu, Peng Hao, Kanok Boriboonsomsin, and Matthew Barth. 2019. “Evaluation of electric vehicle component performance over eco-driving cycles.” _Energy_ 172: 823–839.
  * Guo et al. (2021) Guo, Qiangqiang, Ohay Angah, Zhijun Liu, and Xuegang (Jeff) Ban. 2021. “Hybrid deep reinforcement learning based eco-driving for low-level connected and automated vehicles along signalized corridors.” _Transportation Research Part C: Emerging Technologies_ 124: 102980.
  * Jia and Ngoduy (2016) Jia, Dongyao, and Dong Ngoduy. 2016. “Enhanced cooperative car-following traffic model with the combination of V2V and V2I communication.” _Transportation Research Part B: Methodological_ 90: 172–191.
  * Jiang et al. (2017) Jiang, Huifu, Jia Hu, Shi An, Meng Wang, and Byungkyu Brian Park. 2017. “Eco approaching at an isolated signalized intersection under partially connected and automated vehicles environment.” _Transportation Research Part C: Emerging Technologies_ 79: 290–307.
  * Jiang et al. (2022) Jiang, Xia, Jian Zhang, Qingyang Li, and Tianyi Chen. 2022. “A Multiobjective Cooperative Driving Framework Based on Evolutionary Algorithm and Multitask Learning.” _Journal of Advanced Transportation_ 2022\.
  * Kamal, Taguchi, and Yoshimura (2015) Kamal, Md Abdus Samad, Shun Taguchi, and Takayoshi Yoshimura. 2015. “Intersection Vehicle Cooperative Eco-Driving in the Context of Partially Connected Vehicle Environment.” In _2015 IEEE 18th International Conference on Intelligent Transportation Systems_ , 1261–1266.
  * Kim and Kim (2020) Kim, Minjeong, and Hoe Kyoung Kim. 2020. “Investigation of environmental benefits of traffic signal countdown timers.” _Transportation Research Part D: Transport and Environment_ 85: 102464.
  * Krauß (1998) Krauß, Stefan. 1998. “Microscopic modeling of traffic flow: Investigation of collision free vehicle dynamics.” .
  * Kurczveil, López, and Schnieder (2013) Kurczveil, Tamás, Pablo Álvarez López, and Eckehard Schnieder. 2013\. “Implementation of an Energy Model and a Charging Infrastructure in SUMO.” In _Simulation of Urban MObility User Conference_ , 33–43. Springer.
  * Kuriyama, Yamamoto, and Miyatake (2010) Kuriyama, Motoi, Sou Yamamoto, and Masafumi Miyatake. 2010. “Theoretical study on Eco-Driving Technique for an Electric Vehicle with Dynamic Programming.” In _2010 International Conference on Electrical Machines and Systems_ , 2026–2030.
  * Li et al. (2019) Li, Fangyi, Rui Ou, Xilin Xiao, Kaile Zhou, Wu Xie, Dawei Ma, Kunpeng Liu, and Zhuo Song. 2019. “Regional comparison of electric vehicle adoption and emission reduction effects in China.” _Resources, Conservation and Recycling_ 149: 714–726.
  * Li et al. (2018) Li, Ming, Xinkai Wu, Xiaozheng He, Guizhen Yu, and Yunpeng Wang. 2018. “An eco-driving system for electric vehicles with signal control under V2X environment.” _Transportation Research Part C: Emerging Technologies_ 93: 335–350.
  * Lopez et al. (2018) Lopez, Pablo Alvarez, Michael Behrisch, Laura Bieker-Walz, Jakob Erdmann, Yun-Pang Flötteröd, Robert Hilbrich, Leonhard Lücken, Johannes Rummel, Peter Wagner, and Evamarie Wießner. 2018. “Microscopic Traffic Simulation using SUMO.” In _The 21st IEEE International Conference on Intelligent Transportation Systems_ , IEEE.
  * Ma et al. (2021) Ma, Fangwu, Yu Yang, Jiawei Wang, Xinchen Li, Guanpu Wu, Yang Zhao, Liang Wu, Bilin Aksun-Guvenc, and Levent Guvenc. 2021. “Eco-driving-based cooperative adaptive cruise control of connected vehicles platoon at signalized intersections.” _Transportation Research Part D: Transport and Environment_ 92: 102746.
  * Mintsis, Vlahogianni, and Mitsakis (2020) Mintsis, Evangelos, Eleni I. Vlahogianni, and Evangelos Mitsakis. 2020. “Dynamic Eco-Driving near Signalized Intersections: Systematic Review and Future Research Directions.” _Journal of Transportation Engineering, Part A: Systems_ 146 (4): 04020018.
  * Mintsis et al. (2021) Mintsis, Evangelos, Eleni I. Vlahogianni, Evangelos Mitsakis, and Seckin Ozkul. 2021\. “Enhanced speed advice for connected vehicles in the proximity of signalized intersections.” _European Transport Research Review_ 13 (2).
  * Mnih et al. (2015) Mnih, Volodymyr, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, et al. 2015. “Human-level control through deep reinforcement learning.” _Nature_ 518 (7540): 529–533.
  * Mousa et al. (2020) Mousa, Saleh R., Sherif Ishak, Ragab M. Mousa, Julius Codjoe, and Mohammed Elhenawy. 2020. “Deep Reinforcement Learning Agent with Varying Actions Strategy for Solving the Eco-Approach and Departure Problem at Signalized Intersections.” _Transportation Research Record_ 2674 (8): 119–131.
  * Qu et al. (2020) Qu, Xiaobo, Yang Yu, Mofan Zhou, Chin-Teng Lin, and Xiangyu Wang. 2020. “Jointly dampening traffic oscillations and improving energy consumption with electric, connected and automated vehicles: A reinforcement learning based approach.” _Applied Energy_ 257: 114030.
  * Rakha and Kamalanathsharma (2011) Rakha, Hesham, and Raj Kishore Kamalanathsharma. 2011. “Eco-driving at signalized intersections using V2I communication.” In _2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC)_ , 341–346.
  * Shi et al. (2021) Shi, Haotian, Yang Zhou, Keshu Wu, Xin Wang, Yangxin Lin, and Bin Ran. 2021. “Connected automated vehicle cooperative control with a deep reinforcement learning approach in a mixed traffic environment.” _Transportation Research Part C: Emerging Technologies_ 133: 103421.
  * Shi et al. (2018) Shi, Junqing, Fengxiang Qiao, Qing Li, Lei Yu, and Yongju Hu. 2018. “Application and Evaluation of the Reinforcement Learning Approach to Eco-Driving at Intersections under Infrastructure-to-Vehicle Communications.” _Transportation Research Record_ 2672 (25): 89–98.
  * Talebpour and Mahmassani (2016) Talebpour, Alireza, and Hani S. Mahmassani. 2016. “Influence of connected and autonomous vehicles on traffic flow stability and throughput.” _Transportation Research Part C: Emerging Technologies_ 71: 143–163.
  * Typaldos, Papamichail, and Papageorgiou (2020) Typaldos, Panagiotis, Ioannis Papamichail, and Markos Papageorgiou. 2020. “Minimization of Fuel Consumption for Vehicle Trajectories.” _IEEE Transactions on Intelligent Transportation Systems_ 21 (4): 1716–1727.
  * Uhlenbeck and Ornstein (1930) Uhlenbeck, G. E., and L. S. Ornstein. 1930. “On the Theory of the Brownian Motion.” _Phys. Rev._ 36: 823–841.
  * Wang et al. (2021) Wang, Guan, Jianming Hu, Zhiheng Li, and Li Li. 2021. “Harmonious Lane Changing via Deep Reinforcement Learning.” _IEEE Transactions on Intelligent Transportation Systems_ 1–9.
  * Wang et al. (2020) Wang, Guangjun, Keita Makino, Arek Harmandayan, and Xinkai Wu. 2020. “Eco-driving behaviors of electric vehicle users: A survey study.” _Transportation Research Part D: Transport and Environment_ 78: 102188.
  * Wang and Lin (2020) Wang, Siyang, and Xianke Lin. 2020. “Eco-driving control of connected and automated hybrid vehicles in mixed driving scenarios.” _Applied Energy_ 271: 115233.
  * Watkins and Dayan (1992) Watkins, Christopher JCH, and Peter Dayan. 1992. “Q-learning.” _Machine learning_ 8 (3): 279–292.
  * Wegener et al. (2021) Wegener, Marius, Lucas Koch, Markus Eisenbarth, and Jakob Andert. 2021. “Automated eco-driving in urban scenarios using deep reinforcement learning.” _Transportation Research Part C: Emerging Technologies_ 126: 102967\.
  * Xia, Boriboonsomsin, and Barth (2013) Xia, Haitao, Kanok Boriboonsomsin, and Matthew Barth. 2013. “Dynamic Eco-Driving for Signalized Arterial Corridors and Its Indirect Network-Wide Energy/Emissions Benefits.” _Journal of Intelligent Transportation Systems_ 17 (1): 31–41.
  * Xiong et al. (2018) Xiong, Jiechao, Qing Wang, Zhuoran Yang, Peng Sun, Lei Han, Yang Zheng, Haobo Fu, Tong Zhang, Ji Liu, and Han Liu. 2018. “Parametrized deep q-networks learning: Reinforcement learning with discrete-continuous hybrid action space.” _arXiv preprint arXiv:1810.06394_ .
  * Zhang et al. (2022) Zhang, Jian, Xia Jiang, Suping Cui, Can Yang, and Bin Ran. 2022. “Navigating Electric Vehicles Along a Signalized Corridor via Reinforcement Learning: Toward Adaptive Eco-Driving Control.” _Transportation Research Record_ 0 (0): 03611981221084683.
  * Zhao et al. (2018) Zhao, Weiming, Dong Ngoduy, Simon Shepherd, Ronghui Liu, and Markos Papageorgiou. 2018. “A platoon based cooperative eco-driving model for mixed automated and human-driven vehicles at a signalised intersection.” _Transportation Research Part C: Emerging Technologies_ 95: 802–821.
  * Zhou, Yu, and Qu (2020) Zhou, Mofan, Yang Yu, and Xiaobo Qu. 2020. “Development of an Efficient Driving Strategy for Connected and Automated Vehicles at Signalized Intersections: A Reinforcement Learning Approach.” _IEEE Transactions on Intelligent Transportation Systems_ 21 (1): 433–443.