# Transferable Deep Reinforcement Learning Framework for Autonomous Vehicles
with Joint Radar-Data Communications

Nguyen Quang Hieu, Dinh Thai Hoang, Dusit Niyato,
Ping Wang, Dong In Kim, and Chau Yuen  N. Q. Hieu and D. Niyato are with the
School of Computer Science and Engineering, Nanyang Technological University,
Singapore (e-mail<EMAIL_ADDRESS>dniyato@ntu.edu.sg). D. T.
Hoang is with the School of Electrical and Data Engineering, University of
Technology Sydney, Sydney, NSW 2007, Australia (e-mail:
dinhthaihoang.au@gmail.com). P. Wang is with the Department of Electrical
Engineering and Computer Science, Lassonde School of Engineering, York
University, Toronto, ON M3J 1P3, Canada, (e-mail: pingw@yorku.ca). D. I. Kim
is with the Department of Electrical and Computer Engineering, Sungkyunkwan
University (SKKU), Suwon 16419, South Korea (e-mail: dikim@skku.ac.kr). C.
Yuen is with the Engineering Product Development, Singapore University of
Technology and Design (SUTD), 8 Somapah Road, 487372, Singapore, (e-mail:
yuenchau@sutd.edu.sg).

###### Abstract

Autonomous Vehicles (AVs) are required to operate safely and efficiently in
dynamic environments. For this, the AVs equipped with Joint Radar-
Communications (JRC) functions can enhance the driving safety by utilizing
both radar detection and data communication functions. However, optimizing the
performance of the AV system with two different functions under uncertainty
and dynamic of surrounding environments is very challenging. In this work, we
first propose an intelligent optimization framework based on the Markov
Decision Process (MDP) to help the AV make optimal decisions in selecting JRC
operation functions under the dynamic and uncertainty of the surrounding
environment. We then develop an effective learning algorithm leveraging recent
advances of deep reinforcement learning techniques to find the optimal policy
for the AV without requiring any prior information about surrounding
environment. Furthermore, to make our proposed framework more scalable, we
develop a Transfer Learning (TL) mechanism that enables the AV to leverage
valuable experiences for accelerating the training process when it moves to a
new environment. Extensive simulations show that the proposed transferable
deep reinforcement learning framework reduces the obstacle miss detection
probability by the AV up to $67\%$ compared to other conventional deep
reinforcement learning approaches.

###### Index Terms:

Joint radar-communications, autonomous vehicles, Q-learning, deep
reinforcement learning, transfer learning, MDP.

##  I Introduction

Recent years have witnessed a fast development of Autonomous Vehicles (AVs)
with advanced driving safety technologies. According to the recent Allied
Market Research’s outlook [1], the autonomous vehicle global market is
expected to reach $556.67 billion by 2026 due to significant demands of road
safety. To achieve driving safety, AVs are usually equipped with Artificial
Intelligent (AI) softwares, light detection & ranging (LiDAR), and radar
detection technologies [4]. Furthermore, AVs are recommended to communicate
with each other, e.g., using Vehicle-to-Vehicle (V2V) communications, or with
roadside infrastructures, e.g., using Vehicle-to-Infrastructure (V2I)
communications, to facilitate intelligent traffic management, routing and data
analysis [2, 3]. Aiming for fully-automated future transportation systems, AVs
are expected to be equipped with two main components that are (i) advanced
driver-assistance system (ADAS) and (ii) data communication system [2, 8, 4].

Several research works focusing on vision-based ADAS, i.e., camera and LiDAR,
have been proposed to address driving safety problems for AVs [5, 6]. With the
recent development of computer vision and deep learning, AVs can achieve
highly accurate detection under different requirements of autonomous driving
applications [4]. However, most of these aforementioned works focus on
enhancing the detection accuracy without considering the impact of environment
(e.g., rain, fog, and snow) on the performance and safety of the AVs. Radar,
in contrast, can be equipped on AVs and work in such weather conditions with
effective detection ability ranging from few meters to more than 200 meters
[2].

AVs equipped with data communication systems can exchange information with
roadside units (e.g., base stations and edge computing services) to
efficiently navigate under different weather conditions or avoid unwanted
traffic congestions [2]. Motivated by the fact that the radar detection and
data communication functions can work effectively together under the same
frequency band, e.g., mm-wave band at $71-76$ GHz, many research works have
been proposed to use these functions in a joint manner, namely Joint Radar-
Communications (JRC). Accordingly, this joint design can lead to significant
benefits in terms of size, cost, power consumption, robustness, and
performance [8, 4]. Therefore, JRC, with its abilities of enhancing driving
safety and facilitating intelligent road management, is expected to play a key
role for future AVs.

Although the development of JRC brings major benefits for AVs, they are facing
several technical challenges. In particular, the radar detection and data
communication functions work on the same frequency band, and thus optimizing
resource sharing between these two functions needs to be addressed [8]. In
addition, several environment factors can affect operation of the JRC
functions and thus yield a dynamic optimization problem which is hard to be
solved [7, 19]. Under different environment conditions, e.g., communication
channel, weather, and traffic density, the AVs should be able to control their
JRC functions effectively. Another problem is that the AVs are highly mobile.
When they travel from the current environment to a new environment which has
different data distributions, e.g., different traffic density, weather, and
road state distributions, they should be able to adapt their JRC functions to
suit the new environment.

To address these aforementioned problems, in this work, we first formulate the
dynamic JRC function selection problem of an AV by using a Markov Decision
Process (MDP) framework. The MDP framework enables the AV to make decisions by
considering the current state of surrounding environment. Then, we propose a
deep reinforcement learning algorithm to help the AV learn the environment
dynamics and make optimal JRC decisions without requiring any prior
information from the surrounding environment. Furthermore, we propose a
Transfer Learning (TL) approach that enables the AV to learn quickly an
optimal policy when it has to travel to a new environment. This approach also
enables an AV to share useful knowledge from a learned environment with other
AVs that want to learn optimal policies from this environment. The
contributions of this paper can be summarized as follows.

  * •

We propose an MDP framework for the joint radar-communications system in AVs.
With the MDP framework, the AV can adaptively select the radar detection
function or data communication function based on the current status of
surrounding environment, e.g., weather condition, communication channel state,
traffic condition, and driving speed, to maximize the system performance.

  * •

We develop an algorithm based on Double Deep Q-Network (DDQN) to deal with the
dynamic and uncertainty of the environment and efficiently find the optimal
policy for the AV. The DDQN algorithm with the experience replay memory
technique and deep neural network as a function approximator can address the
slow-convergence problem of conventional reinforcement learning algorithms
(e.g., Q-learning). This feature is especially useful when the AV is required
to obtain quickly the optimal policy under a large number of uncertain
environment factors.

  * •

To make our proposed framework further scalable, we introduce a TL approach,
namely Transfer Learning with Demonstrations (TLwD), to help the AV accelerate
the training process when the AV travels to a new environment. By utilizing a
small amount of demonstration data from an outer environment, combining with
prioritized experience replay [29] and multi-step TD learning [25] techniques,
the AV can obtain an optimal policy faster than that using the conventional
deep reinforcement learning approaches in the new environment. To the best of
our knowledge, our paper is the first work that considers TL setting for joint
radar-communications in autonomous vehicles.

  * •

Finally, we perform extensive simulations with the aims of not only
demonstrating the efficiency of proposed solutions in comparisons with other
conventional methods but also providing insightful analytical results for the
implementation of our framework. To that end, we take into account parameters
of real automotive radar devices [31]. In addition, we use statistics from a
large dataset, namely nuScenes [35], as our simulation parameters. The
simulation results show that our proposed TL approach reduces the miss
detection probability of the AV up to $67\%$ and requires a smaller number of
training iterations to converge to the optimal policy, compared to the other
conventional deep reinforcement learning approaches.

The rest of paper is organized as follows. Related works are reviewed in
Section II. Sections III and IV describe the system model and problem
formulation, respectively. Section V presents the Q-learning and Double Deep
Q-Network algorithms. Then, Transfer Learning with Demonstrations (TLwD)
algorithm is presented in Section VI. After that, the evaluation results are
discussed in Section VII. Finally, conclusions are drawn in Section VIII.

##  II Related Works

Since the AV system equipped with JRC functions can use both radar detection
and data communication functions using the same hardware platform, these
functions share some joint system resources such as antennas, spectrum, and
power. Consequently, one major problem is to optimize the resource sharing
between the radar detection function and data communication function.
Recently, some resource sharing approaches have been proposed to solve the
problem [10, 13, 9, 11, 12, 14, 15]. In [9], the authors proposed a JRC system
that enables the vehicle to transmit data while estimating distance, Doppler
returns and angles of arrival of target vehicles. In [10], the authors
proposed an effective framework to support radar detection and data
transmission for automotive applications. The key idea of this approach is
reserving preamble blocks in the IEEE 802.11ad frame for the radar detection
mode to estimate ranges and velocities of targets, and at the same time uses
data blocks for the data transmission. Similarly, the authors in [12, 11]
exploited the preamble of a communication frame and used it for the radar
detection function. For balancing the resource sharing between radar detection
and data communication, the authors in [12, 11] proposed a fraction parameter
in which the frame contains one fraction for data communication and another
one for radar detection. In [11], the authors illustrated performance metrics
for different fraction parameters, but the optimal value of this parameter was
not discussed. In [12], the optimal value of the fraction parameter was
provided by using convex hull approximation. Although the above works are
effective in specific circumstances, they are fixed-schedule schemes that are
not appropriate to implement in practice. The reason is that the AVs are
highly mobile and when they are running on the road, their surrounding
environments are uncertain and dynamic. In addition, the performance of these
approaches is limited either by the respective communications protocols or
inter-vehicular synchronization.

In [14, 13, 15], the authors proposed time division approaches to optimize the
resource sharing between radar detection function and data communication
function. In [13], a time slotted JRC network with uncoordinated setup is
considered in which the radar detection probability and data communication
throughput were studied. However, the fundamental tradeoff of radar-
communications coexistence with flexible configuration remains unexplored. The
authors in [14] proposed a time allocation approach for a bistatic automotive
radar system aiming to maximize the probability of detection under different
conditions of the communication channel. Simulation results showed that with
different channel conditions, the optimal portions between radar operation
time and data communication operation time are different. In [15], the authors
proposed a time-sharing scheme in which a JRC node can switch between radar
mode and communication mode based on the requirements of radar, i.e., maximum
detection range. Although the approaches in [14, 15] can exploit the
simplicity of switching-based JRC mechanism [8], they are either limited by
the number of considered targets [14] or with fixed settings during operation
time relied on the radar maximum detection range [15].

It is worth noting that most aforementioned solutions focus on optimal time-
sharing between the radar detection function and data communication function
without concerning the dynamic of environment. Optimal time-sharing provides
resource utilization, e.g., power and bandwidth, but the optimal solutions are
fixed during the runtime, that limits practical applications. Alternatively,
these approaches require additional information of environment, e.g., channel
model, interference model, or arrival distributions of vehicles in advance,
which is intractable to obtain, or even infeasible to collect in real-world
applications. The aforementioned approaches also neglect the fact that the AV
can possibly move to new environments in which the assumption of data
distribution in the old environment fails. For example, the AV can travel from
a rural area with a low traffic density, to an urban area with a high traffic
density. Moreover, other environment factors such as road condition and
weather condition may have different distributions in different environments.
Therefore, to achieve robustness in driving safety, AV systems should consider
different data distribution in variuos environment settings [16].

To the best of our knowledge, our previous work in [19] is the first work
considering the joint operation of radar and communication for AVs under high
dynamic environments such as unfavorable weather conditions, noisy
communication channels. However, our previous work has several limitations.
Specially, in [19], we considered that the AV only can choose to either
transmit data or perform radar detection function. However, in practice, the
AV can have multiple operation modes for these two functions based on specific
requirements of autonomous driving applications [31]. Thus, in this work, we
consider the AV with more practical operation modes for the JRC functions
which can provide more flexible operations and thus be able to yield a better
system performance. Furthermore, unlike our previous work [19] which can be
applied in a specific environment for a particular AV, in this work we aim to
develop a more general framework which can help the AV to quickly obtain an
optimal policy when it travels to a new environment.

##  III System Model

Figure 1: An illustration of an AV system equipped with JRC functions.

We consider an AV system equipped with JRC functions as illustrated in Fig. 1.
In the system model, two environments are considered that are source
environment and target environment. The two environments are assumed to be
related but different [20]. The AV is required to learn efficiently in the
source environment and when it travels to the target environment, it should be
able to transfer its experiences from the source environment to learn quickly
a new policy. Since the two environments are different, the optimal policies
learned from these environments are also different [20]. Details of the
differences between the source environment and target environment are
described in the next section.

The AV system includes (i) a rate adaptation module for the data communication
function and (ii) a Frequency Modulated Continuous Wave (FMCW) radar for the
radar detection function. The time horizon is discrete and the AV can choose
either radar detection mode or data communication mode at each time step. On
the one hand, if the AV uses the data communication mode, it will then select
an appropriate data rate $R_{l}$ to transmit data packets to a nearby Base
Station (BS). On the other hand, if the AV uses the radar detection mode, it
will then select a suitable radar mode $B_{k}$ (e.g., short-range or long-
range) to detect objects. It is worth noting that the time division-based
configuration for the JRC functions benefits from the simplicity and low-cost
of implementation [8]. Both radar detection and data communication functions
can be easily implemented using a simple switch without requiring re-design
radar and communication waveform [8]. In the following sections, we discuss
more details on how these modes are performed at the AV.

###  III-A Automotive Joint Radar-Communications Functions

####  III-A1 Data communication function with rate adaption

The data rate adaptation module enables the AV to adjust its data transmission
rates according to the channel quality and thus helps the AV achieve better
throughput[17]. With rate adaptation, the AV can choose one of $L$ data
transmission rates from the set
$\mathbf{R}=\\{R_{1},\ldots,R_{l},\ldots,R_{L}\\}$, where
$R_{1}<\cdots<R_{l}<\cdots<R_{L}$. These rates can be achieved through
different combinations of modulation and coding schemes [18]. The SINR at the
BS, denoted by $\eta$, can be then determined by [17]:

$\eta=\dfrac{P_{R}}{\sigma_{I}+\sigma_{N}},$ (1)

where $P_{R}$ is the received power at the BS, $\sigma_{I}$ and $\sigma_{N}$
are interference power of the other (interfering) signals in the network and
random noise power, respectively. For a given SINR, the BS can decode certain
rates with target bit error rate (BER). For example, for $l=1,\ldots,L$, if
$\delta_{l-1}\leq\eta<\delta_{l}$ with $\delta_{l}$ to be the value of SINR,
only rates $R_{1},R_{2},\ldots,R_{l-1}$, can be decoded at the BS [18]. Thus,
if the received SINR at the BS is less than $\delta_{l}$ and the AV chooses a
data rate $R_{l}$ or higher, the transmitted packets will be dropped. An ACK
message is used by the BS to notify the AV for the successful transmission.

####  III-A2 Radar detection function with multi-mode FMCW radar

Figure 2: Frequency Modulated Continuous Wave (FMCW) mmWave radar with
multiple chirp configurations [31].

FMCW radar can provide multiple detection modes, e.g., short-range detection,
mid-range detection, and long-range detection, for the AV to select during its
running time [31]. The multi-mode setting is especially useful when the AV
operates in different environments with various requirements. For example, a
short-range detection mode can provide a highly accurate range resolution,
which can detect nearby objects. Therefore, this mode is very useful in urban
environments with high traffic density. In particular, each detection mode is
equivalent to a specific configuration as illustrated in Fig. 2. For the FMCW
radar, radar signals are transmitted as chirps in which the shape of a chirp
is decided by sweep bandwidth $b$ and frequency slope $f_{c}$ [3]. This
feature enables multi-mode operations on the radar function, e.g., long-range
detection with low values of $f_{c}$ and $b$, and short-range detection with
high values of $f_{c}$ and $b$. The relationships between the chirp
configuration parameters and the radar performance are given by [31]:

$R_{re}=\dfrac{c_{0}}{2b},\text{and }R_{max}=\dfrac{c_{0}F_{max}}{2f_{c}},$
(2)

where $R_{re}$ and $R_{max}$ are the range resolution and maximum detection
range of the radar, respectively, $c_{0}$ is the speed of light, and $F_{max}$
is the maximum intermediate frequency bandwidth supported. The range
resolution is the minimum distance that the radar can resolve two separated
objects. The maximum detection range is the maximum distance that the AV can
detect reflected signals from the far-off objects. In (2), the values of
$c_{0}$ and $F_{max}$ are constant, while $f_{c}$ and $b$ can be configured
for specific applications, e.g., long/short-range detection [31].

In the system model, the AV can choose one of $K$ radar detection modes from
the set $\mathbf{B}=\\{B_{1},\ldots,B_{k},\ldots,B_{K}\\}$, where each mode
$B_{k}$ corresponds to a pair value of sweep bandwidth $b$ and frequency slope
$f_{c}$, i.e., $(b^{k},f_{c}^{k})$. As a result, each mode $B_{k}$ can be
illustrated by a pair value of range resolution and maximum detection range,
i.e., $(R_{re}^{k},R_{max}^{k})$ [31].

###  III-B Environment Model

At each time step, the AV can choose either radar detection mode or data
communication mode given the current information of surrounding environment.
To model the dynamic of the environment, we consider different factors that
have significant influence on the decisions of the AV. These factors are (i)
the state of the data communication channel (e.g., good or bad channel
condition), (ii) the state of the road (e.g., smooth or rough road), (iii) the
state of the weather (e.g., rainy or clear weather), (iv) the speed state of
the AV (e.g., high or low speed), and (v) the state of nearby vehicles (e.g.,
with or without nearby vehicles). The values of these factors can be obtained
by the AV’s sensing system, e.g., road friction sensor, weather station
instrument, speedometer, and cameras. With the states of these factors,
unexpected events can occur. We consider an unexpected event to be an event
that is out of the range of the AV’s camera and the event can possibly cause
collisions with the AV, e.g., a car coming from another road obscured by a
nearby moving vehicle as illustrated in Fig. 1. Also, under a bad weather
condition (e.g., rainy or foggy) and a bad road condition (e.g., rough road),
if the AV is running in a crowded area (e.g., urban areas), some unexpected
events which cannot be detected by the AV’s vision-based sensing system (e.g.,
camera) can occur [32, 33]. In this situation, the short-range radar detection
mode should be used by the AV to detect more obstructed objects.

To illustrate the dynamic of the environment, we take into account the
probability that an unexpected event occurs given current states of the
environment. Let $c\in\\{c_{1},\ldots,c_{j},\ldots,c_{max}\\}$,
$r\in\\{r_{1},\ldots,r_{j},\ldots,r_{max}\\}$,
$w\in\\{w_{1},\ldots,w_{j},\ldots,w_{max}\\}$,
$v\in\\{v_{1},\ldots,v_{j},\ldots,v_{max}\\}$, and
$m\in\\{m_{1},\ldots,m_{j},\\\ \ldots,m_{max}\\}$ denote the state of the
communication channel, the state of the road, the state of the weather, the
speed state of the AV, and the state of nearby vehicles, respectively, where
the subscript max denote the maximum number of discrete state values. For
example, the speed of the AV can be divided into different discrete state
values such as $v_{1}=15$ (km/h), $v_{2}=45$ (km/h), and $v_{3}=v_{max}=60$
(km/h).

Given a particular state of environment, which is a composite of discrete
state values $(r_{j},w_{j},v_{j},m_{j})$, let $p_{j}^{f}$ denote the
probability that an unexpected event occurs at the state value $f_{j}$ of
state $f$ ($f\in\\{r,w,v,m\\}$). For example, $p_{3}^{v}$ expresses the
probability that an unexpected event occurs at the current speed state value
of the AV is $v_{3}$, e.g., $v_{3}=60$ (km/h). Clearly, the state value of the
communication channel is not considered as a contributing factor to the
occurrence of unexpected events. It is worth noting that these values of
$p_{j}^{f}$ can be obtained from real-world statistical analysis [32, 33].
Then, we can determine the probability that an unexpected event occurs given a
particular state $(r_{j},w_{j},v_{j},m_{j})$ by using the Bayes’ theorem [34].
Let $\tau_{j}^{f}$ be the probability that state $f$ ($f\in\\{r,w,v,m\\}$) is
at state value $f_{j}$. The sum of all individual probabilities is equal to 1,
i.e.,

$\underset{j\in\mathcal{J}}{\sum}\tau_{j}^{f}=1,$ (3)

where $\mathcal{J}=\\{1,2,\ldots,j_{max}\\}$. By using the Bayes’ theorem, the
probability of an unexpected event occuring at the state
$(r_{j},w_{j},v_{j},m_{j})$ is determined by the sum of conditional
probabilities as follows [19]:

$p_{u}=\underset{f\in\mathcal{F}}{\sum}\underset{j\in\mathcal{J}}{\sum}\tau_{j}^{f}p_{j}^{f},$
(4)

where $\mathcal{F}=\\{r,w,v,m\\}$.

In addition to the factors influencing the probability that an unexpected
event occurs as explained above, we need to consider traffic density on the
road. The main reason is that in a high-density traffic area, even if the
radar system can detect objects, but if the objects are too close to each
other, the radar may not be able to differentiate them, and thus unexpected
events still can occur. Several works have considered traffic density as an
important parameter in the field of AVs [17, 35, 36]. In this work, the
traffic density is defined as the number of objects in the maximum detection
range of the AV. At time step $t$, given traffic density $\omega$, the miss
detection ratio of the AV, denoted by $p_{k}(t)$, can be calculated as
follows:

$p_{k}(t)=\dfrac{\omega-N_{k}(t)}{\omega},$ (5)

where $N_{k}(t)$ is the number of objects that the AV can detect by using the
radar detection mode $B_{k}$ at time step $t$. The miss detection ratio
represents the ability to detect closely spaced objects as well as far-off
objects of the AV. A low miss detection ratio results in good detection
ability of the AV and vice versa [41]. In an area with a high traffic density
with closely spaced objects, e.g., urban area, the AV should use the short-
range detection mode with a highly accurate range resolution to reduce the
miss detection ratio. In contrast, in an area with a low traffic density, the
AV should use the long-range detection mode with a low accurate range
resolution to detect more objects which are far-off while maintaining the
small miss detection ratio.

###  III-C System Operation and Performance Metrics

The operation of the AV system with JRC functions can be described as follows.
At each time step, the AV chooses either the radar detection mode or data
communication mode. If the AV chooses the data communication mode at time step
$t$, it then selects a data rate $R_{l}$ to transmit $\hat{d}_{l}(t)$,
$\hat{d}_{l}(t)\in\\{\hat{d}_{0},\hat{d_{1}},\ldots,\hat{d}_{max}\\}$, data
packets to the nearby BS, where $\hat{d}_{max}$ is the maximum number of data
packets that the AV can transmit to the nearby BS. As described in Section
III-A, the value of $\hat{d}_{l}(t)$ depends on the data rate $R_{l}$ of the
AV’s transmitter. The AV can transmit the maximum number of data packets
$\hat{d}_{max}$ when the channel condition is good and the AV selects the
highest data rate $R_{L}$. Otherwise, the number of packets that can be
successfully transmitted to the nearby BS decreases as the channel quality is
low. If the AV chooses the radar detection mode, it then selects a mode
$B_{k}$ to detect objects on the road. According to the selected mode $B_{k}$,
the miss detection ratio $p_{k}(t)$ can be obtained as in (5).

We evaluate the system performance by using two key metrics: (i) system
throughput and (ii) miss detection probability. The system throughput can be
determined by the total number of successfully transmitted data packets to the
nearby BS, i.e.,

$\Psi=\sum_{t=1}^{T}\hat{d}_{l}(t),$ (6)

where $T$ is the time horizon. The miss detection probability can be
calculated as the average miss detection ratio. Thus, the miss detection
probability, denoted by $P_{k}$, can be defined as follows:

$P_{k}=\dfrac{1}{T}\sum_{t=1}^{T}p_{k}(t).$ (7)

Under the dynamic and uncertain environment as well as unexpected events, it
is challenging for the AV to optimally select actions to optimize the system
performance. The problem is even more challenging when the action space of the
AV, i.e., the number of possible actions in the sets $\mathbf{R}$ and
$\mathbf{B}$, increases. Thus, all current static optimization tools in [11,
12, 13, 14, 15] may not be effective in dealing with these practical issues.
Aforementioned challenges motivate us to develop a dynamic framework which can
not only capture the dynamic of environment but also learn incomplete
information as well as uncertainty of unexpected events. Accordingly, we
formulate the dynamic optimization problem as an MDP and then design a novel
deep reinforcement learning algorithm to find the optimal policy requiring no
a priori information about the environment. Furthermore, we propose a new
transfer learning approach to leverage the learned experiences of the AV to
facilitate the training process in the new environment.

##  IV Problem Formulation

To deal with the uncertainty of the environment, we use the MDP [21] framework
to formulate the optimization problem of the system. The MDP is defined by a
tuple $<\mathcal{S},\mathcal{A},\mathcal{T},\rho>$, where $\mathcal{S}$ is the
state space, $\mathcal{A}$ is the action space, $\mathcal{T}$ is the state
transition probabilities, and $\rho$ is the immediate reward function of the
AV system. Therefore, the source and target environments in our problem are
formulated as two MDPs
$\mathcal{M}_{s}=<\mathcal{S}_{s},\mathcal{A}_{s},\mathcal{T}_{s},\rho_{s}>$
and
$\mathcal{M}_{t}=<\mathcal{S}_{t},\mathcal{A}_{t},\mathcal{T}_{t},\rho_{t}>$,
respectively. We consider that the two environments share the same state
space, action space, and immediate reward function [20]. The state transition
probabilites of the two environments are different. The MDPs can be rewritten
as $\mathcal{M}_{s}=<\mathcal{S},\mathcal{A},\mathcal{T}_{s},\rho>$ and
$\mathcal{M}_{t}=<\mathcal{S},\mathcal{A},\mathcal{T}_{t},\rho>$.

###  IV-A State Space

As explained in Section III-B, the state space of the AV system is defined as
follows:

$\mathcal{S}=\Big{\\{}(c,r,w,v,m);c\in\\{c_{1},c_{2},\ldots,c_{max}\\},r\in\\{r_{1},r_{2},\ldots,r_{max}\\},w\in\\{w_{1},w_{2},\ldots,w_{max}\\},\\\
v\in\\{v_{1},v_{2},\ldots v_{max}\\},\text{and
}m\in\\{m_{1},m_{2},\ldots,m_{max}\\}\Big{\\}},$ (8)

where $c,r,w,v$, and $m$ represent the state of the communication channel,
state of the road, state of the weather, speed state of the AV, and state of
nearby vehicles, respectively. The state of the system at time step $t$ is
defined as $s_{t}=(c,r,w,v,m)\in\mathcal{S}$.

###  IV-B Action Space

As described in Section III-C, the possible action at time step $t$ of the AV
can be defined as follows:

$a_{t}=\begin{cases}a_{k}^{r},1\leq k\leq K,&\text{if a radar detection mode
is selected}\\\ a_{l}^{d},1\leq l\leq L,&\text{if a data communication mode is
selected}\\\ \end{cases}$ (9)

where $a_{k}^{r}$ is the action when the AV selects the radar detection mode
$B_{k}$ and $a_{l}^{d}$ is the action when the AV selects the data
communication mode to transmit data at the rate $R_{l}$. Thus, the action
space of the AV can be defined as
$\mathcal{A}=\Big{\\{}a_{t};a_{t}\in\\{a_{k}^{r},a_{l}^{d}\\}\Big{\\}}$.

###  IV-C State Transition Probabilities

The state transition probabilities of the source and target environments are
defined as follows:

$\mathcal{T}_{s}=P(s^{\prime}|p_{u,s},\omega_{s},a),\text{and
}\mathcal{T}_{t}=P(s^{\prime}|p_{u,t},\omega_{t},a),$ (10)

where $p_{u,s}$ and $p_{u,t}$ are the probabilities that an unexpected event
occurs as defined in (4) of the source environment and target environment,
respectively. $\omega_{s}$ and $\omega_{t}$ are the traffic density values as
defined in (5) of the source environment and target environment, respectively.
$s^{\prime}$ is the new state observed by the AV after taking action $a$. Note
that the state transition probabilities illustrate the dynamics of
environments and they are unknown to the AV.

###  IV-D Immediate Reward Function

At time step $t$ given state $s_{t}$, the AV selects an action $a_{t}$ and
receives an immediate reward $\rho_{t}$. The immediate reward $\rho_{t}$ is
designed to maximize jointly the data transmission and radar detection
functions of the AV. Thus, the immediate reward can be defined as follows:

$\rho_{t}(s_{t},a_{t})=\begin{cases}\rho_{1}\hat{d}_{l}(t),\text{ if
$a_{t}=a_{l}^{d}$, given ${\mathit{\bar{X}}}$},\\\ -\rho_{2},\mbox{\ if
$a_{t}=a_{l}^{d}$, given $\mathit{X}$},\\\ -\rho_{3},\mbox{\ if
$a_{t}=a_{k}^{r}$, given $\mathit{\bar{X}}$},\\\ \rho_{4}(1-p_{k}(t)),\mbox{\
if $a_{t}=a_{k}^{r}$, given $\mathit{X}$},\end{cases}$ (11)

where $(\rho_{1},\rho_{2},\rho_{3},\rho_{4})$ are weighting factors,
$\mathit{X}$ denotes the occasion when an unexpected event occurs with
probability $p_{u}$ as defined in (4) and $\mathit{\bar{X}}$ denotes the
occasion when no unexpected event happens. $\hat{d}_{l}(t)$ is the number of
data packets successfully transmitted to a nearby BS, $p_{k}(t)$ is the miss
detection ratio as defined in (5). It is worth noting that, weighting factors
$(\rho_{1},\rho_{2},\rho_{3},\rho_{4})$ can be adjusted to the requirements of
different scenarios. For example, if the AV requires a highly accurate radar
detection and the data communication function is secondary, $\rho_{3}$ and
$\rho_{4}$ can be set at large values while $\rho_{1}$ and $\rho_{2}$ can be
set at small values.

The immediate reward function of the AV in (11) can be described as follows.
When there is no unexpected event and the AV selects a data communication mode
with data rate $R_{l}$, the AV receives a positive reward that is proportional
to the number of data packets successfully transmitted to the BS (the first
condition in (11)). When there is an unexpected event and the AV selects a
data communication mode, it receives a large negative reward, denoted as
$-\rho_{2}$ (the second condition in (11)). When there is no unexpected event,
if the AV selects a radar detection mode, it receives a negative reward,
denoted as $-\rho_{3}$ (the third condition in (11)). When there is an
unexpected event, if the AV selects a radar detection mode, it receives a
reward that is inversely proportional to the miss detection ratio $p_{k}(t)$
(the last condition in (11)). Note that if the AV does not use the radar
detection function for early detection at the beginning of current time slot,
it may still obtain the occurrence of the unexpected event at the end of the
time slot by using its camera and LiDAR sensors.

###  IV-E Optimization Formulation

We first formulate the optimization problem of the AV in the source
environment. The details of how the experiences of the AV can be transferred
to the target environment are described in Section VI. We formulate a dynamic
optimization problem to obtain an optimal policy, denoted by $\pi^{*}$. Given
the current system state, i.e., state of the channel, state of the road, state
of the weather, state of the speed, and state of nearby vehicles, the optimal
policy determines optimal actions to maximize the average expected return,
i.e., the average discounted long-term reward, of the AV system. The
optimization problem is then expressed as follows:

$\max_{\pi}\
R(\pi)=\lim_{T\to\infty}\dfrac{1}{T}\mathbb{E}\Big{\\{}\sum_{t=0}^{T}{\gamma^{t}\rho_{t}(\pi)}\Big{\\}},$
(12)

where $R(\pi)$ is the average expected return under the policy $\pi$,
$\rho_{t}(\pi)$ is the immediate reward under policy $\pi$ at time step $t$ as
defined in (11), $T$ is the time horizon, and $\gamma\in(0,1)$, is the
discount factor. The optimal policy $\pi^{*}$ will allow the AV to make
optimal decisions at any state $s_{t}$, i.e., $a^{*}_{t}=\pi^{*}(s_{t})$. In
practice, it is impossible for the AV to obtain the complete model of the MDP,
i.e., state transition probabilities of the MDP are unknown by the AV in
advance, and thus model-free reinforcement learning approaches are more
suitable for finding the optimal policy for the optimization problem
formulated as in (12). In the next section, we propose to use Q-learning [22]
to obtain the optimal policy for the above optimization problem. After that,
we propose a method based on deep reinforcement learning, namely Double Deep
Q-Network (DDQN) [26], not only to overcome limitations of Q-learning but also
to learn effectively the optimal policy.

##  V Deep Reinforcement Learning Approach For Automotive Joint Radar-
Communication Control

###  V-A JRC Optimal Policy With Q-learning

We first propose to use Q-learning algorithm [22] to obtain the optimal policy
for the AV system in the source environment. In particular, Q-learning
algorithm uses a Q-table as a mapping function between states and actions.
Each state-action value, i.e., Q-value, in the Q-table is iteratively updated
by using Bellman equation as follows:

$Q(s_{t},a_{t})=Q(s_{t},a_{t})+\alpha_{t}\Big{[}\rho_{t}+\gamma\max_{a}Q(s_{t+1},a)-Q(s_{t},a_{t})\Big{]},$
(13)

where $s_{t}=(c_{t},r_{t},w_{t},v_{t})$, $a_{t}=(a^{r}_{k,t},a^{d}_{l,t})$,
and $\alpha_{t}$ is the state, action, and learning rate at the time step $t$,
respectively. When the Q-table is fully updated, the optimal policy for the AV
can be obtained by selecting the highest state-action value in the Q-table at
each given state, i.e.,
$\pi^{*}(s_{t})=\operatorname*{\arg\\!\max}_{a}Q^{*}(s_{t},a)$ where
$Q^{*}(s_{t},\cdot)$ is the state values of the Q-table when the Q-table is
fully updated. To guarantee the convergence, the learning rate $\alpha_{t}$ is
deterministic, non-negative, and satisfies (14) [22]:

$\alpha_{t}\in(0,1],\sum_{t=1}^{\infty}\alpha_{t}=\infty,\mbox{and
}\sum_{t=1}^{\infty}(\alpha_{t})^{2}<\infty.$ (14)

Under the conditions in (14), the Q-learning algorithm converges to the
optimal policy with probability one [22]. By using a table as a simple data
structure to maintain a mapping function between states and actions,
Q-learning usually suffers the curse-of-dimensionality problem [25]. For
example, when the state and action spaces increase, the size of the Q-table
increases quadratically, and thus the time for updating the whole Q-table is
longer, meaning that the time for obtaining an optimal policy is also larger.
To overcome the limitations of Q-learning, deep reinforcement learning with
nonlinear function approximation, i.e., neural networks, has recently
introduced to be a highly-effective solution to overcome limitations of
conventional Q-table [24]. Thus, in the following, we introduce the deep
reinforcement learning-based approach to find the optimal policy for the AV.

###  V-B Proposed Deep Reinforcement Learning Approach With Double Deep
Q-Network for Optimal JRC Policy

Figure 3: Double Deep Q-Network (DDQN) approach for the AV.

To overcome the limitations of conventional Q-learning [22], Deep Q-Network
(DQN) [23] has recently introduced. DQN utilizes (i) an online deep neural
network (online network) as a nonlinear function approximation, (ii) an
experience replay memory as a buffer of historical data, and (iii) a second
target deep neural network (target network) as a periodic copy of the online
network. The use of the online network as a nonlinear function approximation
helps the DQN to overcome the problem related to the large state-action spaces
of Q-learning. In addition, an experience replay memory is utilized to store
past experiences to speed up the process of updating weights in the online
network. Finally, the target network is periodically replaced by the online
network and used for stabilizing the training process. In this work, we
develop a mechanism based on a variation of DQN that is Double DQN (DDQN) [26]
not only to find the optimal policy for the AV but also to improve the
stability of the training process. In particular, the DDQN and DQN algorithms
share the same neural networks’ architecture. However, the difference between
the DDQN and DQN algorithms is that the calculation of target Q-values in the
DDQN algorithm can avoid bias in the action selection [26]. An illustration of
our proposed approach is shown in Fig. 3. In our proposed approach, the DDQN-
empowered AV interacts with the environment through its simulator in the
training process. When the training process completes in the simulator, the AV
can operate in real-world settings with its trained DDQN model.

1 Inputs:

2 $\mathcal{D}^{re}$: initialize replay memory with capacity
$|\mathcal{D}^{re}|$,

3 $\theta$: weights for the initial online network are randomly initialized,

4 $\theta^{\prime}$: weights for the target network are randomly initialized,

5 $\upsilon$: frequency at which to update target network,

6 for  _episode = 1, E_ do

7 Initialize a new state $s_{t}\in\mathcal{S}$

8 for  _t = 1, T_ do

9 With probability $\epsilon$, select a random action $a_{t}$, otherwise
select $a_{t}=\max_{a}Q(s_{t},a;\theta_{t})$

10 Execute action $a_{t}$ and observe reward $\rho_{t}$ and next state
$s_{t+1}$

11 Replace $s_{t}\leftarrow s_{t+1}$

12 Store transition $(s_{t},a_{t},\rho_{t},s_{t+1})$ in $\mathcal{D}^{re}$

13 Sample random mini-batch of $N_{b}$ transitions
$(s_{j},a_{j},\rho_{j},s_{j+1})$ from $\mathcal{D}^{re}$

14 Set $y_{j}^{DDQN}$ as in (15): $y_{j}^{DDQN}=\rho_{j}+\gamma
Q(s_{j+1},\operatorname*{\arg\\!\max}_{a}Q(s_{j+1},a;\theta_{t}),\theta_{t}^{\prime})$

15 Perform a gradient descent step on
$(y_{j}^{DDQN}-Q(s_{j},a_{j},\theta))^{2}$ according to (17), (20), and (21)

16 if  _$t\mbox{ mod }\upsilon=0$_ then

17 $\theta^{\prime}\leftarrow\theta$

18 end if

19

20 end for

21

22 end for

23Outputs: $\pi^{*}(s)=\operatorname*{\arg\\!\max}_{a}Q^{*}(s,a;\theta)$.

Algorithm 1 Double Deep Q-Network-based Algorithm For Finding Optimal Policy

The detailed DDQN algorithm is shown in Algorithm 1 and can be described as
follows. The inputs of the network include channel state, road state, weather
state, speed state, and surrounding vehicle state. The output is the learned
Q-values. The experience replay memory, denoted as $\mathcal{D}^{re}$, with
capacity $|\mathcal{D}^{re}|$, the weights of the online network and target
network, denoted as $\theta$ and $\theta^{\prime}$, are randomly initialized,
i.e., lines 2-5 in Algorithm 1. At each training iteration, the AV selects an
action $a_{t}$ following an $\epsilon$-greedy policy, i.e., selecting $a_{t}$
randomly with probability $\epsilon$, and otherwise selects
$a_{t}=\max_{a}Q(s_{t},a;\theta)$ (line 9). The AV executes the action $a_{t}$
and obtains a reward $\rho_{t}$ and the next state $s_{t+1}$ (line 10). The
tuple of $(s_{t},a_{t},\rho_{t},s_{t+1})$ is stored in $\mathcal{D}^{re}$.
Next, a mini-batch of transitions $(s_{j},a_{j},\rho_{j},s_{j+1})$ is sampled
from $\mathcal{D}^{re}$ to calculate the target Q-values (line 13). The target
$y_{j}^{DDQN}$ at time step $t$ is calculated as follows:

$y_{j}^{DDQN}=\rho_{j}+\gamma
Q(s_{j+1},\operatorname*{\arg\\!\max}_{a^{\prime}}Q(s_{j+1},a^{\prime};\theta_{t}),\theta_{t}^{\prime}).$
(15)

The DDQN algorithm then updates the online network by minimizing the following
loss function at time step $t$:

$\begin{split}L_{t}(\theta_{t})=\mathbb{E}_{(s_{j},a_{j})\sim\mathcal{D}^{re}}\left[(y_{j}^{DDQN}-Q(s_{j},a_{j};\theta_{t}))^{2}\right].\end{split}$
(16)

Differentiating the loss function in (16) with respect to the weights of the
online network, we have the following gradient:

$\nabla_{\theta_{t}}L_{t}(\theta_{t})=\mathbb{E}_{(s_{j},a_{j},s_{j+1})\sim\mathcal{D}^{re}}\Bigl{[}\Bigl{(}y_{j}^{DDQN}-Q(s_{j},a_{j};\theta_{t})\Bigr{)}\nabla_{\theta_{t}}Q(s_{j},a_{j};\theta_{t})\Bigr{]}.$
(17)

In practice, the gradient in (17) can be efficiently calculated by using
Stochastic Gradient Descent (SGD) algorithm [38]. In general, the loss
function in (16) is decayed by a sum over training examples of some per-
example loss functions. For instance, the negative conditional log-likehood of
the training data can be expressed as:

$J_{t}(\theta_{t})=\mathbb{E}_{(s_{j},a_{j},s_{j+1})\sim\mathcal{D}^{re}}L_{t}(\theta_{t})=\dfrac{1}{|\mathcal{D}^{re}|}\sum_{j=1}^{|\mathcal{D}{re}|}L_{t}((s_{j},a_{j},s_{j+1}),\theta_{t}).$
(18)

For the additive cost function in (18), the required gradient descent can be
computed in the following way:

$\nabla_{\theta_{t}}J_{t}(\theta_{t})=\dfrac{1}{|\mathcal{D}^{re}|}\sum_{j=1}^{|\mathcal{D}^{re}|}\nabla_{\theta}L_{t}((s_{j},a_{j},s_{j+1}),\theta_{t}).$
(19)

The computation cost for the operation in (19) is $(O|\mathcal{D}^{re}|)$.
Thus, as the size $|\mathcal{D}^{re}|$ of the replay memory increases, the
time to take a single gradient step becomes prohibitively long. By using SGD,
we can uniformly sample a mini-batch of experiences from the replay memory
$\mathcal{D}^{re}$ at each step of the algorithm. In general, the mini-batch
size can be set to be relatively small number of experiences, e.g., from one
to a few hundreds. As such, the training time is significantly reduced. The
estimation of the gradient using SGD is then calculated as follows:

$g=\dfrac{1}{N_{b}}\nabla_{\theta_{t}}\sum_{j=1}^{N_{b}}L_{t}(\theta_{t}),$
(20)

where $N_{b}$ is the mini-batch size. The SGD algorithm then updates the
online network’s weights as follows:

$\theta_{t+1}\leftarrow\theta_{t}-\nu g,$ (21)

where $\nu$ is the learning rate of the SGD algorithm. After every $\upsilon$
steps, the target network’s weights $\theta^{\prime}$ are replaced by the
online network’s weights $\theta$ (line 16-18 in Algorithm 1). The target
network weights remain unchanged between individual updates.

###  V-C Complexity Analysis for the Double Deep Q-Network Algorithm

In this work, the DDQN algorithm uses two deep neural networks that are online
network and target network. However, we only consider the computations of the
online network because the target network is a periodically replaced by the
online network and thus it does not require significantly additional
computations. The online network consists of an input layer $L_{0}$, two
fully-connected layers $L_{1}$ and $L_{2}$, and an output layer $L_{3}$. Let
$|L_{i}|$ denote the size, i.e., the number of neurons, of layer $L_{i}$. The
complexity of the online network can be formulated as
$|L_{0}||L_{1}|+|L_{1}||L_{2}|+|L_{2}||L_{3}|$. At each training step, a
number of training samples, i.e., transitions, are randomly taken from the
replay memory and fed to the online network for training. Thus, the total
complexity of the training process is
$O\Big{(}TN_{b}(|L_{0}||L_{1}|+|L_{1}||L_{2}|+|L_{2}||L_{3}|)\Big{)}$, where
$N_{b}$ is the size of the training batch and $T$ is the total number of
training iterations. In our paper, the size of $L_{0}$ is the number of state
features, therefore $|L_{0}|=5$. The hidden layers $L_{1}$ and $L_{2}$ have 24
neurons each. The output layer consists of $|\mathcal{A}|$ neurons in which
each neuron corresponds to an action of the AV as explained in Section IV-B.
Clearly, the architecture of the online network is simple. Thus, it can be
deployed at the AV system which is usually equipped with sufficient computing
resources. In the simulation, we show that our proposed algorithm can converge
to the optimal policy much faster than the conventional Q-learning algorithm.

##  VI Transfer Learning Approach For Accelerating Learning Process

In the previous section, we show that the optimal policy of the source
environment can be obtained by using the proposed DDQN algorithm. The optimal
policy then can be applied to the target environment with an assumption that
the two environments share the same data distribution, i.e., state transition
probabilities [27, 20]. However, this assumption may fail in practice due to
the AV’s mobility, i.e., it can often travel from an environment to another
one with different state transition probabilities. For example, the AV can
travel from a rural area with a low traffic density, to an urban area with a
high traffic density. In such a case, factors such as road condition and
weather condition may have different distributions in different environments.
Therefore, to achieve a robust and scalable AV system with JRC functions in
practice, we need a method that can quickly learn the new optimal policy when
the AV moves from one environment to another one. This requirement motivates
us to develop the transfer learning approach that can leverage learned
experiences to facilitate the learning process in the new environment.

Our considered transfer learning technique can be expressed as follows. Given
two different but related environments that are source environment, modelled
with MDP $\mathcal{M}_{s}$, and target environment, modelled with MDP
$\mathcal{M}_{t}$, the transfer learning process aims to learn an optimal
policy $\pi_{t}^{*}$ for the target environment by leveraging exterior
information $\mathcal{D}_{s}$ from $\mathcal{M}_{s}$ as well as interior
information $\mathcal{D}_{t}$ from $\mathcal{M}_{t}$:

$\pi_{t}^{*}=\operatorname*{\arg\\!\max}_{\pi}\mathbb{E}_{(s\sim\mu_{t},a\sim\pi)}\Big{[}Q_{\pi}(s,a)\Big{]},\text{where
}\pi=\phi(\mathcal{D}_{s}\sim\mathcal{M}_{s},\mathcal{D}_{t}\sim\mathcal{M}_{t}).$
(22)

$\pi$ is the function mapping from states to actions in the target environment
and is approximated using deep neural networks trained on both
$\mathcal{D}_{s}$ and $\mathcal{D}_{t}$ [20].

Several transfer learning approaches have been proposed to solve (22) [20]. In
autonomous driving, however, collecting exterior information $\mathcal{D}_{s}$
can be costly and intractable [28]. Thus, we need an algorithm that can
efficiently accelerate the learning with limited amount of exterior
information. In this paper, we develop a transfer learning approach for the AV
system, namely Transfer Learning with Demonstrations (TLwD) [30]. The key idea
of TLwD is to initially pre-train on demonstration data $\mathcal{D}_{s}$
using a combination of temporal difference (TD) and supervised losses. After
pre-training, the TLwD algorithm keeps updating its neural networks with a
combination of demonstration data $\mathcal{D}_{s}$ and self-generated data
$\mathcal{D}_{t}$. It is worth noting that both $\mathcal{D}_{s}$ and
$\mathcal{D}_{t}$ are collections of experiences, i.e., tuples of state,
action, reward, and next state, of the AV system in the source and target
environments, respectively. Our proposed TLwD algorithm is illustrated in Fig.
4.

Figure 4: Transfer Learning with Demonstrations (TLwD) approach for the AV.

1Input:

2 $\mathcal{D}^{re}$: initialized with demonstration dataset,

3 $\theta$: weights for the initial behavior network (random),

4 $\theta^{\prime}$: weights for the target network (random),

5 $\upsilon$: frequency at which to update target net,

6 $T_{pre}$: number of pre-training gradient updates.

7 for  _$t\in\\{1,2,\ldots,T_{pre}\\}$_ do

8 Sample a mini-batch of $N_{b}$ transitions from $\mathcal{D}^{re}$ with
priotization as in (23)

9 Calculate loss $J(Q)$ in (25) using target network

10 Perform a gradient descent step to update $\theta$

11 if  _$t\mbox{ mod }\upsilon=0$_ then

12 $\theta^{\prime}\leftarrow\theta$

13 end if

14

15 end for

16for  _$t\in\\{1,2,\ldots,T\\}$_ do

17 Sample action from behavior policy $a_{t}\sim\pi^{\epsilon Q_{\theta}}$

18 Execute $a_{t}$ and observe $(s_{t+1},\rho_{t})$

19 Store $(s_{t},a_{t},s_{t+1},\rho_{t})$ into $\mathcal{D}^{re}$, overwriting
the oldest self-generated transition if over capacity

20 Sample a mini-batch of $N_{b}$ transitions from $\mathcal{D}^{re}$ with
priotization as in (23)

21 Calculate loss $J(Q)$ in (25) using target network

22 Perform a gradient descent step update $\theta$

23 if  _$t\mbox{ mod }\upsilon=0$_ then

24 $\theta^{\prime}\leftarrow\theta$

25 end if

26 $s_{t}\leftarrow s_{t+1}$

27 end for

Outputs: $\pi^{*}_{t}(s)=\operatorname*{\arg\\!\max}_{a}Q^{*}(s,a;\theta)$

Algorithm 2 Transfer Learning with Demonstrations

Details of our proposed TLwD approach is illustrated in Algorithm 2 and can be
described as follows. First, the replay memory $\mathcal{D}^{re}$ is
initialized with demonstration data $\mathcal{D}_{s}$, the weights of online
network and target network are randomly initialized (line 2-6 in Algorithm 2).
The inputs of the online network include channel state, road state, weather
state, speed state, and state of nearby vehicles. The output is learned
Q-values. The weights of the online network and target networks are pre-
trained as follows (line 7-14). At each pre-train step, a mini-batch of
transitions is sampled from $\mathcal{D}^{re}$ with prioritized-experience
replay scheme [29]. In particular, the probability of sampling transition $i$
can be calculated as follows:

$P(i)=\dfrac{p_{i}^{\alpha}}{\sum_{e}^{e\in|\mathcal{D}^{re}|}p_{e}^{\alpha}},$
(23)

where $p_{i}>0$ is the priority of transition $i$ and $|\mathcal{D}^{re}|$ is
the size of the replay memory. The exponent $\alpha$ determines how much
prioritization is used, with $\alpha=0$ corresponding to the uniform case,
i.e., transition $i$ is sampled randomly. The priority of transition $i$ can
be calculated by $p_{i}=|\zeta_{i}|+\varepsilon$, where $\zeta_{i}$ is the
Temporal Difference (TD) error of transition $i$, $\varepsilon$ is a small
positive constant that prevents the edge-case of transitions not being
revisited once their TD error is equal to 0. With the prioritized-experience
replay mechanism, the TLwD algorithm can sample important transitions in the
demonstration data $\mathcal{D}_{s}$ and thus can significantly reduce the
size of $\mathcal{D}_{s}$ [29, 30]. To account for the change in the
distribution, updates to the network are weighted with importance sampling
weights as follows:

$w_{i}=\left(\dfrac{1}{|\mathcal{D}^{re}|}\cdot\dfrac{1}{P(i)}\right)^{\beta},$
(24)

where $|\mathcal{D}^{re}|$ is the size of the replay buffer and $\beta$
controls the amount of importance sampling with no importance sampling when
$\beta=0$ and full importance sampling when $\beta=1$. $\beta$ is annealed
linearly from $\beta_{0}$ to 1. By using the mini-batch sampled with
prioritization, the online neural network weights are updated by minimizing
the overall loss $J(Q)$, which is the combination of the DDQN loss, the margin
classification loss, multi-step TD error loss, and L2 regulation loss. The
overall loss $J(Q)$ is defined as:

$J(Q)=J_{DDQN}(Q)+\lambda_{1}J_{n}(Q)+\lambda_{2}J_{E}(Q)+\lambda_{3}J_{L2}(Q),$
(25)

where the component losses are calculated as follows. First, the DDQN loss is
calculated as:

$\displaystyle J_{DDQN}(Q)=\left(\rho_{t}+\gamma
Q(s_{t+1},a_{t}^{max};\theta^{\prime})-Q(s,a;\theta)\right)^{2},$ (26)

where $\theta$ and $\theta^{\prime}$ are the parameters of the online network
and target network, respectively, and
$a_{t}^{max}=\operatorname*{\arg\\!\max}_{a}Q(s_{t+1},a;\theta)$. The multi-
step TD loss $J_{n}(Q)$ is calculated as:

$\displaystyle
J_{n}(Q)=\left(R_{t}^{(n)}+\gamma^{n}\max_{a}Q(s_{t+n},a;\theta^{\prime})-Q(s_{t},a_{t};\theta)\right)^{2},$
(27)

where $R_{t}^{(n)}$ is n-step return from a given state $s_{t}$ which can be
calculated as:

$\displaystyle R_{t}^{(n)}=\sum_{k=0}^{n-1}\gamma^{k}\rho_{t+k}.$ (28)

The margin classification loss between data sampled from $\mathcal{D}_{s}$ and
data sampled from $\mathcal{D}_{t}$ is defined as:

$\displaystyle J_{E}(Q)=\max_{a}[Q(s,a)+\Gamma(a_{E},a)]-Q(s,a_{E}),$ (29)

where $a_{E}$ is the action the expert demonstrator, i.e., DDQN with
$\mathcal{D}_{t}$, takes in state $s$ and $\Gamma(a_{E},a)$ is a margin
function that is equal to 0 when $a=a_{E}$ and equal to 1 otherwise. Finally,
the L2 regularization loss $J_{L2}(Q)$ is applied to the weights and biases of
the neural network to prevent over-fitting on the relatively small
demonstration dataset. In addition, $\lambda_{1}$, $\lambda_{2}$, and
$\lambda_{3}$ are weighting parameters of the multi-step TD loss,
classification loss, and regulation loss, respectively.

When the pre-train process completes, the AV starts training process in the
target environment (line 15-26 in Algorithm 2). In particular, the training
processes of the TLwD and DDQN algorithms have three main differences that are
(i) the structure of replay memory $\mathcal{D}^{re}$, (ii) the prioritized-
experience replay mechanism, and (iii) the combined losses $J(Q)$. In the next
section, we will show the advantages of the three above techniques over the
DDQN algorithm in terms of convergence rate.

##  VII Performance Evaluation

###  VII-A Parameter Settings

In this paper, we consider a scenario in which an AV travels from the source
environment to the target environment as shown in Fig. 1. We consider our
problem to be episodic in which the maximum number of time steps in each
episode is 300. The detailed settings for the joint radar-communications
functions and environment can be described as follows.

####  VII-A1 Joint Radar-Communication Functions Settings

a) Data communication function: At time step $t$, if the AV chooses to use the
data communication function, it can transmit data at a low data rate or a high
data rate, i.e., $a_{t}\in\\{a_{1}^{d},a_{2}^{d}\\}$. As described in Section
III-A, when the channel condition is good, the AV can successfully transmit
$\hat{d}_{t}(t)=4$ or $\hat{d}_{t}(t)=2$ packets to a nearby BS by using high
data rate transmission or low data rate transmission, respectively. However,
when the channel condition is bad, the AV can successfully transmit
$\hat{d}_{t}(t)=0$ or $\hat{d}_{t}(t)=2$ packets to a nearby BS by using high
data rate transmission or low data rate transmission, respectively. We assume
that the communication channel is at a good condition (e.g., low interference)
with probability $\tau^{c}_{0}$. Therefore, the channel is at a bad condition
(e.g., high interference) with probability $1-\tau^{c}_{0}$.

b) Radar detection function: For the radar detection function, we consider
parameters obtained from a real automotive radar sensor from [31]. In
particular, at a time step $t$, if the AV uses the radar detection function,
it can choose the long-range detection mode or short-range detection mode,
i.e., $a_{t}\in\\{a_{1}^{r},a_{2}^{r}\\}$. If the AV chooses the long-range
detection mode, i.e., $a_{t}=a_{2}^{r}$, its radar operates with the values of
sweep bandwidth and frequency slope
$(b^{1},f_{c}^{1})=(300\mbox{MHz},10\mbox{MHz/$\mu s$})$. According to (2),
these values of $(b^{1},f_{c}^{1})$ are equivalent to the maximum detection
range and range resolution $(R_{max}^{1},R_{re}^{1})=(225m,0.5m)$,
respectively. In other words, the AV can detect objects that are in a range of
$225m$ and can distinguish two object as long as the distance between two
objects is more than $0.5m$. On the other hand, if the AV chooses the short-
range detection mode, its radar operates with
$(b^{2},f_{c}^{2})=(750\mbox{MHz},15\mbox{MHz/$\mu s$})$. These values of
$(b^{2},f_{c}^{2})$ are equivalent to $(R_{max}^{2},R_{re}^{2})=(45m,0.2m)$.
It can be observed from the values of $(R_{max},R_{re})$ that long-range
detection mode results in a low-quality resolution (i.e., high value of
$R_{re}$), meaning that it is more difficult to detect two closely spaced
objects. In contrast, short-range detection can detect objects in a short
distance, i.e., $45m$, with high-quality resolution, i.e., $0.2m$. A summary
of parameters used in simulation is shown in Table I.

TABLE I: Parameter setting Parameter | Source Environment | Target Environment
---|---|---
$\gamma$ | 0.99
$\epsilon$-greedy | $(1.0\rightarrow 0.01)$
$N_{b}$ | 64
$(\lambda_{1},\lambda_{2},\lambda_{3})$ | $(1.0,1.0,10^{-5})$
$(|\mathcal{D}_{s}|,|\mathcal{D}^{re}|)$ | $(20000,50000)$
$(T_{pre},\upsilon)$ | $(10000,300)$
$(\alpha,\beta)$ | $(0.4,0.6\rightarrow 1.0)$
$(b^{1},f_{c}^{2})$ | $(300\mbox{MHz},10\mbox{MHz/$\mu$s)}$
$(b^{2},f_{c}^{2})$ | $(750\mbox{MHz},15\mbox{MHz/$\mu$s)}$
$(\rho_{1},\rho_{2},\rho_{3},\rho_{4})$ | $(1,100,1,50)$
Car’s length | $\mathcal{N}(4.62,0.18)$
Car’s width | $\mathcal{N}(1.92,0.08)$
Pedestrian’s length | $\mathcal{N}(0.73,0.085)$
Pedestrian’s width | $\mathcal{N}(0.68,0.055)$
$p_{0}^{f}$ $(f\not\in\\{v,w\\})$ | $0.007$ | $0.004$
$p_{1}^{f}$ ($f\not\in\\{v,w\\})$ | $0.07$ | $0.04$
$(p_{1}^{v},p_{0}^{v})$ | $(0.1,0.05)$ | $(0.1,0.05)$
$(p_{1}^{w},p_{0}^{v})$ | $(0.046,0.005)$ | $(0.046,0.005)$
$\tau^{f}_{0}$ | $0.3$ | $0.7$
$\omega$ | $54$ | $27$

####  VII-A2 Environment Settings

We set the state of the communication channel, state of the road, state of the
weather, speed state, and the state of nearby vehicles as $c\in\\{0,1\\}$,
$r\in\\{0,1\\}$, $w\in\\{0,1\\}$, $m\in\\{0,1\\}$, and $v\in\\{0,1\\}$,
respectively. In particular, $c=1$, $r=1$, $w=1$, $v=1$, and $m=1$ represent
unfavorable conditions, i.e. bad channel condition, rough road, rainy weather,
high speed of the AV, and with nearby vehicles, respectively. In contrast,
$c=0$, $r=0$, $w=0$, $v=0$, and $m=0$ represent favorable conditions, i.e.,
good channel condition, smooth road, good weather, low speed of the AV, and
without nearby vehicles, respectively. Note that the generalization of the
states beyond 0 and 1 is straightforward. To model the dynamic of environment,
the probabilities that an unexecpted event occurs at the given speed state and
weather state, denoted as $p_{j}^{v}$ and $p_{j}^{w}$, respectively, in (4)
are taken from the real-world data in [32, 33], and other probabilities are
assumed to be pre-defined. The probabilities that an unexpected event occurs
at low speed and high speed of the AV, denoted as $p_{0}^{v}$ and $p_{1}^{v}$,
respectively, are taken from [32] in which if the AV’s speed exceeds 60 km/h,
the AV’s speed is high and otherwise the AV’s speed is low. Specifically, the
values of $p_{0}^{v}$ and $p_{1}^{v}$ are set to be 0.005 and 0.1,
respectively. Rain can be considered a common unfavorable weather state, and
as in [33], we set $p_{1}^{w}=0.046$ and $p_{0}^{w}=0.005$.

We consider the traffic density and the sizes of objects on the road and the
sidewalk are realistic parameters based on nuScenes dataset [35]. nuScenes is
a large public dataset that provides a benchmark from measurements of camera,
radar and LiDAR sensors on autonomous vehicles. nuScenes dataset contains
well-classified 23 object classes recorded from 242km traveled in Boston and
Singapore. From [35], we extract distributions of the traffic density, length,
and width of the objects and input these parameters into our simulation. In
particular, we consider two most common types of objects in nuScenes that are
car and pedestrian. The average values of the number of cars and the number of
pedestrians in a time step are 20 and 7, respectively [35]. Therefore, we set
the average value of traffic density, denoted as $\omega$, at 27 objects
within the detection range of $(45\times 45)$ meters in cases the AV uses the
short-range radar detection mode. In the case that the AV uses the long-range
radar detection mode, the average value of traffic density is set at
$27\times\frac{225}{45}\times\frac{225}{45}=675$ objects within the detection
range of $(225\times 225)$ meters. In the rest of the paper, we denote
$\omega$ as the average number of objects in the detection range of $(45\times
45)$ meters for the sake of simplicity, and the value of $\omega$ with the
long-range radar detection mode can be calculated as mentioned above. In
addition, the car’s length, car’s width, pedestrian’s length, and pedestrian’s
width are fitted as the normal distributions that are
$\mathcal{N}(4.62,0.18)$, $\mathcal{N}(1.92,0.08)$, $\mathcal{N}(0.73,0.085)$,
and $\mathcal{N}(0.68,0.055)$, respectively [35].

At each time step $t$ in the simulation, we generate $\omega$ objects that are
randomly spaced in the detection range of $(R_{max}\times R_{max})$ meters of
the AV. In the simulation, these parameters are represented as a set of 2D
arrays and the AV can only obtain these parameters if it uses the radar
detection mode in that time step. In addition, we simulate an unexpected event
occurs with probability $p_{u}$ as defined in (4).

####  VII-A3 Neural Networks Settings

For the DDQN algorithm, we adopt parameters based on the common settings for
designing neural networks [23, 38]. Specifically, two fully-connected hidden
layers are implemented together with input layer and output layer. The size of
the hidden layer is 24. The mini-batch size is set at 64. The size of the
experience replay memory is 50,000, and the target network is updated every
300 iterations. For the TLwD parameters, we use the same parameters as in
[30]. The weighting factors between losses in (25) are set at
$(\lambda_{1},\lambda_{2},\lambda_{3})=(1,1,10^{-5})$. The number of pre-train
steps is 10,000. The size of demonstration data and experience replay memory
are 20,000 and 50,000, respectively. The parameters used in the simulation are
summarized in Table I.

####  VII-A4 Transfer Learning Setting

In the transfer learning setting, we first let the AV obtain an optimal policy
and demonstration data of the source environment. Note that the information of
source environment can be pre-computed by other AVs and it can be stored at an
edge server. After that, we let the AV continue learning through updating its
neural networks’ weights in the target environment. The differences between
the source and target environments are shown in Table I. We evaluate our
proposed transfer learning algorithm by comparing with the two baseline
schemes that are DDQN and Direct Policy Reuse. The Direct Policy Reuse (DPR)
is a mechanism in which the policy learned from the source environment is
directly used in the target environment [27, 39]. Note that with the DDQN
algorithm, the AV learns the environment from scratch, i.e., the weights of
neural networks are initialized randomly. The reason that we use the DDQN
algorithm as a baseline scheme is to evaluate the advantages of our proposed
transfer learning approach in the transfer learning setting. In the rest of
the paper, we use the term DDQN (without TL) to denote the results obtain by
the DDQN algorithm without using any transfer learning technique, i.e., the AV
with DDQN algorithm learns the new environment from scratch.

In the next subsection, we evaluate system performance in terms of average
reward, system throughput, and miss detection probability. The average reward
of an episode can be calculated by averaging all the immediate rewards. The
system throughput and miss detection probability are calculated as in (6) and
(7), respectively.

###  VII-B Simulation Results

####  VII-B1 Transfer Learning Experiment

(a)

(b)

Figure 5:  (a) Average reward of the DDQN and Q-learning algorithms in the
source environment, and (b) average reward of the TLwD, DPR, DDQN, and
Q-learning in the target environment.

First, we let the AV obtain the optimal policy in the source environment by
using the DDQN and Q-learning algorithms. In Fig. 5(a), we show the average
reward values of the DDQN and Q-learning algorithms in the source environment.
The result shows that with the same $\epsilon$-greedy scheme, the Q-learning
algorithm is unable to obtain the optimal policy while the DDQN algorithm can
achieve the optimal policy after approximate 150 episodes. This means that the
Q-learning algorithm has a slow convergence due to the curse-of-dimensionality
problem and the considered environment is challenging to learn. When the DDQN
algorithm converges to the optimal policy and achieves stable average reward
values after 400 episodes, we obtain the weights of the neural network,
denoted as $\theta^{*}_{S}$, and the demonstration data $\mathcal{D}_{s}$ from
the source environment. The neural network’s weights $\theta^{*}_{S}$ and
demonstration data $\mathcal{D}_{s}$ are used for initializing DPR and TLwD in
the target environment later. Next, the AV with the obtained optimal policy
and demonstration data is used for the learning process in the target
environment. In the target environment, all algorithms use the same
$\epsilon$-greedy scheme in which the value of $\epsilon$ is reset to 1.0 and
is decreased until $\epsilon=0.01$. In particular, the probability that an
unexpected event occurs $p_{u}$ and the traffic density $\omega$ of the two
environments are different (Table I). Fig. 5(b) shows the average reward
values of the proposed TLwD algorithm, compared to DPR, DDQN, and Q-learning
algorithms. We observe that the average reward values of TLwD and DPR converge
much faster than DDQN. This implies that the transfer learning mechanisms
achieve better performance than that of the traditional deep reinforcement
learning mechanism. The results also show that the Q-learning algorithm is
unable to learn the optimal policy in the target environment. Furthermore, we
can observe that although the two learning curves of TLwD and DPR algorithms
converge from the 60th episode, the TLwD algorithm achieves better performance
in the first 60 episodes with more stable and higher average rewards. The
reason is that the TLwD algorithm leverages the prioritized-experience replay
mechanism that helps to sample important transitions in the experience replay
memory. Note that the experience replay memory $\mathcal{D}^{re}$ in TLwD
algorithm contains two types of data that are the demonstration data obtained
from the source environment and the new generated data from the target
environment. With prioritized-experience replay mechanism, the AV can sample
more important transitions in the demonstration data during early episodes to
accelerate the exploration. In the later episodes, the AV can balance the
sampling ratio between the demonstration data and new generated data with the
sampling probability as defined in (23). In addition, the TLwD algorithm also
utilizes multi-step TD learning, i.e. n-step TD learning as defined in (27),
that helps to accelerate the training process [37]. With the DPR algorithm,
the sampling mechanism is the same as that of DDQN algorithm, i.e., the
transitions are sampled randomly without using the multi-step learning
mechanism. The reason that the DPR outperforms DDQN can be explained as
follows. The weights of neural networks in DPR are already trained with data
in the source environment and then further updated in the target environment.
Although the source and target environments have different state transition
probabilities, their data can share some correlation that benefits the weight
updating process [40]. With DDQN, the weights of neural networks are randomly
initialized and trained from scratch.

In the rest of this section, we consider performances of only three algorithms
that are TLwD, DPR and DDQN in the target environment since the Q-learning
algorithm does not perform well in the transfer learning setting.

####  VII-B2 Performance Evaluation

(a)

(b)

(c)

Figure 6: (a) Average reward, (b) throughput, and (c) miss detection
probability vs. probability that unexpected event occurs with high speed of AV
($p_{1}^{v}$).

To further evaluate the robustness of the proposed TLwD algorithm, we vary the
probability that an unexpected event occurs when the AV moves at a high speed,
i.e., $p_{1}^{v}$ in (4), and show the results of the three algorithms in Fig.
6. It is worth noting that each point in Fig. 6, 7, and 8 is obtained by
averaging metric values in the first 1000 episodes, including the exploration
when the value of $\epsilon$ in $\epsilon$-greedy scheme is decreased from 1.0
to 0.01. As shown in Fig. 6(a), when $p_{1}^{v}$ increases from 0.1 to 0.9,
the TLwD outperforms DPR and DDQN in terms of the average reward. The reason
for the upward trend of TLwD and DPR is that when $p_{1}^{v}$ increases, the
TLwD and DPR algorithms select the radar detection mode more frequently to
avoid miss detection penalties and obtain greater rewards. With the values of
$p_{1}^{v}$ from 0.6 to 0.9, we can observe that the average reward values of
DDQN decrease significantly. The reason is that with the high values of
$p_{1}^{v}$, DDQN takes more episodes to achieve the optimal policy compared
to those of the TLwD and DPR, thus resulting in lower average reward values.
For example, with $p_{1}^{v}=0.8$, DDQN takes 400 episodes to converge to the
optimal policy, while TLwD and DPR take around 150 episodes to achieve similar
reward values. Moreover, the average reward values of TLwD are slightly higher
than that of DPR because in the early episodes, TLwD can outperform DPR as
discussed in Fig. 5(b). The results of throughput and miss detection
probability of the three algorithms are shown in Fig. 6(b) and Fig. 6(c),
respectively. Note that since the AV is able to perform either radar or data
communication at each time step, there is a tradeoff between the throughput
and miss detection probability. In particular, a higher throughput results in
a higher miss detection probability and vice versa. In Fig. 6(b) and Fig.
6(c), we can observe that TLwD obtains more stable results for miss detection
probability and throughput by following the policy that minimizes both miss
detection probability and throughput. With the DPR, the achieved miss
detection probability is higher than that of TLwD. Without using any transfer
learning technique, the DDQN algorithm obtains high throughput and high miss
detection probability at the same time. In particular, with this transfer
learning setting, the miss detection probability obtained by the proposed TLwD
algorithm is much lower than that of the conventional DDQN algorithm, reduced
by up to $67\%$ as shown in Fig. 6(c).

(a)

(b)

(c)

Figure 7: (a) Average reward, (b) throughput, and (b) miss detection
probability vs. probability that the factor $f$ is in favorable condition
($\tau^{f}_{0}$).

Next, we also evaluate the performance of the proposed TLwD algorithm by
varying the probability that the factor $f$ is at state 0 (favorable
condition), i.e., $\tau^{f}_{0}$ in (4). In particular, a higher value of
$\tau^{f}_{0}$ results in a higher probability that an environment state
(e.g., communication channel state, and weather state) is in a favorable
condition (e.g., good communication channel condition and good weather
condition). Without loss of generality, we increase all the values of
($\tau^{r}_{0},\tau^{w}_{0},\tau^{v}_{0},\tau^{m}_{0},\tau^{c}_{0}$) at the
same time, where $\tau^{r}_{0}$, $\tau^{w}_{0}$, $\tau^{v}_{0}$, and
$\tau^{m}_{0}$ are defined in (4), and $\tau^{c}_{0}$ is the communication
channel switching probability. We observe from Fig. 7(a) that when
$\tau^{f}_{0}$ increases from 0.1 to 0.9, average reward values of the three
algorithms reduce significantly. However, the results of the TLwD algorithm
are slightly better than that of DPR and much higher than that of DDQN,
especially with the low values of $\tau^{f}_{0}$, i.e., $\tau^{f}_{0}<0.7$.
The reason for this downward trend is that the increase of $\tau^{f}_{0}$
results in reducing the probability of an unexpected event to occur. With the
low values of $\tau^{f}_{0}$, meaning that unexpected events occur more
frequently, the TLwD and DPR algorithms significantly outperform the DDQN
algorithm, thanks to the transfer learning techniques. In contrast, with high
values of $\tau^{f}_{0}$, i.e., $\tau^{f}_{0}\geq 0.7$, meaning that
unexpected events are less likely to occur, the performance gaps among TLwD,
DPR, and DDQN are shortened. With $\tau^{f}_{0}=0.9$, we observe that the
average reward of DDQN is better than those of TLwD and DPR, thanks to the
policy of DDQN that significantly increases the throughput and thus increases
the miss detection probability at the same time. The above results also reveal
an interesting finding that transfer learning is more efficient when the
system’s dynamic is considerably high, which is equivalent to the scenarios
with small values of $\tau^{f}_{0}$. In Fig. 7(b), with the high values of
$\tau^{f}_{0}$, i.e., $\tau^{f}_{0}\geq 0.7$, the throughput of DDQN increases
significantly while the throughputs of the TLwD and DPR algorithms slightly
increase. Consequently, the miss detection probability of DDQN increases when
$\tau^{f}_{0}\geq 0.7$ and the miss detection probabilities of TLwD and DPR
algorithms still remain at low values.

(a)

(b)

(c)

Figure 8: (a) Average reward, (b) throughput, and (c) miss detection
probability vs. average traffic density ($\omega$).

In Fig. 8, we vary the average traffic density $\omega$ and evaluate the
performance of the proposed TLwD algorithm. Figure 8(a) shows that the average
rewards of the three algorithms decrease when $\omega$ increases from 0.1 to
0.9, and the average rewards obtained by TLwD are always higher than those of
DPR and DDQN. The reason is that the three algorithms are likely to follow the
policies that select the radar detection mode more frequently. As a result,
this leads to the decrease of immediate reward obtained from the fourth and
the fifth conditions in (11). Accordingly, the miss detection probability
values are increased as shown in Fig. 8(c). In particular, the TLwD algorithm
can reduce the miss detection probability by up to $22\%$ and $61\%$ compared
to the DPR and DDQN algorithms, respectively. The results for the throughput
in Fig. 8(b) are similar to the above scenarios (i.e., Fig. 6(b) and Fig.
7(b)) in which the TLwD and DPR algorithms achieve lower throughputs than that
of DDQN.

####  VII-B3 Benefits Of Prioritized-experience Replay Mechanism

Figure 9: Average rewards of TLwD with $(\alpha,\beta)=(0.4,0.6\rightarrow
1.0)$ and TLwD with $(\alpha,\beta)=(0,0)$ (i.e., uniform sampling).

As discussed in the previous results, the prioritized-experience replay
mechanism has a significant impact on the performance of the TLwD algorithm.
We then evaluate the performance of the TLwD algorithm with and without the
prioritized-experience replay mechanism. In particular, the prioritized-
experience replay mechanism can be controlled by adjusting two parameters
$\alpha$ and $\beta$ in (23) and (24). For our prioritized-experience replay
mechanism in the TLwD algorithm, we use the common setting in which the value
of $\alpha$ is fixed at 0.4 and the value of $\beta$ is increased from 0.6 to
1.0 [29], denoted as $(\alpha,\beta)=(0.4,0.6\rightarrow 1.0)$. We denote that
with the values $(\alpha,\beta)=(0,0)$, the TLwD algorithm uses uniform
sampling mechanism in which the transitions in the replay memory are sampled
randomly. Figure 9 shows that the TLwD algorithm with the prioritized-
experience replay mechanism (blue line) acquires faster convergence rate than
that of the TLwD algorithm with the uniform sampling mechanism (orange line).
This result can confirm the advantage of using prioritized-experience replay
technique over traditional uniform sampling technique [29, 37].

##  VIII Conclusions

In this paper, we have developed the dynamic framework for joint radar-
communications system of autonomous vehicles under the dynamic and uncertainty
of surrounding environment. In particular, the dynamic of environment such as
communication channel, weather, road and traffic conditions can be captured
through the MDP framework and then the optimal policy for the AV can be
obtained by using the proposed DDQN algorithm. Furthermore, we have developed
a transfer learning approach, integrating with the proposed DDQN algorithm,
that enables the AV to leverage valuable experiences from similar domains, and
thereby significantly improving the AV’s system performance when it moves to a
new environment. Extensive simulations have demonstrated that the transfer
learning approach has the convergence rate much faster than that of the
conventional deep reinforcement learning approaches in the transfer learning
setting.

## References

  * [1] Autonomous Vehicle Market Outlook: 2026, May 2018. Available Online: https://www.alliedmarketresearch.com/autonomous-vehicle-market. Last Accessed on: March 2021.
  * [2] D. Ma, N. Shlezinger, T. Huang; Y. Liu, and Y. C. Eldar, “Joint radar-communication strategies for autonomous vehicles: Combining two key automotive technologies”, IEEE Signal Processing Magazine, vol. 37, no. 4, pp. 85-97, Jul. 2020.
  * [3] L. Gaudio, M. Kobayashi, G. Caire, and G. Colavolpe, “On the effectiveness of OTFS for joint radar parameter estimation and communication,” IEEE Transactions on Wireless Communications, vol. 19, no. 9, pp. 5951-5965, Sep. 2020.
  * [4] E. Yurtsever, J. Lambert, A. Carballo, and K. Takeda, “A survey of autonomous driving: Common practices and emerging technologies,” IEEE Access, vol. 8, pp. 58443-58469, Mar. 2020.
  * [5] A. I. Maqueda, A. Loquercio, G. Gallego, N. Garcia, and D. Scaramuzza, “Event-based vision meets deep learning on steering prediction for selfdriving cars,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2018, pp. 5419-5427.
  * [6] M. Hnewa and H. Radha, “Object Detection Under Rainy Conditions for Autonomous Vehicles: A Review of State-of-the-Art and Emerging Techniques,” IEEE Signal Processing Magazine, vol. 38, no. 1, pp. 53-67, Jan. 2021.
  * [7] S. Zang, M. Ding, D. Smith, and P. Tyler, “The impact of adverse weather conditions on autonomous vehicles: how rain, snow, fog, and hail affect the performance of a self-driving car,” IEEE Vehicular Technology Magazine, vol. 14, no. 2, pp. 103-111, Mar. 2019.
  * [8] N. C. Luong, X. Lu, D. T. Hoang, D. Niyato, and D. I. Kim, “Radio Resource Management in Joint Radar and Communication: A Comprehensive Survey,” IEEE Communications Surveys & Tutorials, to be published.
  * [9] S. H. Dokhanchi, M. R. B. Shankar, Y. A. Nijsure, T. Stifter, S. Sedighi, and B. Ottersten, “Joint automotive radar-communications waveform design,” 2017 IEEE 28th Annual International Symposium on Personal, Indoor, and Mobile Radio Communications (PIMRC), Oct. 2017, pp. 1-7.
  * [10] P. Kumari, N. Gonzalez-Prelcic, and R. W. Heath, “Investigating the IEEE 802.11ad standard for millimeter wave automotive radar,” in 2015 IEEE 82nd Vehicular Technology Conference (VTC2015-Fall), Sep. 2015, pp. 1–5.
  * [11] S. H. Dokhanchi, B. S. Mysore, K. V. Mishra, and B. Ottersten, “A mmWave automotive joint radar-communications system,” IEEE Transactions on Aerospace and Electronic Systems, vol. 55, no. 3, pp. 1241-1260, Feb. 2019.
  * [12] P. Kumari, S. A. Vorobyov, and R. W. Heath, “Adaptive virtual waveform design for millimeter-wave joint communication–radar,” IEEE Transactions on Signal Processing, vol. 68, pp. 715-730, Nov. 2019.
  * [13] P. Ren, A. Munari, and M. Petrova, “Performance tradeoffs of joint radar-communication networks,” IEEE Wireless Communications Letters, vol. 8, no. 1, pp. 165–168, Feb. 2019.
  * [14] N. Cao, Y. Chen, X. Gu, and W. Feng, “Joint Bi-Static Radar and Communications Designs for Intelligent Transportation,” IEEE Transactions on Vehicular Technology, vol. 69, no. 11, pp. 13060-13071, Nov. 2020.
  * [15] P. Ren, A. Munari, and M. Petrova, “Performance analysis of a time-sharing joint radar-communications network,” 2020 International Conference on Computing, Networking and Communications (ICNC), Feb. 2020, pp. 908-913.
  * [16] J. Kim and C. Park, “End-to-end ego lane estimation based on sequential transfer learning for self-driving cars,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017, pp. 30-38.
  * [17] Y. Yao, X. Chen, L. Rao, X. Liu, and X. Zhou, “LORA: Loss differentiation rate adaptation scheme for vehicle-to-vehicle safety communications”, IEEE Transactions on Vehicular Technology, vol. 66, no. 3, pp. 2499-2512, May 2016.
  * [18] P. S. Khairnar and N. B. Mehta, “Discrete-rate adaptation and selection in energy harvesting wireless systems,” IEEE Transactions on Wireless Communications, vol. 14, no. 1, pp. 219-229, Jul. 2014.
  * [19] N. Q. Hieu, D. T. Hoang, N. C. Luong, and D. Niyato, “iRDRC: An Intelligent Real-time Dual-functional Radar-Communication System for Automotive Vehicles”, IEEE Wireless Communications Letters, vol. 9, no. 12, pp. 2140-2143, Aug. 2020.
  * [20] Z. Zhu, K. Lin, and J. Zhou, “Transfer Learning in Deep Reinforcement Learning: A Survey”, arXiv preprint, arXiv:2009.07888, 2020.
  * [21] R. Bellman, “A markovian decision process”, J. of Math. and Mech., vol. 6, no. 5, pp. 679–684, 1957.
  * [22] C. J. C. H. Watkins and P. Dayan, “Q-learning,” Mach. Learn., vol. 8, no. 3–4, pp. 279–292, 1992.
  * [23] V. Mnih, et al, “Human-level control through deep reinforcement learning”, Nature, vol. 518, pp. 529–533, Feb. 2015.
  * [24] N. C. Luong, D. T. Hoang, S. Gong, D. Niyato, P. Wang, Y-C. Liang, and D. I. Kim, “Applications of deep reinforcement learning in communications and networking: A survey,” IEEE Communications Surveys & Tutorials, vol. 21, no. 4, pp. 3133-3174, May 2019.
  * [25] R. S. Sutton and A. G. Barto, “Reinforcement learning: An introduction,” MIT press, 2018.
  * [26] H. V. Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 30, no. 1, Mar. 2016.
  * [27] C. T. Nguyen, et al, “Transfer Learning for Future Wireless Networks: A Comprehensive Survey,” arXiv preprint, arXiv:2102.07572, 2021.
  * [28] K. Weiss, T. M. Khoshgoftaar, and D. Wang, “A survey of transfer learning,” Journal of Big data, vol. 3, no. 1, p. 9, May 2016.
  * [29] T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized experience replay”, arXiv preprint, arXiv:1511.05952, 2015.
  * [30] T. Hester, et al, “Deep q-learning from demonstrations”, Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, Apr. 2018.
  * [31] V. Dham, “Programming Chirp Parameters in TI Radar Devices”, Application Report SWRA553, Texas Instruments, 2017.
  * [32] C. N. Kloeden, G. Ponte, and J. McLean, “Travelling speed and risk of crash involvement on rural roads”, Australian Transport Safety Bureau, 2001.
  * [33] How Do Weather Events Impact Roads. Accessed: February 2021. [Online]. Available: https://ops.fhwa.dot.gov/weather/q1_roadimpact.htm.
  * [34] M. Deublein, M. Schubert, B. T. Adey, J. Köhler, and M. H. Faber, “Prediction of road accidents: A Bayesian hierarchical approach,” Accident Analysis & Prevention, vol. 51, pp. 274-291, Mar. 2013.
  * [35] A. H. Lang, et al, “nuScenes: A multimodal dataset for autonomous driving,” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 11621-11631.
  * [36] W. Maddern, G. Pascoe, C. Linegar, and P. Newman, “1 year, 1000 km: The Oxford RobotCar dataset,” The International Journal of Robotics Research, vol. 36, no. 1, pp. 3-15, Jan. 2017.
  * [37] M. Hessel, et al, “Rainbow: Combining improvements in deep reinforcement learning”, Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, Apr. 2018.
  * [38] I. Goodfellow, Y. Bengio, and A. Courville, “Deep Learning,” Cambridge, MA, USA: MIT Press, 2016.
  * [39] F. Fernández and M. Veloso, “Probabilistic policy reuse in a reinforcement learning agent,” Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems, May 2006, pp. 720-727.
  * [40] T. Killian, G. Konidaris, and F. Doshi-Velez, “Robust and efficient transfer learning with hidden parameter markov decision processes,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31, no. 1, Feb. 2017.
  * [41] K. U. Mazher, R. W. Heath, K. Gulati, and J. Li, “Automotive Radar Interference Characterization and Reduction by Partial Coordination,” in 2020 IEEE Radar Conference (RadarConf20), Sep. 2020, pp. 1-6.