# Deep Reinforcement Learning Based Multi-Access Edge Computing Schedule for
Internet of Vehicles

Xiaoyu Dai, Kaoru Ota, and Mianxiong Dong  Xiaoyu Dai, Kaoru Ota and Mianxiong
Dong are with the Department of Sciences and Informatics, Muroran Institute of
Technology, Japan (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>it.ac.jp; mx.dong@csse.muroran-it.ac.jp).

###### Abstract

As intelligent transportation systems been implemented broadly and unmanned
arial vehicles (UAVs) can assist terrestrial base stations acting as multi-
access edge computing (MEC) to provide a better wireless network communication
for Internet of Vehicles (IoVs), we propose a UAVs-assisted approach to help
provide a better wireless network service retaining the maximum Quality of
Experience (QoE) of IoVs on the lane. In the paper, we present a Multi-Agent
Graph Convolutional Deep Reinforcement Learning (M-AGCDRL) algorithm which
combines local observations of each agent with a low-resolution global map as
input to learn a policy for each agent. The agents can share their information
with others in graph attention networks, resulting in an effective joint
policy. Simulation results show that the M-AGCDRL method enables a better QoE
of IoTs and achieves good performance.

###### Index Terms:

Intelligent Transport System, Wireless Communication, Markov modeling (state-
based stochastic control)

##  I Introduction

Recently, Internet of Vehicles has been playing an essential role in creating
a safer and more intelligent transportation system in the upcoming 5G (the
fifth generation) network. The concept of IoVs origins from IoTs (Internet of
Things), which provides network connection service between the vehicles on the
road with the other vehicles, pedestrians, and roadside infrastructures. The
massive implementation of IoVs greatly improves the Quality of Experience of
the drivers and the transportation efficiency.

IoVs support five types of network communication, they are Vehicle to
Infrastructure (V2I), Vehicle to Vehicle (V2V), Vehicle to Pedestrian (V2P),
etc.(or broadly the V2X communication) [1], [2]. With the autonomous vehicles
been carried out in more and more countries, the V2X will be of more
significance in the future. However, with the dramatic growth of the data
traffic and bandwidth limitation of the Internet, there are some drawbacks
that can not be neglected, since the enormous traffic data corresponds to
enormous network transmission cost. One of the drawbacks is that current
network bandwidth is not enough to support too many network connection
devices, especially when the traffic is heavy. Moreover, the V2X requires a
very low latency and a relatively high transmission to ensure its regular
running, otherwise it will be rather dangerous.

In order to ensure a extensive area coverage, a low latency, and a relatively
high transmission speed for IoVs, some methods have been put forward.
Companies like Apple and Google have developed applications, Apple CarPlay and
Google Android Auto to help drivers plan route referring to real-time traffic
condition by connecting iPhone or Android Phone to the vehicle.

Some applications support offline route planning considering network
conditions during a traffic jam, or somewhere remote away from the BS. The
defect is obvious since offline route planning lacks plenty of real-time
details, which will largely degrade the users’ experience. [3], [4] come up
with the long term evolution (LTE)-V wireless standard to improve capacity to
improve network performance. In LTE-V, every vehicle shares the same network
resource. The two modes of LTE-V, LTE-V-direct and LTE-V-cell cooperate to
provide wireless network service to vehicles on the roads.

Nevertheless, the LTE base station is stationary, which brings significant
challenges to satisfy the high data rate and low latency, especially in dense
IoVs scenarios. Besides, it costs a lot to construct a BS specifically in
remote areas. Apart from that, it is not necessary or sometimes not possible
to set up a new terrestrial base station timely, especially during the
aftermath of natural disasters.

As wireless communication and smart car technologies become more
sophisticated, IoV has grabbed more and more attention over the past few
years. Not only for manual driving, but also for autonomous vehicles, the
network is widely recognized as possessing considerable value in improving
vehicular safety, mobility, sustainability and efficiency. Meanwhile,
vehicular networks also face great challenges mainly because of the emergence
of those latency-sensitive and computation-intensive vehicular applications,
due to the limited spectrum resources and limited computation resources.

Recently, flying MEC devices such as UAVs have been served as promising
assistants helping provide better wireless networks, especially when equipped
with communication transceivers. Owing to their low cost and high flexibility,
the UAVs are able to provide wireless connectivity for IoVs assisting LTE
network provided by the Base Station.

Deploying some UAVs can perfectly accomplish the task as a temporary wireless
base station. Compared to these conventional methods, UAV-enabled wireless
communications have emerged as a promising way to help ensure stable network
service[5], [6]. Acting as flying MEC devices, those UAVs can significantly
improve network capacity with their high mobility and flexible deployment.
Apart from that, their cost is relatively low; therefore, it can be simply
implemented. By associating with UAVs, better network service can be provided
for devices in vehicular network, which leads to an upgrade of QoE for drivers
in IoVs.

In spite of the leverage of MEC servers and devices, it is stil very
challenging to dynamically support those vehicle application for network
service provider, who should deploy the MEC server in advance, otherwise it
would be difficult to meet the dynamic resource requirements of the vehicular
applications. Technical challenges include the location and number of MEC
servers, and the way multidimensional resources allocate across vehicles. From
driver’s perspective, the key is to decide if, when, and how many tasks should
be offloaded to the MEC server to meet the QoE requirements.

Although MEC-enabled wireless communication shows its superiority on the
network performance of IoVs, it still faces a lot of challenges. For instance,
because of the high speed of vehicles, it is hard to decide when and where to
deploy the UAVs so that we can not only satisfy drivers’ needs but also
minimize the waste of UAV resources. Besides, considering about obtaining the
best QoE continuously, how to schedule the UAVs trajectories is of
extraordinary significance. However, to schedule multiple UAVs has been a
difficult problem, according to recent research works. Most researchers choose
only one UAV to solve their problems, whereas in practical application
scenarios, one UAV is far from enough.

To tackle these challenges, deep DRL was addressed in this paper. DRL combines
DL(deep learning) and RL(reinforcement learning) to create algorithms, which
can be applied in many fields such as robotics, video games, and
transportation and so on. Frameworks based on DRL is capable of many
unsolvable problems due to its unsupervised learning. With the implementation
of DRL, we can find the solution to achieve the best QoE by enormous repeated
training. Meanwhile, for a problem without prior probability, deep
reinforcement learning has the ability to solve it.

The contributions of this paper can be summarized as the below.

1) We proposed a Multi-Agent Graph Convolutional Deep Reinforcement Learning
(M-AGCDRL) algorithm, which applies multiple UAVs working as MEC to assist the
BS, aiming to achieve the utmost capacity of network. Those UAVs can
substantially relieve the burden caused by the excess amount of devices in
vehicular network on the lane.

2) The proposed algorithm combined the graph convolutional networks and actor-
critic architecture to maximum the QoE for the drivers.

3) Simulation results show that the proposed algorithm can improve the QoE for
drivers.

The rest of this paper is organized as follows. Related works are discussed in
Section II. In Section III, the system model is described. In Section IV, we
describe the details of the proposed algorithm. After that, the simulation
results are shown in Section V. At last, our conclusions are drawn in Section
VI.

##  II Related Works

In MEC-assisted vehicular networks, the vehicular applications can get access
to servers through different wireless communications mainly including
dedicated short-range communication (DSRC), Wi-Fi, and cellular. While the
requirement for computing/storage resources is strong, vehicles perform
offloading with MEC servers with a shorter response latency through reducing
data transfer between MEC servers and cloud servers.

In [5], an energy-efficient control algorithm was proposed for UAVs coverage
and connectivity based on deep reinforcement learning to control the UAVs with
minimizing their energy consumption. Coverage, fairness, and energy
consumption are optimized as simulation shows.

An overview of UAV-assisted wireless network through the basic networking
structure and main channel characteristics was proposed in [6]. The paper
discussed communications not only between UAVs and the ground, but also
between UAVs themselves. Besides, due to short-range line-of-sight (LoS)
links, UAV-assisted wireless communications tend to have better communication
channels. Three aspects of UAV trajectory design, UAV energy consumption and
multiple-input multiple-output (MIMO) communications among UAVs are
considered.

Joze Kosmerl and Andrej Vilhar [7] proposed a method for calculating the
optimal proprieties and coordinates of aerial stations and terrestrial base
stations, which shows advances in low altitude platforms (LAPs) in high
bandwidth utilization, especially when terrestrial base stations are under
destroyed after a natural disaster, especially for catastrophes. The algorithm
can deploy minimum number of UAVs to fully cover the target area.

AL-Hourani et al. [8] present an analytical solution for LAP altitude
optimization. An optimal altitude is calculated for providing maximum radio
coverage on the ground. Apart from this, a closed-form formula is proposed to
predict the probability of geometrical LoS from the platform to a ground user.
Considering the fact that air- to-ground radio channel behaves randomly, it
still requires further investigation.

Alzenad et al. [9] proposed an optimal deployment algorithm for UAV-assisted
Base Station wireless network, and the algorithm provides as many the number
of users the network can cover at the least consumption of transmit power. The
three-dimensional deployment algorithm they have proposed decouples the
problem into UAV’s optimal vertical position and horizontal coordinate
separately without any optimal loss.

In [10], the authors proposed a framework to obtain the optimal three-
dimensional positions of multiple UAVs, where they can achieve a maximum
ground coverage and minimum transmit power. In this paper, circle packing
theory is made use of to maximize the zone that UAVs can cover. In the
meantime, the least UAVs are required to assure goal coverage for the
determined ground district.

According to [11], Yong Zeng and Rui Zhang supposed that the UAV would be
flying at a consistent altitude. Under this premise, they developed a model
through optimizing the flying distance and speed of UAV to make the best of
communication energy efficiency under a practical circular UAV trajectory.

In [12], a sense-and-send protocol was come up with, the protocol aims at
coordinating multiple UAVs to carry out a real-time sense and send task. Then
an enhanced DRL algorithm for multiple UAVs is proposed to solve the
decentralized multi-agent trajectory design problem. This paper proved that
DRL could be applied to UAV-assisted wireless communications.

A novel framework proposed in [13] makes full use of UAV acting as an aerial
station to maximize the throughput of users. Such UAV-based wireless network
systems are optimized with regard to the average bits between UAV and users as
well as the duration UAVs hover. UAVs’ 3-dimensional deployment was attained
with the help of the circle packing theory.

To make an interference-aware UAV trajectory scheme, authors in [14] proposed
a DRL algorithm to both maximize energy efficiency and minimize wireless
latency. Also, to minimize the computational complexity of the proposed
algorithm, an upper and a lower bound for the flying altitude are set in
advance.

An UAV-aided cellular communication framework is proposed in [15] to assist
cellular communications hold out against smart jamming. With the help of deep
reinforcement learning, the scheme can achieve a much less error rate through
training the previous anti-jamming relay data.

Machine learning techniques are also utilized to solve UAV trajectory design
and obstacle avoidance problems, authors in [16] applied neural network to
calculate Q-value so that they can avoid a low-quality trajectory and massive
computation of matrix while ensuring the precision.

Mozaffari et al. [17] developed a new concept of 3D cellular networks
integrating base stations and cellular-connected users through UAV. Utilizing
this framework, a method for the minimum number of UAVs converging most is
proposed. Simulation shows the proposed latency-optimal cell association can
improve the spectral efficiency of a UAV-assisted communication.

In [18], Feng Lyu et al. developed an Online UAV Scheduling (OUS) scheme
aiming at obtaining satisfied throughout performance. UAVs are scheduled and
managed overall to assure smooth connections between UAVs and users. Results
show that the scheme can efficiently respond to the QoS demand through sending
as few as UAVs intelligently.

Yuanwei Liu with his partners [19] proposed a novel framework for UAV-assisted
communications with a large number of access capabilities. The framework is
supported by no-orthogonal multiple access. They applied stochastic geometry
to create the coordinates of UAVs and users on the ground, then analyzed the
model with single UAV and multiple UAVs separately.

Figure 1: The MEC-enabled wireless communications architecture.

##  III System Model

The detailed system model in 3-dimensional scenario is described in this
section. We create an environment in which the BS can fully cover IoVs in the
area, and multiple UAVs can be sent alternatively to assist the BS. The UAV-
assisted BS wireless network aims at providing a better network service for
drivers on the road. In this paper, we consider a fixed down-link transmission
rate ${T_{r}}$ is in demand for each IoV. In order to maximize the QoE of
IoVs, multiple UAVs are utilized to support the BS when the network resource
provided by the BS is not abundant enough.

###  III-A System Description

As is illustrated in Fig. 1, we consider a UAV-enabled wireless communications
with a BS and $N$ UAVs. We consider there is one BS in the environment. Assume
that the BS acts as the origin of the system coordinate, and the height of the
BS is $H_{BS}$. The set of the UAVs and vehicles are denoted by
$\mathcal{N}=\\{1,2,...,N\\}$ and $\mathcal{M}=\\{1,2,...,M\\}$, respectively.
Supposing the coordinate of each UAV as
${U_{n}}=[{x_{{u_{n}}}}(t),{y_{{u_{n}}}}(t)]$, where ${x_{{u_{n}}}}(t)$ and
${y_{{u_{n}}}}(t)$ are the coordinate of UAV $N$ at time t. The height of each
UAV is set stationary as $H_{u}$ in this paper. Each vehicle coordinates at
${V_{n}}=[{x_{{v_{m}}}}(t),{y_{{v_{m}}}}(t)]$, where ${x_{{v_{m}}}}(t)$ and
${y_{{v_{m}}}}(t)$ are the coordinate of vehicle $M$ at time t. As the BS
offers the LTE service whose transmission rate has a lot to do with the
distance between the BS and vehicles. The distance from the BS to each vehicle
is expressed by $M$ with $D$, given by

$D=\sqrt{{H_{u}}^{2}+{x_{v_{m}}}^{2}+{y_{{v_{m}}}}^{2}}\text{.}$ (1)

which reflects the straight line distance between them.

Denote all the vehicles associate to the UAV are distributed with equal
bandwidth. Note the total bandwidth of the BS as $W_{BS}$. The BS can be
separated into at most $X$ sub channels. Therefore, the download throughput of
the vehicle M from the BS is

${B_{{v_{m}}}}=\left\\{\begin{array}[]{*{20}{c}}\frac{W_{BS}}{M}\log_{2}(1+\frac{p_{t_{m}}p_{c_{m}}}{\frac{W_{BS}M{n_{0}}}{)}}&M<X\\\
\frac{W_{BS}}{M}\log_{2}(1+\frac{{{p_{{t_{m}}}}{p_{{c_{m}}}}}}{{\frac{{{W_{BS}}}}{X}{n_{0}}}})&M\geq
X&\end{array}\right.\text{.}$ (2)

where $p_{{t_{i}}}$ denotes the transmission power, $p_{{c_{i}}}$ denotes the
channel power and $n_{0}$ means the background density of the additive white
Gaussian noise (AWGN). Besides, UAV $M$’s transmission rare can be signified
as

${R_{{v_{m}}}}={B_{{v_{m}}}}{\log_{2}}\left[{1+\frac{{{p_{{t_{m}}}}{p_{{c_{m}}}}}}{{{\sigma^{2}}}}}\right]\text{.}$
(3)

where $\sigma^{2}={B_{{v_{m}}}}{n_{0}}$.

Since the UAVs are flying in a 3-D space, the probability of LoS connections
for the UAVs is higher than on terrestrial conditions. The LoS probability can
be denoted as

$P_{LoS}(\theta_{k_{n}})=b_{1}(\frac{180}{\pi}\theta_{k_{n}}-\zeta)^{b_{2}}$
(4)

where $\theta_{k_{n}}(t)=sin^{-1}\left[\frac{h_{n}(t)}{d_{k_{n}}(t)}\right]$
is the elevation angle between UAV $n$ and the vehicle $M$. $b_{1}$, $b_{2}$
and $\sigma$ are environment-determined constants. Besides, the Non-Line-of-
Sight (NLoS) probability is given by $P_{NLoS}=1-P_{LoS}$. From $P_{LoS}$ and
$P_{NLoS}$ the channel power from UAV $N$ to UAV $M$ can be denoted as

$g_{k_{n}}(t)=K_{0^{-1}}d_{k_{n}}^{-\alpha}(t)\left[P_{LoS}\mu_{LoS}+P_{NLoS}\mu_{NLoS}\right]$
(5)

where $K_{0}=\left(\frac{2\pi f_{c}}{c}\right)^{2}$, $\alpha$ is constant
which denotes path loss exponen. $\mu_{LoS}$ and $\mu_{NLoS}$ are different
attenuation factors considered for LoS and NLoS links, $f_{c}$ is the carrier
frequency, $c$ is the speed of light. As the received interference from UAV to
vehicle can be mitigated by assigning different spectrum to each cluster, the
received SNR $\Gamma_{v_{m}}(t)$ of vehicle $M$ associated with UAV $N$ at
time $t$ can be denoted as

$\Gamma_{v_{m}}(t)=\frac{p_{t_{m}}p_{c_{m}}}{\sigma^{2}}$ (6)

Since the transmit rate of the vehicles requirements varies from each other,
we denote a SNR threshold $\gamma_{v_{m}}$. If the SNR a vehicle receives
doesn’t reach this threshold, we consider it as a failed transmission, that
is, $\gamma_{v_{m}}\geqslant\gamma_{v_{m}}$.

TABLE I: Notation List Notation | Description | Notation | Description
---|---|---|---
$M$ | Number of vehicles | $H_{u}$ | Height of UAVs
$N$ | Number of UAVs | $W_{BS}$ | Total bandwidth of the BS
$\mathcal{N}$ | Set of vehicles | ${B_{{v_{m}}}}$ | Download throughput from BS
$\mathcal{M}$ | Set of UAVs | $p_{{t_{i}}}$ | Transmission power
$D$ | Distance between BS and vehicle | $p_{{c_{i}}}$ | Channel power
$[{x_{{v_{m}}}}(t),{y_{{v_{m}}}}(t)]$ | Coordinate of UAV $M$ | $n_{0}$ | Background density of AWGN
$[{x_{{u_{n}}}}(t),{y_{{u_{n}}}}(t)]$ | Coordinate of UAV $N$ | ${R_{{v_{m}}}}$ | UAV $M$’s transmission rare

###  III-B Network Resource Shortage Monitoring

As the vehicles move fast in and out all the time, the density of the vehicles
on each lane changes continuously. We set every each intersection $I_{i}$ and
all the lanes $L_{j}$ connected to it as one block. At each time slot $k$, the
density of the block is calculated as $D_{k}=\frac{\sum V}{\sum length(L)}$,
where $length(R)$ represents the length of the lane $L$ at time $t_{k}$. Given
a threshold $\lambda$, when the density of one block exceeds the threshold,
network resource shortage is detected, and UAVs are ready to be deployed to
reduce the burden on BS.

###  III-C Quality-of-Experience Model

QoE measures a customer’s experience with spectacular service, whether he or
she is satisfied or dissatisfied. As customers’ experience varies from each
other, it is essential to set a standard to evaluate the service provided.
Mean opinion score (MOS) is utilized for QoE evaluation, The MOS is
represented on a five alternatives where $5=excellent$, $4=good$, $3=fair$,
$2=poor$, $1=bad$. The definition of QoE is as follows

$MO{S_{{v_{m}}}}(t)={\lambda_{1}}MOS_{{v_{m}}}^{delay}(t)+{\lambda_{2}}MOS_{{v_{m}}}^{rate}(t)\text{.}$
(7)

where${\lambda_{1}}+{\lambda_{2}}=1$, ${\lambda_{1}}$ and ${\lambda_{2}}$ are
coefficients. As the transmission rate is the most important factor, we ignore
the delaying time and simplified the MOS model as

$MO{S_{{v_{m}}}}(t)=MOS_{{v_{m}}}^{rate}(t)\text{.}$ (8)

To maximize the sum MOS of each vehicle during the period time $T_{p}$, which
is defined as follows

$MO{S_{v}}=\sum\limits_{t=0}^{{T_{p}}}{MO{S_{{v_{m}}}}(t)}\text{.}$ (9)

###  III-D Problem Formulation

We are aiming at maximizing the MOS of the drivers on the vehicles in a MEC-
assisted network, the problem can be formulated as

$maxMOS_{total}=\sum_{n=1}^{N}\sum_{k_{n}=1}^{K_{n}}\sum_{t=0}^{T_{n}}MOS_{k_{n}}(t)$
(10)

which means summarize all the UAVs’ MOS during the whole time, and Q-learning
is applied to maximize this value in Section IV and Section V.

Figure 2: The UAV trajectory design.

###  III-E UAV deployment problem

Applications of Q-learning are always set in a grid world where agents are
trained to find the best trajectories to reach the goal. In this paper, each
UAV is considered as an agent, and to get the maximum sum of MOS is the final
goal. Since Q-learning does not perform well in high dimensional environment
because of the restriction of q-table, we attempt to simplify the three
dimensional UAV deployment problem by considering the three dimensional world
into a finite set of discrete spatial points $S_{p}$ in Fig. 2. At the initial
of each time slot, the UAVs can select their flying directions to another 26
points or stay still.

As there are limitations for single-agent DRL framework to solve massive IoVs
problems, we adopt a MEC-assisted multi-agent DRL framework in this paper. In
this way, we can deploy and plan trajectories for multiple UAVs to help assist
the BS with network service.

###  III-F UAV trajectory problem

In this section, we described the trajectories design of UAV to maintain the
QoE of users. As the vehicles move continuously, the UAV’ optimal position
changes at every cycle. As a result, the UAV needs to travel to offer a better
down-link service as the vehicles move. Otherwise, the users’ QoE requirements
will not be satisfied.

Conventionally, wireless network tends to deploy more flying BS, the UAVs to
serve a more portable network experience. However, considering the high
flexibility of UAVs, we can find a trajectory for the UAV to get the optimal
MOS without sending more other UAVs. To obtain the trajectories of the UAVs,
we assume that the UAV can select its flying directions at the beginning of
each cycle, and the vehicles will not roam into the others.

A straightforward method is to decentralize continuous state data. However,
without knowing the decentralization granularity in advance, the state may
cause a large error in reflecting the actual environment. If we want to ensure
accuracy, it may bring about a reduction in the calculation of the matrix.

As shown in Fig. 3, at the beginning of every time slot, each agent carries
out an action ${{\rm{a}}_{t}}\in A$. That is, each agent chooses a direction
according to current state, ${{\rm{s}}_{t}}\in S$ by the policy $J$. Since 27
directions are too much to be calculated by computer, we choose nine points
from 27 points as the action space to speed up the convergence speed of the
training. The action space includes nine actions that can be carried out,
including the eight corners of the cube in Fig. 2 and stay still.

###  III-G Action Space

The position of agents are denoted by
${U_{n}}=[{x_{{u_{n}}}}(t),{y_{{u_{n}}}}(t)]$, indicating the horizontal
position of each UAV at time slot $t$. The altitude of each UAV is set as $H$,
which is decided through UAV deployment algorithm in Section IV. Besides, the
altitude of each UAV keep still all the time through the trajectory design.
Therefore, the sate space can be denoted as
${{\rm{x}}_{{u_{t}}}}{\rm{:\\{0,1,}}...{\rm{,}}{{\rm{X}}_{m}}{\rm{\\}}}$,
${{\rm{y}}_{{u_{t}}}}{\rm{:\\{0,1,}}...{\rm{,}}{{\rm{Y}}_{m}}{\rm{\\}}}$,
where ${{\rm{X}}_{m}}$ and ${{\rm{Y}}_{m}}$ are still the maximum coordinate
of the area.

As for a $10\times 10$ two dimensional panel, the state space includes $80$
points in total for UAV to choose. One UAV can only choose to move one step,
$\frac{1}{2}$ of the cube’s side length in Fig. 2, or stay still. UAVs are
initially deployed where the deployment algorithm has decided.

Figure 3: The illustration of the RL system.

###  III-H State Space

The position of agents are denoted by
${U_{n}}=[{x_{{u_{n}}}}(t),{y_{{u_{n}}}}(t),{h_{{u_{n}}}}(t)]$, which
indicates the position of each UAV in three dimensional space at time slot
$t$. The initial altitude of each UAV is set as $H$, which is parallel to the
BS horizontally. The sate space can be denoted as
${{\rm{x}}_{{u_{t}}}}{\rm{:\\{0,1,}}...{\rm{,}}{{\rm{X}}_{m}}{\rm{\\}}}$,
${{\rm{y}}_{{u_{t}}}}{\rm{:\\{0,1,}}...{\rm{,}}{{\rm{Y}}_{m}}{\rm{\\}}}$,
where ${{\rm{X}}_{m}}$ and ${{\rm{Y}}_{m}}$ are the maximum coordinate of the
area. [x] shows that the further a UAV hovers from the BS, in other words, the
closer a UAV hovers to the marginal of the environment, the larger overall
throughout can be gained. In this paper, we decide the UAVs’ initial position
set on the marginal position for each trial.

As for a three dimensional space of $10\times 10\times 4$, the state space
includes 158 points in total for UAV to choose. One UAV can only select to
move one step, $\frac{\sqrt{2}}{2}$ of the cube’s side length in Fig. 2, or
stay still. UAVs are initially deployed at the farthest points from the BS,
then they are applied with Q-learning algorithm to explore to best initial
position to get the maximum MOS.

###  III-I State Transition Mode

During each time slot $t$, each agent $M$’s state ${s_{t}}$ transits to state
${s_{t+1}}$ after action ${a_{t}}$ is carried out. Action ${a_{t}}$ is
randomly chosen based on the conditional transition probability
$p({s_{t+1}},{r_{t}}|{s_{t}},{a_{t}})$. The Q-learning model aims at achieving
maximum long-term rewards ${G_{t}}$, which can be denoted as follow

${G_{t}}=E|\sum\limits_{n=0}^{\infty}{{\beta^{n}}{r_{t+n}}}|\text{,}$ (11)

where $\beta$ is a discount factor. The problem can be described by a tuple
$<{s_{1}},...,{s_{M}},{a_{1}},...,{a_{M}},{\rm
T},{p_{{r_{1}}}},...,{p_{{r_{M}}}},\beta>$, where

  * •

UAVs are the agents.

  * •

${s_{1}},...,{s_{M}}$ are state spaces of UAV $M$, indicates the position of
UAV $M$.

  * •

${a_{1}},...,{a_{M}}$ are the actions consists of nine possible actions
$(0,0,0,1)$ means upward back right, $(1,1,1,1)$ means upward back left;
$(1,0,0,1)$ represents downward back right, $(0,1,1,1)$ represents downward
back left; $(0,1,0,1)$ indicates upward front right, $(1,0,1,1)$ indicates
upward front left; $(0,0,1,1)$ denotes downward front right, $(0,0,1,1)$
denotes downward front right; finally, $(0,0,0,0)$ implies stay still.

  * •

${p_{{r_{1}}}},...,$ denote the possibility of actions related to rewards for
UAV $M$.

  * •

$\beta\in[0,1)$ is the discount factor. It decides how much proportion future
rewards account for.

We define the reward function as follow,

${R_{t}}=\left\\{{\begin{array}[]{*{20}{c}}{0.8,}&{if{\rm{}}MO{S_{n}}<MO{S_{n+1}}}\\\
{-0.1,}&{if{\rm{}}MO{S_{n}}=MO{S_{n+1}}}\\\
{-0.8,}&{if{\rm{}}MO{S_{n}}>MO{S_{n+1}}}\end{array}}\right.\text{,}$ (12)

where ${MO{S_{n}}}$ and ${MO{S_{n+1}}}$ represents $t$-th cycle and ${t+1}$-th
cycle, respectively. [xx] shows that changing the value of reward does not
make difference on algorithm’s final result, but influences the convergence
rate. In this paper, we assume that every transmission is successful. The
Q-table is iteratively updated during learning in the framework. The BS can
inform the UAVs of the rewards of the last cycle, and it is

${Q_{t+1}}({s_{t}},{a_{t}})\leftarrow(1-\alpha){Q_{t}}({s_{t}},{a_{t}})+\alpha[{R_{t}}+\beta\cdot{\max_{a}}{Q_{t}}({s_{t+1}},a)]$
(13)

###  III-J UAVs deployment algorithm

####  III-J1 Single UAV deployment algorithm

As is mentioned above, Q-learning with single agent in two-dimensional world,
the most fundamental DRL algorithm, has been put into various scenarios. We
firstly expand the two-dimensional world into three-dimensional world with
only one agent, that is, one UAV to assist with the network service.

In the single agent algorithm, there is only one UAV hovering in a constant
environment. It observes the sates $s$ and choose an action $a$, which leads
to a maximum reward $r$. We take every UAV as an independent agent, and the
other UAVs are regarded as part of the environment. Then we can update the
q-table according to the following rule,

${Q_{N}}({s_{t}},{a_{t}})\leftarrow{Q_{t}}({s_{t}},{a_{t}})+\alpha[{R_{t}}+\beta\cdot{\max_{a}}{Q_{N}}({s_{t}},a_{t})]$
(14)

where $\alpha$ is the learning rate.

####  III-J2 Multiple UAVs deployment algorithm

In the multiple agents algorithms, there are more than one UAVs taking actions
simultaneously. In spite of the fact that single agent algorithm takes
advantages in smaller state space, it ignores the other agents’ action
strategies. To solve this problem, we choose an opponent modeling Q-learning
to make a better solution.

The UAVs are deployed randomly close to marginal position initially. Then in
every cycle each UAV $N$ choose an action $a_{t}$ based on the environment and
other UAVs’ observed frequency distribution in state $s_{t}$. After every time
slot, the q-table will be updated.

The process can be described as follows. Firstly, the positions of UAVs are
set to the initial states in environment through the results of deployment.
Secondly, the environment observes current states and judges whether the QoE
reaches the threshold, if it does, take the history into a success trajectory
set; if not, try to reach the threshold until time expires. Then each agent
selects an action and takes it, then the state is set to the new state, and
the process will be repeated until it reaches the terminal.

##  IV Proposed M-AGCDRL algorithm

In the MEC-assisted wireless network of IoTs, the environment state is subject
to dynamic changes at all times due to the infinite channel state. In other
words, it is difficult for the agents to allocate resource through successive
time slots according to the currently observed environment state. Therefore,
in this work, we decide to propose a environment-free DRL algorithm, in which
the environment is uncertain. As the size of the state space and the action
space is infinite, the algorithms such as strategy gradient, actor criticism,
deterministic strategy gradient (DPG) and Deep DPG (DDPG), are often widely
used by researchers. In this article, we combine graph convolutional network
and actor criticism architecture to solve modeled MDP.

###  IV-A The M-AGCDRL architecture

Our architecture uses deep reinforcement learning structures and graph
attention networks to simulate communication. We refer to this architecture as
M-AGCDRL.

Fig. 4 is the schematic diagram of M-AGCDRL. Each agent shared a same
structure with the same weight. This makes it possible to promote the setting
by duplicating the structure for more agents. As is shown in Fig. 4, the
structure is made up of by three parts, which are: a convolutional neural
network for extracting features from the input of each agent; a graph
attention network for sharing information among agents; an actor criticism
algorithm, using to find the best strategy for each agent.

###  IV-B CNN and GAT

Firstly, we extract the feature vector $hc_{i}$ from $w_{i}$ with a CNN. The
observed value $z_{n}$ of all agents $N$ at time $t_{p}$ is input to the
network. Then, the feature vector is passed to a GAT and also sent to its
neighbour agents.

Then, the graph attention network is applied to calculated. Each agent’s local
feature vectors are shared with their adjoining neighbors. We note it the
weighted-edge between two agents $m$ and $n$. $\alpha_{m,n}$ gives the weight
of the agent $m$ from $n$. The attention weight is measured by the following
method. A node $m$ summarizes the feature vector from its neighbor $N_{m}$,
the summed vector $h_{m}$ will be. For the sequential nodes in its neighbor
$N_{m}$, this update of $h_{m}$ is unchanged. In the simulation, a fully
connected network is adopted, and $N_{m}$ is the set of all other agents
excluding $m$. The structure can, nevertheless, be integrated directly into
any other adjacency relationship.

Figure 4: The architecture of the proposed algorithm.

###  IV-C Multi-agent communication

In a multi-agent setting, each agent’s decision can be affected by each
agent’s joint actions in the network. We apply Markov games for modeling the
multi-agent RL problem. It can be thought of as a multi-agent extension of
MDPs.

The aggregate feature vector $h_{m}$ from GAT and the feature vector compose
the input. We introduce the actor-critic algorithm to learn the best strategy.
In path-planning tasks, the critical approach of policy gradient actors is
usually better than DQN.

Each agent’s network includes one actor and one critic. As the CM problem is a
collaborative issue, the rewards are equivalent for each agent. We note that
the parameters of the actors as $t_{1}$, the parameters of the critics as
$t_{2}$. ans the shared reward as $r(t,t_{1})=\sum_{m=1}^{k}\pi(t_{1})(t)$.

###  IV-D Constant monitoring

In this section, continuous monitoring issues are described. We assume the
environment is a three-dimensional grid world $G$. Every unit in this
environment will be monitored by agent. We associate the reward with each
unit. Let $R_{k_{t}}$ be the relevant reward of a unit $u$ in $U$.

We studied the learning effects when using grid maps, real world maps, and
real world and grid maps. Assuming that $\beta$ is the negative decay in each
agent. The reward is reset to $0$ or declines at a rate of $\beta$. If $m$ is
monitored at the time of $t$, then the reward is $0$; otherwise, the reward is
$\beta$.

Since $\beta>=0$, $R_{t_{u}}$ is a non-positive value. $R_{t_{u}}$ is a
negative number of $R_{t_{u}}$, that is, a negative number related to the
penalty of not viewing the unit $u$. The longer the interval of continuous
observation, the greater the cumulative penalty. We set $R_{m}ax$ to the upper
limit of the penalty, to prevent it from increasing indefinitely.

As Fig. 5 shows, consider that every agent obtains the same maximum coverage,
a circle with a radius of $R$ centered on the agent. We consider two cases,
one is that the agent can obtain global information in the whole environment,
another is that the agent can only get local information.

Consistency monitoring aims to acquire the optimal policy $\pi$ which
maximizes the sum reward for all agents over a finite length of time $T$. This
policy maps the state to each agent’s actions. Each agent can choose one of
its neighbors and stay in place.

As the reward is related to the agent’s historical location, the state must
both include the agent’s location and the reward value of every unit in the
present time slot. This guarantees Markov characteristics.

However, as the state space is very large, we use RL to solve this problem. By
studying the effect of RL through the following methods, the diagram tends to
be the low-resolution version of the learning environment. Besides, the grid
map only consists of the location of the present one, the penalty value as
well as the location of obstacles. As for the real world map, it is
approximately the observations designated by those sensors decorated on the
agents.

Figure 5: The maximum coverage of the UAV.

##  V Simulation Results

To quantify our proposed algorithm, we consider two performance metrics: the
MOS and the offloading rate, we perform simulation experiments and evaluate
the performance of the M-AGCDRL algorithm. First, we present the simulation
scenario. Then, the effects of various indicators of the simulation are
analyzed.

###  V-A simulation scene

As it is necessary to obtain all the vehicles’ position at the beginning of
each cycle, the simulation tool SUMO [20] is applied to generate vehicle
trace. Specially, we implement two scenarios: a $10X10$ synthetic traffic grid
and a real-world traffic network from Sapporo city. Concrete IoV trace data is
generated by the help of the simulation tool SUMO. Specifically, we
intercepted a map of Sapporo City where lanes are given a speed limit of
$60km/h$.

With the generated vehicle trace, we implement our algorithms on TensorFlow
using Python 3. Table II shows the parameters of the environment and base
stations. Denote a UAV-assisted wireless network is made up of $M$ vehicles
and $N$ UAVs. The bandwidth of the BS is $B=10MHz$. We consider there are at
most $100$ vehicles moving on the lanes.

Meanwhile, we set the parameter of Q-learning as Table III. The learning rate
is $lr$ = 0.01. The reward decay is $\gamma=0.9$. The threshold $t=0.1$. Then
set $ep=5000$, which means we simulate 5000 episodes.

TABLE II: Parameters of Simulation Parameter | Value
---|---
B | 10MHz
$p_{t_{i}}$ | 4kW
$p_{c_{i}}$ | 40W
N | 5
M | 100
lr | 0.01
$\gamma$ | 0.9
t | 0.1
ep | 5000

###  V-B Numerical Results

Figure 6: The total rewards with 10 MEC server of each episode versus
algorithms under grid world map. Figure 7: The total rewards with 10 MEC
server of each episode versus algorithms under real world map.

Fig. 6 and Fig. 7 show the convergence speed of the proposed algorithm, the
ordinary actor-critic algorithm, as well as the ordinary Q-learning algorithm
under grid world map and real world map, respectively. As we can see from the
figures, the total reward for each of the 1,200 episodes in the learning phase
fluctuates sharply and relatively little during the first few hundred
episodes, then stabilize at a comparatively high level. Once 10,000
experiences are saved, the learning phase of the proposed algorithm starts
updating the parameters. As a result, the total reward per episode fluctuates
dramatically at the beginning then gradually optimized.

Fig. 8 illustrates the impact of the number of MEC devices performing
computational offloading. With the increase in the number of UAVs, the number
of UAVs performing computational offloading also increases, and all of them
tend to be stable at the last. Also, we can find that the proposed method
always performs better.

Figure 8: The number of UAVs performing computation offloading versus
different method.

Fig. 9 shows offloading performance as the number of UAVs gets greater. From
the figure, we can see that at the very beginning when the number of UAVs is
not very large, the the offloading tends to perform better because the number
of channels to the UAVs also increase as the number UAVs increases. As the
limited computing resources, if extravagant UAVs are accessing, the delay will
also rise. In this way, the system finally get to stable.

Figure 9: The number of UAVs performing computation offloading as MEC servers
increase.

Fig. 10 shows the average QoS, in other words, the MOS under the proposed
algorithm, the common actor-critic algorithm and the common Q-learning
algorithm. As available computing resources volume increases, the MOS by our
proposed algorithm and the comparison algorithms increases because of the
increase in the volume of resources assigned to the drivers on the lane.

Figure 10: MOS with UAVs increasing versus algorithm.

##  VI Conclusion

Given the tremendous promise of 5G, it is clear that facilitating circular
communication in 5G cellular networks based on MEC offers significant
advantages in terms of scalability, reliability, mobility support and packet
latency.

In this paper, we have we present the M-AGCDRL algorithm to QoE-driven MEC
schedule problem. We combine the graph convolutional networks and actor-critic
architecture to deal with the modeled MDPs. Through the simulation, we can see
that our proposed algorithm could offloading rate compared with the normal
q-learning algorithm and actor critic algorithm, which could lead to a better
QoE for drivers on the lane. In addition, our algorithm is computationally
efficient and can converge more quickly than the common algorithms. Through
the study, the MEC-assisted wireless network from fundamental theory to
practical implementation can be investigated in a more comprehensive way.

## Acknowledgments

This work is partially supported by JSPS KAKENHI Grant Numbers JP19K20250,
JP20F20080, and JP20H04174, Leading Initiative for Excellent Young Researchers
(LEADER), MEXT, Japan, and JST, PRESTO Grant Number JPMJPR21P3, Japan.
Mianxiong Dong is the corresponding author.

## References

  * [1] F. Lyu, N. Cheng, H. Zhu, H. Zhou, W. Xu, M. Li, and X. S. Shen, “Intelligent context-aware communication paradigm design for IoVs based on data analytics,” _IEEE Network_ , vol. 32, no. 6, pp. 74–82, nov 2018.
  * [2] W. Xu, H. Zhou, N. Cheng, F. Lyu, W. Shi, J. Chen, and X. Shen, “Internet of vehicles in big data era,” _IEEE/CAA Journal of Automatica Sinica_ , vol. 5, no. 1, pp. 19–35, jan 2018.
  * [3] S. Chen, J. Hu, Y. Shi, and L. Zhao, “LTE-v: A TD-LTE-based v2x solution for future vehicular network,” _IEEE Internet of Things Journal_ , vol. 3, no. 6, pp. 997–1005, dec 2016.
  * [4] H. Zhang, X. Tang, R. Banez, P. Ren, L. Song, and Z. Han, “An EPEC analysis for power allocation in LTE-v networks,” in _GLOBECOM 2017 - 2017 IEEE Global Communications Conference_. IEEE, dec 2017.
  * [5] C. H. Liu, Z. Chen, J. Tang, J. Xu, and C. Piao, “Energy-efficient UAV control for effective and fair communication coverage: A deep reinforcement learning approach,” _IEEE Journal on Selected Areas in Communications_ , vol. 36, no. 9, pp. 2059–2070, sep 2018.
  * [6] Y. Zeng, R. Zhang, and T. J. Lim, “Wireless communications with unmanned aerial vehicles: opportunities and challenges,” _IEEE Communications Magazine_ , vol. 54, no. 5, pp. 36–42, may 2016.
  * [7] J. Kosmerl and A. Vilhar, “Base stations placement optimization in wireless networks for emergency communications,” in _2014 IEEE International Conference on Communications Workshops (ICC)_. IEEE, jun 2014.
  * [8] A. Al-Hourani, S. Kandeepan, and S. Lardner, “Optimal LAP altitude for maximum coverage,” _IEEE Wireless Communications Letters_ , vol. 3, no. 6, pp. 569–572, dec 2014.
  * [9] M. Alzenad, A. El-Keyi, F. Lagum, and H. Yanikomeroglu, “3-d placement of an unmanned aerial vehicle base station (UAV-BS) for energy-efficient maximal coverage,” _IEEE Wireless Communications Letters_ , vol. 6, no. 4, pp. 434–437, aug 2017.
  * [10] M. Mozaffari, W. Saad, M. Bennis, and M. Debbah, “Efficient deployment of multiple unmanned aerial vehicles for optimal wireless coverage,” _IEEE Communications Letters_ , vol. 20, no. 8, pp. 1647–1650, aug 2016\.
  * [11] Y. Zeng and R. Zhang, “Energy-efficient UAV communication with trajectory optimization,” _IEEE Transactions on Wireless Communications_ , vol. 16, no. 6, pp. 3747–3760, jun 2017.
  * [12] J. Hu, H. Zhang, and L. Song, “Reinforcement learning for decentralized trajectory design in cellular UAV networks with sense-and-send protocol,” _IEEE Internet of Things Journal_ , vol. 6, no. 4, pp. 6177–6189, aug 2019\.
  * [13] M. Mozaffari, W. Saad, M. Bennis, and M. Debbah, “Wireless communication using unmanned aerial vehicles (UAVs): Optimal transport theory for hover time optimization,” _IEEE Transactions on Wireless Communications_ , vol. 16, no. 12, pp. 8052–8066, dec 2017.
  * [14] U. Challita, W. Saad, and C. Bettstetter, “Interference management for cellular-connected UAVs: A deep reinforcement learning approach,” _IEEE Transactions on Wireless Communications_ , vol. 18, no. 4, pp. 2125–2140, apr 2019.
  * [15] X. Lu, L. Xiao, C. Dai, and H. Dai, “Uav-aided cellular communications with deep reinforcement learning against jamming.”
  * [16] Z. Yijing, Z. Zheng, Z. Xiaoyi, and L. Yang, “Q learning algorithm based UAV path learning and obstacle avoidence approach,” in _2017 36th Chinese Control Conference (CCC)_. IEEE, jul 2017.
  * [17] Mohammad Mozaffari and Ali Taleb Zadeh Kasgari and Walid Saad and Mehdi Bennis and Merouane Debbah, “Beyond 5g with UAVs: Foundations of a 3d wireless cellular network,” _IEEE Transactions on Wireless Communications_ , vol. 18, no. 1, pp. 357–372, jan 2019.
  * [18] F. Lyu, P. Yang, W. Shi, H. Wu, W. Wu, N. Cheng, and X. S. Shen, “Online UAV scheduling towards throughput QoS guarantee for dynamic IoVs,” in _ICC 2019 - 2019 IEEE International Conference on Communications (ICC)_. IEEE, may 2019.
  * [19] Y. Liu, Z. Qin, Y. Cai, Y. Gao, G. Y. Li, and A. Nallanathan, “UAV communications based on non-orthogonal multiple access,” _IEEE Wireless Communications_ , vol. 26, no. 1, pp. 52–57, feb 2019.
  * [20] P. A. Lopez, M. Behrisch, L. Bieker-Walz, J. Erdmann, Y.-P. Flötteröd, R. Hilbrich, L. Lücken, J. Rummel, P. Wagner, and E. Wießner, “Microscopic traffic simulation using sumo,” in _The 21st IEEE International Conference on Intelligent Transportation Systems_. IEEE, November 2018. [Online]. Available: https://elib.dlr.de/124092/

##  VII Biography Section

|  Xiaoyu Dai received the B.E. degree in School of Cyber Science and
Engineering from Wuhan University, China and was a research student with ENeS
lab at Muroran Institute of Technology, Japan, the same year. She is currently
pursuing the M.S. degree in Muroran Institute of Technology, Japan.
---|---
|  Kaoru Ota was born in Aizu-Wakamatsu, Japan. She received M.S. degree in
Computer Science from Oklahoma State University, the USA in 2008, B.S. and
Ph.D. degrees in Computer Science and Engineering from The University of Aizu,
Japan in 2006, 2012, respectively. Kaoru is currently an Associate Professor
and Ministry of Education, Culture, Sports, Science and Technology (MEXT)
Excellent Young Researcher with the Department of Sciences and Informatics,
Muroran Institute of Technology, Japan. From March 2010 to March 2011, she was
a visiting scholar at the University of Waterloo, Canada. Also, she was a
Japan Society of the Promotion of Science (JSPS) research fellow at Tohoku
University, Japan from April 2012 to April 2013. Kaoru is the recipient of
IEEE TCSC Early Career Award 2017, The 13th IEEE ComSoc Asia-Pacific Young
Researcher Award 2018, 2020 N2Women: Rising Stars in Computer Networking and
Communications, 2020 KDDI Foundation Encouragement Award, and 2021 IEEE
Sapporo Young Professionals Best Researcher Award. She is Clarivate Analytics
2019, 2021 Highly Cited Researcher (Web of Science) and is selected as JST-
PRESTO researcher in 2021, Fellow of EAJ in 2022.
---|---
|  Mianxiong Dong received B.S., M.S. and Ph.D. in Computer Science and
Engineering from The University of Aizu, Japan. He is the Vice President and
Professor of Muroran Institute of Technology, Japan. He was a JSPS Research
Fellow with School of Computer Science and Engineering, The University of
Aizu, Japan and was a visiting scholar with BBCR group at the University of
Waterloo, Canada supported by JSPS Excellent Young Researcher Overseas Visit
Program from April 2010 to August 2011. Dr. Dong was selected as a Foreigner
Research Fellow (a total of 3 recipients all over Japan) by NEC C&C Foundation
in 2011. He is the recipient of The 12th IEEE ComSoc Asia-Pacific Young
Researcher Award 2017, Funai Research Award 2018, NISTEP Researcher 2018 (one
of only 11 people in Japan) in recognition of significant contributions in
science and technology, The Young Scientists’ Award from MEXT in 2021,
SUEMATSU-Yasuharu Award from IEIEC in 2021, IEEE TCSC Middle Career Award in
2021. He is Clarivate Analytics 2019, 2021 Highly Cited Researcher (Web of
Science) and Foreign Fellow of EAJ.
---|---