#  Energy Efficient Computation Offloading in Aerial Edge Networks With Multi-
Agent Cooperation

Wenshuai Liu, Bin Li, , Wancheng Xie, Yueyue Dai, ,
and Zesong Fei Wenshuai Liu, Bin Li, and Wancheng Xie are with the School of
Computer and Software, Nanjing University of Information Science and
Technology, Nanjing 210044, China (e-mail<EMAIL_ADDRESS>zuoyeyiwancheng@gmail.com).Yueyue Dai is with the School of Cyber Science and
Engineering and Research Center of 6G Mobile Communications, Huazhong
University of Science and Technology, Wuhan 430074, China (e-mail:
yueyuedai@ieee.org).Zesong Fei is with the School of Information and
Electronics, Beijing Institute of Technology, Beijing 100081, China. (e-mail:
feizesong@bit.edu.cn).

###### Abstract

With the high flexibility of supporting resource-intensive and time-sensitive
applications, unmanned aerial vehicle (UAV)-assisted mobile edge computing
(MEC) is proposed as an innovational paradigm to support the mobile users
(MUs). As a promising technology, digital twin (DT) is capable of timely
mapping the physical entities to virtual models, and reflecting the MEC
network state in real-time. In this paper, we first propose an MEC network
with multiple movable UAVs and one DT-empowered ground base station to enhance
the MEC service for MUs. Considering the limited energy resource of both MUs
and UAVs, we formulate an online problem of resource scheduling to minimize
the weighted energy consumption of them. To tackle the difficulty of the
combinational problem, we formulate it as a Markov decision process (MDP) with
multiple types of agents. Since the proposed MDP has huge state space and
action space, we propose a deep reinforcement learning approach based on
multi-agent proximal policy optimization (MAPPO) with Beta distribution and
attention mechanism to pursue the optimal computation offloading policy.
Numerical results show that our proposed scheme is able to efficiently reduce
the energy consumption and outperforms the benchmarks in performance,
convergence speed and utilization of resources.

###### Index Terms:

Mobile edge computing, unmanned aerial vehicle, computation offloading, deep
reinforcement learning, digital twin.

##  I Introduction

With the rapid advance of Internet of Things (IoT) technology and the
extensive deployment of 5G networks, the envision and planning of future 6G
networks is in progress. It is expected to be empowered with the ability of
providing intelligent and green communication features, and realizing the
ubiquitous connection between massive objectives [1, 2, 3]. In order to
achieve these requirements, the quality of service and application experience
in 6G network need to be significantly enhanced, which involves higher
requirements for the computing resources and communication resources of
various devices [4]. In this sense, mobile edge computing (MEC) as a popular
paradigm naturally arises to alleviate the pressure on resource-constrained
devices where the heterogeneous resource-intensive applications are performed.
Specifically, the devices can offload their computational tasks to edge
servers with powerful computing resources to pursue satisfactory energy
consumption and latency [5, 6].

However, the transmission links of terrestrial MEC servers may be influenced
by blockage, and the mobile users (MUs) located away from the servers are
typically not well covered. Especially in some emergency situations such as
disaster area and hot spot area, the fixed MEC servers are intricate to
deploy. Recently, unmanned aerial vehicle (UAV) has emerged as a promising
technology to assist traditional infrastructure-based MEC networks due to the
representative features of high mobility, flexible deployment, and low cost
[7, 8]. By cooperatively carrying MEC servers, UAVs can fly closer to the MUs
to offload part of the tasks and provide reliable communication links, which
is a fast and cost-effective deployment way for computation offloading with
low delivery delay and high data rate requirements [9, 10]. The dynamicity of
the MEC networks is further enhanced with the participation of UAVs and MUs,
and thus motivates the application of online optimizing in MEC service,
especially the integration of artificial intelligence approaches that
outperforms traditional offline optimization in decision time and adaptability
to the environment.

For the learning-based optimizing techniques, the dynamicity in networks
imposes strict requirements on accurate state sensing, real-time decision-
making, and the precise execution, which presents technical challenges for
implementing the UAV-assisted MEC networks [11, 12]. In view of this, as an
emerging technology in the 6G era, digital twin (DT) has come into the
research vision, which is capable of maintaining the virtual models of
physical entities in digital space and monitoring of the overall system
status. Currently, DT has been widely used in smart city, healthcare, and
intelligent manufacturing [13]. By collecting massive timely data from real
space, DT can also realize efficient model training in dynamic MEC networks,
thus providing the MEC services with more intelligent decisions.

###  I-A Related Work

In recent years, UAV-assisted MEC has attracted growing research interests,
and extensive research efforts have been conducted to enhance network
performance in various scenarios. To exploit the performance of UAV in special
situations, Do-Duy et al. [14] and Tran et al. [15] studied the joint
deployment and resource configuration in UAV-aided disaster emergency
communications by using iterative methods. To pursue the real-time decision of
UAV-assisted networks, Wang et al. [16] integrated deep reinforcement learning
(DRL) method into iterative convex optimization approaches, and Dai et al.
[17] studied the minimization of the execution latency in an edge-cloud
heterogeneous network comprising UAVs and base stations via DRL approach. For
some distributed scenarios, multi-agent reinforcement learning (MARL) emerges
as a mighty technique that trains efficient decentralized models, and thereby
avoids the dependence of centralized DRL on the whole network state. Wang et
al. [18] considered the load balance in multi-UAV-assisted MEC under the
constraints of users’ energy consumption. Peng et al. [19] investigated the
utilization of MARL to solve the dynamic network resource allocation in UAV-
assisted vehicular edge networks. Cai et al. [20] studied the data sensing and
computation offloading problem in a multi-UAV-assisted MEC, and integrated
attention mechanism into MARL algorithm to accelerate the training speed of
the network. To further investigate the cooperation of multiple agents, Ji et
al. [10] exploited the acquisition latency minimization in multi-UAV cellular
networks via MARL. Zhao et al. [21] investigated the cooperation between
multi-UAV and multi-ground MEC servers to minimize the total network utility
and enhance the performance of service.

In view of incomparable merits of DT, the idea of combining DT and MEC has
made many attempts to build a digital twin edge network (DITEN), but the
current research on DITEN is still in its infancy. To fully exploit the
capability of DT network for capturing the real-time state of the network, Sun
et al. [22] investigated the latency minimization problem of DITEN via DRL
method and considered the estimated deviation of DT. Sun et al. [23] developed
an approach to automatically adjust the aggregation frequency of federated
learning in DITEN network. Lu et al. [24] proposed a blockchain-based
federated learning framework that runs in DT for collaborative computing,
improving the reliability and security of the system and enhancing data
privacy. Liu et al. [25] introduced edge collaboration into the DITEN network
and realized intelligent offloading of tasks. As a step forward, Do-Duy et al.
[26] iteratively optimized the joint resources of mobile devices and multiple
edge servers to minimize the total latency in industrial IoT networks, and
Zhang et al. [27] combined DT technology and decomposed MARL into the design
of vehicle edge computing networks. Recently, Li et al. [28] focused on the
DT-aided computation offloading in UAV edge networks where the double deep
Q-network algorithm is explored to reduce the system energy consumption.

###  I-B Contributions and Organization

The above-mentioned excellent works have investigated the UAV/DT networks from
various aspects. However, the scenarios that involve the joint cooperation
among UAVs, base station (BS), and MUs have not been fully exploited.
Especially when MUs are deployed in remote or disaster areas, they are not
able to directly communicate and synchronize status with BS and DT. Therefore,
different from these prior studies, we in this paper propose a real-time
distributed optimization framework for the multi-UAV-assisted MEC networks
based on MARL, where the policy model can be deployed on MUs and UAVs to make
resource allocation decisions, and a DT layer is deployed at the BS to pursue
timely monitoring and centralized training, thereby saving the energy of model
training on resource-limited MUs and UAVs. Particularly, the main
contributions of this paper are presented as follows.

  1. 1.

We proposed a multi-UAV-assisted MEC network with air-ground cooperation to
provide the computing services for the resource-limited MUs, where the
interplay between physical environment and DT layer is taken into account. The
MUs can partially offload their computational task to UAVs or relay to BS for
further computation processing via UAVs. Considering the energy limits of both
MUs and UAVs, a weighted energy consumption minimization problem is formulated
by jointly designing the association, offloading proportion, trajectory
control, and resource allocation on computation and communication.

  2. 2.

To tackle the high complexity of the formulated real-time problem, we
formulate it as a Markov decision process (MDP), and address it by an MARL
framework with heterogeneous agents. the high-dimensional hybrid action
spaces, the multi-agent proximal policy optimization (MAPPO) algorithm is
utilized to train both MUs and UAVs to cooperatively make offloading decisions
in dynamic MEC networks.

  3. 3.

To enhance the performance of training, we adopt Beta distribution and
attention mechanism in actor and critic network, respectively. Simulation
results show the efficient training convergence and effectiveness of our
proposed MAPPO with attention mechanism and Beta distribution (AB-MAPPO)
algorithm in optimizing energy consumption. The proposed algorithm outperforms
other benchmarks, especially in utilization of computational resource, and
flexibility to different deploying scenarios.

The rest of this paper is organized as follows. We first present the system
model and formulate the joint problem in Section II. The design of AB-MAPPO
algorithm is given in Section III. Section IV presents the simulation results.
Finally, we conclude this paper in Section V.

Figure 1: System model.

##  II System Model and Problem Formulation

In this section, we first introduce a multi-UAV-assisted MEC system model,
including network model, mobility model of MUs, DT model, communication model,
and computation model. Then, we formulate the optimization problem to minimize
the weighted energy consumption of MUs and UAVs while ensuring the delay
requirement of the tasks.

###  II-A Network Model

As shown in Fig. 1, we consider a multi-UAV-assisted air-ground MEC network
containing a BS, $M$ UAVs, and $K$ MUs. For simplicity, we denote the set of
MUs by $\mathcal{K}\triangleq\\{1,2,\dots,K\\}$, the set of UAVs by
$\mathcal{M}\triangleq\\{1,2,\dots,M\\}$, and the index of BS by $M+1$. The
MUs typically have limited battery life and computing capability, which are
not capable of completing their resource-intensive tasks efficiently. We
assume that UAVs and BS are equipped with MEC servers to provide computing
service for MUs. The set of servers is denoted by
$\mathcal{M}^{\star}\triangleq\mathcal{M}\cup\\{M+1\\}$. Without loss of
generality, we introduce a time period $T$, which is divided into $N$ time
slots with time of $\delta_{t}$. The set of time slots is denoted as
$\mathcal{N}\triangleq\\{1,2,\dots,N\\}$. Using three-dimensional Cartesian
coordinate system, the locations of MU $k$, UAV $m$ and BS at time slot $n$
are denoted as ${\bf u}_{k}[n]=[x_{k}^{\rm MU}[n],y_{k}^{\rm
MU}[n],0]^{\text{T}}$, ${\bf q}_{m}[n]=[x_{m}^{\rm UAV}[n],y_{m}^{\rm
UAV}[n],H]^{\text{T}}$, and ${\bf
u}_{\text{BS}}=[x_{\text{BS}},y_{\text{BS}},H_{\text{B}}]^{\text{T}}$,
respectively. Note that the UAVs are flying at a fixed altitude $H$. To
establish an efficient mapping of the physical MEC network, a DT layer is
deployed at BS and is equipped with essential functions of data storage,
status synchronization and model training. The main notations in this paper
are summarized in Table I.

###  II-B Mobility Model of MUs

At the beginning of time slot $n=1$, all MUs are randomly located, and the MUs
are moving according to Gaussian-Markov random model [29]. Considering that
each time slot is of short period, the location of MUs are assumed to be
static during one time slot. Specifically, at each time slot $n$, the velocity
of $v_{k}[n]$ and direction $\theta_{k}[n]$ of MU $k$ are respectively given
by

$\displaystyle
v_{k}[n]=\mu_{1}v_{k}[n-1]+(1-\mu_{1})\bar{s}+\sqrt{1-\mu_{1}^{2}}\Phi_{k},$
(1a)
$\displaystyle\theta_{k}[n]=\mu_{2}\theta_{k}[n-1]+(1-\mu_{2})\bar{\theta}+\sqrt{1-\mu_{2}^{2}}\Psi_{k},$
(1b)

where $0\leq\mu_{1},\mu_{2}\leq 1$ are the parameters representing the effect
of previous state, $\bar{s}$ and $\bar{\theta}$ are the average velocity and
direction of all MUs, respectively. Also, $\Psi_{k}$ and $\Phi_{k}$ are
generated by two independent Gaussian distributions with different mean-
variance pairs $(\bar{\xi}_{v_{k}},\varsigma^{2}_{v_{k}})$ and
$(\bar{\xi}_{\theta_{k}},\varsigma^{2}_{\theta_{k}})$ for MU $k$. Therefore,
the coordinate of MUs can be updated as

$\displaystyle x_{k}^{\rm MU}[n]=x_{k}^{\rm
MU}[n-1]+v_{k}[n-1]\cos(\theta_{k}[n-1])\delta_{t},$ (2a) $\displaystyle
y_{k}^{\rm MU}[n]=y_{k}^{\rm
MU}[n-1]+v_{k}[n-1]\sin(\theta_{k}[n-1])\delta_{t}.$ (2b)

###  II-C DT Model of the UAV-assisted MEC Network

In the considered multi-UAV network with MEC service, the DT layer deployed at
BS is responsible for state monitoring and virtual twin mapping. To maintain
the virtual twins, the physical devices need to upload the DT information of
themselves to the DT layer. In this paper, two types of entities are
represented, i.e., MUs and the UAVs. To model the virtual twins of them, we
focus on their key features which efficiently represent their real-time state
corresponding to the optimization scheme.

The virtual twin of each MU $k$ needs to record its location and task
information at each time slot $n$, which is given by

$\displaystyle{\rm DT}_{k}[n]=\\{{\bf
u}_{k}[n],\Omega_{k}[n],\tilde{f}_{k}^{\text{loc}}[n]\\},$ (3)

where $\Omega_{k}[n]$ is the task information of users and
$\tilde{f}_{k}^{\text{loc}}[n]$ is the estimated computational frequency for
MU $k$, which will be illustrated in following subsection.

The DT of each UAV should reflect its service status, involving the resource
allocation and movement, and is denoted as

$\displaystyle{\rm DT}_{m}^{U}[n]=\\{{\bf
q}_{m}[n],\alpha_{k,m}[n],\tilde{f}_{k,m}[n],\forall k\in\mathcal{K}\\},$ (4)

where $\alpha_{k,m}[n]$ and $\tilde{f}_{k,m}^{\text{edge}}[n]$ are defined the
association factor of the network and the estimated computational resource
allocated to MU $k$ by UAV $m$.

TABLE I: Main Notations Notation |  Definition |  Notation |  Definition
---|---|---|---
$K$ |  The number of MUs |  $M$ |  The number of UAVs
$N$ |  The number of time slots per period |  $B$ |  The total bandwidth of the network
$\mathcal{M}$ |  The set of UAVs |  $\mathcal{M}^{\star}$ |  The set of MEC servers
$\mathcal{K}$ |  Offloading proportion of MU $k$ at time slot $n$ |  $\mathcal{N}$ |  The set of time slot
${\bf q}_{m}[n]$ |  The location of MEC server $m$ at time slot $n$ |  ${\bf u}_{k}[n]$ |  The location of MU $k$ at time slot $n$
${\bf u}_{\text{BS}}$ |  The location of BS |  $L_{k}[n]$ |  The size of input data of MU $k$
$\rho_{k}[n]$ |  The offloading proportion of MU $k$ |  $T_{k,m}^{\text{ecmp}}[n]$ |  The actual edge computing time of MU $k$
$C_{k}[n]$ |  Number of CPU cycles required for processing one bit data of MU $k$ at time slot $n$ |  $\alpha_{k,m}[n]$ |  The association variable between MU $k$ and UAV $m$ at time slot $n$
$f_{k,m}^{\text{edge}}[n]$ |  The actual allocated CPU frequency of UAV $m$ to MU $k$ at time slot $n$ |  $f_{k}^{\text{loc}}[n]$ |  The actual CPU frequency of MU $k$ at time slot $n$
$T_{k}^{\text{loc}}[n]$, $T_{k}^{\text{edge}}[n]$ |  The actual local computing time and edge computing time of MU $k$ |  $B_{k,m}[n]$ |  The bandwidth allocated to link of MU $k$ and UAV $m$ at time slot $n$
$\bm{v}_{m}[n]$, ${\bf a}_{m}[n]$ |  The acceleration and velocity of UAV $m$ at time slot $n$ |  $s(t)$, $o_{i}(t)$, $a_{i}(t)$, $r_{i}(t)$ |  The global state, observation and reward of agent $i$ at time step $t$

###  II-D Computation Model

Each MU generates a task $\Omega_{k}[n]=(L_{k}[n],C_{k}[n])$ with a latency
requirement of $\delta_{t}$ at the beginning of each time slot $n$, where
$L_{k}[n]$ and $C_{k}[n]$ denote the size of input data and average number of
central process unit (CPU) cycles for processing one bit data of MU $k$’s
task, respectively. The tasks can be divided into two parts at random and
executed concurrently, with $(1-\rho_{k}[n])L_{k}[n]$ bits of input data for
local computing, and $\rho_{k}[n]L_{k}[n]$ bits offloaded to MEC servers
associated with BS and UAVs for edge computing. Denoting the offloading
association of MU $k$ at time slot $n$ as $\alpha_{k,m}[n]$, we have

$\alpha_{k,m}[n]\in\\{0,1\\},\forall
k\in\mathcal{K},m\in\mathcal{M}^{\star},n\in\mathcal{N}.$ (5)

The details are illustrated as follows.

####  II-D1 Local computing

With $\alpha_{k,0}[n]=1$, MU $k$ computes total task by itself locally,
indicating that $\rho_{k}[n]=0$.

According to prior MEC research [17], by adopting dynamic voltage frequency
scaling (DVFS) technique, the estimated frequency of MU $k$ at time slot $n$
in DT can be denoted as

$\tilde{f}_{k}^{\text{loc}}[n]=\min\\{{Y_{k}^{\text{loc}}[n]/\delta_{t}},f^{\text{loc}}_{\max}\\},$
(6)

where $f^{\text{loc}}_{\max}$ denotes the maximum CPU frequency of MUs,
$Y_{k}^{\text{loc}}[n]=(1-\rho_{k}[n])L_{k}[n]C_{k}[n]$ denotes the total
computing cycles of MU $k$’s task. It means that for energy efficient
objective, the task is processed with the minimum CPU frequency required to
complete it according to DVFS, thus we have
$\tilde{f}_{k}^{\text{loc}}[n]=Y_{k}^{\text{loc}}[n]/\delta_{t}$, and
$\tilde{f}_{k}^{\text{loc}}[n]$ is limited by the maximum CPU frequency
$f^{\text{loc}}_{\max}$.

It is noteworthy that the DT layer can’t fully reflect the state of MUs and
UAVs due to some issues such as the hysteresis of status synchronization and
the transmission error. Thus, a deviation is introduced to model the estimated
error of DT and can be utilized to verify the robustness of the system. The
deviation may occur from computational frequency [22, 25, 26], and location
data. Denoting $\tilde{f}_{k}^{\text{loc}}[n]$ as the estimated CPU frequency
of MU $k$, the estimated local computing time of MU $k$ at time slot $n$ can
be expressed as
$\tilde{T}_{k}^{\text{loc}}[n]=Y_{k}^{\text{loc}}[n]/\tilde{f}_{k}^{\text{loc}}[n]$.
By denoting $\hat{f}_{k}^{\text{loc}}[n]$ as the estimated deviation of actual
frequency
$f_{k}^{\text{loc}}[n]=\tilde{f}_{k}^{\text{loc}}[n]+\hat{f}_{k}^{\text{loc}}[n]$,
the computing latency gap of MU $k$ between DT and actual value can be
expressed as

$\Delta
T_{k}^{\text{loc}}[n]=\frac{-Y_{k}^{\text{loc}}[n]{\hat{f}_{k}^{\text{loc}}[n]}}{\tilde{f}_{k}^{\text{loc}}[n](\tilde{f}_{k}^{\text{loc}}[n]+{\hat{f}_{k}^{\text{loc}}[n]})}.$
(7)

As a result, the actual local computing time of MU $k$ can be calculated by

$T_{k}^{\text{loc}}[n]=\tilde{T}_{k}^{\text{loc}}[n]+\Delta
T_{k}^{\text{loc}}[n].$ (8)

####  II-D2 Edge computing

MUs can request UAVs for offloading their tasks to MEC servers at UAVs or BSs.
The procedures are respectively illustrated as follows.

  * •

UAV computing: A part of MU $k$’s task is transmitted to UAV $m$, and executed
by the MEC server on UAV. In this case, we have $\alpha_{k,m}[n]=1$ and
$\alpha_{k,M+1}[n]=0$.

  * •

BS computing: Considering the complex ground environment such as obstacle
blocking, the MU-BS links are too poor to support transmission. Therefore, a
certain part of task is first transmitted to UAV $m$, and further relayed to
BS for executing. To this end, we have $\alpha_{k,m}[n]=\alpha_{k,M+1}[n]=1$.

We adopt probabilistic line-of-sight (LoS) channel to represent UAV-MU links.
The probability of geometrical LoS between the UAV and MUs depends on the
environment status and the elevation angle. Denoting the elevation angle as
$\vartheta_{k}[n]$, and the LoS probability of MU $k$ and UAV $m$ at time slot
$n$ as $\mathbb{P}(\text{LoS},\vartheta_{k}[n])$, which is approximated to
following form

$\mathbb{P}(\text{LoS},\vartheta_{k}[n])=\frac{1}{1+a\text{exp}\left(-b(\vartheta_{k}[n]-a)\right)},$
(9)

where $a$ and $b$ are the parameters related to environment, and
$\vartheta_{k}[n]$ is given by

$\vartheta_{k}[n]=\frac{180}{\pi}\arctan\left(\frac{H}{\|{\bf u}_{k}[n]-{\bf
q}_{m}[n]\|}\right).$ (10)

Additionally, the non-line-of-sight (NLoS) channel probability can be
expressed as
$\mathbb{P}(\text{NLoS},\vartheta_{k}[n])=1-\mathbb{P}(\text{LoS},\vartheta_{k}[n])$.
Therefore, the expected channel power gain of from MU $k$ to UAV $m$ is given
by

$\displaystyle h_{k,m}[n]$
$\displaystyle=\frac{\beta_{0}\mathbb{P}(\text{LoS},\vartheta_{k}[n])}{\|{\bf
u}_{k}[n]-{\bf
q}_{m}[n]\|^{\tilde{\iota}}}+\frac{\nu\beta_{0}\mathbb{P}(\text{NLoS},\vartheta_{k}[n])}{\|{\bf
u}_{k}[n]-{\bf q}_{m}[n]\|^{\tilde{\iota}}}$
$\displaystyle=\frac{\nu\beta_{0}\mathbb{P}(\text{NLoS},\vartheta_{k}[n])}{\|{\bf
u}_{k}[n]-{\bf q}_{m}[n]\|^{\tilde{\iota}}},$ (11)

where $\tilde{\iota}$ is the path loss exponent, $\nu$ is the NLoS
attenuation, $\beta_{0}$ is the channel power gain at the reference distance
of 1 m. Similar to [30], we assume that the change of LoS probability
$\mathbb{P}(\text{LoS},\vartheta_{k}[n])$ in each time slot is negligible.
Furthermore, channel from UAV $m$ to BS can be modeled by quasi-static block
fading LoS link [31], i.e.,

$h_{m}^{\text{rel}}[n]=\frac{\beta_{0}}{\|{\bf q}_{m}[n]-{\bf
u}_{\text{BS}}\|^{2}}.$ (12)

We consider the orthogonal frequency division multiple access scheme for data
transmission. In each time slot, MU $k$ first requests for offloading, then
UAV $m$ allocates an orthogonal frequency bandwidth of $B_{k,m}[n]$ for
transmission. Therefore, the transmit rates between MU $k$ and UAVs, and UAV-
BS are given by

$R_{k}[n]=\sum\limits_{m=1}^{M}\alpha_{k,m}[n]B_{k,m}[n]\log_{2}\left(1+\frac{p_{k}h_{k,m}[n]}{B_{k,m}[n]N_{0}}\right),$
(13)
$\begin{split}R_{m}^{\text{rel}}[n]=B_{u}\log_{2}\left(1+\frac{p_{m}h_{m}^{\text{rel}}[n]}{B_{u}N_{0}}\right),\end{split}$
(14)

where $p_{k}$ and $p_{m}$ denote the uplink transmit power of MU $k$ and UAV
$m$, respectively, $B_{u}$ is the bandwidth allocated to UAVs for relaying,
and $N_{0}$ is the noise power density.

Since each MU can associate with at most one UAV or BS in each time slot $n$,
and the bandwidth should be only allocated to links between associated
devices, we have

$\displaystyle\sum\limits_{m=0}^{M}\alpha_{k,m}[n]=1,\forall
k\in\mathcal{K},n\in\mathcal{N},$ (15)
$\displaystyle\sum\limits_{m=1}^{M}\sum\limits_{k=1}^{K}B_{k,m}[n]\leq
B,\forall n\in\mathcal{N},$ (16) $\displaystyle B_{k,m}[n]\geq 0,\forall
k\in\mathcal{K},\forall m\in\mathcal{M},n\in\mathcal{N},$ (17)
$\displaystyle\lceil B_{k,m}[n]/B\rceil=\alpha_{k,m}[n],\forall
k\in\mathcal{K},m\in\mathcal{M},n\in\mathcal{N},$ (18)

where $B$ denotes the available channel bandwidth. Then the size of total task
transmitted to UAV $m$ by MUs and the CPU cycles of MU $k$’s task executed by
UAV $m$ can be respectively calculated as

$\displaystyle
L_{m}^{\text{trans}}[n]=\sum\limits_{k=1}^{K}\alpha_{k,m}[n]\rho_{k}[n]L_{k}[n],$
(19) $\displaystyle
Y_{k,m}^{\text{edge}}[n]=\alpha_{k,m}[n](1-\alpha_{k,M+1}[n])\rho_{k}[n]L_{k}[n]C_{k}[n].$
(20)

Following prior research [32, 17], we assume that the computation result with
small size can be downloaded to MU with a negligible transmission latency.
Additionally, BS is equipped with energy supply and powerful MEC server, hence
the computing time and energy at BS are not considered. Thus, the total
estimated edge computing latency of MU $k$ can be expressed as

$\displaystyle T_{k}^{\text{edge}}[n]=$
$\displaystyle\sum\limits_{m=1}^{M}\alpha_{k,m}[n]\left(T_{k}^{\text{off}}[n]+\alpha_{k,M+1}[n]T_{k,m}^{\text{rel}}[n]\right.$
$\displaystyle\left.+(1-\alpha_{k,M+1}[n])T_{k,m}^{\text{ecmp}}[n]\right),$
(21)

where $T_{k}^{\text{off}}[n]=L_{k}[n]\rho_{k}[n]/R_{k,m}[n]$ denotes the
transmission delay of MU $k$,
$T_{m}^{\text{rel}}[n]=L_{k}[n]\rho_{k}[n]/R_{k,m}^{\text{rel}}[n]$ denotes
the time of UAV $m$ for relaying the task to BS, and
$\tilde{f}_{k,m}^{\text{edge}}[n]$ is the estimated CPU frequency of server
$m$ allocated to MU $k$ at time slot $n$. Accordingly, the computing latency
gap of UAV $m$ between DT and actual value can be expressed as

$\Delta
T_{k,m}^{\text{ecmp}}[n]=\frac{-Y_{k,m}^{\text{edge}}[n]{\hat{f}_{k,m}^{\text{edge}}[n]}}{\tilde{f}_{k,m}^{\text{edge}}[n](\tilde{f}_{k,m}^{\text{edge}}[n]+{\hat{f}_{k,m}^{\text{edge}}[n]})},$
(22)

where $\hat{f}_{k,m}^{\text{edge}}[n]$ is the estimated deviation of
$f_{k,m}^{\text{edge}}[n]$. Denoting
$\tilde{T}_{k,m}^{\text{ecmp}}[n]=Y_{k,m}^{\text{edge}}[n]/\tilde{f}_{k,m}^{\text{edge}}[n]$
as the estimated time of UAV $m$ for computing the task of MU $k$, the actual
edge computing time of MU $k$ at each time slot $n$ can be calculated by

$T_{k,m}^{\text{ecmp}}[n]=\tilde{T}_{k,m}^{\text{ecmp}}[n]+\Delta
T_{k,m}^{\text{ecmp}}[n],$ (23)

and the actual CPU frequency allocated to MU $k$ by UAV $m$ is given by
$f_{k,m}^{\text{edge}}[n]=\tilde{f}_{k,m}^{\text{edge}}[n]+\hat{f}_{k,m}^{\text{edge}}[n]$.
Since it is inappropriate to allocate the CPU frequency to MUs which are not
choosing to offload to the UAVs, i.e., computing at local or BS, we have

$\displaystyle\lceil\tilde{f}_{k,m}^{\text{edge}}[n]/f_{\max}^{\text{edge}}\rceil=$
$\displaystyle\alpha_{k,m}[n](1-\alpha_{k,M+1}[n]),$ $\displaystyle\forall
k\in\mathcal{K},m\in\mathcal{M},n\in\mathcal{N}.$ (24)

For the location data, a random noise is imposed on the observation of
devices, and thus has an influence on the decisions of algorithm, which will
illustrated in Section III.

###  II-E Energy Consumption Model

As the entities (MUs and UAVs) can upload their essential status information
to DT layer via UAVs with specific bandwidth, the energy consumption can be
evaluated by the DT layer. The energy models of MUs and UAVs are illustrated
as follows.

####  II-E1 MU energy consumption

The energy consumption of MU $k$ comprises the local computing energy and
transmit energy, which are denoted as $E_{k}^{\text{loc}}[n]$ and
$E_{k}^{\text{off}}[n]$, respectively. Therefore, we have

$E_{k}^{\text{loc}}[n]=\kappa f_{k}^{\text{loc}}[n]^{2}Y_{k}^{\text{loc}}[n],$
(25) $E_{k}^{\text{off}}[n]=p_{k}[n]T_{k}^{\text{off}}[n],$ (26)

where $\kappa$ is the effective capacitance coefficient determined by chip
architecture.

####  II-E2 UAV energy consumption

The energy consumption of UAVs is imposed by flying and computation. In each
time slot $n$, UAV $m$ is assumed to fly under the limits of the maximum speed
$v_{\max}$ and the maximum acceleration $a_{\max}$, which indicates that

$\|{\bf a}_{m}[n]\|=\frac{\|{\bf v}_{m}[n+1]-{\bf v}_{m}[n]\|}{\delta_{t}}\leq
a_{\max},\forall m\in\mathcal{M},n\in\mathcal{N},$ (27) ${\bf q}_{m}[n+1]={\bf
q}_{m}[n]+{\bf v}_{m}[n]\delta_{t}+\frac{1}{2}{\bf
a}_{m}[n]\delta_{t}^{2},\forall m\in\mathcal{M},n\in\mathcal{N},$ (28) $\|{\bf
v}_{m}[n]\|=\frac{\|{\bf q}_{m}[n+1]-{\bf q}_{m}[n]\|}{\delta_{t}}\leq
v_{\max},\forall m\in\mathcal{M},n\in\mathcal{N}.$ (29)

Subsequently, we adopt the UAV’s propulsion energy model introduced by [33],
the flying energy of UAV $m$ is then expressed as follows

$\displaystyle E_{m}^{\text{fly}}[n]=$
$\displaystyle\Bigg{[}\frac{1}{2}d_{0}\rho sA\|{\bf
v}_{m}[n]\|^{3}+P_{0}\left(1+\frac{3\|{\bf
v}_{m}[n]\|^{3}}{U_{\text{tip}}^{2}}\right)$
$\displaystyle+P_{i}\left(\sqrt{1+\frac{\|{\bf
v}_{m}[n]\|^{4}}{4v_{0}^{4}}}-\frac{\|{\bf
v}_{m}[n]\|^{2}}{2v_{0}^{2}}\right)\Bigg{]}\delta_{t},$ (30)

where $P_{0}$ and $P_{i}$ are the blade profile power and induced power in
hovering status, respectively, $U_{{\text{tip}}}$ is the tip speed of the
rotor blade, $v_{0}$ is the mean rotor velocity, $d_{0}$ is fuselage drag
ratio, $s$ denotes the rotor solidity, $\rho$ denotes the air density, and $A$
denotes the rotor disc area.

Note that the communication energy of UAV is relatively small compared to the
energy consumption imposed by the intensive computation and flying, which can
be neglected. The computation energy of UAV $m$ is calculated as

$E_{m}^{\text{edge}}=\sum\limits_{k=1}^{K}\kappa\alpha_{k,m}[n]f_{k,m}^{\text{edge}}[n]^{2}Y_{k,m}^{\text{edge}}[n].$
(31)

As a consequence, the overall energy consumption of MUs and UAVs are
calculated by

$E_{k}^{\text{MU}}[n]=E_{k}^{\text{loc}}[n]+E_{k}^{\text{off}}[n],$ (32)
$E_{m}^{\text{UAV}}[n]=E_{m}^{\text{fly}}[n]+E_{m}^{\text{edge}}[n].$ (33)

###  II-F Problem Formulation

Considering the deficient energy resource of MUs and limited energy budget of
UAVs, we pursue a multi-UAV-assisted MEC network with air-ground cooperation
design, aiming at minimizing the weighted energy consumption of MUs and UAVs
with latency sensitive tasks, by jointly optimizing the MU association
$\textbf{A}\triangleq\\{\alpha_{k,m}[n],\forall
n\in\mathcal{N},k\in\mathcal{K},m\in\mathcal{M}^{\star}\cup\\{0\\}\\}$, the
offloading proportion $\bm{\varrho}\triangleq\\{\rho_{k}[n],\forall
n\in\mathcal{N},k\in\mathcal{K}\\}$, the CPU frequency allocation
$\textbf{F}\triangleq\\{\tilde{f}_{k,m}^{\text{edge}}[n],\forall
m\in\mathcal{M},k\in\mathcal{K},n\in\mathcal{N}\\}$, the bandwidth allocation
$\textbf{B}\triangleq\\{B_{k,m}[n],\forall
m\in\mathcal{M},k\in\mathcal{K},n\in\mathcal{N}\\}$, and the UAV velocity
$\textbf{V}\triangleq\\{{\bf v}_{m}[n],\forall
m\in\mathcal{M},n\in\mathcal{N}\\}$. Therefore, the weighted energy
minimization problem is formulated as

$\displaystyle\mathcal{P}1$
$\displaystyle\min\limits_{\textbf{A},\textbf{B},\textbf{F},\bm{\varrho},\textbf{V}}\quad\varpi\sum\limits_{n=1}^{N}\sum\limits_{m=1}^{M}E_{m}^{\text{UAV}}[n]+\sum\limits_{n=1}^{N}\sum\limits_{k=1}^{K}E_{k}^{\text{MU}}[n]$
(34a) s.t
$\displaystyle\eqref{c:alpha_zeroone},\eqref{c:alpha_1},\eqref{c:band_alloc},\eqref{c:Bge0},\eqref{c:band_alloc_on},\eqref{c:freq_alloc_on},\eqref{c:a},\eqref{c:u},\eqref{c:v},$
(34b) $\displaystyle 0\leq\rho_{k}[n]\leq 1,\forall
k\in\mathcal{K},n\in\mathcal{N},$ (34c)
$\displaystyle\lceil\rho_{k}[n]\rceil=1-\alpha_{k,0}[n],\forall
k\in\mathcal{K},n\in\mathcal{N},$ (34d)
$\displaystyle\sum\limits_{k=1}^{K}f_{k,m}^{\text{edge}}[n]\leq
f_{\max}^{\text{edge}},\forall m\in\mathcal{M},n\in\mathcal{N},$ (34e)
$\displaystyle{T}_{k}^{\text{loc}}[n]\leq\delta_{t},\forall
k\in\mathcal{K},\forall m\in\mathcal{M},n\in\mathcal{N},$ (34f)
$\displaystyle{T}_{k}^{\text{edge}}[n]\leq\delta_{t},\forall
k\in\mathcal{K},\forall m\in\mathcal{M},n\in\mathcal{N},$ (34g)
$\displaystyle\|{\bf q}_{i}[n]-{\bf q}_{j}[n]\|^{2}\geq
d_{\rm{min}}^{2},\forall i,j\in\mathcal{M},i\neq j,n\in\mathcal{N},$ (34h)

where $\varpi$ is the weight factor, $d_{\rm{min}}$ denotes the minimum safe
distance between the UAVs. Constraints (5) and (15) ensure the feasibility of
the association status, constraints (16)-(18) guarantee that the bandwidth is
only allocated to valid links, constraints (27)-(29) denote the velocity and
acceleration limits of UAVs, constraints (34c) and (34d) indicate the range
and the validness of offloading proportion, constraints (24) and (34e) denotes
feasibility of computational resource allocation, constraints (34f) and (34g)
denote the computational latency constraints, and constraint (34h) limits the
minimum safe distance between UAVs.

It can be readily derived that $\mathcal{P}1$ is a mixed integer non-convex
and combinational problem with huge amount of highly decoupled variables.
Furthermore, the proposed problem needs to be timely solved in each time slot
$n$ due to the randomly generated tasks and the movement of devices.
Nonetheless, it is very intricate to solve the problem by typical iterative
techniques such as alternative optimization and genetic algorithm. In the
following, we propose an efficient algorithm for this complicated problem by
harvesting the MARL approach.

##  III Proposed DRL Approach: AB-MAPPO

In this section, the essential elements of MDP in multi-agent DRL are first
illustrated. Then, we endeavor to develop an MAPPO approach with attention
mechanism and Beta distribution to address problem $\mathcal{P}1$.

###  III-A MDP Elements Formulation

In MARL setup, problem $\mathcal{P}1$ can be modeled as an MDP with the set of
agents $\mathcal{I}\triangleq\\{1,\dots,K+M\\}$, state space $\mathcal{S}$,
action space
$\mathcal{A}=\mathcal{A}_{1}\times\mathcal{A}_{2}\times\dots\times\mathcal{A}_{K+M}$.
The agents can enhance their policy by interacting with the environment in
discrete steps. In each step $t$, each agent $i$ obtains the current
observation $o_{i}(t)$ from the global environment state
$s(t)\triangleq\\{o_{i}(t),\forall i\in\mathcal{I}\\}$, takes action
$a_{i}(t)\in\mathcal{A}_{i}$, then obtains a reward $r_{i}(t)$, and the
environment transfers to a new state $s(t+1)$.

The MDP involve two types of agents, i.e., MUs and UAVs. At the beginning of
each step $t$, the MUs generate actions, and then request for offloading
according to the actions. Afterwards, the UAVs obtain the actions and serve
the MUs. The status information is periodically synchronized to DT if is
required, and rewards are evaluated centrally by DT. The formulation of MDP
elements are illustrated as follows.

####  III-A1 MDP elements of MUs

The MDP elements of MUs involve the observation $o_{k}(t)$, action $a_{k}(t)$
and reward $r_{k}(t)$, which are presented as follows.

  * •

Observation: From the perspective of privacy, MUs can only obtain the
positions of themselves ${\bf u}_{k}[n]$ by Global Positioning System, and the
positions of MEC servers ${\bf q}_{m}[n]$ from the broadcast of the DT layer.
The position data accordingly has a random deviation, which is also imposed to
the observations of agents. Furthermore, the task information $\Omega_{k}[n]$
can be observed. As the task load on UAVs is highly relevant to MUs’
offloading decisions, the previous computation load of UAVs are also involved.
Therefore, the observation of MU $k$ at step $t$ can be expressed as

$o_{k}(t)=\Big{\\{}{\bf u}_{k}[n],{\bf
q}_{m}[n],\Omega_{k}[n],Y_{k,m}^{\text{edge}}[n-1],\forall
m\in\mathcal{M}\Big{\\}}.$ (35)

  * •

Action: The decisions of MUs involve offloading association A, and offloading
proportion $\bm{\varrho}$. Therefore, for each MU $k$, the action is
decomposed by

$a_{k}(t)=\Big{\\{}\alpha_{k,m}[n],\rho_{k}[n],\forall
m\in\mathcal{M}^{*}\cup\\{0\\}\Big{\\}}.$ (36)

  * •

Reward: It can be noticed from $\eqref{obj}$ that the MUs need to consider
their own influence on the total weighted energy consumption and the load on
UAV servers. Hence, the reward of each MU $k$ should involve the energy
consumption of both MU $k$ itself and the UAV associated by MU $k$. The reward
of each MU agent $k$ is given by

$\displaystyle r_{k}(t)=$
$\displaystyle-\sum\limits_{m=1}^{M}\alpha_{k,m}[n]\left(\varpi
E_{m}^{\text{UAV}}[n]+P^{t}(T_{k}^{\text{loc}}[n],T_{m}^{\text{edge}}[n])\right)$
$\displaystyle-E_{k}^{\text{MU}}[n],$ (37)

where

$\displaystyle P^{t}(T_{k}^{\text{loc}}[n],T_{m}^{\text{edge}}[n])=$
$\displaystyle\frac{\mu_{t}}{\delta_{t}}\Big{(}\text{ReLU}(T_{k}^{\text{loc}}[n]-\delta_{t})$
$\displaystyle+\text{ReLU}(T_{m}^{\text{edge}}[n]-\delta_{t})\Big{)}$ (38)

denotes the penalty for unsatisfaction of latency constraints, $\mu_{t}$ is a
penalty factor, and the $\text{ReLU}(\cdot)$ is the rectified linear unit
function.

####  III-A2 MDP element of UAVs

Note that each UAV needs to make decision after MUs give their association.
Herein, MDP elements are denoted as follows

  * •

Observation: The UAV agent $m$ observes the locations of themselves, locations
of all MUs as well as the UAVs, and task information from associated MUs.
Denoting $-m$ as the index set of UAVs except for UAV $m$, we have

$o_{K+m}(t)=\Big{\\{}\rho_{k}[n],\Omega_{k}[n],{\bf u}_{k}[n],{\bf
q}_{m}[n],{\bf q}_{-m}[n]\Big{\\}}.$ (39)

  * •

Action: After receiving the requests, UAVs need to allocate the bandwidth,
configure the computational frequency, and adjust velocity according to the
observations. Therefore, the action of UAV agent $m$ is given by

$a_{K+m}(t)=\bigg{\\{}B_{k,m}[n],\tilde{f}_{k,m}^{\text{edge}}[n],{\bf
a}_{m}[n],\forall k\in\mathcal{K}\bigg{\\}}.$ (40)

  * •

Reward: After receiving, relaying and executing the tasks from MUs, the UAVs
get rewards from environments. The reward of each UAV $m$ needs to consider
the energy consumption of both itself and the served MUs, i.e.,

$\displaystyle r_{K+m}(t)=$
$\displaystyle-\sum\limits_{k=1}^{K}\alpha_{k,m}[n]\left(E_{k}^{\text{MU}}[n]+P^{t}({T}_{k}^{\text{loc}}[n],{T}_{m}^{\text{edge}}[n])\right)$
$\displaystyle+\varpi E_{m}^{\text{UAV}}[n]+P^{o}({\bf q}_{m}[n])+P^{c}({\bf
q}_{m}[n])$ $\displaystyle+P^{d}({\bf q}_{m}[n]),$ (41)

where

$\displaystyle P^{o}({\bf q}_{m}[n])=\mu_{o}\|{\bf q}_{m}[n]-\text{clip}({\bf
q}_{m}[n],0,W)\|,$ (42)

denotes the penalty when UAVs try to fly out of the square boundary with width
$W$, and $\mu_{o}$ is a penalty factor.

$\displaystyle P^{d}({\bf q}_{m}[n])=$ $\displaystyle\frac{1}{W}\bigg{(}\|{\bf
q}_{m}[n]-\frac{1}{|\mathcal{K}_{m}|}\sum\limits_{k=1}^{K}\alpha_{k,m}[n]{\bf
w}_{k}[n]\|$ $\displaystyle-d_{\rm th}\bigg{)}$ (43)

guides the distance from UAV $m$ to MUs, $d_{\rm th}$ is a distance threshold,
and $|\mathcal{K}_{m}|$ is the set of MUs associated with UAV $m$. In
addition,

$\displaystyle P^{c}({\bf q}_{m}[n])$
$\displaystyle=\mu_{c}\sum\limits_{j=1,j\neq m}^{M}\min\Big{\\{}\|{\bf
q}_{m}[n]-{\bf q}_{j}[n]\|-d_{\min},0\Big{\\}}/d_{\min}$ (44)

is the penalty for disobeying the safety distance $d_{\min}$ between UAVs, and
$\mu_{c}$ is corresponding penalty factor. It is assumed that the UAVs will
stop at the boundary if they try to fly out of it, and thus ${\bf
q}_{m}[n]\leftarrow\text{clip}({\bf q}_{m}[n-1]+{\bf
v}_{m}[n-1]\delta_{t}+\frac{1}{2}{\bf a}_{m}[n-1]\delta_{t}^{2},0,W)$.

Figure 2: Training framework of AB-MAPPO.

###  III-B MAPPO-based DRL Approach With CTDE

As a variant of PPO specialized for multi-agent settings, MAPPO is one of the
state-of-the-art MARL algorithms [34]. Based on on-policy scheme, each agent
has an actor, a critic, and a replay buffer. As DT is capable of evaluating
the global state of the environment, the centralized training and
decentralized executing (CTDE) scheme can be adopted. In MAPPO with CTDE, the
critics evaluate the centralized state-value function of global environment
state. Specifically, the agents repeatedly input the observations $o_{i}(t)$
into actor networks and get $a_{i}(t)$ and $r_{i}(t)$ for an episode, storing
the experiences into buffer. Then in the end of an episode, the agents update
their policies. Firstly, they sample some batches of experiences
$\\{o_{i}(t),a_{i}(t),r_{i}(t),s(t),\text{pr}_{i}(t)\\},\forall
i\in\mathcal{I}$ from their buffers, where the $\text{pr}_{i}(t)$ denotes the
log-probability for sampling the action $a_{i}(t)$. Then, in each update, the
actor and critic update the parameters with the loss of policy and global
state-value, respectively. The loss of actor $i$ is calculated by

$\displaystyle
L^{\text{actor}}(\theta_{i})=\mathbb{E}_{\pi_{\theta_{i}}}\Bigg{\\{}\min\bigg{[}\frac{\pi_{\theta_{i}}(a_{i}(t)|o_{i}(t))}{\pi_{\theta_{i}^{{}^{\prime}}}(a_{i}(t)|o_{i}(t))}\hat{A}_{i}(t),$
$\displaystyle\text{clip}\left(\frac{\pi_{\theta_{i}}(a_{i}(t)|o_{i}(t))}{\pi_{\theta_{i}^{{}^{\prime}}}(a_{i}(t)|o_{i}(t))},1-\epsilon,1+\epsilon\right)\hat{A}_{i}(t)\bigg{]}\Bigg{\\}},$
(45)

where $\pi_{\theta_{i}^{{}^{\prime}}}$ and $\pi_{\theta_{i}}$ denote the old
and current policy, respectively. $\hat{A}_{i}(t)$ is the estimation of
advantage function $A_{i}(t)=Q_{i}(s(t),a_{i}(t))-V_{i}(s(t))$. In PPO series,
the generalized advantage estimation (GAE) is adopted to improve the
performance, which is defined as follows

$\displaystyle\hat{A}_{i}(t)=$
$\displaystyle\sum\limits_{l=0}^{\infty}(\gamma\lambda)^{l}\Big{(}R_{i}(t+l)+\gamma
V_{i}\big{(}s(t+1+l)\big{)}$ $\displaystyle-V_{i}(s(t+l))\Big{)},$ (46)

where $\gamma$ is the discount factor, $\lambda$ is the parameter of GAE for
bias-variance tradeoff in estimation, and
$V_{i}(s(t))=\sum\limits_{l=0}^{\infty}\gamma^{l}R_{i}(t+l)$ is the cumulative
discounted reward, which also represents the state-value function. Denoting
$V^{\xi_{i}}(s(t))$ as the state-value function estimated by the critic of
agent $i$, the loss of critic $i$ is given by

$L_{i}^{\text{critic}}(\xi_{i})=\frac{1}{2}\Big{[}V^{\xi_{i}}\big{(}s(t)\big{)}-V_{i}\big{(}s(t)\big{)}\Big{]}^{2},$
(47)

where $\xi_{i}$ is the parameter of $i$-th critic network. Therefore, the
actor and critic can be updated according to (45) and (47), respectively.

It is worth noting that the constraints need to be satisfied by action-
remapping as follows. First, the output vectors of actor networks in MAPPO are
sampled from distributions and are divided according to the sequence for
concatenating the variables of actions. Then, the divided vectors are scaled
into their origin domain. Furthermore, some constraints need to be tackled by
normalization or rounding operation. For each MU $k$, the index of its
associated UAV is with respect to the maximum value of the divided vector
corresponding to $\\{\alpha_{k,m}[n],\forall
m\in\mathcal{M}^{*}\cup\\{0\\}\\}$, and thus $\\{\alpha_{k,m}[n],\forall
m\in\mathcal{M}\cup\\{0\\}\\}$ is obtained. If $\alpha_{k,0}[n]=0$,
$\alpha_{k,M+1}[n]$ will be decided by rounding its entry in the divided
vector. For constraints (18) and (34d) on $B_{k,m}[n]$ and $\rho_{k}[n]$, we
multiply their divided vectors by
$[\alpha_{k,1}[n],\alpha_{k,2}[n],\ldots,\alpha_{k,M}[n]]$ and
$1-\alpha_{k,0}[n]$ as action masks. And then the constraints (16) and (34e)
are satisfied by normalizing their divided vectors.

###  III-C MAPPO-based Training Framework

As displayed in Fig. 2, during the proposed DT-assisted CTDE process, the MUs
and UAVs perform computation offloading according to the actions given by the
actor networks of their agents in physical environment, send their own
experiences and synchronizing system status to the DT layer. Then, the DT
layer evaluates the global environment state by the observations of agents,
updates the buffers, and gets the predicted values. After updating the actors
and critics, the parameters of actor networks are downloaded to UAVs and MUs.
Note that the network parameters can be shared between homogeneous agents
[34]. To fully exploit the performance of MAPPO approach, we introduce Beta
distribution and attention, which are stated as below.

####  III-C1 Beta policy

The above-mentioned actions are typically continuous and bounded, such as
${\bf\varrho}\in[0,1]$ and ${\bf a}_{k}[n]\in[-a_{\max},a_{\max}]$. However,
conventional action sampling from Gaussian distribution in policy networks
will unavoidably introduce an estimation bias of policy gradient, since the
boundary effects will be imposed by force clipping the values of out-of-bound
actions. To tackle this problem, we adopt Beta distribution instead of
Gaussian distribution in terms of the output of policy networks, which has the
following form

$f(x,\alpha,\beta)=\frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}x^{\alpha-1}\Big{(}1-x\Big{)}^{\beta-1},$
(48)

where $\alpha$ and $\beta$ are the parameters of Beta distribution. Since (48)
has also a bounded domain, it is appropriate to sample bounded actions.
Moreover, the Beta distribution can be more close to uniform distribution than
Gaussian by initializing, and thus the agents can more abundantly explore the
action space at initial stage of training.

Algorithm 1 Proposed AB-MAPPO algorithm

1: Initialize $n=1$, episode length epl, PPO epochs Pe, and maximum episodes
${\rm e}^{\max}$.

2: for agent $i\in\mathcal{I}$ do

3: Initialize actor networks $\theta_{i}$, critic networks $\xi_{i}$,replay
buffer $\bm{B}_{i}$;

4: end for

5: for Episode = 1,$\dots$, ${\rm e}^{\max}$ do

6: for $t$ = 1,$\dots$, epl do

7: Obtain $o_{k}(t),\forall k\in\mathcal{K}$ from the environment;

8: Execute the action $a_{k}(t)$, $\forall k\in\mathcal{K}$;

9: Obtain $o_{K+m}(t),\forall m\in\mathcal{M}$ from the environment;

10: Execute the action $a_{K+m}(t),\forall m\in\mathcal{M}$;

11: Calculate log-probability $\text{pr}_{i}(t),\forall i\in\mathcal{I}$;

12: Synchronize status information to the DT layer if is required;

13: if $n$ mod $N$ = 0 then

14: Synchronize observations $o_{i}(t)$, actions $a_{i}(t)$ to the DT layer;

15: Evaluate reward $r_{i}(t),\forall i\in\mathcal{I}$ in DT layer;

16: Store transition
$\text{Tr}_{i}(t)=${$o_{i}(t),a_{i}(t),r_{i}(t),s(t),\text{pr}_{i}(t)$},
$\forall i\in\mathcal{I}$ into buffer $B_{i}$;

17: end if

18: Update $n=n$ mod $N$ \+ 1

19: end for

20: for epoch = $1,\dots\rm Pe$ do

21: for agents $i\in\mathcal{I}$ do

22: Update actor $\theta_{i}$ and critic $\xi_{i}$ according to (45) and (47)
by $\forall\text{Tr}_{i}(t)\in\bm{B}_{i}$;

23: end for

24: end for

25: Download the actor networks from DT layer to UAVs and MUs;

26: end for

####  III-C2 Attention mechanism

With the increasing number of agents, the ever-increasing dimensions of
environment state make it difficult for the critic network with a simple
fully-connected layer to tackle the input, thereby leading to slower or even
difficult convergence of critic network and being harmful to the behavior of
actor network. However, for each agent, other agents may impose the impact
with different intensities on the state-value, and thus the distinguished
attention should be paid to them, which involves the nearby devices and the
devices with higher computational load. Therefore, we introduce the multi-head
attention unit before the multi-layer perceptron (MLP) of critic networks to
improve the training process. The vectors of observations for two types of
agents are first passed through their corresponding 3-layer MLPs to get
feature values $e_{i}$. Then, all the feature values of agents are sent to the
attention heads to get the attention values $x_{i}$ by

$\alpha_{i,j}=\text{Softmax}\left(\frac{e_{j}^{\text{T}}W_{\text{key}}^{\text{T}}W_{q}e_{i}}{\sqrt{d_{\text{key}}}}\right),$
(49) $x_{i}=\sum\limits_{j\neq i}\alpha_{i,j}W_{v}e_{j},$ (50)

where $e_{j}$ is the feature value of another agent $j$, $d_{\text{key}}$ is
the variance of $e_{j}^{\text{T}}W_{\text{key}}^{\text{T}}W_{q}e_{i}$. The
matrix $W_{\text{key}}$ transforms $e_{j}$ into a key, the matrix $W_{v}$
transforms $e_{j}$ into a value, and the matrix $W_{q}$ transforms $e_{i}$
into a query. Finally, $x_{i}$ and $o_{i}(t)$ are sent to the MLP to get the
estimated state-value $V^{\xi_{i}}(s(t))$.

###  III-D Complexity Analysis

Based on above-mentioned discussions, we propose the AB-MAPPO algorithm, which
is summarized in Algorithm 1. The complexity of attention module is
$\mathcal{O}(I^{2}V)$, where $V$ is the length of feature-value vectors,
according to [35]. For an MLP, the computational complexity of the $j$-th
layer is $\mathcal{O}(Z_{j-1}Z_{j}+Z_{j}Z_{j+1})$, where $Z_{j}$ is the number
of neurons for $j$-th layer. Hence, the computational complexity of a
$J$-layer MLP is calculated by
$\mathcal{O}\left(\sum_{j=2}^{J-1}Z_{j-1}Z_{j}+Z_{j}Z_{j+1}\right)$. The actor
networks are all 3-layer MLPs, and each of two critic networks for MU and UAV
agents have two 3-layer MLPs for the observations of different types agents
and one 3-layer MLP after attention module for the output of value function.
Therefore, the overall computational complexity for training is calculated by
the sum of complexity imposed by actor and critic networks
$\mathcal{O}\left({\rm e}^{\max}({\rm Pe}I^{2}V+{\rm
epl}\sum_{j=2}^{J-1}Z_{j-1}Z_{j}+Z_{j}Z_{j+1})\right)$, and for one-step
execution is just calculated by
$\mathcal{O}\left(\sum_{j=2}^{J-1}Z_{j-1}Z_{j}+Z_{j}Z_{j+1}\right)$.

TABLE II: Environment Parameters Parameters |  Value
---|---
Channel bandwidth $B$ |  50 MHz
Channel power gain $\beta$ |  -30 dB
Environment parameters $a$ and $b$ |  15,0.5
Time period $T$ |  60 s
Time slot $\delta_{t}$ |  1 s
Transmit power of MUs $p_{k}^{\text{MU}}$ |  0.2 W
Transmit power of UAVs $p_{m}^{\text{UAV}}$ |  0.5 W
Path loss exponent $\tilde{\iota}$ |  2.2
NLoS attenuation $\nu$ |  0.2
Noise power density $N_{0}$ |  -127 dBm/Hz
Effective capacitance coefficient $\kappa$ |  $10^{-27}$
Maximum CPU frequency of MU $k$ $f_{\max}^{\text{loc}}$ |  1 GHz
Maximum CPU frequency of UAV $m$ $f_{\max}^{\text{edge}}$ |  10 GHz
Maximum velocity of UAV $v_{\max}$ |  30 m/s
Maximum acceleration $a_{\max}$ |  5 $\text{m/s}^{2}$
UAV settings $P_{0},P_{i}$ |  39.04 W, 79.07 W
UAV settings $U_{\text{tip}},v_{0},A$ |  120 m/s, 3.6 m/s, 0.5030 $\text{m}^{2}$
Weight factor $\varpi$ |  0.001

##  IV Numerical Results

In this section, we evaluate the performance of the proposed AB-MAPPO
algorithm. The simulation settings and numerical results are presented as
follows.

###  IV-A Simulation Settings

In our simulations, a 1000 $\times$ 1000 m rectangular area is considered,
where the MUs are randomly distributed in region with $x,y\in[0,1000]$ m, the
BS is set at (-500 m, 0 m, 10 m), and the UAVs are flying at $H=200$ m. Note
that since the energy consumption of UAVs and MUs are sensitive and the BS
typically has abundant power supply, the energy consumption of the BS
involving computation and DT construction is not considered. Other parameter
settings of simulation and hyperparameters of MAPPO are summarized in Table II
and Table III, according to prior works [16, 17, 36, 33].

The baseline algorithms for comparison with our proposed AB-MAPPO are listed
as follows:

  * •

B-MAPPO: MAPPO without attention mechanism and with Beta distribution.

  * •

AG-MAPPO: MAPPO with Gaussian distribution and attention mechanism.

  * •

MADDPG: MADDPG is an off-policy algorithm with determinate policy [18]. The
exploration noise is set as 0.5, the buffer size is set as 20000, and other
parameters are the same to the proposed scheme.

  * •

Randomized: The MUs and UAVs execute randomly generated actions by the
algorithm.

TABLE III: Hyperparameters Parameters of AB-MAPPO Parameters | Value
---|---
Total number of steps | 80k
Length of an episode epl | 300
Learning rate of actor | 0.0003
Discount factor of UAVs | 0.95
Discount factor of MUs | 0.8
Penalty factors $\mu_{o}$, $\mu_{t}$, and $\mu_{c}$ | 0.1
Number of heads in attention unit | 4
Number of mini-batches | 1
PPO epoch Pe | 5
Learning rate of critic | 0.0003
Number of hidden layers of MLPs | 1
Hidden size of MLPs | 128
Optimizer | Adam
$d_{\rm th}$ | 300 m

###  IV-B Convergence of the MARL Training Algorithm

(a) The average reward of MUs.

(b) The average reward of UAVs.

Figure 3: The convergence of the rewards of agents.

We first evaluate the convergence performance of the proposed AB-MAPPO
algorithm in Fig. 3(a) and Fig. 3(b). The average episode reward of all MUs is
evaluated with comparison of RL baselines under $M=10$ UAVs and $K=60$ MUs. It
can be seen that the average episode reward of UAVs obviously increases during
the training process and then reaches to the stable values. Intuitively, the
proposed AB-MAPPO scheme, B-MAPPO scheme, AG-MAPPO scheme, and MADDPG scheme
converge at around 30k, 45k, 60k and 60k steps, respectively. As expected, the
proposed scheme outperforms other baselines, with the fastest convergence and
reward. Specifically, the proposed AB-MAPPO scheme converge faster than the
B-MAPPPO scheme, and the MAPPO schemes with Beta distribution achieve greater
rewards than the AG-MAPPO scheme. The reason can be explained as follows: 1)
The attention mechanism makes the critic networks quickly concentrate on the
significant parts of the global state, thereby accelerating the convergence.
2) With the Beta distribution, the agents explore more uniformly at the
initial stage of training, retain better exploration ability, and converge to
better solutions, which also presents the superiority of Beta distribution
over Gaussian distribution and the MADDPG’s exploration noise.

Figure 4: The convergence of weighted energy consumption.

Figure 5: The improvement of fairness of MUs.

Subsequently, Fig. 5 presents the convergence of the average weighted energy
consumption of users with $K=60$ MUs and $M=10$ UAVs. It indicates that the
energy consumption has also been optimized with the increase of reward during
the training process. Intuitively, the proposed AB-MAPPO scheme achieves lower
energy consumption and has the most rapid convergence. In terms of stability,
the energy consumption of the proposed AB-MAPPO scheme goes down more
smoothly, the AG-MAPPO and the B-MAPPO schemes have higher fluctuation during
training, and the MADDPG scheme is more tortuous. The above results verify the
effectiveness and reliability of our proposed scheme.

To verify the fairness of service, we present the varying of Jain’s fairness
index of MUs in Fig. 5 during training. It is worth noting that the Jain’s
fairness index of service is denoted as
$\frac{\sum\limits_{k=1}^{K}E_{k}^{\text{MU}}[n])^{2}}{(K\sum\limits_{k=1}^{K}(E_{k}^{\text{MU}}[n])^{2}}$.
We can find that the fairness of MUs gradually grows during the training
process and the fairness of proposed AB-MAPPO is higher and more stable
against the AG-MAPPO scheme. Another observation is that the fairness of both
schemes slightly decreases at first and then ascends, while the AG-MAPPO
scheme converges faster but has lower fairness. The ascending of fairness is
due to the fact that the policy of MUs gradually becomes stable, indicating
that both the quality and reliability of service have been improved.

###  IV-C Comparison of Benchmarks Under Different Settings

Figure 6: The impact of MUs on average weighted energy consumption of MUs.

Figure 7: The impact of number of UAVs on average weighted energy consumption
of MUs.

To verify the impact of number of MUs on the energy expense of the network, we
then evaluate the average weighted energy consumption of MUs, i.e.,
$\sum\limits_{k=1}^{K}\left(\varpi\sum\limits_{m=1}^{M}\alpha_{k,m}[n]E_{m}^{\text{UAV}}[n]+E_{k}^{\text{loc}}[n]\right)/K$,
versus the number of MUs in Fig. 7 with $M=10$ UAVs. From this figure, we
observe that the energy cost gradually increases as the number of MUs grows,
the proposed AB-MAPPO scheme outperforms the baselines. Additionally, the gaps
between RL-based schemes have the tendency of becoming larger, the randomized
scheme keeps at the highest energy consumption, and the performance gain of
attention mechanism increases as the number of MUs grows. Since the average
communication resource declines as the number of MUs grows, the latency and
energy for transmission of MUs increase, and the MEC servers can process less
amount of tasks. Therefore, the MUs tend to computing locally, thereby leading
to the ascending of the energy for local computing. Furthermore, the
randomized scheme uniformly generates actions, and thus has poor ability to
appropriately utilize the abundant resource at BS and UAVs.

Figure 8: The impact of bandwidth on energy consumption of MUs.

Figure 9: The impact of weight factor on energy consumption of MUs and UAVs.

Fig. 7 presents the comparison of the proposed scheme and baselines on average
weighted energy consumption of $K=70$ MUs under different number of UAVs. As
shown in this figure, when more UAVs participate in the service, the average
weighted energy of MUs gradually declines. It can be seen that the proposed
AB-MAPPO scheme has the lowest energy cost, and the B-MAPPO scheme is slightly
higher than it. The likely reason is that the Beta distribution can
efficiently improve the exploration of UAVs in large action space. Moreover,
the MADDPG scheme converges to worse solutions than that of the MAPPO scheme,
which indicates that the algorithm is relatively difficult to tackle the
increasingly challenging actions.

Figure 10: The impact of computational resource.

To gain further insights, Fig. 9 depicts the average weighted energy
consumption of MUs versus the channel bandwidth $B$. It can be seen that the
energy consumption gradually decreases when the bandwidth increases.
Furthermore, the proposed AB-MAPPO scheme obtains the best performance, and
when the bandwidth becomes larger, the gaps between RL schemes have the
tendency of decreasing. This is because more abundant communication resource
enables the MUs to offload more tasks to UAVs and BS, since lager bandwidth
imposes looser constraints on the optimization problem, which may introduce
larger penalty to hinder the exploration of agents. This in turn verifies that
the proposed AB-MAPPO scheme outperforms the baselines in tackling the
problems with strong constraints.

###  IV-D The Impact of Environment Settings on the Performance

Figure 11: The impact of DT deviation.

In Fig. 9, we examine the impact of weight factor $\varpi$ on the optimized
energy consumption of UAVs and MUs, under $K=60$ MUs and $M=10$ UAVs. It can
be observed that as the weight factor $\varpi$ increases from 0 to 0.009, the
energy consumption of UAVs sharply decreases at first, and then becomes
smoother. Meanwhile, the energy of MUs increases sharply. The reason for this
trend is that as $\varpi$ increases, the energy consumption of UAVs is more
emphasized and more tasks are computed locally by MUs, which means that more
computing energy is consumed by MUs and less energy is consumed by UAVs.

To evaluate the impact of computational resource on the energy expense of
users, we present the average weighted energy consumption of users versus the
maximum computational frequency of users and UAVs in Fig. 10(a) and Fig.
10(b), respectively. It can be readily observed that as the computational
frequency of users and UAVs increases, the energy consumption gradually
decreases, and keeps at about constant values. The likely reason is that the
increase of computational resource leads to an increase of satisfaction with
computational service. Hence, the policy of MUs on offloading proportion is
accordingly adjusted to balance the energy consumption and the penalty for the
improvement of reward. Then, the abundant computational resource makes the
latency requirements satisfied, and the policies become stable.

Figure 12: The impact of maximum task size of users.

We then evaluate the deviation of DT on system performance in Fig. 11. We can
find that as the deviation rate of DT increases, the average energy
consumption of MUs has the tendency of increasing. This can be explained by
the fact that the DT deviation reduces the accuracy of observations and the
actions of agents. As such, the improvement of the policies for agents is
influenced, and thus the reward and the energy consumption become worse.
Another observation is that as the deviation of DT increases from 0 to 0.25,
the weight energy consumption rises by about 15%, which verifies the
robustness of the proposed MARL approach.

(a) Scenario 1 with $K=12$, $M=3$, and $T=50$ s

(b) Scenario 2 with $K=12$, $M=3$, and $T=50$ s

(c) Scenario 3 with $K=60$, $M=10$, and $T=60$ s

(d) Scenario 4 with $K=60$, $M=10$, and $T=60$ s

Figure 13: The trajectories of UAVs with different scenarios.

Fig. 12 shows weighted energy consumption versus the maximum task size
$D_{\max}$ and evaluates the without DT scheme (“labeled as w/o DT”), while
fixing the minimum value $5\times 10^{5}$ for with different number of MUs.
The w/o DT scheme is evaluated by adding the random deviation rate in
[-0.5,0.5] on the normalized variables of locations and computational
frequency. This is under the following consideration. The DT-assisted
framework enables UAVs to directly synchronize the estimated information with
the DT layer at BS. Hence, the estimated deviation is cut down by avoiding the
frequent queries for exchanging the environment information between UAVs, and
thus the devices can obtain more timely status of the system. Therefore, the
estimated deviation is set higher to evaluate the w/o DT schemes, whose
performance is intuitively worse than that of DT-assisted schemes.
Additionally, it can be readily observed that more energy is consumed as the
task size grows and the bandwidth decreases. It can be explained by the fact
that more computation resource is required to satisfy the latency requirements
of MUs, and since the bandwidth decreases, the average communication resource
decreases. As such, the transmission expense ascends and the MUs tend to
compute more tasks locally, thereby leading to the increasing of energy
consumption.

Fig. 13 displays the trajectories of MUs and UAVs in different scenarios. In
Fig. 13 (a), the MUs are relatively crowded in a part of region, and the UAVs
start from random locations, with the width of region 500 m and $T=50$ s. It
can be observed that the trained UAVs are capable of flying closer to MUs and
keep hovering on the crowded MUs to pursue higher transmit rate. In Fig. 13
(b), the UAVs start at the corner of the region, and the MUs are relatively
dispersed. At the beginning, the UAVs can quickly fly to the MUs, and then
cooperate with each other to serve the associated MUs. Additionally, the UAVs
are close to most of associated MUs rather than fully involving the remote
ones, which indicates that UAVs can skillfully the long-term reward. In Figs.
13 (c) and (d), we present the trajectory with large-scale simulations with
$K=60$ users and $M=10$ UAVs. It can be seen that the UAVs can move to the
regions with more MUs and cooperatively cover the region of MUs to provide
better service and maintain the fairness. It verifies that in the situation
with larger scale, the reward function can guide the UAVs to rapidly access
the MUs.

##  V Conclusion

In this paper, we proposed a multi-UAV-assisted MEC network with air-ground
cooperation where DT is applied to enhance the offloading service. We
formulated a weighted sum energy consumption minimization problem by jointly
optimizing the offloading decision, bandwidth, flying trajectory,
communication resource and computation resource. To tackle this challenging
problem, we modeled our problem as an MDP where MUs and UAVs act as agents to
collectively interact with the environment to receive distinctive
observations. Considering the high-dimensional hybrid action space, the MAPPO
algorithm with attention mechanism and Beta distribution was leveraged to
efficiently obtain the optimal policy. Simulation results verified that the
proposed scheme can significantly reduce energy consumption of the network
compared with other benchmarks. Additionally, the combination of UAV and BS
offloading strategies can take fully advantage of communication and
computational resources, as well as adapt to the time-varying network
environments.

## References

  * [1] D. C. Nguyen, M. Ding, P. N. Pathirana, A. Seneviratne, J. Li, D. Niyato, O. Dobre, and H. V. Poor, “6G internet of things: A comprehensive survey,” _IEEE Internet Things J._ , vol. 9, no. 1, pp. 359–383, Jan. 2022\.
  * [2] F. Guo, F. R. Yu, H. Zhang, X. Li, H. Ji, and V. C. M. Leung, “Enabling massive IoT toward 6G: A comprehensive survey,” _IEEE Internet Things J._ , vol. 8, no. 15, pp. 11 891–11 915, Aug. 2021.
  * [3] K. B. Letaief, Y. Shi, J. Lu, and J. Lu, “Edge artificial intelligence for 6G: Vision, enabling technologies, and applications,” _IEEE J. Sel. Areas Commun._ , vol. 40, no. 1, pp. 5–36, Jan. 2022.
  * [4] B. Ji, Y. Wang, K. Song, C. Li, H. Wen, V. G. Menon, and S. Mumtaz, “A survey of computational intelligence for 6G: Key technologies, applications and trends,” _IEEE Trans. Ind. Informat._ , vol. 17, no. 10, pp. 7145–7154, Oct. 2021.
  * [5] X. Chen, C. Wu, Z. Liu, N. Zhang, and Y. Ji, “Computation offloading in beyond 5G networks: A distributed learning framework and applications,” _IEEE Wireless Commun._ , vol. 28, no. 2, pp. 56–62, Apr. 2021.
  * [6] X. Cao, B. Yang, C. Huang, C. Yuen, Y. Zhang, D. Niyato, and Z. Han, “Converged reconfigurable intelligent surface and mobile edge computing for space information networks,” _IEEE Netw._ , vol. 35, no. 4, pp. 42–48, Jul. 2021.
  * [7] B. Li, Z. Fei, and Y. Zhang, “UAV communications for 5G and beyond: Recent advances and future trends,” _IEEE Internet Things J._ , vol. 6, no. 2, pp. 2241–2263, Apr. 2019.
  * [8] F. Jiang, K. Wang, L. Dong, C. Pan, W. Xu, and K. Yang, “AI driven heterogeneous MEC system with UAV assistance for dynamic environment: Challenges and solutions,” _IEEE Netw._ , vol. 35, no. 1, pp. 400–408, Jan. 2021.
  * [9] B. Liu, Y. Wan, F. Zhou, Q. Wu, and R. Q. Hu, “Resource allocation and trajectory design for MISO UAV-assisted MEC networks,” _IEEE Trans. Veh. Technol._ , vol. 71, no. 5, pp. 4933–4948, May 2022.
  * [10] J. Ji, K. Zhu, and L. Cai, “Trajectory and communication design for cache-enabled UAVs in cellular networks: A deep reinforcement learning approach,” _IEEE Trans. Mobile Comput._ , to be published, doi:10.1109/TMC.2022.3181308.
  * [11] Y. Wu, K. Zhang, and Y. Zhang, “Digital twin networks: A survey,” _IEEE Internet Things J._ , vol. 8, no. 18, pp. 13 789–13 804, Sep. 2021\.
  * [12] S. Mihai, M. Yaqoob, D. V. Hung, W. Davis, P. Towakel, M. Raza, M. Karamanoglu, B. Barn, D. Shetve, R. V. Prasad, H. Venkataraman, R. Trestian, and H. X. Nguyen, “Digital twins: A survey on enabling technologies, challenges, trends and future prospects,” _IEEE Commun. Surveys Tuts._ , vol. 24, no. 4, pp. 2255–2291, Fourthquarter 2022.
  * [13] Y. Dai and Y. Zhang, “Adaptive digital twin for vehicular edge computing and networks,” _J. Commun. Inf. Networks_ , vol. 7, no. 1, pp. 48–59, Mar. 2022\.
  * [14] T. Do-Duy, L. D. Nguyen, T. Q. Duong, S. R. Khosravirad, and H. Claussen, “Joint optimisation of real-time deployment and resource allocation for UAV-aided disaster emergency communications,” _IEEE J. Sel. Areas Commun._ , vol. 39, no. 11, pp. 3411–3424, Nov. 2021.
  * [15] D.-H. Tran, V.-D. Nguyen, S. Chatzinotas, T. X. Vu, and B. Ottersten, “UAV relay-assisted emergency communications in IoT networks: Resource allocation and trajectory optimization,” _IEEE Trans. Wireless Commun._ , vol. 21, no. 3, pp. 1621–1637, Nov. 2022.
  * [16] L. Wang, K. Wang, C. Pan, W. Xu, N. Aslam, and A. Nallanathan, “Deep reinforcement learning based dynamic trajectory control for UAV-assisted mobile edge computing,” _IEEE Trans. Mobile Comput._ , vol. 21, no. 10, pp. 3536–3550, Oct. 2022.
  * [17] B. Dai, J. Niu, T. Ren, Z. Hu, and M. Atiquzzaman, “Towards energy-efficient scheduling of UAV and base station hybrid enabled mobile edge computing,” _IEEE Trans. Veh. Technol._ , vol. 71, no. 1, pp. 915–930, Jan. 2022.
  * [18] L. Wang, K. Wang, C. Pan, W. Xu, N. Aslam, and L. Hanzo, “Multi-agent deep reinforcement learning-based trajectory planning for multi-UAV assisted mobile edge computing,” _IEEE Trans. on Cogn. Commun. Netw._ , vol. 7, no. 1, pp. 73–84, Mar. 2021.
  * [19] H. Peng and X. Shen, “Multi-agent reinforcement learning based resource management in MEC- and UAV-assisted vehicular networks,” _IEEE J. Sel. Areas Commun._ , vol. 39, no. 1, pp. 131–141, Jan. 2021.
  * [20] T. Cai, Z. Yang, Y. Chen, W. Chen, Z. Zheng, Y. Yu, and H.-N. Dai, “Cooperative data sensing and computation offloading in UAV-assisted crowdsensing with multi-agent deep reinforcement learning,” vol. 9, no. 5, pp. 3197–3211, Sep.-Oct. 2022.
  * [21] N. Zhao, Z. Ye, Y. Pei, Y.-C. Liang, and D. Niyato, “Multi-agent deep reinforcement learning for task offloading in UAV-assisted mobile edge computing,” _IEEE Trans. Wireless Commun._ , vol. 21, no. 9, pp. 6949–6960, Mar. 2022.
  * [22] W. Sun, H. Zhang, R. Wang, and Y. Zhang, “Reducing offloading latency for digital twin edge networks in 6G,” _IEEE Trans. Veh. Technol._ , vol. 69, no. 10, pp. 12 240–12 251, Oct. 2020.
  * [23] W. Sun, S. Lei, L. Wang, Z. Liu, and Y. Zhang, “Adaptive federated learning and digital twin for industrial internet of things,” _IEEE Trans. Ind. Informat._ , vol. 17, no. 8, pp. 5605–5614, 2021.
  * [24] Y. Lu, X. Huang, K. Zhang, S. Maharjan, and Y. Zhang, “Low-latency federated learning and blockchain for edge association in digital twin empowered 6G networks,” _IEEE Trans. Ind. Informat._ , vol. 17, no. 7, pp. 5098–5107, Jul. 2021.
  * [25] T. Liu, L. Tang, W. Wang, Q. Chen, and X. Zeng, “Digital-twin-assisted task offloading based on edge collaboration in the digital twin edge network,” _IEEE Internet Things J._ , vol. 9, no. 2, pp. 1427–1444, Jan. 2022.
  * [26] T. Do-Duy, D. Van Huynh, O. A. Dobre, B. Canberk, and T. Q. Duong, “Digital twin-aided intelligent offloading with edge selection in mobile edge computing,” _IEEE Wireless Commun. Lett._ , vol. 11, no. 4, pp. 806–810, Apr. 2022.
  * [27] K. Zhang, J. Cao, and Y. Zhang, “Adaptive digital twin and multiagent deep reinforcement learning for vehicular edge computing and networks,” _IEEE Trans. Ind. Informat._ , vol. 18, no. 2, pp. 1405–1413, Feb. 2022\.
  * [28] B. Li, Y. Liu, L. Tan, H. Pan, and Y. Zhang, “Digital twin assisted task offloading for aerial edge computing and networks,” _IEEE Trans. Veh. Technol._ , vol. 71, no. 10, pp. 10 863–10 877, Oct. 2022.
  * [29] Q. Liu, L. Shi, L. Sun, J. Li, M. Ding, and F. Shu, “Path planning for UAV-mounted mobile edge computing with deep reinforcement learning,” _IEEE Trans. Veh. Technol._ , vol. 69, no. 5, pp. 5723–5728, May 2020.
  * [30] Z. Yang, S. Bi, and Y.-J. A. Zhang, “Online trajectory and resource optimization for stochastic UAV-enabled MEC system,” _IEEE Trans. Wireless Commun._ , vol. 21, no. 7, pp. 5629–5643, Jul. 2022.
  * [31] Y. Xu, T. Zhang, Y. Liu, D. Yang, L. Xiao, and M. Tao, “UAV-assisted MEC networks with aerial and ground cooperation,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 12, pp. 7712–7727, Dec. 2021.
  * [32] C. Zhao, J. Liu, M. Sheng, W. Teng, Y. Zheng, and J. Li, “Multi-UAV trajectory planning for energy-efficient content coverage: A decentralized learning-based approach,” _IEEE J. Sel. Areas Commun._ , vol. 39, no. 10, pp. 3193–3207, Oct. 2021.
  * [33] Y. Zeng, J. Xu, and R. Zhang, “Energy minimization for wireless communication with rotary-wing UAV,” _IEEE Trans. Wireless Commun._ , vol. 18, no. 4, pp. 2329–2345, Apr. 2019.
  * [34] C. Yu, A. Velu, E. Vinitsky, Y. Wang, A. Bayen, and Y. Wu, “The surprising effectiveness of PPO in cooperative, multi-agent games,” _arXiv preprint arXiv:2103.01955_ , 2021. [Online]. Available: http://arxiv.org/abs/2103.01955
  * [35] Z. Gao, L. Yang, and Y. Dai, “Large-scale computation offloading using a multi-agent reinforcement learning in heterogeneous multi-access edge computing,” _IEEE Trans. Mobile Comput._ , to be published, doi:10.1109/TMC.2022.3141080.
  * [36] J. Hu, H. Zhang, L. Song, R. Schober, and H. V. Poor, “Cooperative internet of UAVs: Distributed trajectory design by multi-agent deep reinforcement learning,” _IEEE Trans. Commun._ , vol. 68, no. 11, pp. 6807–6821, Nov. 2020.


# Self-Compression in Bayesian Neural Networks

###### Abstract

Machine learning models have achieved human-level performance on various
tasks. This success comes at a high cost of computation and storage overhead,
which makes machine learning algorithms difficult to deploy on edge devices.
Typically, one has to partially sacrifice accuracy in favor of an increased
performance quantified in terms of reduced memory usage and energy
consumption. Current methods compress the networks by reducing the precision
of the parameters or by eliminating redundant ones. In this paper, we propose
a new insight into network compression through the Bayesian framework. We show
that Bayesian neural networks automatically discover redundancy in model
parameters, thus enabling self-compression, which is linked to the propagation
of uncertainty through the layers of the network. Our experimental results
show that the network architecture can be successfully compressed by deleting
parameters identified by the network itself while retaining the same level of
accuracy.

Index Terms—  Bayesian deep learning, edge devices, self-compression.

##  1 Introduction

Deep Neural Networks (DNNs) have achieved outstanding results in a variety of
domains, including computer vision and natural language processing [1, 2]. One
of the key factors responsible for these successes is the ability of modern
DNNs to learn a large number of parameters [2]. However, an increase in the
number of parameters is directly linked to higher computational complexity,
the requirement for substantial computational resources, and additional
storage for the learned parameters. The availability of dedicated
computational hardware (e.g., GPUs) and novel training techniques (e.g.,
distributed training) have stimulated and supported the growth of DNNs.
However, resource-constrained systems, including mobile and edge devices (with
limited storage and energy), cannot support these DNNs [3, 4].

Recently, research communities have focused on exploring compression
techniques to reduce the memory usage and computational requirements of DNNs.
Some well-known current strategies include pruning and quantization of neural
networks. Pruning aims at eliminating (most) redundant connections from the
network, while quantization is characterized by the use of low-precision
representation for parameters and activations. Traditionally, most
implementations of DNNs used single-precision floating-point format (FP32).
However, recently, half-precision (FP16), and low-precision including INT8,
INT4, and binary formats have been explored [5].

Various quantization techniques have been proposed to make DNNs perform faster
and fit larger networks on edge devices with limited storage capacity and
energy budget [6, 7, 8]. An unfortunate consequence of quantization is the
reduced accuracy, which can be tackled by increasing the network size,
performing quantization only on parameters (and not on activations), or fine-
tuning and re-training the network. Although many real-world applications can
benefit from various quantization techniques, the reduced accuracy offsets the
benefits of low precision and significantly increasing training and testing
time.

Pruning techniques reduce the total number of parameters and corresponding
computations using an “importance” measure. The pruning process is based on
the assumption that the trained networks may have redundant parameters that do
not contribute to the output of the network and can be removed without
compromising the performance [9]. Parameter pruning techniques are generally
aimed at introducing sparsity during training or removing parameters or
kernels from trained networks using a threshold [10, 11, 12, 13, 14, 15]. The
importance or threshold values are chosen heuristically. Moreover, inducing
sparsity through approximate techniques, such as $l_{1}$-norm constraint, may
not be effective. Therefore, the pruning process remains a challenging task
due to the manual elimination of kernels or setting up a threshold value for
the parameter importance [10, 11, 12, 13, 14, 15].

Fig. 1: Sixteen convolutional kernels (randomly selected out of total one
hundred) for the Bayesian eVI CNN trained on the MNIST dataset are presented.
On the top of each kernel, Frobenius norm and standard deviation obtained from
the kernel’s posterior distribution are shown. The top row shows eight kernels
with Frobenius norm less than $10^{-7}$, which can be removed without
compromising the network’s classification accuracy

.

In this paper, we propose an automated process of network compression by
capitalizing on the recent work in Bayesian neural networks [16, 17]. We
leverage uncertainty information available in Bayesian techniques to realize
self-compression in convolutional neural networks (CNNs). Consequently, “less
important” kernels are automatically identified by the Bayesian CNN and can be
removed without compromising the performance accuracy. Our proposed Bayesian
CNNs perform at the same accuracy level as vanilla CNNs, however, at reduced
storage and computational requirements.

##  2 Bayesian Self-Compression

###  2.1 Extended Variational Inference

In the Bayesian framework, the unknown parameters $\Omega$ are treated as
random variables, and a prior distribution $\mathcal{}{p(\mathrm{\Omega})}$ is
defined over these parameters. Using Bayes’ Theorem and available dataset
$\cal{D}$, the posterior distribution $p(\mathrm{\Omega}|{\cal{D}})$ and
subsequently the predictive distribution are found. Exact Bayesian inference
on network parameters is mathematically intractable due to the functional form
of these networks and large parameter space [18, 19, 20].

Variational Inference (VI) is scalable density approximation technique based
on optimization which avoids intractable integration operations [18, 20]. VI
proposes an approximating distribution
$\mathcal{}{q_{\boldsymbol{\theta}}(\mathrm{\Omega})}$ and minimize the
Kullback-Leibler (KL) divergence between the proposed distribution and the
true posterior $p(\mathrm{\Omega}|{\cal{D}})$:

$\mathbf{KL}(q_{\boldsymbol{\theta}}(\mathrm{\Omega})\big{|}\big{|}p(\Omega\big{|}\mathcal{D}))=\int
q_{\boldsymbol{\theta}}(\mathrm{\Omega})\log\frac{q_{\boldsymbol{\theta}}(\mathrm{\Omega})}{p(\mathrm{\Omega})p(\cal{D}|\mathrm{\Omega})}d\mathrm{\Omega}.$
(1)

Using (1), an optimization problem in the form of well-known Evidence Lower
Bound (ELBO) [18] can be formulated as:

$\mathbf{L}(\boldsymbol{\theta})=\mathbf{KL}\left(q_{\boldsymbol{\theta}}(\mathrm{\Omega})\big{|}\big{|}p(\Omega)\right)-\mathbb{E}_{q_{\boldsymbol{\theta}}(\mathbf{\Omega})}\left[\log\left(p(\mathcal{D}|\Omega)\right)\right].$
(2)

Recently, Dera _et al._ proposed a VI framework, referred to as the extended
Variational Inference (eVI), which propagates the first two moments of the
variational posterior through layers of a CNN [16, 17]. Authors defined the
tensor normal distribution over kernels as the prior distribution, and used
(2) to approximate the posterior distribution of kernels
$\mathcal{N}\left(\boldsymbol{\mu},\sigma^{2}\boldsymbol{I}\right)$. The
estimated $\sigma^{2}$ provided a measure of confidence or uncertainty
attached to the learned kernels during and after the training [17, 16].
Bayesian eVI CNNs showed improved robustness to noise and adversarial attacks
[17, 16]. In this work, we establish that the Bayesian eVI approach leads to
self-compress and improves network performance.

###  2.2 Self-Compression with Bayesian eVI

The proposed method can be considered as a “self-aware” compression process
for neural networks. During training, a set of kernels is identified as
redundant by the Bayesian eVI framework. The redundant kernels are removed
through an iterative process, and a compressed network is realized. The
Frobenius norm is used to rank each kernel’s redundancy or equivalently its
importance. The Frobenius norm of a kernel $K$ of size $d\times d$ is defined
as:

$\left\|K\right\|_{F}=\sqrt{\text{Tr}\left(KK^{T}\right)},$ (3)

where Tr is the trace operator and T represents transpose. The Frobenius norm
helps us understand the contribution of a kernel to the performance of the
Bayesian eVI CNN.

Fig. 2: Frobenius norm of 100 kernels for (a) Bayesian eVI CNN and (b) vanilla
CNN trained on the MNIST dataset are presented. The zoomed sub-figures show
zero kernels ($\left\|.\right\|_{F}\leq 10^{-7}$). The test accuracy of both
CNN was 97%. We consider that a Bayesian CNN can perform at the same accuracy
level with 28 zero kernels, which are a candidate for removal in the next
training cycle.

The MNIST handwritten digits dataset and Fashion-MNIST (F-MNIST) dataset were
used to train multiple Bayesian and vanilla CNNs [21, 22]. The architecture of
all CNNs included one convolution layer followed by the rectified linear unit
(ReLU) activation, one max-pooling layer, and one fully-connected layer [23].
The kernel size was set to $5\times 5$ in the convolutional layer of all CNNs.
The test accuracy served as the metric to evaluate and compare the performance
of Bayesian eVI and vanilla CNNs. In the case of Bayesian CNN, the variances
of the predictive distribution provided a measure of the confidence in the
classification decision.

We used a different number of kernels in the convolutional layer of both
Bayesian and vanilla CNNs. For the MNIST dataset, our first set of CNNs was
trained using $N=100$ kernels. For the F-MNIST, our experiments started with
$N=128$ kernels. The kernels are then reduced based on the kernel’s importance
metric computed from the Frobenius norm while the variance information is used
to evaluate the network’s confidence in the values obtained during training
for each kernel.

##  3 Results and Discussion

###  3.1 MNIST Dataset

The first set of experiments included training both Bayesian and vanilla CNNs
using $N=100$ kernels. Both CNNs achieved $97\%$ test accuracy. In Fig. 1, we
present sixteen randomly selected kernels of a trained Bayesian eVI CNN on
MNIST dataset. Fig. 2 shows Frobenius norms of all hundred kernels for both
CNNs. We note that for the Bayesian CNN, there are 28 kernels with Frobenius
norm equal to zero ($\left\|K\right\|_{F}\leq 10^{-7}$). However, for vanilla
CNN, the number is only 2. We refer to the kernels that have
$\left\|K\right\|_{F}\leq 10^{-7}$ as zero kernels.

In the case of Bayesian eVI CNN, the average variance $\sigma^{2}$ was $0.006$
for zero kernels (28 in total) and $0.007$ for non-zero kernels. These low
variance values for zero and non-zero kernels indicate the network’s high
level of confidence in these kernels’ values. However, no kernel variance
information is available for vanilla CNNs. Through experiments and empirical
evidence, we found that the reduction of 28 kernels in the Bayesian eVI CNN
and two kernels in the vanilla CNN did not affect the test accuracy.

In the second set of experiments using the MNIST dataset, we trained both the
Bayesian eVI and vanilla CNNs with a different number of kernels, i.e.,
$N=64,32,\text{and},25$. Frobenius norms of kernels for all three cases and
both CNNs are presented in Fig. 3. In all three cases, Bayesian CNNs have more
zero kernels as compared to vanilla CNNs with comparable or higher accuracy.
For the case of $N=64$, both CNNs have 97% test accuracy. For the Bayesian
CNN, 11 zero kernels were identified, however, for the vanilla CNN, the number
was only 5. Similarly, for the case of N=25, Bayesian CNN has 96% test
accuracy, and vanilla CNN has 94% test accuracy. We identified 4 zero kernels
for the Bayesian CNN and none for the vanilla CNN.

Fig. 3: The Frobenius norms of convolutional kernels for the Bayesian eVI and
Vanilla CNNs with $N=64,32,\text{and},25$ for the MNIST dataset are presented.
The zoomed sub-figures show kernels with small Frobenius norms. In all cases,
Bayesian eVI CNN can discover zero kernels ($\left\|.\right\|_{F}\leq
10^{-7}$). We consider that a Bayesian eVI CNNs provide a principled way to
identify and iteratively remove zero kernels, i.e., self-compression.

###  3.2 Fashion MNIST Dataset

In Fig. 4, the Frobenius norms of kernels for both CNNs with
$N=128,64,\text{and},46$ are presented. For all three cases, Bayesian eVI CNN
achieved the same or better test accuracy; however, only Bayesian eVI CNN was
able to identify a set of kernels that could be potentially removed without
adversely affecting the test accuracy.

Fig. 4: The Frobenius norms of convolutional kernels for Bayesian eVI and
Vanilla CNNs with $N=128,64,\text{and},46$ for the F-MNIST dataset are
presented. The zoomed sub-figures show kernels with small Frobenius norm
values. We note that for all cases, Bayesian eVI CNN can discover zero kernels
and leads to self-compression.

###  3.3 Iterative Compression of CNNs

Based on experiments with MNIST and F-MNIST datasets and Bayesian eVI and
vanilla CNNs, we developed an iterative self-compression technique. We used
the number of zero kernels identified by the Bayesian eVI CNN as our guide to
remove these kernel in an iterative process. The number of identified zero
kernels were removed in the next training cycle. Fig. 5 shows the test
accuracy of both Bayesian eVI and vanilla CNNs for a range of the number of
kernels $N$ for both datasets. We note that Bayesian eVI CNN retains high
accuracy when zero kernels are removed; however, the accuracy of vanilla CNNs
starts to drops at a higher rate.

Bayesian eVI CNNs are able to identity zero kernels (i.e., kernels with
$\left\|.\right\|_{F}\leq 10^{-7}$) and provide a principled mechanism for the
self-compression, i.e., iteratively removing convolutional kernels that are
irrelevant. It is important to highlight that the identified zero kernels had
low variance values, i.e., the network showed a high level of confidence in
these kernels. The capability to identify zero kernels (with high confidence)
is exhibited by Bayesian eVI CNNs and is referred to as the self-compression.

The self-compression ability of the Bayesian eVI CNNs stems from propagating
variational posterior distribution across layers of a neural network during
the training process. The availability of the variance information through the
variational posterior during the training process allows Bayesian eVI networks
to discover important and unimportant (zero) kernels. Vanilla CNNs, on the
other hand, owing to lack of information about uncertainty (or variance) tend
to use all available kernels to achieve a high accuracy.

Fig. 5 shows that at the same level of test accuracy, Bayesian eVI CNNs
require significantly less number of kernels as compared to vanilla CNNs. The
Bayesian eVI CNN performed at the same accuracy level as a vanilla CNN using
only half kernels (32 vs. 64 kernels) for the MNIST dataset. Similarly, for
the F-MNIST dataset, both Bayesian eVI and vanilla CNNs achieved 83% accuracy.
However, the former used only 40 kernels as compared to 72 for the vanilla
CNN. Moreover, the process of discovery of unimportant (zero) kernels is
inherently part of Bayesian eVI CNNs. On the other hand, the process of
compression or pruning for vanilla CNNs requires manual selection of threshold
and possibly a compromise on the test accuracy and performance.

Fig. 5: Test accuracy vs. the number of convolutional kernels $N$ for the
Bayesian eVI CNN (orange) and a vanilla CNN (blue) are presented. We note that
the Bayesian eVI CNN is able to perform at the same accuracy level as that of
vanilla CNN using almost half the number of convolutional kernels (32 vs. 64
for the MNIST dataset and 40 vs. 72 for the Fashion-MNIST dataset). (a) Test
accuracy for the MNIST Dataset. (b) Test accuracy for the F-MNIST dataset.

###  3.4 The Size of the Parameter Space

Previously proposed Bayesian approaches for neural networks resulted in a
larger parameter space (double the number of parameters as compared to vanilla
network) due to the probability distribution defined over the parameters [19,
16]. However, the recently proposed Bayesian eVI CNNs require only one
additional variance parameter $\sigma^{2}$ for each convolutional kernel owing
to the use of diagonal tensor normal distribution [17, 16]. A Bayesian eVI CNN
has $(d^{2}+1)N$ parameters as compared to a vanilla CNN that has $d^{2}N$
parameters, where $d$ is the size of a kernel and $N$ represents the total
number of convolutional kernels.

Storage-wise, the additional number of parameters does not appear to be an
issue given the self-compression capability. For example, in our
implementation (with kernel size $d=5$), the storage for the convolution
parameters after compression for both models is shown in table 1.

Table 1: Storage (KB) after Compression Dataset | Bayesian eVI CNN | Vanilla CNN
---|---|---
MNIST | 3.25 | 6.25
F-MNIST | 4.06 | 7.03

##  4 Conclusion

The high performance of current machine learning algorithms has associated
high computational costs and storage requirements. We exploit the uncertainty
information inherent in Bayesian neural networks and propose a method for
self-compression of convolutional neural networks. We used the recently
proposed extended Variational Inference (eVI) framework that offers advantages
of Bayesian neural networks, albeit at a minimal increase in the number of
parameters. We showed that Bayesian eVI CNNs were able to identify redundant
kernel during training, which can be removed in an iterative process to
realize self-compression. The Bayesian eVI CNNs were able to perform at the
same accuracy level as that of vanilla CNNs using almost half the number of
convolutional kernels. The proposed process of self-compression is part of the
training, and resulting Bayesian eVI CNNs are ready for deployment on the edge
devices.

##  5 Acknowledgement

This work was supported by the National Science Foundation Awards NSF
ECCS-1903466. NSF CCF-1527822, NSF OAC-2008690. Giuseppina Carannante is
supported by the US Department of Education through a Graduate Assistance in
Areas of National Need (GAANN) program Award Number P200A180055. We are also
grateful to UK EPSRC support through EP/T013265/1 project NSF-EPSRC: ShiRAS.
Towards Safe and Reliable Autonomy in Sensor Driven Systems.

## References

  * [1] Tero Karras, Samuli Laine, and Timo Aila,  “A Style-Based Generator Architecture for Generative Adversarial Networks,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 4401–4410.
  * [2] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever,  “Language Models are Unsupervised Multitask Learners,”  OpenAI Blog, vol. 1, no. 8, pp. 9, 2019.
  * [3] Rajat Raina, Anand Madhavan, and Andrew Y Ng,  “Large-Scale Deep Unsupervised Learning using Graphics Processors,”  in Proceedings of the 26th annual international conference on machine learning, 2009, pp. 873–880.
  * [4] Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc’aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, et al.,  “Large Scale Distributed Deep Networks,”  in Advances in neural information processing systems, 2012, pp. 1223–1231.
  * [5] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David,  “Binaryconnect: Training Deep Neural Networks with Binary Weights during Propagations,”  in Advances in neural information processing systems, 2015, pp. 3123–3131.
  * [6] Alexander Goncharenko, Andrey Denisov, Sergey Alyamkin, and Evgeny Terentev,  “Fast Adjustable Threshold for Uniform Neural Network Quantization,”  arXiv preprint arXiv:1812.07872, 2018.
  * [7] Yoni Choukroun, Eli Kravchik, and Pavel Kisilev,  “Low-Bit Quantization of Neural Networks for Efficient Inference,”  arXiv preprint arXiv:1902.06822, 2019.
  * [8] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi,  “Xnor-Net: Imagenet Classification using Binary Convolutional Neural Networks,”  in European Conference on Computer Vision. Springer, 2016, pp. 525–542.
  * [9] Song Han, Huizi Mao, and William J Dally,  “Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding,”  arXiv preprint arXiv:1510.00149, 2015.
  * [10] Vadim Lebedev and Victor Lempitsky,  “Fast Convnets using Group-Wise Brain Iamage,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2554–2564.
  * [11] Hao Zhou, Jose M Alvarez, and Fatih Porikli,  “Less is More: Towards Compact CNNs,”  in European Conference on Computer Vision. Springer, 2016, pp. 662–677.
  * [12] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li,  “Learning Structured Sparsity in Deep Neural Networks,”  in Advances in neural information processing systems, 2016, pp. 2074–2082.
  * [13] Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung,  “Structured Pruning of Deep Convolutional Neural Networks,”  ACM Journal on Emerging Technologies in Computing Systems (JETC), vol. 13, no. 3, pp. 32, 2017.
  * [14] Adam Polyak and Lior Wolf,  “Channel-Level Acceleration of Deep Face Representations,”  IEEE Access, vol. 3, pp. 2163–2175, 2015.
  * [15] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf,  “Pruning Filters for Efficient Convnets,”  arXiv preprint arXiv:1608.08710, 2016.
  * [16] Dimah Dera, Ghulam Rasool, Nidhal C. Bouaynaya, Adam Eichen, Stephen Shanko, Jeff Cammerata, and Sanipa Arnold,  “Bayes-SAR Net: Robust SAR Image Classification with Uncertainty Estimation using Bayesian Convolutional Neural Network,”  in Proceedings of the IEEE International Radar Conference, 2020\.
  * [17] Dimah Dera, Ghulam Rasool, and Nidhal Bouaynaya,  “Extended Variational Inference for Propagating Uncertainty in Convolutional Neural Networks,”  in 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP). IEEE, 2019, pp. 1–6.
  * [18] Alex Graves,  “Practical Variational Inference for Neural Networks,”  in Advances in neural information processing systems, 2011, pp. 2348–2356.
  * [19] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra,  “Weight Uncertainty in Neural Networks,”  in Proceedings of the 32nd International Conference on International Conference on Machine Learning, (ICML), 2015, vol. 37, pp. 1613–1622.
  * [20] Kumar Shridhar, Felix Laumann, Adrian Llopart Maurin, and Marcus Liwicki,  “Bayesian Convolutional Neural Networks,”  arXiv preprint arXiv:1806.05978, 2018.
  * [21] Han Xiao, Kashif Rasul, and Roland Vollgraf,  “Fashion-mnist: a Novel Image Dataset for Benchmarking Machine Learning Algorithms,”  arXiv preprint arXiv:1708.07747, 2017.
  * [22] L. Deng,  “The MNIST Database of Handwritten Digit Images for Machine Learning Research,”  IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 141–142, 2012.
  * [23] Vinod Nair and Geoffrey E. Hinton,  “Rectified Linear Units Improve Restricted Boltzmann machines,”  in Proceedings of the 27th International Conference on International Conference on Machine Learning, 2010, pp. 807–814.