# Cooperative Task Offloading and Block Mining in Blockchain-based Edge
Computing with Multi-agent Deep Reinforcement Learning

Dinh C. Nguyen, Ming Ding, Pubudu N. Pathirana, Aruna Seneviratne, , Jun Li, ,
and H. Vincent Poor  Dinh C. Nguyen and Pubudu N. Pathirana are with School of
Engineering, Deakin University, Waurn Ponds, VIC 3216, Australia (e-mails:
{cdnguyen, pubudu.pathirana}@deakin.edu.au).Ming Ding is with the Data61,
CSIRO, Australia (email: ming.ding@data61.csiro.au).Aruna Seneviratne is with
School of Electrical Engineering and Telecommunications, University of New
South Wales (UNSW), NSW, Australia (e-mail: a.seneviratne@unsw.edu.au). Jun Li
is with School of Electrical and Optical Engineering, Nanjing University of
Science and Technology, Nanjing 210094, China (e-mail: jun.li @njust.edu.cn).
H. Vincent Poor is with the Department of Electrical Engineering, Princeton
University, Princeton, NJ 08544 USA (e-mail: poor@princeton.edu). This work
was supported in part by the CSIRO Data61, Australia, and in part by U.S.
National Science Foundation under Grant CCF-1908308. The work of Jun Li was
supported by National Natural Science Foundation of China under Grant
61872184.

###### Abstract

The convergence of mobile edge computing (MEC) and blockchain is transforming
the current computing services in mobile networks, by offering task offloading
solutions with security enhancement empowered by blockchain mining.
Nevertheless, these important enabling technologies have been studied
separately in most existing works. This article proposes a novel cooperative
task offloading and block mining (TOBM) scheme for a blockchain-based MEC
system where each edge device not only handles data tasks but also deals with
block mining for improving the system utility. To address the latency issues
caused by the blockchain operation in MEC, we develop a new Proof-of-
Reputation consensus mechanism based on a lightweight block verification
strategy. A multi-objective function is then formulated to maximize the system
utility of the blockchain-based MEC system, by jointly optimizing offloading
decision, channel selection, transmit power allocation, and computational
resource allocation. We propose a novel distributed deep reinforcement
learning-based approach by using a multi-agent deep deterministic policy
gradient algorithm. We then develop a game-theoretic solution to model the
offloading and mining competition among edge devices as a potential game, and
prove the existence of a pure Nash equilibrium. Simulation results demonstrate
the significant system utility improvements of our proposed scheme over
baseline approaches.

###### Index Terms:

Blockchain, mobile edge computing, task offloading, block mining, deep
reinforcement learning.

##  1 Introduction

Recent advances in Internet of Things (IoT) have promoted the proliferation of
numerous mobile applications that mostly rely on edge devices (EDs), e.g.,
laptops, tablets, and smartphones, to collect data from IoT sensors to serve
end users. To meet the increasing users’ computation demand, mobile edge
computing (MEC) has been proposed as a promising technique to improve the
computation experience of EDs, by offloading computationally intensive tasks
to a nearby MEC server located at a base station (BS) [1]. Mapping each
offloading process to a specific application, multiple distributed EDs
naturally share computation and communication resources of the BS to handle
data tasks without device’s battery depletion. Task offloading with MEC thus
becomes a viable solution to satisfy various EDs’ computation demands and
enhances the quality of experience (QoE) of end users.

However, the design of an efficient task offloading scheme for MEC systems
still faces non-trivial challenges. Each ED always aims to maximize its
individual utility by occupying as many edge resources (e.g., channel
spectrum, CPU frequency) as possible, which is likely to cause network traffic
congestion and user interference. The heterogeneous resource requirements of
multiple IoT data tasks, e.g., different resource allocations needed for
handling different data tasks, and the heterogeneous features of real-time IoT
data tasks, e.g., computation deadlines and data task sizes, pose challenges
for the design of offloading strategies for all EDs. Moreover, the lack of
prior information on system statistics in practical multi-user MEC systems,
e.g., channel state and edge computational resource state, makes it
challenging to derive an optimal offloading solution for each ED. Therefore,
it is of the utmost importance to develop a intelligent and self-organized
offloading scheme to guide the offloading actions of all EDs in the
distributed MEC systems.

Furthermore, the dynamic communications between IoT devices, EDs, and the MEC
server and the migration of IoT data tasks across the MEC network potentially
cause security vulnerabilities. Recent works [1, 2, 3] have mainly focused on
computation offloading designs for task scheduling and resource allocation,
with the lack of considering security aspects in MEC networks.  Fortunately,
blockchain has been envisioned as a strong candidate to enhance security of
MEC systems [4]. In fact, blockchain is able to provide high degrees of
security and trust for MEC by employing the community verification among edge
nodes via mining mechanisms such as Delegated Proof of Stake (DPoS) [4]
without requiring any central authority.

In this context, the use of blockchain is highly desirable to support edge
computing systems [5, 6, 7]. Specifically, blockchain decentralizes the MEC
system where edge nodes can communicate with each other via the peer-to-peer
network over the decentralized data ledger. Different from traditional MEC
systems that often rely on a central server to coordinate the MEC operation,
blockchain helps build decentralized edge communications without the need for
a single authority, which eliminates the risks of single-point failure. This
feature is very useful in practical application scenarios, e.g., decentralized
edge data sharing and decentralized edge data caching in MEC networks. Another
motivation behind the integration of blockchain in MEC is its immutability
that makes edge data records, e.g., IoT data, unchangeable once they are
stored on the ledger [8]. By deploying immutable transaction ledgers, EDs can
establish reliable communications to perform heterogeneous networking and
computation, such as large-scale IoT collaborations or mobile edge computing
over trustless IoT environments. Moreover, blockchain provides transparency
for MEC networks, where blockchain allows the copy of data records to
replicate across edge nodes for public validation, which in return enhances
data integrity. This feature is particularly suitable for MEC ecosystems where
openness and fairness are required. For example, blockchains can offer
transparent ledger solutions to support open and secure data delivery and
payment for EDs in a fashion such that EDs can trace and monitor transactions.

Moreover, each ED joins the block mining process to maintain the operation of
blockchain in MEC. The key purpose of mining is to verify the data
transactions, aiming to guarantee the security for the involved edge networks.
Accordingly, in the blockchain-based MEC system, EDs perform the mining, and
data blocks are secured and chained via an immutable ledger. With more devices
mining the blockchain, the security of the edge network increases accordingly.

###  1.1 Related Works

Recently, many edge task offloading solutions have been proposed [9, 10, 11],
but these works mostly considered offloading scenarios with a single agent
using traditional convex optimization tools. Deep reinforcement learning (DRL)
techniques such as deep Q-learning (DQN) have emerged as a promising
alternative, by modelling the offloading problem as a Markov decision process
(MDP) with using deep neural network (DNN) for function approximation [12, 13,
14]. However, these works only used a single agent to handle the entire
offloading process which could not work well in large-scale distributed MEC
environments. An interesting alternative is to use multi-agent-DRL (MA-DRL)
[15] for supporting intelligent task offloading in MEC networks [16]. The work
in [17] proposed a non-cooperative MA-DRL scheme where EDs could build their
offloading policy independently. Another study in [18] also suggested an MA-
DRL approach for joint data offloading and resource allocation in multiple
independent edge clouds. Furthermore, a multi-agent Q-learning algorithm was
developed in [19] for a joint computation offloading and resource allocation
scheme in edge computing.

In terms of reputation-based DPoS mining design, the work in [4] focused
primarily on addressing the secure block verification issues in the DPoS
mechanism using contract theory. The paper in [20] suggested a fair voting
scheme for the DPoS mechanism via vague set theory. Specifically, this work
leveraged a general model of transforming vague sets into fuzzy sets to
calculate the comprehensive evaluation indices for agent node selection. The
paper in [21] proposed a contract theory-based optimization scheme for
transaction relaying and DPoS based block verification. The authors of this
work focused on formulating two mathematical models: value of transaction
relaying and value of block verification, and developed the optimal contract
to maximize utility of the miners. Further, the work in [8] proposed a
lightweight blockchain-based information trading framework to model the
interactions between traffic administration and vehicles in the reputation-
based DPoS mechanism via a budgeted auction approach. This study paid
attention to the optimization of the mining profit by using a truthful
budgeted selection and pricing algorithm. However, a design for low-latency
block verification in the reputation-based DPoS mechanism has not been
developed.

Moreover, the research related to task offloading and blockchain mining in MEC
networks has been conducted recently. A blockchain-empowered computation
offloading scheme was presented in [16] where blockchain was mainly used for
data integrity in the offloading. The authors in [22, 23, 24] studied edge
offloading schemes for blockchain mining tasks with edge clouds, aiming to
enhance the quality of service (QoS) for efficient block mining. Blockchain
was also utilized in [25] to support resource trading for the edge task
offloading, while the work in [26] considered a cooperative blockchain-MEC
system with an actor-critic DRL algorithm. The study in [27] optimized the
edge computation offloading and resource allocation via a double-dueling deep
Q network. The authors in [28] considered a cooperative computation offloading
framework for blockchain-based IoT networks. An MA-DRL algorithm was designed
which could allow IoT devices to collaboratively explore the offloading
environments in order to minimize long-term offloading costs. Another work
[29] considered a blockchain-based energy trading scheme to manage the energy
trading process toward building a secure energy trading system in Industry
4.0. In [30], the problem of resource trading for blockchain-based IoT was
studied by using a two-level Stackelberg game with a credit-based payment with
smart contracts. Game theory was also applied in [31] to minimize the economic
cost of industrial IoT devices. However, in most existing works [22, 23, 24,
25, 26, 27], [29, 30, 31], the design and optimization of task offloading and
blockchain mining were implemented separately, which would result in a sub-
optimal performance.

TABLE I: The comparison of the existing works and our scheme. Design features | Schemes
---|---
[13] | [16] | [17] | [18] | [26] | [27] | [28] | [31] | Our scheme
Task offloading with blockchain |  | ✓ |  |  | ✓ | ✓ | ✓ | ✓ | ✓
Intelligent edge task offloading | ✓ |  | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Cooperative edge task offloading |  |  |  |  |  |  | ✓ | ✓ | ✓
Lightweight blockchain design |  |  |  |  |  |  |  |  | ✓
Joint offloading and mining design |  |  |  |  |  |  |  |  | ✓

###  1.2 Motivations and Our Key Contributions

Despite the recent research efforts in blockchain-MEC designs, there are
several limitations in existing works, as highlighted below:

  * •

In distributed blockchain-based MEC systems, traditional single-agent DRL
algorithms like DQN [13], [14], [22], [26], [32] face critical challenges
caused by diversified and time-varying local environments. In more details,
during the training process of DQN, each agent only observes its local
information and cannot know the updates from other agents due to non-
collaboration. This makes it hard to ensure the stability and convergence of
the agents’ algorithm [33]. Moreover, this breaks the Markov properties
required by the Q-learning algorithm and thus, DQN may not be capable of
learning the cooperative offloading policies of EDs. Moreover, the non-
cooperative multi-agent DRL solutions [17, 18, 19] may not be able to learn
the cooperative policy; and thus resource usage over the edge network is not
efficient which limits the overall offloading performance, e.g., offloading
utility.

  * •

In addition, in most existing blockchain-based MEC schemes [22, 23, 24, 25,
26], the design and optimization of task offloading and blockchain mining are
done separately, which would result in sub-optimal performance. Moreover, the
problem of high network latency caused by blockchain mining in the edge
offloading system has not been addressed so far [27, 28], [30], [31]. To
improve the overall performance, a joint offloading and blockchain design is
needed for realizing efficient blockchain-based MEC systems.

Motivated by the aforementioned limitations, we propose a novel cooperative
task offloading and blockchain mining (TOBM) scheme for blockchain-based MEC
systems enabled by a new MA-DRL solution. Different from existing works [23,
24, 25, 26, 27], [29, 30, 31], we here focus on maximizing the overall system
utility as the sum of offloading utility and mining utility. More
specifically, each ED handles data tasks collected from its IoT sensors and
deals with block mining tasks simultaneously. To reduce the network latency
caused by the blockchain integration in the MEC system, we design a new Proof-
of-Reputation (PoR) mining mechanism enabled by a lightweight block
verification solution. In particular, we develop a novel distributed DRL-based
algorithm using a multi-agent deep deterministic policy gradient (MA-DDPG)
approach to optimize the overall system utility. The proposed MA-DDPG approach
enables the efficient learning of the mutual policy among cooperative EDs in
the dynamic environment and high-dimensional system state space. Indeed, the
proposed MA-DDPG scheme allows EDs to learn mutually the cooperative
offloading and mining policy which helps enhance the computation efficiency
and thus improves the system utility. To enhance the convergence performance
in model training and solve the nonstationary issues caused by the
concurrently learning process of all EDs in the multi-agent environment, a
centralized learning and decentralized execution solution is adopted. As such,
the proposed MA-DRL algorithm is first trained at the centralized MEC server,
and the learned model is then executed at EDs in a distributed manner. In
fact, the benefits of the MA-DDPG algorithm in edge computing have been proved
in recent works for MEC-based industry 4.0 [34], smart ocean federated
learning IoTs [35], and smart grid [36]. However, its potential in blockchain-
MEC system has not been explored so far. The comparison of our paper and the
related works via some key features is summarized in Table I. In a nutshell,
the unique contributions of this article are highlighted as follows:

  1. 1.

We propose a novel cooperative TOBM scheme in a blockchain-based MEC system to
enable a joint design of task offloading and blockchain mining for improving
the overall system utility.

  2. 2.

The details of task offloading are presented, where EDs cooperatively offload
their IoT data tasks to the MEC server. Moreover, we propose a new PoR mining
mechanism enabled by a lightweight block verification strategy, in order to
solve latency issues caused by the blockchain adoption in the MEC system.

  3. 3.

In the TOBM scheme, each ED as an intelligent agent to learn cooperatively
policies, by jointly considering offloading decision, channel selection,
transmit power allocation, and computational resource allocation with respect
to both offloading and mining states for maximizing the system utility. Then,
we propose a novel distributed DRL-based approach using an MA-DDPG algorithm
to solve the proposed problem based on a centralized learning and
decentralized execution strategy.

  4. 4.

We further develop a game-theoretic solution to model the competition among
EDs in offloading and mining as a potential game. We then analyze the
properties of the formulated game and prove the existence of a pure Nash
equilibrium (NE).

  5. 5.

We conduct extensive numerical simulations and compare with the existing
schemes to verify the effectiveness of the proposed scheme.

###  1.3 Paper Organization

The remainder of this paper is organized as follows. Section 2 introduces the
system model along with the analysis of network model edge task offloading
model. The PoR blockchain consensus mechanism is proposed in Section 3. Based
on the offloading and mining design, a joint system utility problem is
formulated in Section 4 which is then modelled by a cooperative offloading
game. A new MA-DRL algorithm is proposed to solve the formulated offloading
game by using an MA-DDPG algorithm. The simulation results are provided in
Section 5 and the comparison with other related offloading schemes is also
discussed. Finally, Section 6 concludes this article and highlights possible
future directions.

##  2 System Model

In this section, we introduce the network model of the blockchain-based MEC
system, and then present the task offloading model.

###  2.1 Network Model

We consider a cooperative TOBM architecture in the blockchain-based MEC system
as illustrated in Fig. 1. The BS is equipped with an MEC server to provide
computation services for EDs. We denote the set of EDs as
$\mathcal{N}=\\{1,2,...,N\\}$. For the sake of simplicity, we assume that each
ED $n\in\mathcal{N}$ has an IoT data task $Y_{n}$ to be executed [9, 10],
which can be defined by a tuple
$Y_{n}=(C_{n},D_{n},\tau_{n}),n\in\mathcal{N}$. Herein, $C_{n}$ denotes the
total computational resource (i.e., the number of the CPU cycles) to
accomplish the task $Y_{n}$. Also, $D_{n}$ expresses the size of the input
data, and $\tau_{n}$ specifies the maximum permissible latency to accomplish
task $Y_{n}$. In addition to the task offloading function, each ED also
participates in the block mining by using a PoR consensus mechanism. The key
network components of the blockchain-based MEC system are described as
follows:

Figure 1: The cooperative task offloading and block mining architecture in the
blockchain-based MEC system.

  * •

IoT Sensors: IoT sensors such as cameras, smart meters, and wearables are
responsible for sensing physcial environments, e.g., entertainment, logistics,
transportation and healthcare monitoring, and generating data which need to be
computed to serve end users. IoT sensors can act as lightweight blockchain
nodes to securely communicate and transmit data to their nearby ED via the
blockchain network.

  * •

Edge Devices: Each ED such as a laptop or a powerful smartphone manages a
group of IoT sensors under its coverage. Based on the QoE requirements, EDs
can use their computational capability to process data tasks locally or
offload to a nearby MEC server via wireless links. EDs also participate in
block mining, i.e., transaction verification and block generation, via a PoR
consensus mechanism where IoT sensor users vote to select representative EDs
to run the mining process. The details of our blockchain mining design are
explained in Section 3.

  * •

MEC Server: In our considered blockchain-based MEC system, there is a single
MEC sever located at a BS to handle computationally extensive data tasks
offloaded from EDs. By analyzing the task profile such as task sizes, channel
conditions, and available resource, EDs can make offloading decisions so that
the MEC server can allocate its resources for computation under QoE
requirements.

  * •

Blockchain: A blockchain network is deployed over the MEC system where each ED
acts as a blockchain miner. In this paper, we propose a PoR framework and
focus on analyzing the block verification latency that is a significant factor
in evaluating the efficiency of a blockchain network.  The use of our proposed
PoR scheme allows EDs to participate in the block mining with an enhanced
mining utility which contributes to the system utility improvement in the
blockchain-based MEC system.

Figure 2: Illustration of the blockchain operation in our MEC network: (1) An
IoT sensor user sends a transaction to its associated ED for triggering the
IoT data task offloading, (2) A new block is created to represent the verified
transaction, (3) A mined block is appended to the blockchain.

The blockchain framework for our MEC network is illustrated in Fig. 2, and its
operational concept is explained via the following steps:

  * •

Step 1: The IoT sensor user first creates a transaction with metadata (i.e.
user ID), user signature and timestamp from its wallet account and sends it to
the associated ED. The user then submits a transaction to the ED for a certain
request, such as IoT data task offloading.

  * •

Step 2: The ED releases its available resources and processes the request from
the user. For example, an ED can use a smart contract [37], a self-executing
software running on blockchain, to automatically perform transaction
authentication, user verification, or resource trading. Moreover, the ED
collaborates with other mining members to aggregate the transactions offloaded
from IoT users to build a block after a certain period of time. Then, the EDs
participate in the mining to verify the block using a consensus mechanism,
e.g., PoR.

  * •

Step 3: After the mining, if all miners achieve an agreement on the verified
block, this block with its signature is then appended to the chain of blocks
in chronological order. Finally, all network entities receive this block and
synchronize the copy of the blockchain.

###  2.2 Edge Task Offloading Model

Here, we present the communication model and the computing model for the edge
task offloading.

####  2.2.1 Communication Model

We denote $\mathcal{K}=\\{1,...,K\\}$ as the set of available sub-bands at the
BS. We define a task offloading policy, which also incorporates the uplink
sub-band scheduling, by a binary variable $x_{n}^{k}$,
($n\in\mathcal{N},k\in\mathcal{K}$). Here, $x_{n}^{k}=1$ indicates that the
task $Y_{n}$ from ED $n$ is offloaded to the MEC server via sub-band $k$, and
$x_{n}^{k}=0$ otherwise. Each computation task can be either executed locally
at the ED or offloaded to the MEC server under a feasible offloading policy:

$\sum_{k\in\mathcal{K}}x_{n}^{k}\leq 1,n\in\mathcal{N}.\vspace{-0.06in}$ (1)

In line of the above discussion, we define the task offloading policy
$\boldsymbol{X}$ that contains all the task offloading variables $x_{n}^{k}$
as
$\boldsymbol{X}=\\{x_{n}^{k}|x_{n}^{k}=1,n\in\mathcal{N},k\in\mathcal{K}\\}$.
Besides, we denote
$\mathcal{N}_{n}=\\{n\in\mathcal{N}|\sum_{k\in\mathcal{K}}x_{n}^{k}=1\\}$ as
the set of EDs offloading their tasks to the MEC server.

Moreover, we consider that each ED and the BS have a single antenna for uplink
communications. We denote $h_{n}^{k}$ as the uplink channel gain between the
ED $n$ and the BS on sub-band $k$. Let
$\boldsymbol{P}=\\{p_{n}^{k}|0<p_{n}^{k}\leq P_{n}^{k},n\in\mathcal{N}_{n}\\}$
denote the transmit power policy of EDs, where $p_{n}^{k}$ is the transmit
power of ED $n$ when offloading the task $Y_{n}$ to the BS via the channel
$k$, subject to a maximum budget $P_{n}^{k}$. We also assume that the MEC
system has a total operational frequency band $B_{MEC}$ that is divided into
$K$ sub-bands of an equal size $W=B_{MEC}/K$ [Hz]. Then, the transmission data
rate of the ED $n$ can be calculated as

$R_{n}=Wlog_{2}\left(1+\frac{p_{n}^{k}h_{n}^{k}}{\sigma^{2}+\sum_{j\in\mathcal{N}_{n},j\neq
n}\left(x_{j}^{k}p_{j}^{k}h_{j}^{k}\right)}\right),$ (2)

where $\sigma^{2}$ is the background noise variance and the second term at the
denominator is the interference among mobile users in the same channel. We
also denote $x_{n}=\sum_{k\in\mathcal{K}}x_{n}^{k},\forall n\in\mathcal{N}$.
Thus, the required time that the ED $n$ upload its task input $D_{n}$ via the
uplink is specified as

$T_{n}^{up}=\frac{D_{n}}{R_{n}},\forall n\in\mathcal{N}.$ (3)

Accordingly, the energy consumption of ED $n$ for offloading the task $Y_{n}$
is specified as

$E_{n}^{up}=p_{n}T_{n}^{up}=p_{n}\frac{D_{n}}{R_{n}},\forall n\in\mathcal{N},$
(4)

where $p_{n}=\sum_{k\in\mathcal{K}}p_{n}^{k},\forall n\in\mathcal{N}$.

####  2.2.2 Computing Model

We consider two computing modes: local execution and edge offloading.

\- Local execution: Let $f_{n}^{l}$ denote the computational resource of ED
$n$ (in CPU cycles/s) allocated to execute the data task, which should not
exceed its total computation capacity $F_{n}$. Thus, we can define the policy
of computational resource allocation of EDs as
$\boldsymbol{F}=\\{f_{n}^{l}|0<f_{n}^{l}\leq F_{n},n\in\mathcal{N}\\}$. The
time consumed to execute the task input $D_{n}$ (with $C_{n}$ in CPU cycles)
at an ED $n$ is expressed as

$T_{n}^{l}=\frac{C_{n}}{f_{n}^{l}}.$ (5)

Moreover, the energy consumption of an ED $n$ when executing its task locally
is specified as

$E_{n}^{l}=\kappa(f_{n}^{l})^{2}C_{n},$ (6)

where $\kappa$ is the energy coefficient depending on the chip architecture
[9] and $C_{n}$ is the CPU workload of ED $n$.

\- Edge offloading: For the offloading case, the MEC server at the BS can
provide computation services to multiple EDs concurrently. Compared to the
local device, the MEC server has much more powerful computation capacity
$f^{e}$ (in CPU cycles/s) and more stable power supply. The execution time of
task $Y_{n}$ at the MEC server can be calculated as

$T_{n}^{ex}=\frac{C_{n}}{f^{e}},\forall n\in\mathcal{N}.$ (7)

Similar to [9, 10], we do not model the downloading part due to the small size
of data results compared to the offloading data.

In summary, the latency cost consumed by the ED $n$ when offloading its task
$Y_{n}$ is given by

$T_{n}^{off}=T_{n}^{up}+T_{n}^{ex}=\left(\frac{D_{n}}{R_{n}}+\frac{C_{n}}{f^{e}}\right),\forall
n\in\mathcal{N}.$ (8)

Moreover, the energy cost consumed by the ED $n$ when offloading its task
$Y_{n}$ is only associated with the data transmission, which is given by

$E_{n}^{off}=E_{n}^{up}=p_{n}\frac{D_{n}}{R_{n}},\forall n\in\mathcal{N}.$ (9)

##  3 Blockchain Consensus Design

In the blockchain-based MEC system, a crucial component is blockchain
consensus that aims to mine the blocks of transactions (i.e., IoT data
records) and add them to the blockchain. To handle the transactions, the EDs
work as blockchain miners to perform mining. In the blockchain-based edge
offloading environment, latency is one of the most important factors
determining the efficiency of a blockchain system. Given a consensus
algorithm, when the number of transactions to the blockchain increases, the
consensus workload to validate and append them into the blockchain will
increase significantly. In current consensus schemes, e.g., DPoS [38], each
miner node must implement a repeated verification process across the miner
network, which results in unnecessary consensus latency and network bandwidth
waste. A possible solution is to reduce the number of miner nodes to reduce
the consensus latency, but it potentially compromises the security of
blockchain because of the high probability of adding compromised transactions
from malicious nodes [39]. To solve these mining issues, here we propose a new
lightweight Proof of Reputation (PoR) consensus mechanism for our blockchain
system. Compared to the DPoS scheme, we make an improvement in the miner
selection based on a reputation score evaluation approach. Moreover, instead
of using a repeated verification among miner nodes, we implement a lightweight
block verification solution that allows each miner only needs to verify once
with another node during the consensus process, which would significantly
reduce the verification latency and save network bandwidth. There are two main
parts of our PoR consensus, including miner node selection and block
verification, as illustrated in Fig. 3.

###  3.1 Miner Node Selection

In this phase, the IoT users first calculate the reputation score of EDs and
then select the miner nodes to implement the mining process.

####  3.1.1 Reputation Calculation

In our MEC system, IoT sensors’ users participate in the delegate selection
process to vote the mining candidates among EDs for performing blockchain
consensus. In this regard, each IoT user votes its preferred ED with the most
reputation. Here, the reputation of an ED is measured by its mining utility
with respect to mining latency. That is, an ED exhibits a lower mining latency
will have a better mining utility which increases its reputation. To this end,
we define a mining utility function of each ED as:

$J_{n}^{mine}=\left[e^{1-\frac{T_{n}^{PoR}}{\tau_{n}}}-1\right]^{+},$ (10)

where $T_{n}^{PoR}$ is the mining latency of the ED $n$ (its detail will be
explained in the following sub-section), $\tau_{n}$ denotes the task execution
latency constraint. $[y]^{+}=\max\\{y,0\\}$ implies that the reputation of an
ED is set to 0 if the mining latency $T_{n}^{PoR}$ is exceeded to its task
execution constraint $\tau_{n}$.

####  3.1.2 Miner Selection

Based on the calculated reputation score, each sensor user will vote for the
ED candidates as the miners based on their reputation ranking. The top EDs
with highest reputation scores are selected to become edge miners (EMs) to
perform mining, as indicated in Fig. 3. Also, similar to the traditional DPoS
framework [38], in our PoR mechanism, each of the active EMs takes turn to act
as a block manager during its time slot to coordinate the consensus process.
In other words, there is one manager in each consensus process. In the next
time slot, another active EM will undertake this manager role.

###  3.2 Lightweight Block Verification

Figure 3: The proposed PoR consensus in our blockchain-based MEC system.

In this phase, the block manager first produces an unverified block $B$ that
contains several offloading transactions collected in a given amount of time.
Then, the manager broadcasts this created block to all EMs within the miner
network for verification. Different from the traditional DPoS scheme [38]
which relies on a repeated verification process among miners, here we
implement a lightweight verification solution that allows each miner to only
verify once with another node during the consensus process. Algorithm 1
presents how our proposed block verification procedure is performed. In lines
4 to 9, the block manager divides the block $B$ consisting of transactions
into $N$ transaction parts $Tr_{n}$, $n\in\mathcal{N}$ that will be assigned
to each mining member $EM$ within the miner group. Each miner $EM$ will also
be assigned a unique random number $R_{n}$. In lines 11 to 21, an $EM$ selects
any miner $s$ ($s\in\mathcal{N}\backslash n$) within the miner group to
implement the verification for its assigned transaction part $Tr_{n}$. If 51%
of the EMs respond positive verification, and the sum of random numbers $Sum$
calculated by all EMs is equal to the initial number set $Rnd$, the block
manager accepts the verified block $B^{\prime}$ and adds it to the blockchain
with a signature. For instance, in Fig. 3, the EM $4$ works as a manager to
create the block $C$ and append it into the blockchain. Otherwise, the manager
discards it from the network (in lines 22 to 27).

Algorithm 1 Procedure of the proposed PoR consensus

1: Input: the unverified block $B$, a set of EMs $\mathcal{N}$

2: Output: the verified block $B^{\prime}$

3: Initialization: Select an unverified block $B$, group the selected EMs $EM$
in the list $Array[n]$, $n\in\mathcal{N}$, initiate public key $Array[n].PK$
and block manager $BM$

4: Divide the block $B$ into $N$ parts $Tr_{n}$, $Sum\leftarrow 0$

5: for $n=1,...,N$ do

6: Set a part of block $Tr_{n}\rightarrow Array[n].content$

7: Assign a random number $R_{n}\leftarrow Random()$

8: Calculate a signature as $Sig_{n}\leftarrow
Hash(Array[n].content,Array[n].PK,timestamp)$

9: end for

10: Specify the total random number $Rnd=N(1+\frac{N-1}{2})$

11: for $n=1,...,N$ do

12: Run a random function $s=Random.randrange(1,N,1)$

13: if $s\neq n$ then

14: Select a random different EM within the list

15: Send the $Tr_{n}$ to $EM_{s}$: $EM\rightarrow
EM_{s}:(Tr_{n},Array[n].PK,Sig_{n},timestamp)$

16: Verify the transaction $Tr_{n}$

17: if ($Array[n].PK_{EM_{s}}==EM_{s}^{EM.PK})\cap(Verify(Sig_{n})\leftarrow
true$) then

18: $Sum\leftarrow Sum+R_{n}$

19: end if

20: end if

21: end for

22: if $Sum==Rnd$ then

23: Accept the block $B$ as a verified one ($B^{\prime}$) and send it back to
the block manager $BM$

24: The manager $BM$ appends the verified block $B^{\prime}$ into the
blockchain network: $BM\rightarrow*:(BM_{PK},B^{\prime},Sign_{B},timestamp)$

25: else

26: Discard the block $B$ from the blockchain

27: end if

###  3.3 Latency of Block Verification

In this sub-section, we calculate the verification latency incurred by the
mining. For simplicity, we assume that the transaction part $Tr_{n}$ (which
also expresses the size) is the same for all EMs. Each EM is willing to
allocate a certain CPU resource $\phi_{n}$ (in CPU cycles) for the
verification of transaction part $n$. Then, the CPU resource allocation policy
for block verification of EDs can be defined as
$\boldsymbol{G}=\\{\phi_{n}|0<\phi_{n}\leq\Phi_{n},n\in\mathcal{N}\\}$, where
$\Phi_{n}$ is the CPU resource budget of the ED $n$. Further, the size of
verified transaction result for the $Tr_{n}$ is denoted by $Tr_{n}^{re}$.
Hence, the transaction verification task can be expressed as a tuple
($Tr_{n},\phi_{n},Tr_{n}^{re}$).

Conceptually, the block verification process in our proposed PoR mechanism at
an EM experiences four steps: (1) unverified block transmission from the block
manager to the EMs, (2) local block verification at the EM, (3) broadcasting
of the verification result among two EMs, and (4) transmission of verification
result feedback from the EMs to the manager. For a miner EM $n$, the time
required to complete these steps is expressed as:

$T_{n}^{PoR}=\frac{Tr_{n}}{r_{n}^{d}}+\frac{\phi_{n}}{F_{n}}+\xi
Tr_{n}|L^{2}|+\frac{Tr_{n}^{re}}{r_{n}^{u}},n\in\mathcal{N},$ (11)

where $r_{n}^{u}$ and $r_{n}^{d}$ are uplink and downlink transmission rates
between the miner $n$ and the block manager. Here, the transmission time of an
unverified transaction part $Tr_{n}$ from the block manager to the miner is
$\frac{Tr_{n}}{r_{n}^{d}}$, while the local verification time of this
transaction is $\frac{\phi_{n}}{F_{n}}$. Moreover, similar to [4], the time
for transaction broadcasting among two miners is a function of transaction
size $Tr_{n}$ and network scale $Tr_{n}|L^{2}|$ (which means two miners for
transaction verification), which is defined as $\xi Tr_{n}|M^{2}|$. Here,
$\xi$ is a pre-defined parameter of broadcasting verification result and
comparison among two miners, which can be acquired from the previous
verification records [4]. Besides, $\frac{Tr_{n}^{re}}{r_{n}^{u}}$ is the
verification feedback time

Meanwhile, in the traditional DPoS scheme [38], each miner has to implement a
repeated verification process among all miners for the block $B$, instead of
dividing into separate transaction parts like our proposed PoR model.
Therefore, the verification latency of the DPoS consensus at an EM $n$ is
expressed as [4]:

$T_{n}^{DPoS}=\frac{B}{r_{n}^{d}}+\frac{\phi_{n}^{B}}{c^{B}_{n}}+\xi
B|L^{N}|+\frac{B^{re}}{r_{n}^{u}},$ (12)

where $\phi_{n}^{B}$ is the CPU resource occupied to verify the block $B$
under the computation budget $c_{n}^{B}$. $B^{re}$ denotes the size of
verified result of the block $B$. $|L^{N}|$ expresses the whole miner network
which means all miners $n$ join the repeated block verification in each
consensus process, instead of two-miner verification in our PoR scheme. By
comparison of equations 11 and 12, it can be seen that the proposed PoR scheme
needs less time for block verification in comparison with the traditional DPoS
scheme, for the same block size and number of miners. Moreover, our mining
scheme can save much network bandwidth due to less message exchange during the
consensus process. The benefits of our proposed PoR mechanism are verified in
the following sections.

##  4 System Utility Formulation and Proposed MA-DRL Algorithm

In this section, we present the system utility formulation for our proposed
TOBM scheme based on the joint consideration of offloading utility and mining
utility as presented in the previous sections. Then, we derive the system
utility optimization problem as a cooperative game and propose a new MA-DRL
algorithm to solve it.

###  4.1 System Utility Formulation

In this paper, we formulate the system utility for the TOBM scheme by taking
both offloading utility and mining utility into account.

####  4.1.1 Offloading Utility

Here, we focus on formulating the QoE-aware offloading utility function. In an
MEC system, the offloading’s QoE is mainly characterized by their task
computation time, i.e., $T_{n}$ and energy consumption, i.e., $E_{n}$.
Specifically, $T_{n}$ and $E_{n}$ can be specified as

$T_{n}=T_{n}^{off}x_{n}+T_{n}^{l}(1-x_{n}),\forall n\in\mathcal{N},$ (13)
$E_{n}=E_{n}^{off}x_{n}+E_{n}^{l}(1-x_{n}),\forall n\in\mathcal{N}.$ (14)

Accordingly, we define a QoE-aware utility function to measure the offloading
utility that is specified as a trade-off between the time and energy
consumption of the task compared with local execution

$J_{n}^{off}=\lambda^{T}_{n}\left(\frac{T_{n}^{l}-T_{n}}{T_{n}^{l}}\right)+\lambda^{E}_{n}\left(\frac{E_{n}^{l}-E_{n}}{E_{n}^{l}}\right),$
(15)

where $\lambda^{T}_{n},\lambda^{E}_{n}\in[0,1]$ (with
$\lambda^{T}_{n}+\lambda^{E}_{n}=1,\forall n\in\mathcal{N}$) are set by ED $n$
to show the preference on time and energy cost when computing the task
$Y_{n}$. If the task is emergency, the ED can increase the weighting factor of
time consumption. Meanwhile, if the ED is operating with low battery, the
factor of energy consumption should be preferred.

It is noting that here, the offloading utility function $J_{n}^{off}$ reflects
the improvement in QoE over local execution, that is measured by
$\left(\frac{T_{n}^{l}-T_{n}}{T_{n}^{l}}\right)$ and
$\left(\frac{E_{n}^{l}-E_{n}}{E_{n}^{l}}\right)$, respectively. Specifically,
when the ED $n$ executes the task locally, the offloading utility equals 0
(i.e., $J_{n}^{off}=0$). If the computation cost (i.e., latency and energy
consumption) of the offloading mode is lower than that of the local execution
mode, the offloading utility ($J_{n}^{off}$) can be positive, which indicates
the offloading’s QoE improvement. However, if offloading too many tasks, the
EDs can suffer from higher latency due to the traffic congestion, which would
reduce the QoE. As a result, the offloading utility ($J_{n}^{off}$) can be
negative.

To this end, we formulate the QoE-aware offloading utility for the MEC system.
Given the offloading decision policy $\boldsymbol{X}$, the transmission power
policy $\boldsymbol{P}$, and the computational resource allocation policy
$\boldsymbol{F}$, we define the offloading utility as the weighted sum of all
MDs’ offloading utilities $J_{n}^{off}$, denoted as:

$J^{off}=\sum_{n\in\mathcal{N}}J_{n}^{off}(\boldsymbol{X},\boldsymbol{P},\boldsymbol{F}),$
(16)

####  4.1.2 Mining Utility

We adopt the mining utility built in equation 10 to analyse the efficiency of
the mining in the blockchain-enabled task offloading. Given the CPU resource
allocation policy $\boldsymbol{G}$, each MEC server yields an utility
$J_{n}^{mine}$ when performing the mining process. Then, we can specify the
total mining utility of EDs in the MEC system as

$J^{mine}=\sum_{n\in\mathcal{N}}J_{n}^{mine}(\boldsymbol{G}).$ (17)

Accordingly, the system utility of each MD $J_{n}$ can be expressed as

$J_{n}=J^{off}_{n}+J^{mine}_{n}.$ (18)

####  4.1.3 System Utility Formulation

In this paper, our objective is maximize the total system utility as the sum
of the offloading utility and the mining utility for the proposed TOBM scheme:

$\displaystyle\underset{\boldsymbol{X},\boldsymbol{P},\boldsymbol{F},\boldsymbol{G}}{\operatorname*{maximize}}$
$\displaystyle J^{off}+J^{mine}$ (19a) subject to $\displaystyle
x_{n}^{k}\in\\{0,1\\},\forall n\in\mathcal{N},k\in\mathcal{K},$ (19b)
$\displaystyle\sum_{k\in\mathcal{K}}x_{n}^{k}\leq 1,n\in\mathcal{N},$ (19c)
$\displaystyle 0<p_{n}^{k}\leq P_{n}^{k},\forall
n\in\mathcal{N}_{n},k\in\mathcal{K},$ (19d) $\displaystyle 0<f_{n}^{l}\leq
F_{n},\forall n\in\mathcal{N},$ (19e) $\displaystyle
0<\phi_{n}\leq\Phi_{n},\forall n\in\mathcal{N}_{n}$ (19f) $\displaystyle
T_{n}\leq\tau_{n},\forall n\in\mathcal{N}.$ (19g)

Here, the constraints (19b) and (19c) imply that each task can be either
executed locally or offloaded to the MEC server via a sub-channel. (19d) shows
the transmission power constraint of each ED. The constraint (19e) states that
each ED $n$ must allocate a positive computational resource to execute the
computing task, but not exceed the total computation budget $F_{n}$. Each ED
$n$ also must allocate a positive CPU resource for the block verification
under a maximum CPU capability $\Phi_{n}$, as indicated in (19f). Constraint
(19g) ensures that each data task needs to be completed under a delay
threshold.

The key intuition behind this integrated calculation is that in the
blockchain-based MEC system, each ED needs to simultaneously perform task
offloading and block mining. In this context, the evaluation of the system
quality, e.g., overall utility performance, must consider both offloading
utility and mining utility. Indeed, an ED needs to minimize its offloading
latency and energy consumption to maintain the quality of task offloading
service, while also minimizing its mining latency to maintain the quality of
block mining service. Therefore, we come up with a final solution to satisfy
both services, aiming to optimize the overall system utility performance.

The problem 19 is non-convex and centralized. As a dynamic TOBM problem (due
to varying channel conditions and task sizes) and high-dimensional system
state space (due to the increase of EDs), the use of traditional optimization
approaches results in high computational complexity which would hinder the
applicability of the proposed model in practical blockchain-based MEC
scenarios. Moreover, most of current solutions [32, 14, 13] rely on single-
agent learning which suffers from some critical shortcomings:

  * •

Dimensionality: The cardinalities of DNN input and output are generally
proportional to the number of EDs, and thus the use of centralized learning to
obtain the optimal policy for all EDs is challenging. Moreover, exploration in
high-dimensional state space is inefficient especially when the number of EDs
increases exponentially, which makes the learning in the blockchain-based MEC
system impractical.

  * •

Information transmission: Since the centralized learning always requires full
information of all EDs (e.g., data task state, resource state, block size
state) for decision making, the information transmission becomes challenging
with the increase of EDs.

Due to the powerlessness of centralized learning algorithms in the multi-agent
environment like our considered blockchain-based MEC system, we propose to use
a distributed MA-DRL scheme to solve our cooperative TOBM problem, as
presented in the following sub-sections.

###  4.2 Cooperative Learning Formulation

First, we convert the objective function 19a from a system utility
maximization problem to a reward maximization problem. To do this, we
formulate the task offloading problem using a multi-agent version of MDP, also
known as a Markov game which is denoted by a tuple
$<\mathcal{N,\mathcal{S},\mathcal{A},\mathcal{O}}>$. Here, each ED $n$ is
considered as an intelligent agent to learn its optimal policy by observing
the TOBM environment and collaborating with other agents, aiming to achieve
the optimal system utility. Then, we have $\mathcal{N}=\\{1,2,...,N\\}$ as the
set of EDs (or agents). Moreover, $\mathcal{S}=\\{s_{1},s_{2},...,s_{N}\\}$ is
defined as the set of states, $\mathcal{A}=\\{a_{1},a_{2},...,a_{N}\\}$ is a
set of agent actions, and $O=\\{o_{1},o_{2},...,o_{N}\\}$ denotes a set of
observations for agents. We assume that the considered cooperative TOBM scheme
operates on discrete time horizon with each time slot $t$ equal and non-
overlapping, and the communication parameters keep unchanged during each time
slot. Now we define each item in the tuple at each time slot $t$ as follows.

####  4.2.1 State

The environment states in the cooperative TOBM network include five
components: task state $S_{task}(t)$, channel state $S_{channel}(t)$, power
state $S_{power}(t)$, resource state $S_{res}(t)$, and transaction state
$S_{trans}(t)$. Therefore, the system state is defined as a matrix:

$\mathcal{S}(t)=\\{S_{task}(t),S_{channel}(t),S_{power}(t),S_{res}(t),S_{trans}(t)\\},$
(20)

where each state vector is explained as follows. $S_{task}(t)$ is defined as
$S_{task}(t)=[D_{n}(t),C_{n}(t)],n\in\mathcal{N}$ where $D_{n}(t)$ represents
the computation task size of the ED $n$ and $C_{n}(t)$ is the required input
CPU cycles number to complete the task data $D_{n}(t)$. $S_{channel}(t)$ is
defined as:

$S_{channel}(t)=c^{k}_{n}(t)=\begin{bmatrix}&c_{1,1}&\cdots&c_{1,K}\\\
&\vdots&\ddots&\vdots\\\ &c_{N,1}&\cdots&c_{N,K}\end{bmatrix},$ (21)

where $c^{k}_{n}(t)$ indicates whether the sub-channel $k$ is used by ED $n$
at time slot $t$. If yes, $c^{k}_{n}(t)=1$, otherwise $c^{k}_{n}(t)=0$. Also,
$S_{power}(t)$ is defined as:

$S_{power}(t)=p^{k}_{n}(t)=\begin{bmatrix}\footnotesize&p_{1,1}&\cdots&p_{1,K}\\\
&\vdots&\ddots&\vdots\\\ &p_{N,1}&\cdots&p_{N,K}\end{bmatrix},$ (22)

where $p^{k}_{n}(t)$ represents the ED $n$’s transmit power level in the $k$th
sub-channel, which is a continuous variable and satisfies $0<p^{k}_{n}(t)\leq
P^{k}_{n}$. Moreover, the resource state $S_{res}(t)$ is expressed as

$S_{res}(t)=\\{v_{1}(t),v_{2}(t),...,v_{N}(t)\\},$ (23)

where $v_{n}(t)$ contains the states of current available computational
resource $f^{l}_{n}(t)$ and CPU resource $\phi_{n}(t)$ of the ED $n$. Lastly,
the transaction state $S_{trans}(t)$ is defined as

$S_{trans}(t)=\\{Tr_{1}(t),Tr_{2}(t),...,Tr_{N}(t)\\},$ (24)

where $Tr_{n}(t)$ is the transaction state of ED $n$.

####  4.2.2 Action

By observing the system states, each ED needs to make actions in each time
step to deal with the task execution and block mining, including offloading
decision, channel selection, transmit power selection, computational resource
allocation, and CPU resource allocation. Accordingly, the action space can be
expressed as

$\mathcal{A}(t)=\\{x_{n}^{k}(t),k(t),p^{k}_{n}(t),f^{l}_{n}(t),\phi_{n}(t)\\},$
(25)

where each action compopent is explained as follows:

  * •

Offloading decision $x_{n}^{k}(t)$: $x_{n}^{k}(t)\in\\{0,1\\}$,
($n\in\mathcal{N},k\in\mathcal{K}$). Each ED $n$ makes decision to execute the
task locally $x_{n}^{k}(t)=0$ or offload it to the MEC server $x_{n}^{k}(t)=1$
via the channel $k$, based on the current task state $S_{task}(t)$.

  * •

Channel selection $k(t)$: $k(t)=[1,2,...,K]$. Each ED $n$ selects one of the
available channels to offload the task to the MEC server, based on the current
channel state $S_{channel}(t)$.

  * •

Transmit power selection $p^{k}_{n}(t)$: $p^{k}_{n}(t)\in(0,P_{n}^{k}]$,
($n\in\mathcal{N}_{n},k\in\mathcal{K}$). Each ED $n$ chooses a transmit power
value to transmit the data task to the MEC server with respect to the current
task state $S_{task}(t)$ and channel state $S_{channel}(t)$.

  * •

Computational resource allocation $f^{l}_{n}(t)$:
$f^{l}_{n}(t)=[f^{l}_{1}(t),f^{l}_{2}(t),...,f^{l}_{N}(t)]$. Each ED $n$
allocates part of its computational resource to execute the task with respect
to the current resource state $S_{res}(t)$ and task state $S_{task}(t)$.

  * •

CPU resource allocation $\phi_{n}(t)$:
$\phi_{n}(t)=[\phi_{1}(t),\phi_{2}(t),...,\phi_{N}(t)]$. Each ED $n$ allocates
part of its CPU resource to verify the blockchain transaction, based on the
current resource state $S_{res}(t)$ and transaction state $S_{trans}(t)$.

####  4.2.3 System Reward Function

The system reward at one time slot $t$ is the sum of the rewards of all EDs.
Each ED $n$ will get a reward $r(s_{n}(t),a_{n}(t))$ in a certain state
$s_{n}(t)$ after executing each possible action $a_{n}(t)$. In our paper, the
system reward function should be positively correlated to the objective
function in the optimization problem 19, aiming to maximize the system utility
of all EDs. Then, we can specify the system reward function of our offloading
network at each time slot $t$ as

$r(s(t),a(t))=\sum_{n\in\mathcal{N}}r(s_{n}(t),a_{n}(t))=J(t),$ (26)

where $J(t)=J^{off}(t)+J^{mine}(t)$ is the total system utility of the
blockchain-based MEC system.

Figure 4:  The proposed MA-DDPG architecture.

###  4.3 Proposed MA-DRL Algorithm for Cooperative TOBM

In the cooperative TOBM problem in our blockchain-based MEC system,
conventional single-agent [32, 14, 13] or independent multi-agent [17, 18, 19]
solutions are unable to obtain the cooperative policies of EDs due to the
nonstationary and partially observable environment. Indeed, when policies of
other agents change (i.e., due to computation mode preference), the ED (agent)
$n$ observation $O_{n}$ can be changed (nonstationary), which makes the
obtained reward $r_{n}(t)$ different from the accumulated reward from its
actual state-action pair. Moreover, in independent multi-agent learning
schemes, the agent $n$ only has the local information and cannot know the
updates from other agents due to non-collaboration. This would affect the
agent $n$’s reward $r_{n}(t)$ and make the learning algorithms hard to ensure
stable convergence [40]. Therefore, we adopt a centralized learning and
decentralized execution solution to implement our MA-DRL algorithm for the
proposed TOBM scheme.

####  4.3.1 Preliminaries of Reinforcement Learning

In RL, an agent takes some actions to obtain rewards through the trial and
error procedure according to a predefined MDP, aiming to accumulate experience
as much as possible to construct an optimal policy. Specifically, the state-
action function can be updated using the agent $n$’s experience tuple
$(s_{n}(t),a_{n}(t),r_{n}(t),s_{n}(t+1))$ at each time step $t$ as

$Q(s_{n}(t),a_{n}(t))\leftarrow Q(s_{n}(t),a_{n}(t))+\alpha\sigma(t),$ (27)

which is called as the Q-learning algorithm [32], where $\sigma(t)$ is a
temporal difference (TD) error that would be zero for the optimal Q-value,
$\alpha$ is the learning rate, and $\gamma$ is the discount factor between
$(0,1)$.

####  4.3.2 Proposed MA-DDPG Algorithm

Here, we present an MA-DRL approach using an MA-DDPG algorithm to solve the
cooperative TOBM problem in the blockchain-based MEC system, as illustrated in
Fig. 4. Different from RL, DRL uses a DNN as the non-linear approximator to
sample the loss function at each training step in order to alleviate the
computational complexity for the large-scale offloading problem. Here, agents
cooperatively offload their tasks to the MEC server and perform mining to form
a shared learning environment consisting of all EDs and the MEC server. In the
centralized training step, the information of state-action of all EDs is
aggregated by the MEC server to train the DRL model where each agent can
obtain the global view of the learning environment to learn collaboratively
with other agents. This makes the learning environment stationary and thus
enhances the convergence performance. After training at the MEC server, the
learned parameters are downloaded to each of EDs to execute the model for
decision making based on its own locally observed information.

We denote $\pi=\\{\pi_{1},\pi_{2},...,\pi_{N}\\}$ as the set of all agent
policies and $\theta=\\{\theta_{1},\theta_{2},...,\theta_{N}\\}$ as the
parameter set of corresponding policies. Every agent updates its parameters
$\theta_{n}$ to obtain the optimal policy
$\pi^{*}_{\theta_{n}}=argmax_{\theta_{n}}J(\theta_{n})$, where $J(\theta_{n})$
is the objective function (also the expected reward) of agent $n$ as defined
in equation 26. MA-DDPG is a deterministic policy gradient-based off-policy
actor-critic operating over continuous action spaces in a multi-agent
environment. Here, the actor generates deterministic action $a$ over time
slots with a behavior network and the critic evaluates the behavior of the
actor with a target network. In the training, the actor updates the behavior
network by computing the gradient of the objective function $J(\theta_{n})$ as

$\bigtriangledown_{\theta_{n}}J(\pi_{n})=\mathds{E}_{\textbf{o},a\sim\mathcal{D}}\left[\bigtriangledown_{\theta_{n}}Q_{n}^{\pi}(\textbf{o},a_{1},...,a_{N}).\bigtriangledown_{\theta_{n}}\pi_{n}(a_{n}|o_{n})\right],$
(28)

with ($\textbf{o}=\\{o_{1},...,o_{N}\\}$) as the observation set,
$Q_{n}^{\pi}(\textbf{o},a_{1},...,a_{N})$ is a centralized action-value
function of the agent $n$ with $a_{1},a_{2},...,a_{N}$ as the actions of all
agents and is learned separately for each $n\in\mathcal{N}$. Also,
$\mathcal{D}$ is the memory buffer for experience replay, containing multiple
episode samples ($\textbf{o},a,r,\textbf{o}^{\prime}$). Moreover, the critic
updates the behavior Q-function $Q_{n}^{\pi}()$ in a fashion that minimizes
the loss function, which is written as

$L(\theta_{n})=\mathds{E}_{\textbf{o},a,r,\textbf{o}^{\prime}}\left[(y_{n}-Q_{n}^{\pi}(\textbf{o},a_{1},a_{2},...,a_{N}))^{2}\right],\vspace{-0.06in}$
(29)

where $y_{n}=r_{n}+\gamma
Q^{\pi}_{n}(\textbf{o}^{\prime},a_{1}^{\prime},a_{2}^{\prime},...a_{N}^{\prime})|_{a^{\prime}_{n}=\pi^{\prime}_{n}(o_{n})}$
is the is TD target and $\pi^{\prime}_{n}(o_{n})$ defines the target policies
with delayed parameters $\theta^{\prime}_{n}$. The training procedure is
summarized in Algorithm 2. Here, the procedure consists of two main phases,
the planning phase and the updating phase. In the planning phase, we use an
$\epsilon$-greedy policy to balance the exploration and exploitation for
updating the Q function (line 7). At each time epoch, each ED executes an
action and estimates the system reward, i.e., system utility, and stores
training information in the replay memory (lines 8-10). After each action, the
ED moves to the next step, updates the critic and actor networks as well as
corresponding target networks (lines 11-17). The training is iterated until
achieving the desired system reward performance.

We can see that the update of $\theta_{n}$ of the target network in the policy
gradient method $\bigtriangledown_{\theta_{n}}$ would guide how the agent ED
acts correctly to obtain the optimal policy. It is based on the fact that that
the value of DNN in the target actor network is fixed for several iterations,
and the weights $\theta_{n}$ of the DNN in the actor behavior network are
updated. In other words, all agent EDs in the blockchain-based MEC system can
maximize their expected function $J(\theta_{n})$ (i.e., user utility) and
obtain stable policies even via interactions between EDs and the environment.
This makes the learning environment stationary even when the policies
$\pi_{n}$ change, which would enhance the quality of policy evaluation for
improving the overall system utility. After the training is completed, EDs
download the learned policy network parameters from the MEC server and update
the target network parameters for the actor and critic as

$\theta_{j}^{\prime}\leftarrow\zeta\theta_{j}+(1-\zeta)\theta_{j}^{\prime},$
(30)

where $\zeta\in(0,1)$ is the update step.

####  4.3.3 Computational Complexity Analysis

In our MA-DRL algorithm, the training process is implemented in the MEC server
with sufficient computational resource. Therefore, we mainly focus on the
computational complexity of the execution process at EDs. Here, a DRL agent
with a DNN is established for each ED and all DNNs run in parallel across the
MEC network. As a result, the overall complexity of the multi-agent system can
be determined by the complexity of a single DNN at an ED. We assume that are
$K$ neurons at the input layer of the DNN for each ED, and $Z$ as the number
of neurons at the output layer. Also, the hidden layer is $L$, and the number
of neurons at hidden layers is $H$. Accordingly, the computation cost at a DNN
is $(KH+(L-1)HH+HZ)$ =$\mathcal{O}(H(K+(L-1)H+Z))$. Also, the complexity of
using activation function is $\mathcal{O}(HL)$. Hence, the total complexity is
$\mathcal{O}(H(K+HL-H+Z+L))$ which can be simplified as
$\mathcal{O}(H(K+HL+Z))$.

Algorithm 2 The MA-DDPG training procedure in the blockchain-based MEC system

1: Input: Replay memory $\mathcal{D}$, time budget $T$, exploration
probability $\epsilon$, discount factor $\gamma$, update step $\zeta$

2: Output: The optimal policy $\pi^{*}_{\theta_{n}}$ and maximum reward
$r^{*}(s,a)$

3: Initialization: Initialize the deep Q network $Q(s,a)$ with random weight
$\theta$ and $\theta^{\prime}$, initialize the exploration probability
$\epsilon\in(0,1)$

4: for episode = 1,…, M do

5: Initialize the state
$s_{0}\leftarrow\\{S_{task}(t),S_{channel}(t),S_{power}(t)\\}|_{t=0}$

6: for $t=1,2,...,T$ do

7: For each agent ED $j\in\mathcal{N}$, select a random action $a_{j}(t)$ with
probability $\epsilon$, otherwise $a_{j}(t)=\pi_{\theta_{j}}(s_{j}(t))$

8: Execute actions $a(t)=(a_{1}(t),a_{2}(t),...,a_{N}(t))$ by performing
offloading decision $x_{n}^{k}(t)$, channel selection $k(t)$, transmit power
selection $p^{k}_{n}(t)$, computational resource allocation $f^{l}_{n}(t)$,
and CPU resource allocation $\phi_{n}(t)$

9: Observe the system reward $r(t)$ via 26 and the new state
$\textbf{s}^{\prime}$

10: Store $(s(t),a(t),r(t),\textbf{s}^{\prime}(t))$ into the memory
$\mathcal{D}$

11: for $agent~{}j=1$ to $N$ do

12: Sample random mini-batch of transitions
($s_{j},a_{j},r_{j},\textbf{s}^{\prime}_{j}$) from $\mathcal{D}$

13: Set $y_{j}=r_{j}+\gamma
Q^{\pi}_{j}(\textbf{s}^{\prime}_{j},a_{1}^{\prime},a_{2}^{\prime},...a_{N}^{\prime})|_{a^{\prime}_{j}=\pi^{\prime}_{j}(o_{j})}$

14: Update behavior critic by minimizing the loss:
$L(\theta_{j})=\frac{1}{S}\sum_{j}\left[y_{j}-Q_{j}^{\pi}({s}_{j},a_{1},a_{2},...,a_{N})\right]^{2}$

15: Update actor by using the sampled policy gradient:
$\bigtriangledown_{\theta_{j}}J(\pi_{j})=\frac{1}{S}\left[\bigtriangledown_{\theta_{j}}Q_{j}^{\pi}(\textbf{s}_{j}^{\prime},a_{1},a_{2},...,a_{N}).\bigtriangledown_{\theta_{j}}\pi_{j}(a_{j}|s_{j})\right]$

16: end for

17: Update the target network parameters for each agent via 30

18: end for

19: end for

###  4.4 Cooperative Game-theoretic Solution

We next develop a game-theoretic approach for the proposed TOMB problem, where
each ED can act as a game player to react to other players’ decisions for
maximizing its utility [41], [42]. After a number of steps, all the EDs self-
organize into a mutual equilibrium state, i.e., the Nash equilibrium, at which
no ED can further increase its utility by unilaterally altering its strategy.

We first define the game as
$G=\\{\mathcal{N},\\{\mathcal{A}_{n}\\}_{n\in\mathcal{N}},\\{J_{n}\\}_{n\in\mathcal{N}}\\}$,
where $\mathcal{N}$ is the set of rational game players, $\mathcal{A}_{n}$ is
the strategy set for player $n$, and $J_{n}$ is the utility function of ED
$n$. Let denote $a_{n}$ as the offloading decision profile of the player $n$
over the wireless sub-bands $\mathcal{K}$, from 18 we can rewrite as

$J_{n}(a_{n})=J_{n}^{off}I_{\\{a_{n}=1\\}}+J_{n}^{mine},$ (31)

where $I_{z}$ is an indicator function. If $z$ is true, $I_{z}=1$; otherwise,
$I_{z}=0$. Based on 18 and 31, the utility of a player $n$ in the cooperative
game can be expressed as

$J(a_{n},a_{-n})=\begin{cases}J_{n}^{mine},&a_{n}=0\\\
J_{n}^{off}+J_{n}^{mine},&a_{n}=1\end{cases},$ (32)

where $a_{-n}$ is the offloading decisions of all players exept $n$.
Accordingly, by considering the influence of other players on the utility
optimization of a player $n$ in the cooperative game, we can express the game-
theoretic utility function as

$\scriptstyle J(a_{n},a_{-n})=\begin{cases}\scriptstyle
J_{n}^{mine},&\scriptstyle a_{n}=0\\\
\scriptstyle\sum_{m\\#n}\left(J_{n}^{off}(a_{n})+J_{m}^{off}(a_{m})\right)I_{\\{a_{m}=a_{n}\\}}\\\
\scriptstyle+J_{n}^{mine}(a_{n}),&\scriptstyle a_{n}=1\end{cases}$ (33)

where $m\in\mathcal{N}\backslash\\{n\\}$.
Theorem 1: The collaborative game
$G=\\{\mathcal{N},\\{\mathcal{A}_{n}\\}_{n\in\mathcal{N}},\\{J_{n}\\}_{n\in\mathcal{N}}\\}$
has a pure Nash equilibrium (NE) and guarantees the finite improvement
property.
Proof: We first prove that the proposed game G is an exact (cardinal)
potential game with potential function

$\Psi(a_{n},a_{-n})=\frac{1}{2}\sum_{n\in\mathcal{N}}\sum_{m\\#n}(J_{n}^{off}+J_{m}^{off})I_{\\{a_{m}=a_{n}\\}}I_{\\{a_{n}=1\\}}\\\
+\sum_{n\in\mathcal{N}}J_{n}^{mine}I_{\\{a_{n}=1\\}}+\sum_{n\in\mathcal{N}}J_{n}^{mine}I_{\\{a_{n}=0\\}},$
(34)

such that

$\Psi(a^{\prime}_{n},a_{-n})-\Psi(a_{n},a_{-n})=J(a^{\prime}_{n},a_{-n})-J(a_{n},a_{-n}),\\\
\forall a_{n},a_{-n}\in\mathcal{A}_{n}.$ (35)

We now consider three cases as follows:
Case 1: $a_{n}>0,a_{n}^{\prime}>0$

$\Psi(a_{n}^{\prime},a_{-n})-\Psi(a_{n},a_{-n})\\\
=\frac{1}{2}\sum_{m\\#n}\left(J_{n}^{off}(a_{n}^{\prime})-J_{n}^{off}(a_{m})\right)I_{\\{a_{m}=a_{n}^{\prime}\\}}\\\
+\frac{1}{2}\sum_{m\\#n}\left(J_{n}^{off}(a_{m})-J_{n}^{off}(a_{n}^{\prime})\right)I_{\\{a_{n}^{\prime}=a_{m}\\}}+J_{n}^{mine}(a_{n}^{\prime})\\\
-\frac{1}{2}\sum_{m\\#n}\left(J_{n}^{off}(a_{n})-J_{n}^{off}(a_{m})\right)I_{\\{a_{m}=a_{n}\\}}\\\
-\frac{1}{2}\sum_{m\\#n}\left(J_{n}^{off}(a_{m})-J_{n}^{off}(a_{n})\right)I_{\\{a_{n}=a_{m}\\}}+J_{n}^{mine}(a_{n})-J_{n}^{mine}(a_{n})\\\
=\sum_{m\\#n}\left(J_{n}^{off}(a_{n}^{\prime})+J_{m}^{off}(a_{m})\right)I_{\\{a_{m}=a_{n}^{\prime}\\}}+J_{n}^{mine}(a_{n}^{\prime})\\\
-\sum_{m\\#n}\left(J_{n}^{off}(a_{n})+J_{m}^{off}(a_{m})\right)I_{\\{a_{m}=a_{n}\\}}-J_{n}^{mine}(a_{n})\\\
=J(a^{\prime}_{n},a_{-n})-J(a_{n},a_{-n}).$ (36)

Case 2: $a_{n}=0,a_{n}^{\prime}>0$

$\Psi(a_{n}^{\prime},a_{-n})-\Psi(a_{n},a_{-n})\\\
=\frac{1}{2}\sum_{m\\#n}\left(J_{n}^{off}(a_{n}^{\prime})-J_{n}^{off}(a_{m})\right)I_{\\{a_{m}=a_{n}^{\prime}\\}}\\\
+\frac{1}{2}\sum_{m\\#n}\left(J_{n}^{off}(a_{m})-J_{n}^{off}(a_{n}^{\prime})\right)I_{\\{a_{n}^{\prime}=a_{m}\\}}+J_{n}^{mine}(a_{n}^{\prime})-J_{n}^{mine}\\\
=\sum_{m\\#n}\left(J_{n}^{off}(a_{n}^{\prime})+J_{m}^{off}(a_{m})\right)I_{\\{a_{m}=a_{n}^{\prime}\\}}+J_{n}^{mine}(a_{n}^{\prime})-J_{n}^{mine}\\\
=J(a^{\prime}_{n},a_{-n})-J(a_{n},a_{-n}).$ (37)

Case 3: $a_{n}>0,a_{n}^{\prime}=0$

Similar to Case 2, it is also straightforward to prove that
$\Psi(a_{n}^{\prime},a_{-n})-\Psi(a_{n},a_{-n})=J(a^{\prime}_{n},a_{-n})-J(a_{n},a_{-n})$.

Therefore, the game $G$ is an exact potential game with the potential function
given in 34. Finally, according to the potential game theory [41], our
proposed collaborative game $G$ has an NE and possesses the finite improvement
property.

##  5 Simulations and Performance Analysis

In this section, we perform extensive simulations to verify the performance of
the proposed TOBM scheme.

###  5.1 Simulation Setting

We here leverage the widely used mobile wireless dataset provided by Shanghai
Telecom111 http://www.sguangwang.com/dataset/telecom.zip for numerical
simulations. We select maximum 500 mobile phones as EDs and an MEC server in a
sub-area of Shanghai city with the geographical distribution as illustrated in
Fig. 5. Moreover, IoT sensor data traces from location services [43] collected
during 6 months in 2014 are selected as data tasks for task offloading
simulations. Inspired by [11, 32, 14, 13], our simulation parameters are
configured as in Table II.

TABLE II: Simulation parameters.
---
Figure 5: Illustration of the MEC network with an MEC server and distributed
mobile phones.

(a) Comparison of average system rewards with different learning rates.

(b) Comparison of average system rewards with different algorithms.

Figure 6: Evaluation of training performance.

Moreover, the channel gains $h_{n}^{k}$ are generated using distance-dependent
path-loss model $L[dB]=140.7+36.7log_{10}d_{[km]}$. For the proposed multi-
agent DRL algorithm, the discount factor $\gamma$ equals 0.85 and the replay
memory capacity and training batch size are set to $10^{5}$ and 128,
respectively. The update step $\zeta$ in the critic-actor training is set to
0.8. The used DNN structure has three hidden layers (64, 32 and 32 neurons)
with ReLU as the activation function and Adam as the optimizer. For blockchain
mining, we set up 10 transactions per block and vary the numbers of mining
nodes (i.e., EDs) from 2 to 100. The other mining parameters are included in
Table II. The results from simulation are averaged from 50 runs of numerical
simulations.

###  5.2 Evaluation of Training Performance

We first evaluate the training performance of our proposed algorithm. To prove
the advantage of the proposed cooperative MA-DDPG algorithm, we compare its
performance with the state-of-the-art non-cooperative schemes, including DDPG,
actor-critic [26] and DQN [18]. Here, DDPG and actor-critic are policy-based
algorithms where each ED agent only observes the local information and does
not the information of other EDs during the training. Meanwhile, DQN is a
value-based algorithm where each ED also has no information of other EDs.

Fig. 6(a) shows the performance of average system reward with different
learning rates $\alpha$. It can be seen that the learning rate affects the
learning rewards over the training episodes. That is, when the learning rate
decreases, the convergence performance of the proposed algorithm decreases due
to slow learning speed. Based on our experimental results, the learning value
$\alpha=0.01$ yields the best reward performance and has good convergence rate
and thus we use it in the following system simulations and evaluations.

Fig. 6(b) shows the learning curves of the average system reward with the
increase of episodes for an MEC system with 50 EDs. It is clear that our MA-
DDPG scheme is more robust and yields the best performance in terms of average
system reward, compared to the baseline schemes. This is because the proposed
scheme allows EDs to learn mutually the cooperative policy which helps reduce
the channel congestion and user interference, and enhance computational
resource efficiency for improving the overall system reward. Meanwhile, in the
DQN and actor-critic schemes, EDs greedily access the wireless channel
spectrum to maximize their own utility as much as possible without
collaborating with each other, which increases the possibility of channel
collision and thus results in higher offloading latency. Consequently, the
average system reward becomes worse. Although the DDPG scheme shows a better
reward performance than these two schemes, it still remains a non-stationary
learning issue and its average reward is lower than that of the MA-DDPG
scheme.

###  5.3 Evaluation of Task Offloading Utility

(a) Average offloading utility with different numbers of EDs.

(b) Average offloading utility with different IoT data task sizes.

Figure 7: Evaluation of task offloading performance.

Next, Fig. 7(a) indicates the performance of average offloading utility versus
the different numbers of offloaded EDs. It can be seen that when the number of
EDs is small ($<60$), the average offloading utility increases with the number
of EDs because in this case, the MEC system can support sufficient spectrum
and computing resources to handle all tasks offloaded from EDs. However, after
exceeding some thresholds (e.g., $N=60$ EDs), the system utility decreases
because the higher the number of offloaded EDs, the higher the competition of
resource usage (i.e., channel spectrum). Note that the configured number of
sub-bands $K=30$ is relatively low with respect to the increase in the number
of EDs, thus the channel bandwidth allocated for each ED decreases when there
are more EDs in the system. In turn, this increases the offloading latency and
thus, degrades the overall offloading utility. Nevertheless, our MA-DDPG
scheme still achieves the best utility performance due to its cooperative
offloading policies among EDs compared to the other schemes with selfish
learning. For instance, in the case of 100 EDs, the average offloading utility
of the MA-DDPG scheme is 22.5%, 37.5%, and 43.6% higher than those of the
DDPG, actor-critic, and DQN schemes, respectively. These results also imply
that as the EDs number increases, the cooperative policy learned by MA-DDPG
becomes more important in the cooperative edge task offloading.

Moreover, we evaluate the effect of different task sizes on the average
offloading utility as shown in Fig. 7(b). We find that a higher task size
results in a higher offloading utility. Specifically, when the task size is
relatively high ($>2$MB), the offloading utility increases significantly. The
reason is that the edge computation mode becomes more efficient than the local
execution mode in terms of lower computing cost in handling the larger-size
tasks due to the high MEC capability. This leads to the increase of the user
utility $J_{n}$ which thus enhances the system-inter utility $J$.
Particularly, when the task size increases, the advantage of the proposed MA-
DDPG scheme over the baselines become more profound, with the larger
performance gaps and better utilities. For example, as the task size is 3.5
MB, the proposed MA-DDPG scheme achieves 30%, 24.5% and 21.7% higher
offloading utilities compared with the DDPG, actor-critic, and DQN schemes,
respectively.

###  5.4 Evaluation of Blockchain Performance

Here, we evaluate the performance of our proposed PoR consensus scheme via
numerical simulations using Python programming and compare it with the
traditional DPoS scheme [38] via the verification block latency and bandwidth
usage metrics.

(a) Comparison of block verification latency.

(b) Comparison of network bandwidth cost.

Figure 8: Evaluation of blockchain performance. Figure 9: Comparison of system
utility with non-cooperative schemes.  Figure 10: Comparison of system utility
with cooperative schemes.

We first show the block verification latency performance versus different
numbers of mining nodes with the block size fixed at 50 KB and compare with
the traditional DPoS scheme [38]. As illustrated in Fig. 8(a), our proposed
PoR scheme requires significantly less time for mining blocks, compared to the
DPoS scheme thanks to the optimized block verification procedure. Although the
time required for block verification increases with the increasing number of
miners, our solution still achieves a much better performance than that of the
DPoS scheme. This result confirms our lightweight consensus design that is
thus well suitable for large-scale blockchain-based MEC systems.

(a) Average system utility with different numbers of EDs.

(b) Average system utility with different IoT data task sizes.

Figure 11: Comparison of learning and game-theoretic approaches.

Next, Fig. 8(b) indicates the simulation result in terms of network bandwidth
cost spent by the mining process for different data block sizes from 50 KB to
500 KB in the edge blockchain network. Due to the optimized block exchange
procedure where each ES only needs to contact with one different miner for
transaction verification, instead of using a repeated process, our PoR scheme
can save much network bandwidth resources, compared to the DPoS scheme [38].

###  5.5 Evaluation of Overall System Utility Performance

In this subsection, we evaluate the performance of the overall system utility
of our TOBM scheme enabled by the joint consideration of offloading utility
and mining utility. The performances of our cooperative scheme with our PoR
mining design and other non-cooperative schemes with PoR and DPoS mining are
illustrated in Fig. 9. Unsurprisingly, our TOBM scheme with a PoR mining
design achieves the best overall system utility. The reasons for this
observation are two-fold. First, our offloading scheme with a cooperative MA-
DDPG algorithm outperforms other non-cooperative offloading schemes in terms
of a better offloading utility, as evidenced in Fig. 7. Second, our PoR design
yields a lower mining latency which consequently increases the mining utility,
as explained in Section 3.1.1. As a result, our scheme with a cooperative
offloading design and a lightweight mining design achieves a much better
overall system utility than the other non-cooperative offloading schemes with
DPoS design. Moreover, due to its better mining utility, our PoR design
contributes to better overall system utilities in each non-cooperative
offloading scheme, compared to the use of DPoS design.

Furthermore, we compare the system utility performance of our cooperative TOBM
scheme with other cooperative schemes, including a cooperative scheme with
DPoS design [27] and a cooperative scheme without mining design [26]. As shown
in Fig. 10, our TOBM scheme with PoR design achieves a better system utility
than the cooperative scheme with DPoS design, thanks to the better mining
utility of our PoR framework. Meanwhile, the cooperative scheme in [26] has
lowest system utility due to the lack of consideration of mining design.

###  5.6 Comparison of Learning and Game-theoretic Approaches

Next, we compare the utility performance between the MA-DRL scheme and the
game-theoretic scheme, where the DQN scheme is used as the baseline, after
averaging the results from 5 simulations. As shown in Fig. 11(a), the game-
theoretic approach can achieve the optimal system utility, compared to the MA-
DRL scheme, when increasing the number of MDs. This is because in the game
approach, each MD can obtain the full knowledge of other MDs’ information such
as the information of offloading decisions and mining status via the
collaborative interactions. This allows each MD to determine the optimal
computation offloading strategy to achieve the converged point of NE.
Meanwhile, the MA-DRL scheme can also achieve reasonably close results, where
the physical parameters of MDs are time-varying, and each MD can compute the
approximately optimal computation offloading strategy without requiring any
prior information about other MDs. Similar performances can also be seen in
Fig. 11(b), when increasing the size of task inputs.

##  6 Conclusions and Future Works

In this article, we have proposed a novel cooperative TOBM scheme to enable a
joint design of task offloading and blockchain mining in blockchain-based MEC
systems. First, we have proposed a new cooperative offloading framework that
enables EDs to learn offloading policies in a collaborative manner. Then, we
have designed a new PoR mining scheme enabled by a lightweight block
verification strategy. To this end, we have formulated a joint offloading and
mining optimization problem which is solved by an MA-DRL algorithm. We then
derived a game-theoretic solution to model the competition among EDs in
offloading and mining as a potential game, and proved the existence of a pure
Nash equilibrium. Simulation results have clearly showed the significant
advantages of our proposed scheme over the existing schemes in terms of higher
system rewards with better offloading utility and lower blockchain costs which
thus enhance the overall system utility.

Our proposed approach has potential for future intelligent mobile networks,
where EDs are able to build distributed intelligent solutions via our
cooperative DRL model for enabling intelligent computation, communications and
network control [44]. In future work, it is of interest to consider fair
resource allocation strategies for simultaneously supporting the edge
computation and blockchain services. The tradeoff between mining security and
latency should be also studied to strike a beneficial balance between these
two important design factors before integrating into MEC. Moreover, resource
trading solutions should be developed to enable reliable energy purchase for
resource-constrained edge nodes in blockchain-based MEC systems.

## References

  * [1] H. Guo, J. Liu, J. Ren, and Y. Zhang, “Intelligent Task Offloading in Vehicular Edge Computing Networks,” _IEEE Wireless Communications_ , vol. 27, no. 4, pp. 126–132, Aug. 2020.
  * [2] H. Guo and J. Liu, “UAV-Enhanced Intelligent Offloading for Internet of Things at the Edge,” _IEEE Transactions on Industrial Informatics_ , vol. 16, no. 4, pp. 2737–2746, Apr. 2020.
  * [3] P. A. Apostolopoulos, G. Fragkos, E. E. Tsiropoulou, and S. Papavassiliou, “Data Offloading in UAV-assisted Multi-access Edge Computing Systems under Resource Uncertainty,” _IEEE Transactions on Mobile Computing_ , pp. 1–1, 2021.
  * [4] J. Kang, Z. Xiong, D. Niyato, D. Ye, D. I. Kim, and J. Zhao, “Toward Secure Blockchain-Enabled Internet of Vehicles: Optimizing Consensus Management Using Reputation and Contract Theory,” _IEEE Transactions on Vehicular Technology_ , vol. 68, no. 3, pp. 2906–2920, 2019.
  * [5] Z. Xiong, Y. Zhang, D. Niyato, P. Wang, and Z. Han, “When Mobile Blockchain Meets Edge Computing,” _IEEE Communications Magazine_ , vol. 56, no. 8, pp. 33–39, Aug. 2018.
  * [6] S. Guo, X. Hu, S. Guo, X. Qiu, and F. Qi, “Blockchain Meets Edge Computing: A Distributed and Trusted Authentication System,” _IEEE Transactions on Industrial Informatics_ , vol. 16, no. 3, pp. 1972–1983, Mar. 2020.
  * [7] J. Kang, R. Yu, X. Huang, M. Wu, S. Maharjan, S. Xie, and Y. Zhang, “Blockchain for Secure and Efficient Data Sharing in Vehicular Edge Computing and Networks,” _IEEE Internet of Things Journal_ , vol. 6, no. 3, pp. 4660–4670, Jun. 2019.
  * [8] J. Guo, X. Ding, and W. Wu, “Reliable Traffic Monitoring Mechanisms Based on Blockchain in Vehicular Networks,” _IEEE Transactions on Reliability_ , pp. 1–11, 2021.
  * [9] C. Wang, C. Liang, F. R. Yu, Q. Chen, and L. Tang, “Computation Offloading and Resource Allocation in Wireless Cellular Networks With Mobile Edge Computing,” _IEEE Transactions on Wireless Communications_ , vol. 16, no. 8, pp. 4924–4938, Aug. 2017.
  * [10] Z. Kuang, L. Li, J. Gao, L. Zhao, and A. Liu, “Partial Offloading Scheduling and Power Allocation for Mobile Edge Computing Systems,” _IEEE Internet of Things Journal_ , vol. 6, no. 4, pp. 6774–6785, Aug. 2019.
  * [11] J. Zhao, Q. Li, Y. Gong, and K. Zhang, “Computation Offloading and Resource Allocation For Cloud Assisted Mobile Edge Computing in Vehicular Networks,” _IEEE Transactions on Vehicular Technology_ , vol. 68, no. 8, pp. 7944–7956, Aug. 2019.
  * [12] D. C. Nguyen, P. N. Pathirana, M. Ding, and A. Seneviratne, “Deep Reinforcement Learning for Collaborative Offloading in Heterogeneous Edge Networks,” in _2021 IEEE/ACM 21st International Symposium on Cluster, Cloud and Internet Computing (CCGrid)_. Melbourne, Australia: IEEE, May 2021, pp. 297–303.
  * [13] Y. Liu, H. Yu, S. Xie, and Y. Zhang, “Deep Reinforcement Learning for Offloading and Resource Allocation in Vehicle Edge Computing and Networks,” _IEEE Transactions on Vehicular Technology_ , vol. 68, no. 11, pp. 11 158–11 168, Nov. 2019.
  * [14] D. C. Nguyen, P. N. Pathirana, M. Ding, and A. Seneviratne, “Privacy-Preserved Task Offloading in Mobile Blockchain with Deep Reinforcement Learning,” _IEEE Transactions on Network and Service Management_ , vol. 17, no. 4, pp. 2536–2549, Jul. 2020.
  * [15] Y. Yu, S. C. Liew, and T. Wang, “Multi-Agent Deep Reinforcement Learning Multiple Access for Heterogeneous Wireless Networks with Imperfect Channels,” _IEEE Transactions on Mobile Computing_ , pp. 1–1, Feb. 2021.
  * [16] X. Xu, X. Zhang, H. Gao, Y. Xue, L. Qi, and W. Dou, “BeCome: Blockchain-Enabled Computation Offloading for IoT in Mobile Edge Computing,” _IEEE Transactions on Industrial Informatics_ , vol. 16, no. 6, pp. 4187–4195, Jun. 2020.
  * [17] J. Heydari, V. Ganapathy, and M. Shah, “Dynamic Task Offloading in Multi-Agent Mobile Edge Computing Networks,” in _2019 IEEE Global Communications Conference (GLOBECOM)_ , Waikoloa, HI, USA, Dec. 2019, pp. 1–6.
  * [18] Y. Zhang, B. Di, Z. Zheng, J. Lin, and L. Song, “Joint Data Offloading and Resource Allocation for Multi-Cloud Heterogeneous Mobile Edge Computing Using Multi-Agent Reinforcement Learning,” in _2019 IEEE Global Communications Conference (GLOBECOM)_ , Waikoloa, HI, USA, Dec. 2019, pp. 1–6.
  * [19] X. Liu, J. Yu, and Y. Gao, “Multi-agent Reinforcement Learning for Resource Allocation in IoT networks with Edge Computing,” _arXiv preprint arXiv:2004.02315_ , 2020.
  * [20] G. Xu, Y. Liu, and P. W. Khan, “Improvement of the DPoS Consensus Mechanism in Blockchain Based on Vague Sets,” _IEEE Transactions on Industrial Informatics_ , vol. 16, no. 6, pp. 4252–4259, Jun. 2020\.
  * [21] L. Jiang, S. Xie, S. Maharjan, and Y. Zhang, “Joint Transaction Relaying and Block Verification Optimization for Blockchain Empowered D2D Communication,” _IEEE Transactions on Vehicular Technology_ , vol. 69, no. 1, pp. 828–841, Jan. 2020.
  * [22] X. Qiu, L. Liu, W. Chen, Z. Hong, and Z. Zheng, “Online Deep Reinforcement Learning for Computation Offloading in Blockchain-Empowered Mobile Edge Computing,” _IEEE Transactions on Vehicular Technology_ , vol. 68, no. 8, pp. 8050–8062, Aug. 2019.
  * [23] S. Guo, Y. Dai, S. Guo, X. Qiu, and F. Qi, “Blockchain Meets Edge Computing: Stackelberg Game and Double Auction Based Task Offloading for Mobile Blockchain,” _IEEE Transactions on Vehicular Technology_ , vol. 69, no. 5, pp. 5549–5561, May 2020.
  * [24] M. Liu, F. R. Yu, Y. Teng, V. C. M. Leung, and M. Song, “Computation Offloading and Content Caching in Wireless Blockchain Networks With Mobile Edge Computing,” _IEEE Transactions on Vehicular Technology_ , vol. 67, no. 11, pp. 11 008–11 021, Nov. 2018.
  * [25] Z. Zhang, Z. Hong, W. Chen, Z. Zheng, and X. Chen, “Joint Computation Offloading and Coin Loaning for Blockchain-Empowered Mobile-Edge Computing,” _IEEE Internet of Things Journal_ , vol. 6, no. 6, pp. 9934–9950, Dec. 2019.
  * [26] J. Feng, F. Richard Yu, Q. Pei, X. Chu, J. Du, and L. Zhu, “Cooperative Computation Offloading and Resource Allocation for Blockchain-Enabled Mobile-Edge Computing: A Deep Reinforcement Learning Approach,” _IEEE Internet of Things Journal_ , vol. 7, no. 7, pp. 6214–6228, Jul. 2020.
  * [27] F. Guo, F. R. Yu, H. Zhang, H. Ji, M. Liu, and V. C. M. Leung, “Adaptive Resource Allocation in Future Wireless Networks With Blockchain and Mobile Edge Computing,” _IEEE Transactions on Wireless Communications_ , vol. 19, no. 3, pp. 1689–1703, Mar. 2020.
  * [28] Z. Li, M. Xu, J. Nie, J. Kang, W. Chen, and S. Xie, “NOMA-Enabled Cooperative Computation Offloading for Blockchain-Empowered Internet of Things: A Learning Approach,” _IEEE Internet of Things Journal_ , pp. 1–1, 2020.
  * [29] M. Li, D. Hu, C. Lal, M. Conti, and Z. Zhang, “Blockchain-Enabled Secure Energy Trading With Verifiable Fairness in Industrial Internet of Things,” _IEEE Transactions on Industrial Informatics_ , vol. 16, no. 10, pp. 6564–6574, Oct. 2020.
  * [30] Z. Yang, K. Liu, Y. Chen, W. Chen, and M. Tang, “Two-Level Stackelberg Game for IoT Computational Resource Trading Mechanism: A Smart Contract Approach,” _IEEE Transactions on Services Computing_ , pp. 1–1, 2020.
  * [31] W. Chen, Z. Zhang, Z. Hong, C. Chen, J. Wu, S. Maharjan, Z. Zheng, and Y. Zhang, “Cooperative and Distributed Computation Offloading for Blockchain-Empowered Industrial Internet of Things,” _IEEE Internet of Things Journal_ , vol. 6, no. 5, pp. 8433–8446, Oct. 2019.
  * [32] J. Wang, J. Hu, G. Min, W. Zhan, Q. Ni, and N. Georgalas, “Computation Offloading in Multi-Access Edge Computing Using a Deep Sequential Model Based on Reinforcement Learning,” _IEEE Communications Magazine_ , vol. 57, no. 5, pp. 64–69, May 2019.
  * [33] L. Busoniu, R. Babuska, and B. De Schutter, “A Comprehensive Survey of Multiagent Reinforcement Learning,” _IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)_ , vol. 38, no. 2, pp. 156–172, Mar. 2008.
  * [34] Z. Cao, P. Zhou, R. Li, S. Huang, and D. Wu, “Multiagent Deep Reinforcement Learning for Joint Multichannel Access and Task Offloading of Mobile-Edge Computing in Industry 4.0,” _IEEE Internet of Things Journal_ , vol. 7, no. 7, pp. 6201–6213, Jul. 2020.
  * [35] D. Kwon, J. Jeon, S. Park, J. Kim, and S. Cho, “Multi-Agent DDPG-based Deep Learning for Smart Ocean Federated Learning IoT Networks,” _IEEE Internet of Things Journal_ , vol. 7, no. 10, pp. 9895–9903, Apr. 2020.
  * [36] S. Wang, J. Duan, D. Shi, C. Xu, H. Li, R. Diao, and Z. Wang, “A Data-driven Multi-agent Autonomous Voltage Control Framework Using Deep Reinforcement Learning,” _IEEE Transactions on Power Systems_ , vol. 35, no. 6, pp. 4644–4654, Apr. 2020.
  * [37] D. Nguyen, P. Pathirana, M. Ding, and A. Seneviratne, “Secure Computation Offloading in Blockchain based IoT Networks with Deep Reinforcement Learning,” _IEEE Transactions on Network Science and Engineering_ , pp. 1–1, 2021.
  * [38] W. Sun, J. Liu, Y. Yue, and P. Wang, “Joint Resource Allocation and Incentive Design for Blockchain-Based Mobile Edge Computing,” _IEEE Transactions on Wireless Communications_ , vol. 19, no. 9, pp. 6050–6064, Sep. 2020.
  * [39] F. Yang, W. Zhou, Q. Wu, R. Long, N. N. Xiong, and M. Zhou, “Delegated Proof of Stake With Downgrade: A Secure and Efficient Blockchain Consensus Algorithm With Downgrade Mechanism,” _IEEE Access_ , vol. 7, pp. 118 541–118 555, 2019.
  * [40] R. Lowe, Y. WU, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch, “Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 6379–6390.
  * [41] H. Guo and J. Liu, “Collaborative Computation Offloading for Multiaccess Edge Computing Over Fiber–Wireless Networks,” _IEEE Transactions on Vehicular Technology_ , vol. 67, no. 5, pp. 4514–4526, May 2018\.
  * [42] Z. Ning, P. Dong, X. Wang, X. Hu, J. Liu, L. Guo, B. Hu, R. Kwok, and V. C. M. Leung, “Partial Computation Offloading and Adaptive Task Scheduling for 5G-enabled Vehicular Networks,” _IEEE Transactions on Mobile Computing_ , pp. 1–1, 2020.
  * [43] S. Yang, K. Xu, L. Cui, Z. Ming, Z. Chen, and Z. Ming, “EBI-PAI: Towards An Efficient Edge-Based IoT Platform for Artificial Intelligence,” _IEEE Internet of Things Journal_ , pp. 1–1, 2020.
  * [44] D. C. Nguyen, M. Ding, P. N. Pathirana, A. Seneviratne, J. Li, D. Niyato, O. Dobre, and H. V. Poor, “6G Internet of Things: A Comprehensive Survey,” _IEEE Internet of Things Journal_ , 2021.

|  Dinh C. Nguyen (Member, IEEE) is currently working toward the Ph.D. degree
with the School of Engineering, Deakin University, Victoria, Australia. His
research interests focus on wireless communications, federated learning, deep
reinforcement learning, blockchain, and edge computing. He has published over
20 papers as the first author at the top-tier IEEE journals and conferences,
such as IEEE Transactions on Mobile Computing, IEEE Wireless Communications
Magazine, IEEE Communications Surveys and Tutorials, IEEE Internet of Things
Journal, IEEE GLOBECOM, ICC, and CCGrid conferences. He has been a recipient
of the prestigious Data61 PhD scholarship, CSIRO, Australia. He has been the
TPC member of top-tier conferences including IEEE GLOBECOM 2021.
---|---
|  Ming Ding (Senior Member, IEEE) received the B.S. and M.S. degrees (with
first-class Hons.) in electronics engineering from Shanghai Jiao Tong
University (SJTU), Shanghai, China, and the Doctor of Philosophy (Ph.D.)
degree in signal and information processing from SJTU, in 2004, 2007, and
2011, respectively. From April 2007 to September 2014, he worked at Sharp
Laboratories of China in Shanghai, China as a Researcher/Senior
Researcher/Principal Researcher. Currently, he is a senior research scientist
at Data61, CSIRO, in Sydney, NSW, Australia. His research interests include
information technology, data privacy and security, machine learning and AI,
etc. He has authored over 140 papers in IEEE journals and conferences, all in
recognized venues, and around 20 3GPP standardization contributions, as well
as a Springer book “Multi-point Cooperative Communication Systems: Theory and
Applications”. Also, he holds 21 US patents and co-invented another 100+
patents on 4G/5G technologies in CN, JP, KR, EU, etc. Currently, he is an
editor of IEEE Transactions on Wireless Communications and IEEE Wireless
Communications Letters. Besides, he has served as Guest Editor/Co-Chair/Co-
Tutor/TPC member for many IEEE top-tier journals/conferences and received
several awards for his research work and professional services.
---|---
|  Pubudu N. Pathirana (Senior Member, IEEE) was born in 1970 in Matara, Sri
Lanka, and was educated at Royal College Colombo. He received the B.E. degree
(first class honors) in electrical engineering and the B.Sc. degree in
mathematics in 1996, and the Ph.D. degree in electrical engineering in 2000
from the University of Western Australia, all sponsored by the government of
Australia on EMSS and IPRS scholarships, respectively. He was a Postdoctoral
Research Fellow at Oxford University, Oxford, a Research Fellow at the School
of Electrical Engineering and Telecommunications, University of New South
Wales, Sydney, Australia, and a Consultant to the Defence Science and
Technology Organization (DSTO), Australia, in 2002. He was a visiting
professor at Yale University in 2009. Currently, he is a full Professor and
the Head of Discipline, Mechatronics, Electrical and Electronic Engineering
and the Director of Network Sensing and Biomedical Engineering(NSBE) research
group at the School of Engineering, Deakin University, Geelong, Australia. His
current research interests include Bio-Medical assistive device design, human
motion capture, mobile/wireless and IoT networks, rehabilitation robotics and
signal processing.
---|---
|  Aruna Seneviratne (Senior Member, IEEE) is currently a Foundation Professor
of telecommunications with the University of New South Wales, Australia, where
he holds the Mahanakorn Chair of telecommunications. He has also worked at a
number of other Universities in Australia, U.K., and France, and industrial
organizations, including Muirhead, Standard Telecommunication Labs, Avaya
Labs, and Telecom Australia (Telstra). In addition, he has held visiting
appointments at INRIA, France. His current research interests are in physical
analytics: technologies that enable applications to interact intelligently and
securely with their environment in real time. Most recently, his team has been
working on using these technologies in behavioral biometrics, optimizing the
performance of wearables, and the IoT system verification. He has been awarded
a number of fellowships, including one at British Telecom and one at Telecom
Australia Research Labs.
---|---
|  Jun Li (M’09-SM’16) received Ph. D degree in Electronic Engineering from
Shanghai Jiao Tong University, Shanghai, P. R. China in 2009. From January
2009 to June 2009, he worked in the Department of Research and Innovation,
Alcatel Lucent Shanghai Bell as a Research Scientist. From June 2009 to April
2012, he was a Postdoctoral Fellow at the School of Electrical Engineering and
Telecommunications, the University of New South Wales, Australia. From April
2012 to June 2015, he is a Research Fellow at the School of Electrical
Engineering, the University of Sydney, Australia. From June 2015 to now, he is
a Professor at the School of Electronic and Optical Engineering, Nanjing
University of Science and Technology, Nanjing, China. He was a visiting
professor at Princeton University from 2018 to 2019. His research interests
include network information theory, game theory, distributed intelligence,
multiple agent reinforcement learning, and their applications in ultra-dense
wireless networks, mobile edge computing, network privacy and security, and
industrial Internet of things. He has co-authored more than 200 papers in IEEE
journals and conferences, and holds 1 US patents and more than 10 Chinese
patents in these areas. He was serving as an editor of IEEE Communication
Letters and TPC member for several flagship IEEE conferences. He received
Exemplary Reviewer of IEEE Transactions on Communications in 2018, and best
paper award from IEEE International Conference on 5G for Future Wireless
Networks in 2017.
---|---
|  H. Vincent Poor (S’72, M’77, SM’82, F’87) received the Ph.D. degree in EECS
from Princeton University in 1977. From 1977 until 1990, he was on the faculty
of the University of Illinois at Urbana-Champaign. Since 1990 he has been on
the faculty at Princeton, where he is currently the Michael Henry Strater
University Professor. During 2006 to 2016, he served as the dean of
Princeton’s School of Engineering and Applied Science. He has also held
visiting appointments at several other universities, including most recently
at Berkeley and Cambridge. His research interests are in the areas of
information theory, machine learning and network science, and their
applications in wireless networks, energy systems and related fields. Among
his publications in these areas is the forthcoming book Machine Learning and
Wireless Communications, (Cambridge University Press). Dr. Poor is a member of
the National Academy of Engineering and the National Academy of Sciences and
is a foreign member of the Chinese Academy of Sciences, the Royal Society, and
other national and international academies. He received the IEEE Alexander
Graham Bell Medal in 2017.
---|---