#  Play to Earn in Augmented Reality with Mobile Edge Computing over Wireless
Networks: A Deep Reinforcement Learning Approach

Terence Jie Chua, Wenhan Yu, and Jun Zhao The authors are all with Nanyang
Technological University, Singapore. Email<EMAIL_ADDRESS>A 7-page short
version containing partial results is accepted for the 2023 EAI GameNets
($12^{th}$ EAI International Conference on Game Theory and Advanced
Communication Networks [1]). IEEE Communications Society (ComSoc) writes “It
is possible to submit the journal and conference version at the same time” at
https://www.comsoc.org/publications/journals/ieee-transactions-wireless-
communications/conference-vs-journal

###### Abstract

Play-to-earn (P2E) games have been gaining popularity as they enable players
to earn in-game tokens which can be translated to real-world profits. With the
advancements in augmented reality (AR) technologies, AR play-to-earn games
become compute-intensive. In-game graphical scenes need to be offloaded from
mobile devices to an edge server for computation. In this work, we consider an
optimization problem where the Mobile edge computing Service Provider (MSP)’s
objective is to reduce downlink transmission latency of in-game graphics, the
latency of uplink data transmission, and the worst-case (greatest) battery
charge expenditure of user equipments (UEs), while maximizing the worst-case
(lowest) UE resolution-influenced in-game earning potential through optimizing
the downlink UE-Mobile edge computing Base Station (UE-MBS) assignment and the
uplink transmission power selection. The downlink and uplink transmissions are
executed asynchronously. We propose a multi-agent, loss-sharing (MALS)
reinforcement learning model to tackle the asynchronous and asymmetric
problem. We then compare the MALS model with other baseline models and show
its superiority over other methods. Finally, we conduct multi-variable
optimization weighting analyses and show the viability of using our proposed
MALS algorithm to tackle joint optimization problems.

###### Index Terms:

Play-to-earn, mobile edge computing, deep reinforcement learning, resource
allocation.

##  I Introduction

Background.  Play-to-earn (P2E) games are a new breed of video games that
allow players to earn virtual-world rewards or cryptocurrency by participating
in gameplay. By combining entertainment with economic incentives, these games
offer players the opportunity to turn their leisure time into a profitable
endeavor. While P2E games are not restricted to Metaverse virtual worlds, the
concept of play-to-earn has gained popularity within the Metaverse and
blockchain gaming communities. Traditionally, players have little ownership of
their in-game earnings, tokens, and characters as they belong to the game
developers. However, blockchain contracts now allow players to own their in-
game tokens and possessions, granting them real-world value [2]. Players can
profit by playing these games, earning in-game tokens, acquiring possessions,
and trading them for real-world currencies [2]. In this paper, we introduce
the concept of “earning potential”, which represents the maximum earnings a
player can achieve based on factors such as in-game graphics resolution. The
earning potential is a key part of our problem formulation.

###  I-A Problem

Compute-intensive Mobile Augmented Reality (MAR).  Compute-intensive play-to-
earn games, like Polkacity [3] and Reality Clash [4], utilize high-resolution
graphics and complex interactions, driven by the increasing adoption of Mobile
Augmented Reality (MAR) [5]. However, current mobile devices struggle to meet
the demands of augmented reality (AR) applications, affecting gameplay
smoothness in AR P2E games. Offloading game computation to edge computing
servers is a potential solution. In the context of play-to-earn games,
inefficient edge computing allocation and selection can lead to poor graphics,
high latency, and signal interference, negatively impacting earning potential,
gameplay smoothness, and play duration.

Formulated Problem. Developing an optimal strategy for maximizing P2E
profitability, minimizing device energy consumption, and reducing transmission
latency in the P2E games is complex. Key considerations include player
profitability, in-game smoothness, and device battery levels. We formulate the
problem in a way where the Mobile edge computing Service Provider (MSP) aims
to minimize the (i) downlink transmission latency of graphical data, (ii)
uplink transmission latency, (iii) and worst-case (greatest) UE battery charge
expenditure, while maximizing (iv) the players’ worst-case (lowest) in-game
earning potential influenced by graphic resolution, by optimizing the UE-MBS
allocation, and UE uplink transmit power. Our goal is to enhance player
retention by improving in-game fluidity, experience, profitability, and
battery life for extended play. We prioritize maximizing the worst-case
earning potential and minimizing the worst-case battery charge expenditure
across players, ensuring that no player’s interests are neglected.

###  I-B Our Approach and Rationale

To address these challenges, we propose a novel Multi-Agent Loss-Sharing
(MALS) reinforcement learning (RL)-based orchestrator. It optimizes UE-MBS
allocation and uplink transmission power to maximize the utility function of
the MSP. Our approach involves a discrete-action space RL agent for the
downlink (DL) UE-MBS allocation and a continuous-action space RL agent for UE
uplink (UL) power selection. In the process of RL reward assignment, we break
down the overarching objective function into parts so that the agents are
assigned objectives within their control, avoiding confusion and providing
clear goals. The DL and UL agents are asymmetric, with separate objectives and
action space types. They execute asynchronously, with the DL agent performing
UE-MBS allocation, followed by the UL agent’s power selection. Although these
processes seem distinct and asynchronous, optimizing both stages concurrently
is crucial.

Reinforcement learning approach over convex optimization.  We proposed a
complex play-to-earn mobile edge computing problem, which is non-convex due to
the mixed-integer programming of UE-MBS allocation and uplink power
transmission. Additionally, considering cumulative in-game profitability and
UE battery charge expenditure adds sequential complexity, deterring solutions
to be found at each transmission iteration.

Play-to-earn under mobility. The existence of P2E games on mobile devices
would mean that players are expected to be on the move. As players move, their
distance from the MBSs changes. This changing distance results in varying
channel gain and hence effective rate of data transfer. In our proposed play-
to-earn framework, we have to take into account players’ (UEs’) mobility.

###  I-C Existing Literature and Our Contributions

In this subsection, we first present relevant literature and then highlight
the contributions of this paper. We divide related studies into the following
categories: 1) AR over wireless networks, 2) deep reinforcement learning for
task offloading, 3) deep reinforcement learning tackling MINLP problems, and
4) multi-agent deep reinforcement learning.

AR over wireless networks. Numerous studies have investigated resource
management and optimization challenges related to virtual, augmented, or
extended reality over wireless networks, as evidenced by various works [6, 7,
8, 9]. Chen et al. [6] focus on addressing a resource allocation problem and
propose a distributed machine learning algorithm to tackle it. In a different
vein, Wang et al. [7] examine the optimization of VLC access points (VAP) and
user-base station association to enhance virtual reality experiences in indoor
wireless network environments.Liu et al. [8] propose an MAR over wireless
network scenario and utilize convex optimization techniques to solve an object
detection for MAR and latency of computation offloading problem. On the other
hand, Wang et al. [9] focus on reducing the energy consumption of users
utilizing mobile augmented reality systems. Their objective is achieved
through the optimization of both the configuration of the MAR (Mobile
Augmented Reality) setup and the allocation of radio resources.

Deep reinforcement learning for task offloading. There are several works
utilizing deep reinforcement learning to optimize task offloading [10, 11].
[10] introduces a novel approach using deep reinforcement learning (DRL) to
address the offloading problem in large-scale heterogeneous mobile edge
computing (MEC), incorporating LSTM network layer and candidate network set to
enhance the DQN algorithm based on the MEC environment. [11] propose a RL-
SARSA algorithm which utilizes reinforcement learning to address the resource
management problem in the edge server, enabling optimal offloading decisions
that minimize system cost, including energy consumption and computing time
delay.

Deep RL for MINLP. In addition to the works above, there are several existing
works [12, 13, 14, 15] that adopt deep RL approaches to tackle mixed-integer
non-linear programming problems, as these problems are non-convex and may not
yield good solutions with convex optimization approaches. [12] present a
cooperative edge caching scheme designed to enhance content placement and
delivery within vehicular edge computing and networks. Their approach involves
trilateral collaboration among a macro-cell station, roadside units, and smart
vehicles. To address the joint optimization problem, the researchers formulate
it as a double time-scale Markov decision process (DTS-MDP) and adopt a
nature-inspired method based on the deep deterministic policy gradient (DDPG)
framework. Through this framework, they obtain a computationally efficient
suboptimal solution. [13] address the joint binary offloading and system
resource allocation problem by formulating it as a multi-stage stochastic
MINLP problem. To overcome the coupling between decisions in different time
frames, they propose LyDROO, a novel framework that integrates Lyapunov
optimization and deep reinforcement learning (DRL) for effective solution.
[14] introduce the ACDQN algorithm, a deep reinforcement learning approach, to
jointly optimize computation offloading policy and channel resource allocation
in a NOMA-assisted MEC network under dynamic network conditions. The algorithm
effectively reduces computational overhead by considering partial computation
offloading and a hybrid multiple access scheme that combines NOMA and
orthogonal multiple access to cater to different user needs. [15] introduce a
novel approach using model-free deep reinforcement learning for online
computation offloading in blockchain-enabled mobile edge computing. Their
method takes into account both mining tasks and data processing tasks. By
formulating the problem as a Markov decision process, the researchers utilize
deep reinforcement learning in conjunction with an adaptive genetic algorithm.
Their objective is to maximize long-term offloading performance while
effectively handling dynamic environments and reducing computational
complexity. Nevertheless, these works do not consider a multi-agent
transmission scenario.

Multi-Agent Reinforcement Learning. In contrast to these abovementioned works,
there are several works which consider the concurrent optimization of several
variables within their objective function and propose multi-agent
reinforcement learning approaches to handle them. Guo et al. [16] utilize a
Centralized Training and Decentralized Execution (CTDE) framework to tackle
handover and power selection. Similarly, He et al. [17] adopt the CTDE [18]
framework to address the user to channel allocation and transmission power
selection problem. In our work, we utilize multiple agents, but traditional
MARL algorithms (e.g., [18, 19, 16, 17]) are not suitable for our proposed
problem. These algorithms are based on centralized training and decentralized
execution (CTDE) approach, where each agent has its own Critic considering all
other agents’ actions in a time step. However, these adaptations of MARL do
not efficiently handle our problem. The simultaneous action selection and
consideration of all agent actions lead to sparse observation spaces that do
not capture sequential agent actions effectively. Additionally, our problem
involves distinct agents with discrete and continuous actions. The existing
MARL approaches are rendered impractical in tackling our proposed problem as
the UL and DL transmissions in our work are considered asynchronously and the
actions taken by our agents are asymmetric. Existing MARL approaches rely on
agents sharing all states and actions through the Critic.

Contributions. We make the following contributions to the field:

  * •

_Joint Optimization:_ We formulate a multi-variable optimization problem for
play-to-earn over mobile edge computing.

  * •

_Multi-Agent-Loss-Sharing: _We propose a novel asymmetric and asynchronous
multi-agent loss-sharing reinforcement learning-based orchestrator to tackle a
novel asynchronous and asymmetric wireless communication problem.

  * •

_Comparison of our proposed methods against baseline models:_ We compare our
(i) proposed method MALS with baseline methods such as (ii) independent dual-
agent (IDA) and (iii) CTDE [18] reinforcement learning algorithm. The
comparison shows the superiority of our proposed method in handling
asynchronous executions.

  * •

_Analyses of weighted utility objective functions:_ We provide in-depth
analyses of how the different weights on each factor in the joint optimization
functions influence reward and other variables. These analyses provide
insights into solving multi-agent joint-optimization problems.

The paper is organized as follows: Section II presents the play-to-earn in
augmented reality (AR) with mobile edge computing optimization problem, while
Section III discusses its formulation using reinforcement learning. Our novel
MALS deep reinforcement learning model is introduced in Section IV, along with
a detailed algorithm. Thorough experiments in Section V compare different
methods, highlighting the superiority of the MALS algorithm for handling
asymmetric and asynchronous problems. We also analyze the weighting of
variables in utility functions to demonstrate the feasibility across various
optimization scenarios. Finally, Section VI provides the paper’s conclusion.

Differences between this Journal and Our Conference Paper [1]. This journal
builds upon the work in [1], focusing on the worst-case earning potential in
the downlink stage and the worst-case battery charge expenditure in the uplink
stage for optimization, unlike the conference version which considered the sum
of potentials/expenditure. Additionally, a multi-variable optimization
weighting analysis is included to demonstrate the algorithm’s suitability for
joint optimization problems, and more detailed rationale and explanations.

##  II System Model

Problem Scenario.  Consider the real-time downlink (DL) and uplink (UL)
transmission of $N$ players from a set $\mathcal{N}=\\{1,2,...,N\\}$ players
(UE) moving about. In each complete transmission iteration, each UE
$i\in\mathcal{N}$ begins downloading P2E in-game scenes from an MBS. We
consider interference based on a Non-Orthogonal Multiple Access (NOMA) system
in both the DL and UL transmission. After the DL of in-game graphical data, we
consider the UL transmission of the same set of $N$ players’ (UE) changes to
in-game graphical scenes, to its allocated MBS $(\mathcal{M}=\\{1,2,...,M\\}$.
Each UE $i\in\mathcal{N}$ will upload their in-game graphical scenes to their
allocated MBS. We use iterations to refer to a complete round of data
transmissions. Each iteration includes a downlink and an uplink data
transmission. To simplify an already complex scenario, we consider the
scenario in which users only make movement after each iteration of downlink
and uplink data transmissions.

Figure 1: System model illustrating the interaction between the UEs and the
MBSs, agents’ objectives and variables-of-control.

###  II-A Downlink Communication

Each of the MBS from $\mathcal{M}=\\{1,2,...,M\\}$ will be allocated to some
UEs from $\mathcal{N}=\\{1,2,...,N\\}$. We define the size of the in-game
graphics to be downloaded from the MBSs to the UEs as
$D^{t}=\\{D^{t}_{1},D^{t}_{2},...,D^{t}_{N}\\}$, where $D^{t}_{i}$ represents
the in-game data size to be downloaded from an MBS to UE $i\in\mathcal{N}$ at
iteration $t$. In addition, the UE-MBS association is defined as:
$\boldsymbol{c}^{t}=(c_{1}^{t},...,c_{N}^{t})$. Each of the $c_{i}^{t}$ where
$i\in\mathcal{N}$ is allocated to an MBS $v$ where $v\in\mathcal{M}$ at
iteration $t$. We adopt the Non-Orthogonal Multiple Access (NOMA) system as
this work’s propagation model since it allows multiple users to share the same
frequency band [20]. In a NOMA multiple access channel scheme, users (UEs) can
download in-game graphics from MBSs efficiently and concurrently. When a UE
initiates a request for downloading in-game graphics, the allocated MBS
employs NOMA principles to allocate power levels and transmit the data to
multiple users simultaneously. We define each MBS to be on a single distinct
channel. Each MBS determines the power allocation based on the channel
conditions of each of its assigned UE. Each MBS then superimposes the encoded
graphics data for all its assigned UEs into a composite signal, incorporating
the power allocation scheme. The composite signal is transmitted over the
wireless channel, and each UE receives the signal along with interference from
other UEs assigned to the same MBS. Through successive interference
cancellation (SIC), each UE performs interference removal by canceling the
contributions of other UEs’ signals based on the power allocation and channel
conditions. Each UE decodes their own intended graphics data while mitigating
the interference from other UEs’ signals. Once decoded, the UEs can perform
necessary decompression and rendering operations to display the downloaded in-
game graphics.

The sequence of user devices (UEs) in the downlink (DL) is established by
organizing them in a descending manner, with respect to their channel-to-noise
ratios. We use $\bar{d}_{1},\bar{d}_{2},\ldots,\bar{d}_{N}$ to denote the
ordering, as referenced in [21]:

$\displaystyle\frac{|g^{t}_{\bar{d}_{1},v}|^{2}}{(\sigma_{\bar{d}_{1},v}^{t})^{2}}\geq\frac{|g^{t}_{\bar{d}_{2},v}|^{2}}{(\sigma_{\bar{d}_{2},v}^{t})^{2}}\geq\ldots\geq\frac{|g^{t}_{\bar{d}_{N},v}|^{2}}{(\sigma_{\bar{d}_{N},v}^{t})^{2}}.$
(1)

Consequently, the downlink rate $r_{i}^{\text{d},t}$ for UE $i$ is given by:

$\displaystyle
r_{i}^{\text{d},t}(\boldsymbol{c}^{t})=B_{v}\log_{2}\left(1+\frac{p^{\text{d},t}_{c_{i}^{t},i}|g_{i,v}^{t}|^{2}}{\sum\limits_{j=1}^{\iota^{\prime}-1}p^{\text{d},t}_{c_{i}^{t},\bar{d_{j}}}|g_{i,v}^{t}|^{2}+B_{v}(\sigma_{i,v}^{t})^{2}}\right),$
(2)

where $\iota^{\prime}$ fulfils $c^{t}_{\iota^{\prime}}=i$, indicating that UE
$i$ holds the $\iota^{\prime}$th position among the UEs assigned to MBS $v$ in
terms of the magnitude of channel-to-noise ratio. $(\sigma_{i,v}^{t})^{2}$
represents the power spectral density of the Additive White Gaussian Noise
experienced by UE $i$ over the channel by MBS $v$ at time step $t$. $B_{v}$
corresponds to the bandwidth of MBS $v$. $p^{\text{d},t}_{c_{i}^{t},i}$
represents the downlink transmission power provided by MBS $c_{i}^{t}$ to
enable the transmission of in-game graphical scenes to UE $i$ during
transmission step $t$ (the superscript ”d” denotes downlink). $g_{i,v}^{t}$
denotes the channel gain between UE $i$ and MBS $v$ at transmission step $t$.

A higher downlink data transmission rate will reduce the downlink data
transmission latency for a given in-game data size to be downloaded. The
downlink transmission delay can be defined as:

$\displaystyle\ell^{\text{d},t}_{i}=\frac{D^{t}_{i}}{r^{\text{d},t}_{i}}$ (3)

where $D^{t}_{i}$ is the size of in-game data to be downloaded from an MBS to
UE $i$ at transmission step $t$ and the resolution-impacted earning potential
function $\omega(r^{\text{d},t}_{i})$ is influenced by the UE $i$’s signal-to-
noise-ratio (SINR). Intuitively, a more efficient UE to MBS allocation results
in a higher UE $i$ SINR, UE $i$ downlink transmission rate
$r^{\text{d},t}_{i}$ (data transmitted per unit of time), and lower downlink
transmission delay $\ell^{\text{d},t}_{i}$, at iteration $t$. A higher data
transmitted per unit of time is translated to improved graphic resolution per
unit of time, resulting in an enhancement in players’ in-game visuals, which
translates to better in-game performance and hence, better resolution-
influenced earning potential.

###  II-B Uplink Communication

In the uplink leg of the communication model, each player (UE) will upload its
graphical changes to the previously downloaded P2E in-game scene data, to its
assigned MBS. In the context of Play-to-Earn games, these in-game graphical
changes could be induced by UEs’ physical-world actions. These graphical
changes to the scenes are denoted
$F^{t}=\\{F^{t}_{1},F^{t}_{2},...,F^{t}_{N}\\}$.

In the uplink (UL) NOMA system, the players (UEs) assigned to a MBS are
ordered in descending order based on their received signals. The resulting
indices of the UEs are denoted as
$\bar{u}_{1},\bar{u}_{2},\ldots,\bar{u}_{N_{v}^{t}}$. According to this
arrangement, the power levels and channel gains of the UEs satisfy the
following inequality [20]:

$\displaystyle p^{\text{u},t}_{\bar{u}_{1}}|g_{\bar{u}_{1},v}^{t}|^{2}\geq
p^{\text{u},t}_{\bar{u}_{2}}|g_{\bar{u}_{2},v}^{t}|^{2}\geq\ldots\geq
p^{\text{u},t}_{\bar{u}_{N_{v}^{t}}}|g_{\bar{u}_{N_{v}^{t}},v}^{t}|^{2},$ (4)

where $g_{i,v}^{t}$ represents the channel gain between UE $i$ and MBS $v$ at
iteration $t$. This study assumes equivalent UL and DL channel conditions.
Based on the findings presented in [21], the uplink rate $r_{i}^{\text{u},t}$
for UE $i$ is defined as follows:

$\displaystyle
r_{i}^{\text{u},t}(\boldsymbol{c}^{t},\boldsymbol{p}^{\text{u},t})=B_{v}\log_{2}\left(1+\frac{p^{\text{u},t}_{c_{i}^{t},i}|g_{i,v}^{t}|^{2}}{\sum_{j=\eta+1}^{N}p^{\text{u},t}_{c_{i}^{t},\bar{u}_{j}}|g_{\bar{u}_{j},v}^{t}|^{2}+B_{v}(\nu_{v}^{t})^{2}}\right),$
(5)

where $\eta$ is determined such that $\bar{u}_{\eta}=i$, indicating that UE
$i$ is in the $\eta$-th position among the set of UEs, $\mathcal{N}_{v}^{t}$,
ranked in terms of the magnitude of channel-to-noise ratio. In Equation 5,
$(\nu_{v}^{t})^{2}$ is the power spectral density of the Additive White
Gaussian Noise experienced at MBS $v$. The term $p^{\text{u},t}_{c_{i}^{t},i}$
(with the superscript ”u” denoting uplink) corresponds to the transmission
power of UE $i$ for facilitating the uplink transmission of graphical changes
in-game data during iteration $t$.

The uplink transmission latency of UE $i$ at transmission step $t$ can be
formulated as:

$\displaystyle\ell^{\text{u},t}_{i}=\frac{F^{t}_{i}}{r^{\text{u},t}_{i}}$ (6)

where $F^{t}_{i}$ is the uplink in-game graphics data size of UE $i$ at
transmission step $t$, $r^{\text{u},t}_{i}$ is the uplink data transmission
rate of UE $i$ at transmission step $t$. UE $i$’s energy expenditure for the
uplink transmission of in-game graphics data at transmission step $t$ can be
formulated as:

$\displaystyle
E^{t}_{i}=p^{\text{u},t}_{c_{i}^{t},i}\cdot\ell^{\text{u},t}_{i}.$ (7)

We define $Q^{t}_{i}$ to be the UE $i$ energy consumption as a percentage of
the UE $i$’s initial battery amount, at transmission step $t$. $Q^{t}_{i}$ can
be defined as: $Q^{t}_{i}=\frac{E^{t}_{i}}{Bat^{0}_{i}}\cdot 100$. In this
context, $Bat^{0}_{i}$ represents the UE $i$’s initial battery amount while
$E^{t}_{i}$ represents UE $i$’s energy consumption, at transmission step $t$.
Utilizing a UE energy consumption as a percentage of the UE’s initial battery
charge is a natural choice. This is because normalized values are more
conducive for training reinforcement learning agents. The control of uplink
transmission power influences the uplink transmission latency, which
influences fluidity. On the other hand, the selection of uplink transmission
power also influences UE battery expenditure.

###  II-C UE Mobility Model

We consider the network area to be a two-dimensional coordinate system,
$x$-dimension and $y$-dimension, where the UEs and MBSs are positioned. At
each given iteration step $t$, UE $i$ is located at
$(\mathcal{X}^{t}_{i},\mathcal{Y}^{t}_{i})$. The locations of UEs are
restricted to an area, $\mathcal{Y}^{t}_{i}\in[0,Y_{max}]$ and
$\mathcal{X}^{t}_{i}\in[0,X_{max}]$. $\mathcal{X}^{t}_{i}$ refers to UE $i$’s
position in the $x$-dimension at transmission step $t$, while
$\mathcal{Y}^{t}_{i}$ refers to UE $i$’s position in the $y$-dimension at
transmission step $t$. $X_{max}$ is the defined boundary in the $x$-dimension
while $Y_{max}$ is the defined boundary in the $y$-dimension. After each round
of data transmission (downlink followed by uplink), UEs will take steps $x$
and $y$ in both $x$-dimension and $y$-dimension, with the step size uniformly
sampled from $(-x_{max},x_{max})$ and $(-y_{max},y_{max})$, respectively.
$x_{max}$ refers to the maximum allowed distance a UE can move in a single
transmission iteration in the $x$-dimension. $y_{max}$ refers to the maximum
allowed distance a UE can move in a single transmission iteration in the
$y$-dimension. The $x$-dimension position $\mathcal{X}^{t}_{i}$ can be defined
as:

$\displaystyle\mathcal{X}^{t}_{i}=\begin{cases}0,&\text{if
}\mathcal{X}^{t-1}_{i}+x\leq 0,\\\ X_{max},&\text{if
}\mathcal{X}^{t-1}_{i}+x\geq X_{max},\\\
\mathcal{X}^{t-1}_{i}+x,&\text{otherwise,}\end{cases}$ (8)

Likewise, the $y$-dimension position $\mathcal{Y}^{t}_{i}$ can be defined as:

$\displaystyle\mathcal{Y}^{t}_{i}=\begin{cases}0,&\text{if
}\mathcal{Y}^{t-1}_{i}+y\leq 0,\\\ Y_{max},&\text{if
}\mathcal{Y}^{t-1}_{i}+y\geq Y_{max},\\\
\mathcal{Y}^{t-1}_{i}+y,&\text{otherwise,}\end{cases}$ (9)

###  II-D Problem formulation

To sum up, the MSP’s DL objective is to find the optimal UE-MBS allocation
$\boldsymbol{c}^{t}$ which minimizes the total downlink latency
$\ell^{\text{d},t}_{i}$ while maximizing the worst-case (lowest) resolution-
influenced UE earning potential
$\min_{i\in\mathcal{N}}\sum_{t=1}^{T}\omega(r^{\text{d},t}_{i})$ across all
UEs. The number of transmission steps in an episode is denoted by $T$. We
formulate our downlink utility function as:

$\displaystyle\min_{\boldsymbol{c}^{t}}$
$\displaystyle\left[q\cdot\left(\frac{1}{N}\sum_{i\in\mathcal{N}}\sum_{t=1}^{T}\ell^{\text{d},t}_{i}\right)-(1-q)\cdot
b\cdot\min_{i\in\mathcal{N}}\sum_{t=1}^{T}\omega(r^{\text{d},t}_{i})\right],$
(10) $\displaystyle s.t.~{}~{}$ $\displaystyle c_{i}^{t}\in\mathcal{M},\forall
t\in\mathcal{T},~{}\forall i\in\mathcal{N},$ (11) $\displaystyle
p^{\text{u}}_{min}\leq p^{\text{u},t}_{i}\leq p^{\text{u}}_{max},\forall
t\in\mathcal{T},~{}\forall i\in\mathcal{N}.$ (12)

where $\mathcal{T}$ represents the set of all transmission steps in an
episode. Our resolution-impacted in-game earning potential is motivated by a
recent work [22] and is modeled as:
$\omega(r^{\text{d},t}_{i})=P\cdot\ln\left(1+{r^{\text{d},t}_{i}}\right)$,
where $P$ is the player-specific profitability factor (using such logarithmic
function as the utility also appears in the classical work on crowdsourcing
[23]). The logarithmic function has a increasing first-order derivative and
decreasing second-order derivative property. When applied to the P2E profit
model, the increasing first-order derivative implies that UE in-game profits
increases as the downlink transmission rate increases, while the decreasing
second-order derivative implies that the rate of profit increment decreases as
the downlink transmission rate increases. It is natural to model our profit
model as such as a larger downlink data (resolution) per unit time improves
UEs’ visibility within the games, allowing them to perform better and reap
higher profits. However, as the resolution improves, its impact on UE’s
visibility and hence earnings, decreases, resulting in a slowing rate of
profit increment with increasing downlink transmission rate. The constraint
(11) ensures that each UE is assigned to one MBS in each transmission step.
$q$ is a factor which determines the weighting on
$\sum_{i\in\mathcal{N}}\sum_{t=1}^{T}\ell^{\text{d},t}_{i}$ and
$b\cdot\min_{i\in\mathcal{N}}\sum_{t=1}^{T}\omega(r^{\text{d},t}_{i})$. $b$ is
a factor that places both terms
$\sum_{i\in\mathcal{N}}\sum_{t=1}^{T}\ell^{\text{d},t}_{i}$ and
$b\cdot\min_{i\in\mathcal{N}}\sum_{t=1}^{T}\omega(r^{\text{d},t}_{i})$ on the
same measurement unit. Constraint (12) ensures that the uplink transmission
power by UE falls between the feasible range of output powers.

On the other hand, MSP’s UL objective is to obtain the optimal UE uplink power
$p^{\text{u},t}$ that minimizes the total latency of the data uplink
transmissions $\ell^{\text{u},t}_{i}$ and worst-case (greatest) UE device
energy expenditure $Q^{t}_{i}$. We formulate our uplink utility function as:

$\displaystyle\min_{\boldsymbol{p^{t}}}$
$\displaystyle\left[h\cdot\left(\frac{1}{N}\sum_{i\in\mathcal{N}}\sum_{t=1}^{T}\ell^{\text{u},t}_{i}\right)+(1-h)\cdot
f\cdot\max_{i\in\mathcal{N}}\sum_{t=1}^{T}Q^{t}_{i}\right],$ (13)
$\displaystyle s.t.~{}~{}$ $\displaystyle\sum_{t=1}^{T}Q^{t}_{i}\leq
100,\forall i\in\mathcal{N},$ (14) $\displaystyle p^{\text{d}}_{min}\leq
p^{\text{d},t}_{i}\leq p^{\text{d}}_{max},\forall t\in\mathcal{T},~{}\forall
i\in\mathcal{N}.$ (15)

where $\ell^{\text{u},t}_{i}$ is the player (UE) $i$’s uplink data
transmission delay at transmission iteration $t$. $h$ is a factor that
determines the weighting on
$\sum_{i\in\mathcal{N}}\sum_{t=1}^{T}\ell^{\text{u},t}_{i}$ and
$f\cdot\max_{i\in\mathcal{N}}\sum_{t=1}^{T}Q^{t}_{i}$. Scaling factor $f$ in
the objective function is to place both terms
$\sum_{i\in\mathcal{N}}\sum_{t=1}^{T}\ell^{\text{u},t}_{i}$ and
$f\cdot\max_{i\in\mathcal{N}}\sum_{t=1}^{T}Q^{t}_{i}$ on the same measurement
unit. Constraint (14) ensures that the UE battery consumption has to be less
than 100% (or battery charge (%) has to be greater than 0) for the uplink
transmission. Finally, constraint (15) ensures that the MBS downlink power
output falls within a feasible range.

To sum up, we aim to minimize the UL and DL transmission latency, worst-case
P2E earning potential, worst-case UE battery charge expenditure. We write our
overarching (overall) objective as:

$\displaystyle\min\limits_{\boldsymbol{c}^{t},\boldsymbol{p}^{t}}\Bigg{\\{}w_{1}\left[q\cdot\left(\frac{1}{N}\sum_{i\in\mathcal{N}}\sum_{t=1}^{T}\ell^{\text{d},t}_{i}\right)-(1-q)\cdot
b\cdot\min_{i\in\mathcal{N}}\sum_{t=1}^{T}\omega(r^{\text{d},t}_{i})\right]+$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}w_{2}\left[h\cdot\left(\frac{1}{N}\sum_{i\in\mathcal{N}}\sum_{t=1}^{T}\ell^{\text{u},t}_{i}\right)+(1-h)\cdot
f\cdot\max_{i\in\mathcal{N}}\sum_{t=1}^{T}Q^{t}_{i}\right]\Bigg{\\}}$ (16a)
$\displaystyle s.t.~{}\text{C1}:\sum_{t=1}^{T}Q^{t}_{i}\leq 100,\forall
i\in\mathcal{N},$ (16b)
$\displaystyle~{}~{}~{}~{}~{}\text{C2}:c_{i}^{t}\in\mathcal{M},\forall
t\in\mathcal{T},~{}\forall i\in\mathcal{N},$ (16c)
$\displaystyle~{}~{}~{}~{}~{}\text{C3}:p^{\text{d}}_{min}\leq
p^{\text{d},t}_{i}\leq p^{\text{d}}_{max},\forall t\in\mathcal{T},~{}\forall
i\in\mathcal{N},$ (16d)
$\displaystyle~{}~{}~{}~{}~{}\text{C4}:p^{\text{u}}_{min}\leq
p^{\text{u},t}_{i}\leq p^{\text{u}}_{max},\forall t\in\mathcal{T},~{}\forall
i\in\mathcal{N}.$ (16e)

##  III Reinforcement learning settings

We assign two reinforcement learning agents, one (DL agent) controlling the
downlink transmission UE-MBS allocation, and the other (UL agent) controlling
the uplink transmission power selection. The DL agent and UL agent serve the
overarching objective (16a). These reinforcement learning agents will be
deployed as an overarching cloud orchestrator to manage resource requirements
for the mobile edge computing (MEC) between MBSs and UEs. The DL and UL RL
agents are first trained offline using simulations or in a controlled
environment. The trained RL agents are then integrated into the MSP. This
involves implementing the RL agents as software that can run on the server
infrastructure. In the downlink transmission, the MBS sends channel gain
information between UEs and MBS, as well as in-game scene data size, to the
MSP cloud server. The DL agent at the cloud server makes decisions based on
this information and communicates the UE-MBS allocation to the MBSs.
Similarly, in the uplink transmission, the UEs transmit information about
their battery life to the MSP cloud. The UL agent at the cloud server makes
decisions based on this information and communicates the UE uplink power
selection to the UEs. These information transmissions occur through dedicated
channels.

###  III-A State

For the DL agent’s observation state $s^{\text{d}}$, we choose to include 1)
channel gain between UEs and MBSs: $g^{t}_{v,i}$, 2) in-game scene DL data
size at each transmission iteration $t$: $D^{t}_{i}$, as $D^{t}_{i}$
influences latency $\ell^{\text{d},t}_{i}$, and $g^{t}_{v,i}$ influences both
data DL transmission latency $\ell^{\text{d},t}_{i}$ and worst-case (lowest)
resolution-influenced in-game earning potential
$\min_{i\in\mathcal{N}}\sum_{t=1}^{T}\omega(r^{\text{d},t}_{i})$.

For the UL agent’s observation state $s^{\text{u}}$, we choose to include 1)
channel gain between UEs and MBSs: $g^{t}_{v,i}$, as it influences data UL
transmission rate, latency $\ell^{\text{u},t}_{i}$ and uplink transmission
battery consumption $Q^{t}_{i}$. As we consider the cumulative battery amount
consumed by UEs and are enforcing that each UE has positive battery-charge to
continue the uplink transmission at each transmission iteration step $t$, we
have to include 2) UE battery life (%):
$\left[\frac{Bat^{t}_{i}}{Bat^{0}_{i}}\cdot 100\right]$ , as UE battery life
changes in each iteration.

###  III-B Action

In the DL communication model, the DL agent’s action involves deciding the UE
to MBS assignment. The UE to MBS allocation action at transmission step $t$ is
defined as: $a^{\text{d},t}=\textbf{c}^{t}=\\{c_{1}^{t},...c_{N}^{t}\\}.$ In
the UL communication model, the UE power for uplink data transmission at
transmission step $t$ is defined as:
$a^{\text{u},t}=\textbf{p}^{\text{u},t}=\\{p^{\text{u},t}_{1},...,p^{\text{u},t}_{N}\\}$.
$p^{\text{u},t}_{i}$ represents UE $i$’s selected power for the uplink data
transmission at transmission step $t$.

###  III-C Reward

For the uplink communication model, the reward issued to the UL agent at
transmission step $t$ is defined as:

$\displaystyle
R^{\text{u},t}=\begin{cases}-h\cdot\frac{\sum_{i\in\mathcal{N}}\ell^{\text{u},t}_{i}}{N}+\left(1-h\right)\cdot
f\cdot\min_{i\in\mathcal{N}}\sum_{\tau=1}^{t}Q^{t}_{i},&\text{if
}\sum_{\tau=1}^{t}Q^{t}_{i}\leq 100,~{}\forall i\in\mathcal{N}\\\
\varrho,&\text{otherwise.}\end{cases}$ (17)

The intuition for the downlink and uplink reward assignment follows the
utility functions in Equations (10) and (13), respectively, and the
overarching objective (16a). For the downlink communication model, the reward
given to the DL agent at transmission step $t$ is given as such:

$\displaystyle
R^{\text{d},t}=\begin{cases}-q\cdot\frac{\sum_{i\in\mathcal{N}}\ell^{\text{d},t}_{i}}{N}+\left(1-q\right)\cdot
b\cdot\min_{i\in\mathcal{N}}\sum_{\tau=1}^{t}\omega(r^{\text{d},t}_{i})+\varkappa
R^{\text{u},t},&\text{if }\sum_{\tau=1}^{t}Q^{t}_{i}\leq 100,~{}\forall
i\in\mathcal{N}\\\ \varrho,&\text{otherwise.}\end{cases}$ (18)

$\varrho$ is a very large penalty and $\tau$ denotes the summation of index.
Note that the DL and UL agents are given only portions of the overarching
objective that they control as this is to avoid “confusing” agents with
rewards that they do not control. The downlink reward comprises of a weighted
component of the uplink reward as the DL agent’s allocation decisions have
consequences on the UL transmission latency and battery consumption as the UL
transmission latency and energy consumption is controlled by the composition
of UL transmission power selection and the UE-MBS allocation. By allocating
the DL agent with a component of the UL reward, the DL agent can have a more
comprehensive view of both the DL and UL process and optimize for both the DL
and UL rewards. On the other hand, the selection of UL transmission power, and
UE-MBS allocation influences UEs’ battery life and whether further
transmissions can be executed. Hence both the DL and UL agents are also issued
a large penalty $\varrho$ should any UE have no remaining battery charge for
transmission. In such circumstance, the episode ends and no further
transmission continues. Following the logic above, in our work, we do not
present the UL agent with rewards pertaining to downlink transmission latency
as downlink transmission latency is not within the UL agent’s control. The
decomposition of rewards is inspired by [24], and is known to be an effective
method for training of reinforcement learning agents. Hence, from the above,
we note that the UL and DL agents are deeply related, and the asynchronous
execution of DL and UL transmissions must be optimized concurrently.

##  IV  Our Multi-Agent Loss-Sharing (MALS) Reinforcement Learning Algorithm

Novelty of our MALS algorithm. Given that the MSP has multiple variables to
optimize, rudimentary reinforcement learning agents face challenges in
achieving their objectives and may encounter difficulties in achieving
convergence. One reason is that independent agents act in their own interest,
choosing the best actions to maximize their own rewards, even if it is at the
expense of other agents. In a mixed-cooperative scenario as proposed in our
study where each agent acts in its interest, agents’ greedy actions may result
in sub-optimal action solutions learned across all agents. To tackle this
issue, we proposed a novel multiple-input, multiple-objective (head) critic
which provides actors with actor-centric guidance, facilitating the actors in
breaking down complex objectives into simpler ones. A (multi) dual-head critic
refers to a critic model with (multiple) two distinct neural network input
layers, each to accommodate to the input state of each agent, while sharing a
common neural network backbone. The critic head serving the uplink actor
beacons the UL actor in (i) minimizing the UL transmission delay and worst-
case UE battery consumption in the uplink transmissions. The critic head
serving the downlink actor beacons the DL actor in (ii) minimizing the DL
transmission delay, maximizing the UE resolution-influenced in-game earning
potential, minimizing the UL transmission delay and worst-case UE battery
consumption in the uplink transmissions. To ensure that each agent involved
learns a jointly optimal solution, the loss value of each critic’s head is
shared (weighted sum) for the joint update of the critic model. We refer to
the above approach as Multi-Agent Loss-Sharing (MALS).

In the following section, we will introduce the PPO algorithm, its mechanism
as it is the backbone algorithm in which our MALS algorithm is built upon.

Figure 2: Reinforcement Learning Multi-Agent-Loss-Sharing Proximal Policy
Optimization (MALS-PPO) structure and model update.

###  IV-A Proximal Policy Optimization (PPO)

Our focus lies in the development of a multi-agent RL model that addresses
tasks with asymmetrical characteristics in an asynchronous manner. In this
context, ensuring policy stability becomes crucial. OpenAI introduced the
Proximal Policy Optimization (PPO) algorithm [25] as an improvement over the
traditional policy gradient approach. PPO achieves better sample efficiency by
utilizing separate policies for trajectory sampling and optimization, while
also incorporating a policy constraint to enhance stability.

To summarize, PPO exhibits two key features within its policy network (Actor):
(1) Enhanced sample efficiency: PPO employs two distinct policies, namely the
evaluation policy denoted as $\pi_{\theta}$ and the data-sampling policy
denoted as $\pi_{\theta^{{}^{\prime}}}$, to improve the efficiency of sampling
trajectories and optimization. The evaluation policy is controlled by the
parameter $\theta$, while the data-sampling policy is controlled by
$\theta_{{}^{\prime}}$.

(2) Policy constraint. After switching the data sampling policy from
$\pi_{\theta}$ to $\pi_{\theta^{{}^{\prime}}}$, an issue still remains.
Although the expected value of the evaluation and data sampling policy is
similar, their variances are starkly distinct. A policy constraint is
implemented in the form of threshold clipping and the state-action pair
expectation is defined as:
$\mathbb{E}_{(s^{t},a^{t})\sim\pi_{\theta^{{}^{\prime}}}}[\min\\{\frac{\pi_{\theta}(a^{t}|s^{t})}{\pi_{{\theta^{\prime}}}(a^{t}|s^{t})},\operatorname{clip}(\frac{\pi_{\theta}(a^{t}|s^{t})}{\pi_{{\theta^{\prime}}}(a^{t}|s^{t})},1-\epsilon,1+\epsilon)\\}\cdot
A^{t}]$, where $A^{t}$ is the advantage value. Gradient ascent is then used to
improve the policy:

$\displaystyle\Delta\theta=\mathbb{E}_{(s^{t},a^{t})\sim\pi_{\theta^{{}^{\prime}}}}[\Delta\min\\{\frac{\pi_{\theta}(a^{t}|s^{t})}{\pi_{{\theta^{\prime}}}(a^{t}|s^{t})},\operatorname{clip}(\frac{\pi_{\theta}(a^{t}|s^{t})}{\pi_{{\theta^{\prime}}}(a^{t}|s^{t})},1-\epsilon,1+\epsilon)\\}\cdot
A^{t}].$ (19)

With regards to the critic network, all actors under the PPO algorithm share a
similar critic network model. Hence, the authors of [25] formulate the critic
loss to be:

$\displaystyle L(\phi)=(V_{\phi}(s^{t})-A^{t}+\gamma
V_{\phi_{targ}}(s^{t}))^{2}.$ (20)

$\phi$ is the parameter of the critic model. The state-values
$V_{\phi}(s^{t})$ and $V_{\phi_{targ}}(s^{t})$ are estimated with the critic
model and the target critic model, respectively. Essentially, the critic model
utilizes the loss function $L(\phi)$ to update the critic network.

###  IV-B Multi-Agent-Loss-Sharing (MALS)

What is Multi-Agent-Loss-Sharing? We proposed a novel Multi-Agent-Loss-Sharing
(MALS) algorithm based on actor-critic (AC) reinforcement learning. In this
algorithm, each agent has an independent actor, similar to traditional
interacting, independent dual agents (IDA). However, in MALS, agents share a
critic model with multiple heads, allowing for personalized advantage values
calculation and cooperative training. The sum of critic losses from both heads
facilitates concurrent improvement of both agents’ models and ensures
cooperative training. MALS can accommodate asymmetric actors of different
action spaces. We adopt PPO as the underlying algorithm of MALS and utilize
two actors, one which has a discrete UE-MBS allocation action, and the other
which has a continuous power selection action.

Why not apply the CTDE framework? In traditional Multi-Agent Reinforcement
Learning (MARL) using the Centralized Training and Decentralized Execution
(CTDE) framework [18], each agent’s actor operates independently during
execution, while the shared critic model considers all agents’ actions during
training. However, CTDE is unsuitable for our asymmetric problem with distinct
action spaces for DL and UL agents (discrete-valued UE-MBS allocation and
continuous-valued UE uplink transmission power, respectively). Additionally,
our asynchronous problem, where DL and UL agents alternate action executions,
differs from the simultaneous action execution assumption in MARL, leading to
computational inefficiency in CTDE [18]. Moreover, our asymmetric and
asynchronous MEC resource allocation problem creates dependency between agents
in a Partially Observable Markov Decision Process (POMDP). In a conventional
single-agent environment, an agent’s actions only affect the environment and
rewards. However, in our two-agent asynchronous environment, an agent’s
transition probabilities are influenced by the other agent’s decisions. This
interdependence requires a new reinforcement learning structure to handle the
asynchronous setting and the concurrent orchestration of UL and DL stages.

Transmission mechanism: In a single iteration with one DL and one UL
transmission, the downlink state $s^{\text{d},t}$ will be observed by the DL
agent. The DL agent will then choose an action $a^{\text{d},t}$ based on its
policy and observed state. This action causes an impact on the environment,
and a DL reward $R^{\text{d},t}$ is issued to the DL agent in accordance to
the merit of its actions. The altered environment consequently influences the
UL state $s^{\text{u},t}$ observed by UL agent. Based on its policy, the UL
agent will choose an action $a^{\text{u},t}$ which corresponds to the current
observed state. This action by the UL agent influences the environment, and
yields the UL agent a corresponding reward $R^{\text{u},t}$ and the next DL
state $s^{\text{d},t+1}$. After several iterations, the rewards
($R^{\text{d},t},R^{\text{u},t}$) with states
($s^{\text{d},t},s^{\text{u},t}$) and their corresponding next states
($s^{\text{d},t+1},s^{\text{u},t+1}$) will be sent to the multi-head critic.
The critic will then utilize these rewards, states, and next iteration states
to calculate state-value $V$ and advantage value $A$ to update the actor
network parameters. This transmission mechanism is illustrated in Fig 2.

Asymmetric Actor: In MALS, there are two Actors: one arranges UE-MBS
allocation in the downlink transmission, and the other selects the UE power
for the uplink transmission. The UE-MBS allocation is a discrete case
selection and hence probabilities are computed for each action based on the DL
agent policy [26]. The transmission power selection in the UL stage falls
along a continuum and hence a probability distribution over the output power
values are learnt by the UL agent.

Discrete Action through Softmax. In the discrete DL actor neural network, we
adopt a softmax activation function in the final layer. Softmax is a commonly
used activation function for the output layer of a neural network in multi-
class classification tasks. The softmax function transforms the raw outputs of
the network into a probability distribution over multiple actions. It maps the
output values to a range between 0 and 1, ensuring that the sum of all
probabilities is equal to 1. This allows the network to assign a probability
to each action, indicating the likelihood of the input belonging to that
particular action. The action with the highest probability is then chosen as
the predicted class, where the UE-MBS allocation follows
$c_{i}^{t}=\text{arg}\max_{v}\left[\text{prob}(a^{t}_{i}|s^{t})\right]~{}\forall
v\in\mathcal{M},\forall t\in\mathcal{T},~{}\forall i\in\mathcal{N}$. By using
softmax, the neural network can provide a meaningful and interpretable output
for multi-action selection problems, enabling the identification of the most
probable UE-MBS allocation among the available options.

Continuous Action through sampling. The continuous PPO algorithm parameterizes
a probability distribution over the continuous action space, allowing the
agent to sample actions based on this distribution. The distribution is
typically chosen to be a Gaussian distribution with a mean and standard
deviation. The continuous output is then clamped to values between 0 and 1,
and eventually scaled to a valid range with a continuous action space adapter,
where $p^{\text{u}}_{min}$ is the minimal value, and $p^{\text{u}}_{max}$
being the maximum value, as per [27].

Actor network update. In MALS, $\theta^{\text{d}}$ and $\theta^{\text{u}}$ are
the network weights of DL agent and UL agent, and the gradients
$\Delta\theta^{\text{d}}$, $\Delta\theta^{\text{u}}$ of the DL agent and UL
agent are:

$\displaystyle\Delta\theta^{\text{u}}=\mathbb{E}_{(s^{\text{u},t},a^{\text{u},t})\sim\pi_{{\theta^{\text{u}}}^{\prime}}}[\triangledown
f^{t}(\theta^{\text{u}})A^{\text{u}}(s^{\text{u},t})],$ (21)
$\displaystyle\Delta\theta^{\text{d}}=\mathbb{E}_{(s^{\text{d},t},a^{\text{d},t})\sim\pi_{{\theta^{\text{d}}}^{\prime}}}[\triangledown
f^{t}(\theta^{\text{d}})A^{\text{d}}(s^{\text{d},t})],$ (22)

where
$f^{t}(\theta)=\min\\{\frac{\pi_{\theta}(a^{t}|s^{t})}{\pi_{{\theta^{\prime}}}(a^{t}|s^{t})},\operatorname{clip}(\frac{\pi_{\theta}(a^{t}|s^{t})}{\pi_{{\theta^{\prime}}}(a^{t}|s^{t})},1-\epsilon,1+\epsilon)\\}$
in the case of PPO, in which $\epsilon$ is the clipping parameter, and
$A(\cdot)$ is the advantage function of both UL and DL agents.

Advantage function. We utilize the state-value function ($V$) and the
truncated version of the Generalized Advantage Estimation (GAE) [28] method to
calculate the advantage values. The policy is then executed for $\bar{T}$ time
steps, where $\bar{T}$ is the number of steps in a trajectory. A trajectory of
$\bar{T}$ number of steps represents the path of an agent through the state
space up till horizon $\bar{T}$. The advantage value in our work can be
calculated as:

$\displaystyle
A^{\text{u},t}=\delta^{\text{u},t}+(\gamma\lambda)\delta^{\text{u},t+1}+...+(\gamma\lambda)^{\bar{T}-1}\delta^{\text{u},t+\bar{T}-1},\text{
for}~{}\delta^{\text{u},t}=R^{\text{u},t}+\gamma
V_{\phi_{targ}}(s^{\text{u},t+1})-V_{\phi_{targ}}(s^{\text{u},t}),$ (23)
$\displaystyle
A^{\text{d},t}=\delta^{\text{d},t}+(\gamma\lambda)\delta^{\text{d},t+1}+...+(\gamma\lambda)^{\bar{T}-1}\delta^{\text{d},t+\bar{T}-1},\text{
for}~{}\delta^{\text{d},t}=R^{\text{d},t}+\gamma
V_{\phi_{targ}}(s^{\text{d},t+1})-V_{\phi_{targ}}(s^{\text{d},t}),$ (24)

where $\bar{T}$, $\gamma$, and $\lambda$ denote the trajectory length,
discount factor, and GAE parameter, respectively. $\phi$ and $\phi_{targ}$ are
the parameters of the critic model and the target model, respectively.

Multi-head Critic: We proposed an asymmetric problem in our work, where each
agent optimizes different variables and has different objectives. A multi-head
critic caters to the asymmetric actors by providing personalized calculations
of state-values $V_{\phi}^{u},V_{\phi}^{d}$ using a neural network with two
heads. Each critic head produces a loss value corresponding to the state-value
that it produces, and the loss values of the critic heads are weighted-summed
($\kappa_{1}$ and $\kappa_{2}$) to give a conclusive loss value as follows:

$\displaystyle L^{\text{u}}(\phi)=(V_{\phi}(s^{\text{u},t})-\gamma
V_{targ}^{\text{u},t}))^{2},$ (25) $\displaystyle
L^{\text{d}}(\phi)=(V_{\phi}(s^{\text{d},t})-\gamma
V_{targ}^{\text{d},t}))^{2},$ (26) $\displaystyle L(\phi)=\kappa_{1}\times
L^{\text{u}}(\phi)+\kappa_{2}\times L^{\text{d}}(\phi).$ (27)

The specific values of $\kappa_{1}$ and $\kappa_{2}$ will be specified in the
experiments. The multi-head critic’s model parameter is updated with the loss
in Equation (27) to improve the critics model in guiding the asymmetric
actors. The MALS algorithm is detailed in Algorithm 1. The above explanations
and equations, along with their connections, are clearly illustrated in Figure
2.

Algorithm 1 MALS-PPO (proposed RL structure)

0: critic model $\phi$ and target model $\phi_{targ}$, downlink actor model
$\theta^{\text{d}}$ and sampling model $(\theta^{\text{d}})^{\prime}$, uplink
actor model $\theta^{\text{u}}$ and sampling model
$(\theta^{\text{u}})^{\prime}$, initial state $s^{\text{d},1}$.

1: $s^{\text{d},t}\leftarrow s^{\text{d},1}$;

2: for iteration $t=1,2,...$ do

3: DL agent executes action according to
$\pi_{(\theta^{\text{d}})^{\prime}}(a^{\text{d},t}|s^{\text{d},t})$;

4: Get reward $R^{\text{d},t}$ and uplink state $s^{\text{u},t}$;

5: UL agent executes action according to
$\pi_{(\theta^{\text{u}})^{\prime}}(a^{\text{u},t}|s^{\text{u},t})$;

6: Get reward $R^{\text{u},t}$ and next step downlink state
$s^{\text{d},t+1}$;

7: DL agent executes action according to
$\pi_{(\theta^{\text{d}})^{\prime}}(a^{\text{d},t+1}|s^{\text{d},t+1})$;

8: Get reward $R^{\text{d},t+1}$ and uplink state $s^{\text{u},t+1}$;

9: if iteration $t\geq 2$ then

10: Sample
($s^{\text{d},t},a^{\text{d},t},s^{\text{u},t},a^{\text{u},t},R^{\text{d},t},R^{\text{u},t},s^{\text{d},t+1},s^{\text{u},t+1}$)
iteratively till end of episode;

11: end if

12: $s^{\text{d},t}\leftarrow s^{\text{d},t+1}$, $s^{\text{u},t}\leftarrow
s^{\text{u},t+1}$, $a^{\text{d},t}\leftarrow a^{\text{d},t+1}$,
$R^{\text{d},t}\leftarrow R^{\text{d},t+1}$;

13: Compute advantages $\\{A^{\text{d},t},A^{\text{u},t}\\}$

14: Compute target values {$V^{\text{d},t}_{targ},V^{u,t}_{targ}$}

15: for $k$ = $1,2,...,K$ do

16: Assign samples into training groups;

17: for each training group do

18: Compute UL and DL actor model gradients:
$\triangledown\theta^{\text{d}},\triangledown\theta^{\text{u}}$

19: Apply gradient ascent on $\theta^{\text{d}}$ using
$\triangledown\theta^{\text{d}}$ by Eq. (22)

20: Apply gradient ascent on $\theta^{\text{u}}$ using
$\triangledown\theta^{\text{u}}$ by Eq. (21)

21: Update the critic model with loss from Eq. (27)

22: end for

23: end for

24: Update the critic target model parameters $\phi_{targ}$ with the critic
model parameters $\phi$ every $C$ steps where $C$ is the critic model update
interval;

25: end for

##  V Experiments

We conduct several experiments and showcase the superiority of our MALS RL
structure in handling asymmetric and asynchronous problems. We first discuss
the numerical settings for the experiments we conduct. We then show how the
proposed MALS algorithm is effective in improving resource management when
compared against commonly adopted metrics. We then compare the proposed MALS
model with baseline algorithms (i) independent dual agent (IDA) [29] and (ii)
Centralized Training and Decentralized Execution multi-agent model (CTDE)
[30]. Finally, we vary the weighting on each variable in the DL and UL agents’
objective functions to study the performance of the MALS model on varying
variable weights and their implications on the model performance.

Figure 3: Independent Dual Agent (IDA) (left) and Centralized Training
Decentralized Execution (CTDE) (right).

###  V-A Baseline algorithms

We study how (i) our proposed model (MALS model structure) compared to other
reinforcement learning-based structures. We adopt two reinforcement learning-
based model structures; the most intuitive (ii) independent dual agent (IDA)
(which employs two individual agents in which information transfer is only via
the states), and (iii) CTDE (which is the framework used for multi-agent
reinforcement learning [18]). We pit the baseline algorithms (IDA and CTDE)
against our proposed MALS model in tackling the asymmetric and asynchronous
task. The three models - Independent Dual Agent (IDA), Centralized Training
and Decentralized Execution (CTDE), and Multi-agent Loss Sharing (MALS) - all
have two separate actors influencing the environment. The IDA model has two
separate actors and two critics, with each actor of each agent outputting
action $a^{\text{d}}$, then $a^{\text{u}}$, in each iteration. The environment
is influenced by the actions of the two agents, and returns state
$s^{\text{d}}$ (after $a^{\text{d}}$) and $s^{\text{u}}$ (after
$a^{\text{u}}$) to its corresponding actors. Each agent’s critic updates with
trajectories of the states and its agent’s reward assignment $R^{\text{u}}$
and $R^{\text{d}}$. The CTDE model also has two separate actors, but they
share one critic model. The environment returns $s^{\text{d}}$ and
$s^{\text{u}}$ to the corresponding actors, and a single, common reward $R$ is
returned to each agent. Note that the reward $R$ assigned for the CTDE model
is not decomposed, and corresponds to the overarching objective in equation
16a. The sole critic receives the states $s^{\text{d}}$ and $s^{\text{u}}$ and
reward $R$ and updates its model. On the other hand, our proposed MALS model
has two actors and each outputs actions $a^{\text{d}}$ and $a^{\text{u}}$,
correspondingly that influence the environment, with each agent receiving
separate rewards $R^{\text{d},t}$ and $R^{\text{u},t}$. States $s^{\text{d}}$
and $s^{\text{u}}$ and separate rewards are sent to the central, multi-headed
critic for computation of separate advantage $A^{\text{d},t}$ and
$A^{\text{u},t}$ values. Finally, the computed advantage values for each agent
is used to update their actor. Illustrations of the model structures are shown
in Figure 2 and 3. Note that we do not compare our proposed model with a
multi-input, multi-headed, single agent. The states presented to the agent at
each transmission is different and a single agent is not able to accommodate
differing states sizes.

###  V-B Configuration

(a) Average downlink delay.

(b) Average UE earning potential.

(c) Average uplink delay.

(d) Average battery consumption.

Figure 4: Key metrics performance obtained by MALS across training steps for
the 4 MBS, 8 UE configuration.

(a) Downlink reward for 4 MBS, 6 UE.

(b) Downlink reward for 4 MBS, 7 UE.

(c) Downlink reward for 4 MBS, 8 UE.

(d) Uplink reward for 4 MBS, 6 UE.

(e) Uplink reward for 4 MBS, 7 UE.

(f) Uplink reward for 4 MBS, 8 UE.

Figure 5: Uplink and downlink agent rewards obtained for MALS model,
Independent Dual Agent model, CTDE model, across configuration settings and
seed 0 to 9. Bands within the graph indicate the range of reward values
obtained across the seeds.

We design an experiment with three scenarios: 4 MBSs and 6 to 8 UEs to test
our proposed Multi-Agent-Loss-Sharing (MALS) reinforcement learning-based
method. $T$ which is defined as the number of transmission steps within an
episode is set as $100$. We limit our experiments to configurations with no
more than 4 MBSs and 8 UEs. Even this configuration poses a significant
computational challenge for our RL agents. The complexity of discrete actions
increases exponentially with the number of MBSs and UEs, reaching $4^{8}$ in
our most complex scenario per iteration. Since we run the transmission process
for 100 iterations, where the actions selected in each iteration influences
subsequent iterations, the overall complexity becomes $4^{8^{100}}$.
Consequently, adding more MBSs or UEs would render the computation infeasible
due to the exponentially expanding action space. Additionally, since the UL
and DL agents influence each other, introducing more complex configurations
beyond our proposed limits would make the RL training process difficult and
prevent the agents from achieving convergence. Exhaustive search is an
infeasible method to optimize the UE-MBS assignment as the sequential nature
of our proposed problem renders the complexity to be $(M^{N})^{T}$, where $M$
is the number of MBS, $N$ is the number of UEs, and $T$ is the number of
transmission iterations in an episode. The bandwidth $B_{v}$ is set as $10$
Ghz [31] as P2E applications are bandwidth-intensive and a wider band permits
higher data transfer rates. We have designed all MBS to have similar frequency
band as (1) we are only considering a single application (P2E), (2) and
sharing a similar frequency band simplifies the network planning and
deployment process and enables seamless handover between MBSs as UEs move. The
noise $\sigma^{2}$ is set as $100$ dBm/Hz. The output power by the MBS for the
downlink transmission is set to be within $(1.5,2.0)$ Watt. We initialize the
locations of the MBSs and UEs to be random, within a 100m by 100m area. The
P2E in-game graphic size $D_{i}^{t}$ of each UE at every step is uniformly
distributed between $800$ and $1000$ Mb. We set $P$, $b$ and $f$ to be 10, 1
and 1, respectively, after empirical tuning, and $h$ and $q$ to be 0.5. $b$,
$f$, $h$ and $q$ are defined in subsection II-D.
$p^{\text{d}}_{min},p^{\text{d}}_{max},p^{\text{u}}_{min},p^{\text{u}}_{max}$
are set to be 5, 20, 3, 10 watts, respectively. We set the values of both
$\kappa_{1}$ and $\kappa_{2}$ from Equation (27) to be $0.5$ and $\varkappa$
from Equation (18) to be 0.3. $\varrho$ is set to $-50$. We set the maximum
distance UEs can move in each iteration ($x_{max}$ and $y_{max}$) to be 10m.
The ADAM optimizer [32] is utilized for the algorithms we study. We trained
the models for 1,000,000 iterations (steps) for all congestion settings. We
conducted the training and simultaneous evaluation of the models for each
configuration at different seed settings: seed 0 to seed 9.

###  V-C Channel Attenuation

We adopt a Rician fading as the small-scale fading, in terms of channel gain,
following the works of [33]. It is given by:

$\displaystyle g_{i,v}^{t}=\sqrt{\beta_{n}^{t}}\zeta_{i,v}^{t},$ (28)

where

$\displaystyle\beta_{i}^{t}=\beta_{0}(\mathcal{L}_{i}^{t})^{-\alpha},$ (29)
$\displaystyle\zeta_{i,v}^{t}=\sqrt{\frac{K}{K+1}}\bar{g}+\sqrt{\frac{1}{K+1}}\tilde{g},$
(30)
$\displaystyle\mathcal{L}_{i}=\sqrt{(\mathcal{X}_{i}-\mathcal{X}_{v})^{2}+(\mathcal{Y}_{i}-\mathcal{Y}_{v})^{2}}.$
(31)

UE $i$’s position is indicated by $(\mathcal{X}_{i},\mathcal{Y}_{i})$. The
distance between UE $i$ and MBS $v$ is denoted as $\mathcal{L}_{i}^{t}$, while
$\beta_{i}^{t}$ represents UE $i$’s large-scale channel gain during iteration
$t$. The components $\bar{g}$ represents the Line-Of-Sight (LOS) component
while $\tilde{g}$ represents the Non-LOS (NLOS) components. We assign variable
$\tilde{g}$ to follow a standard complex normal distribution with a mean of 0
and a standard deviation of 1. $\beta$ is the channel gain at a
$\mathcal{L}_{0}=1$m. In this context, the Rician factor $K$ and the path-loss
exponent $\alpha$ is simulated to be 3 and 2, respectively.

###  V-D Result analyses

####  V-D1 MALS model convergence

In training our MALS model, all three configurations (i) 4 MBS, 6 UE, (ii) 4
MBS, 7 UE and (iii) 4 MBS, 8 UE showed a general increment of achieved test-
time reward for both the downlink and the uplink agents (shown in Figure 5)
and improvement in the underlying metrics (shown in Figure 4, 10 and 9). The
solid line in the charts represents the mean reward of the different seed
settings, for each configuration. The color hue bands around the solid line
represent the minimum (lower bound) and maximum (upper bound) of the rewards
received. The training of RL agents converged quicker in the 4 MBS, 6 UE
configuration as opposed to the 4 MBS, 7 UE and 4 MBS, 8 UE as there is less
complexity in its scenario, allowing the model to converge quicker and achieve
a higher eventual reward. Conversely, a more complex configuration like 4 MBS,
8 UE converges slower due to the higher complexity of the action space and
having more UEs to manage. Overall, the improvement in both the downlink and
uplink reward signifies that the overall utility function for both agents are
improving, indicating that the training of the downlink agent and uplink agent
improves its UE-MBS allocation and UL power allocation, respectively.

Downlink. It is observed that as training progresses, there is a decrease in
in-game downlink transmission delay (shown in Figure 4(a), 10(a), 9(a)) and
increase in worst-case (lowest) earning potential of the UEs (shown in Figure
4(b), 10(b), 9(b)) with each training step. These observations directly
reflect that the downlink agent is allocating UE-MBS more efficiently,
achieving a higher downlink data transfer rate which decreases the downlink
latency and improves the worst-case (lowest) earning capability of the UEs. In
addition, the improvement of worst-case battery charge expenditure and uplink
transmission delay (shown in Figure 10(c), 9(c) and Figure 10(d), 9(d),
respectively), as training progress, is also in part attributed to better UE-
MBS allocation, which results in less battery charge expended and reduced
uplink transmission latency. The improvement of the metrics, as training
progress, contribute to an improving reward obtained by the DL agent.

Uplink. Similarly, we note that as training progresses, a more optimal use of
UE transmission output power for the uplink transmission results in a lower
average uplink delay (shown in Figure. 4(c)) and lower worst-case (greatest)
battery charge expenditure. This is reflected in Figure 4(d). These
improvements in uplink delay and lower worst-case battery charge expenditure
contributes to higher reward obtained by the UL agent. Furthermore, a higher
downlink reward is also in part, attributed to more efficient uplink battery
charge expenditure by the UL agent, as an efficient UL power selection enables
continual iterations of transmissions, and lesser likelihood for receiving
battery charge depletion penalty.

####  V-D2 Comparison with other reinforcement learning model structures

Our proposed model (MALS) achieves a smooth convergence for each
configuration, exhibiting a narrow range of downlink and uplink rewards,
across different seeds (shown in Figure 5). We observe that our proposed model
obtains a significantly higher reward for both the uplink and downlink
rewards. The prowess of the MALS model is more notable in more complex
configurations (i.e., 4 MBS, 7 UE and 4 MBS, 8 UE) where it outperforms other
baseline models and obtain significantly higher rewards, when compared to both
the IDA and CTDE baseline models. The narrower bands indicate that the
proposed MALS model is more stable across the different seed settings, for
each of the configurations.

(a) Earning potential wrt $q$.

(b) Downlink delay wrt $q$.

(c) Battery consumption wrt $q$.

(d) Uplink delay values wrt $q$.

(e) Earning potential wrt $h$.

(f) Downlink delay wrt $h$.

(g) Battery consumption wrt $h$.

(h) Uplink delay wrt $h$.

Figure 6: Metric values with respect to (wrt) different $q$ and $h$.

On the other hand, the (ii) IDA model converged at a much lower reward for
both the UL and DL agents (shown in Figure 5), across all configurations, and
took longer number of training steps to achieve similar rewards obtained by
MALS earlier in the training process. We observe that the independent dual
agents model obtained significantly lower downlink and uplink rewards across
all configurations, when compared to both the proposed MALS model and the CTDE
model. Furthermore, the variance of both downlink and uplink rewards obtained
is much larger for the independent dual agents model. This is notable across
all configurations. This indicates instability and hyper-sensitivity of the
model when handling different seed settings and minute changes in environment.
In addition, IDA shows signs of training instability. From Figure 5(d), 5(e),
5(f), we observe that the IDA model have a slight improvement in the uplink
rewards in the early training steps, followed by a subsequent decline in
rewards obtained, across all configurations (indicated by the lower bands).
Some seeds show an evident decline in uplink reward obtained as training
proceeds. This shows a convergence failure for some of these seeds in the
independent dual-agent model. It is apparent that the downlink agent’s rewards
are prioritised over the uplink agent’s rewards in this case, which renders
the IDA model unsuitable for real-world deployment.

The (iii) CTDE model achieves slightly poorer performance to our proposed
methods (MALS) in each configuration, but better than the IDA model for both
the downlink and uplink rewards (shown in Figure 5(a), 5(d)), but achieved
lackluster performance for the 4 MBS, 7 UE and 4 MBS, 8 UE configurations for
both the downlink and uplink rewards (shown in Figure 5(b), 5(c), 5(e), 5(f)).
The CTDE produces a large variance of downlink and uplink reward outputs
across the different seeds, albeit not as severe as the IDA model, as the
model still converges in all seeds. However, the large variance and sub-par
rewards obtained renders the CTDE inferior to our proposed methods due to its
poorer performance, instability and unreliability.

####  V-D3 Weighting Emphasis On Joint Utility function

When dealing with joint optimization problems, it is essential for us to
observe how our proposed model handles the varying weights on variables and
their influence on variables of interest, as this provides future research
insights on multi-agent joint utility function design.

Firstly, we vary the weight $q$ in the downlink reward $R^{\text{d},t}$ and
vary the weight $h$ in the uplink reward $R^{\text{u},t}$, that are defined in
subsection III-C, with values from the set of {0, 0.25, 0.50, 0.75, 1}. While
we vary $q$, we keep $h$ at a value of 0.5, and vice versa. We then take the
average results of the last 20,000 training steps of the different variables
of interest (downlink delay, worst-case (lowest) earning potential, uplink
transmission delay, worst-case (greatest) battery charge expenditure). These
averaged results are then compared with a random policy that assigns the
downlink UE-MBS allocation and uplink power output randomly. We compare our
proposed model with a random policy to show how our proposed model adds value
in optimizing UE-MBS allocation and power selection. Please refer to the
Appendix (shown in Fig. 7 and 8) for the complete comparison with the
reinforcement learning model structures discussed (IDA and CTDE). We do not
focus on the weighted analysis of IDA and CTDE as they have been shown to
produce unreliable performance with high variances.

Downlink weight analysis.  The earnings of our proposed MALS model
consistently outperform random allocation across all $q$ weight values, except
for $q=1$ (refer to Figure 6(a)). This indicates that our model significantly
improves the earning capability of the UEs. It is worth noting that the random
policy agent allows higher earnings for UEs at $q=1$ compared to our model.
This is because higher $q$ values imply more emphasis on minimizing downlink
delay and less emphasis on the worst-case (lowest) UE earning capability. At
$q=1$, the downlink agent in our proposed model focuses solely on minimizing
downlink and uplink latency, as well as worst-case uplink battery expenditure,
without considering the maximization of the worst-case (lowest) UE earning
potential. Additionally, it is observed that the worst-case (lowest) UE
earning potential decreases as $q$ increases across each scenario
configuration in our proposed method. This decrease is expected since higher
$q$ values prioritize minimizing downlink latency over maximizing the worst-
case (lowest) UE earning potential.

We observe that our proposed downlink latency in various configurations and
different $q$ values is significantly lower compared to the random policy,
except for the 4 MBS, 8 UE configuration at $q=0$ (refer to Figure 6(b)). It
is not surprising to see poorer performance in this specific configuration, as
at $q=0$, there is no weight emphasis on minimizing downlink delay and the
entire focus is on maximizing the worst-case (lowest) UE earning potential.
Another noteworthy observation is that the overall downlink latency for each
configuration decreases as $q$ increases. This trend can be attributed to the
fact that a higher value of $q$ increases the weight emphasis on minimizing
agent downlink delay. Extensive results are shown in Fig. 7

Uplink weight analyses.  Our findings reveal that, across different $h$ values
and configurations, the worst-case (highest) battery charge expenditure and
uplink delay obtained by our proposed model is significantly lower than that
of the random policy (refer to Figure 6(g) and 6(h)). This outcome highlights
the substantial contribution of our proposed model in the power allocation
process. Upon closer examination of the changes in the worst-case (highest)
battery charge expenditure with respect to the weight $h$, we observe a
general increase in the worst-case battery charge consumption for each
configuration, and decrease in uplink delay, as $h$ increases. This behavior
can be attributed to the higher weight emphasis placed on minimizing uplink
latency by the uplink agent as $h$ increases, at the expense of worst-case
(highest) battery charge expenditure. Extensive results are shown in Figure 8.

##  VI Conclusion

In this work, we consider a play-to-earn mobile edge computing framework and
formulate an asymmetric (discrete-continuous) and asynchronous (alternating DL
and UL) multi-variable optimization where the Mobile edge computing Service
Provider’s objective is to minimize in-game graphics downlink, uplink data
transmission latency, and worst-case (greatest) battery charge expenditure,
while maximizing the UEs’ in-game resolution-influenced worst-case (lowest)
earning potential. We then propose a novel multi-agent loss-sharing (MALS) RL
model to tackle the abovementioned asynchronous and asymmetric problem, and
demonstrate its superiority in performance over other methods. Finally, we
conduct joint optimization weighting analyses and show the viability of
utilizing our proposed MALS algorithm to tackle joint-optimization problems
across different variable weighting emphases.

##
References

  * [1] T. J. Chua, W. Yu, and J. Zhao, “Play to earn in the metaverse over wireless networks with deep reinforcement learning,” _submitted to the 2023 IEEE International Conference on Communications (ICC)_. [Online]. Available: https://personal.ntu.edu.sg/JunZhao/ICC2023MALS.pdf
  * [2] R. Browne, “Cash grab or innovation? the video game world is divided over NFTs,” Dec 2021. [Online]. Available: https://www.cnbc.com/2021/12/20/cash-grab-or-innovation-the-video-game-world-is-divided-over-nfts.html
  * [3] “Polka city.” [Online]. Available: https://www.polkacity.io/
  * [4] “Reality clash.” [Online]. Available: https://realityclash.com/
  * [5] N. Roos, “Best Play-to-Earn games with NFT or crypto,” November 2022. [Online]. Available: https://www.playtoearn.online/games/
  * [6] M. Chen, W. Saad, and C. Yin, “Resource management for wireless virtual reality: Machine learning meets multi-attribute utility,” in _GLOBECOM 2017-2017 IEEE Global Communications Conference_. IEEE, 2017, pp. 1–7.
  * [7] Y. Wang, M. Chen, Z. Yang, W. Saad, T. Luo, S. Cui, and H. V. Poor, “Meta-reinforcement learning for immersive virtual reality over THz/VLC wireless networks,” in _ICC 2021-IEEE International Conference on Communications_. IEEE, 2021, pp. 1–6.
  * [8] Q. Liu, S. Huang, J. Opadere, and T. Han, “An edge network orchestrator for mobile augmented reality,” in _IEEE INFOCOM 2018-IEEE Conference on Computer Communications_. IEEE, 2018, pp. 756–764.
  * [9] H. Wang and J. Xie, “User preference based energy-aware mobile ar system with edge computing,” in _IEEE INFOCOM 2020-IEEE Conference on Computer Communications_. IEEE, 2020, pp. 1379–1388.
  * [10] H. Lu, C. Gu, F. Luo, W. Ding, and X. Liu, “Optimization of lightweight task offloading strategy for mobile edge computing based on deep reinforcement learning,” _Future Generation Computer Systems_ , vol. 102, pp. 847–861, 2020.
  * [11] T. Alfakih, M. M. Hassan, A. Gumaei, C. Savaglio, and G. Fortino, “Task offloading and resource allocation for mobile edge computing by deep reinforcement learning based on sarsa,” _IEEE Access_ , vol. 8, pp. 54 074–54 084, 2020.
  * [12] G. Qiao, S. Leng, S. Maharjan, Y. Zhang, and N. Ansari, “Deep reinforcement learning for cooperative content caching in vehicular edge computing and networks,” _IEEE Internet of Things Journal_ , vol. 7, no. 1, pp. 247–257, 2019.
  * [13] S. Bi, L. Huang, H. Wang, and Y.-J. A. Zhang, “Lyapunov-guided deep reinforcement learning for stable online computation offloading in mobile-edge computing networks,” _IEEE Transactions on Wireless Communications_ , vol. 20, no. 11, pp. 7519–7537, 2021.
  * [14] T. P. Truong, T.-V. Nguyen, W. Noh, S. Cho _et al._ , “Partial computation offloading in noma-assisted mobile-edge computing systems using deep reinforcement learning,” _IEEE Internet of Things Journal_ , vol. 8, no. 17, pp. 13 196–13 208, 2021.
  * [15] X. Qiu, L. Liu, W. Chen, Z. Hong, and Z. Zheng, “Online deep reinforcement learning for computation offloading in blockchain-empowered mobile edge computing,” _IEEE Transactions on Vehicular Technology_ , vol. 68, no. 8, pp. 8050–8062, 2019.
  * [16] D. Guo, L. Tang, X. Zhang, and Y.-C. Liang, “Joint optimization of handover control and power allocation based on multi-agent deep reinforcement learning,” _IEEE Transactions on Vehicular Technology_ , vol. 69, no. 11, pp. 13 124–13 138, 2020.
  * [17] C. He, Y. Hu, Y. Chen, and B. Zeng, “Joint power allocation and channel assignment for NOMA with deep reinforcement learning,” _IEEE Journal on Selected Areas in Communications_ , vol. 37, no. 10, pp. 2200–2210, 2019.
  * [18] R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” _Advances in Neural Information Processing Systems_ , vol. 30, 2017.
  * [19] T. Rashid, M. Samvelyan, C. Schroeder, G. Farquhar, J. Foerster, and S. Whiteson, “Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning,” in _International conference on machine learning_. PMLR, 2018, pp. 4295–4304.
  * [20] L. Dai, B. Wang, Z. Ding, Z. Wang, S. Chen, and L. Hanzo, “A survey of non-orthogonal multiple access for 5g,” _IEEE communications surveys & tutorials_, vol. 20, no. 3, pp. 2294–2323, 2018.
  * [21] ——, “A survey of non-orthogonal multiple access for 5g,” _IEEE communications surveys & tutorials_, vol. 20, no. 3, pp. 2294–2323, 2018.
  * [22] J. Feng and J. Zhao, “Resource allocation for augmented reality empowered vehicular edge metaverse,” _arXiv preprint arXiv:2212.01325_ , 2022.
  * [23] D. Yang, G. Xue, X. Fang, and J. Tang, “Crowdsourcing to smartphones: Incentive mechanism design for mobile phone sensing,” in _Annual International Conference on Mobile Computing and Networking (MobiCom)_ , 2012, pp. 173–184.
  * [24] H. Van Seijen, M. Fatemi, J. Romoff, R. Laroche, T. Barnes, and J. Tsang, “Hybrid reward architecture for reinforcement learning,” _Advances in Neural Information Processing Systems_ , vol. 30, 2017.
  * [25] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017.
  * [26] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour, “Policy gradient methods for reinforcement learning with function approximation,” _Advances in Neural Information Processing Systems_ , vol. 12, 1999.
  * [27] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017.
  * [28] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, “High-dimensional continuous control using generalized advantage estimation,” _arXiv preprint arXiv:1506.02438_ , 2015.
  * [29] K. Zhang, Z. Yang, and T. Başar, “Multi-agent reinforcement learning: A selective overview of theories and algorithms,” _Handbook of Reinforcement Learning and Control_ , pp. 321–384, 2021.
  * [30] J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson, “Counterfactual multi-agent policy gradients,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 32, no. 1, 2018.
  * [31] P. Yang, Y. Xiao, M. Xiao, and S. Li, “6G wireless communications: Vision and potential techniques,” _IEEE Network_ , vol. 33, no. 4, pp. 70–75, 2019.
  * [32] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [33] I. Yildirim, A. Uyrus, and E. Basar, “Modeling and analysis of reconfigurable intelligent surfaces for indoor and outdoor applications in future wireless networks,” _IEEE Transactions on Communications_ , vol. 69, no. 2, pp. 1290–1301, 2021.

Appendix: Model performance and weighting comparisons between models

(a) Downlink delay with 6 UEs.

(b) Earning potential with 6 UEs.

(c) Up-link delay with 6 UEs.

(d) Battery usage with 6 UEs.

(e) Downlink delay with 7 UEs.

(f) Earning potential with 7 UEs.

(g) Up-link delay with 7 UEs.

(h) Battery usage with 7 UEs.

(i) Downlink delay with 8 UEs.

(j) Earning potential with 8 UEs.

(k) Up-link delay with 8 UEs.

(l) Battery usage with 8 UEs.

Figure 7: Complete results - Metric values across different $q$ values.

(a) Downlink delay with 6 UEs.

(b) Earning potential with 6 UEs.

(c) Up-link delay with 6 UEs.

(d) Battery usage with 6 UEs.

(e) Downlink delay with 7 UEs.

(f) Earning potential with 7 UEs.

(g) Up-link delay with 7 UEs.

(h) Battery usage with 7 UEs.

(i) Downlink delay with 8 UEs.

(j) Earning potential with 8 UEs.

(k) Up-link delay with 8 UEs.

(l) Battery usage with 8 UEs.

Figure 8: Complete results - Metric values across different $h$ values.

Fig. 7 and 8 show how the performance metrics vary across different $q$ and
$h$ values. The changes in $q$ and $h$ corresponds to changing weight emphasis
between variables. Value of $h$ is $0.5$ when $q$ is varying, and vice versa.
Figure 10 and 9 shows the obtained metric values by the uplink and downlink
agents in the 4 MBS, 7 UE and 4 MBS, 6 UE configuration, respectively.

(a) Average downlink delay.

(b) Average UE earning potential.

(c) Average uplink delay.

(d) Average battery consumption.

Figure 9: Key metrics performance obtained by MALS across training steps for
the 4 MBS, 6 UE configuration.

(a) Average downlink delay.

(b) Average UE earning potential.

(c) Average uplink delay.

(d) Average battery consumption.

Figure 10: Key metrics performance obtained by MALS across training steps for
the 4 MBS, 7 UE configuration.