algorithm achieves around 15% sum-rate gains and a 20% reduction in vehicular
user outages compared to baseline approaches.

The problem of dynamic spectrum access in CRNs is investigated in [133]
through combining DRL and evolutionary game theory. In particular,
uncooperative multi-agent DQN is considered in which the agents are the SUs
whose action is discrete, corresponding to selecting the access channel. The
state space is discrete, which includes two main parts; the channel selected
by the agent and the utility obtained after transmission on the selected
channel. The reward function is defined in terms of evolutionary game theory.
Simulation results indicate the performance enhancement of their proposed
algorithm over the case without learning in terms of average system capacity.

Another interesting work is presented in [134] in which the authors propose a
multi-agent DQN-based DRL algorithm to address the problem of optimum multi-
user access control in Non-Terrestrial Networks (NTNs). In their model, UEs
are the independent agents that report their experiences and local
observations to a centralized trainer controller located at the backhaul
network. The latter will then utilize the collected experiences to update the
global DQN parameter. The agent’s state space is continuous, comprised of the
connected NT-BS of UEs at the previous time slots, the RSS of UEs, the number
of connected UEs of each NT-BS, and the transmission rate of UEs. The action
space is discrete, representing the binary indicator functions of UEs, while
the reward is a function of the transmission rate of UEs. Experimental
performance results show that their proposed scheme is efficient in addressing
the fundamental issues in the deployment of NTNs infrastructure, and it
outperforms the traditional algorithms in terms of both the data rate and the
number of handovers.

In [135], the authors propose a multi-agent DQN-based DRL scheme to address
the problem of dynamic joint spectrum access and mode selection (SAMS) in
cognitive radio networks (CRNs). The agents are the secondary users (SUs)
whose action space is discrete, corresponding to selecting the access channel
and access mode. The state space of each SU agent is discrete, comprised of
the action taken by the $m$th SU agent, the ACKs of all SUs agents, and the
ACK of the $m$th SU agent. The reward function is discrete, which is ”1” if
the action selection process is successful; otherwise, there is a collision,
and the agent will receive a ”0” reward.

Tomovic et al. [136] propose a single-agent DRL model based on the integration
of Double deep $Q$-learning architecture and RNN to address the problem of DSA
in multi-channel wireless networks. In particular, the agent is the SU node,
whose action space is discrete, representing the selection of a channel for
sensing. The state space is also discrete, comprised of a history of the
binary observations and history of taken actions. The reward function is a
discrete binary function, which is ”1” if the observation is ”1” and ”0”
otherwise. Simulation results show that their proposed method is able to find
a near-optimal policy in a smaller number of iterations, and it can support a
wide range of communication environment conditions.

In other work in [137], the authors propose both a single-agent and multi-
agent deep actor-critic DRL-based framework for dynamic multi-channel access
in wireless networks. In their system, the agents are the users whose action
space is discrete, corresponding to selecting a channel. The observation space
is also discrete, which is defined based on the status of the channel and
collision status. The reward function is discrete, which is ”+1” if the
selected channel is good; otherwise, it is ”-1”. Using simulation results, the
authors show that their proposed actor-critic framework outperforms the DQN-
based algorithm, random access, and the optimal policy when there is full
knowledge of the channel dynamics.

The problem of DSA for the CRN is studied in [138] based on an uncoordinated
and distributed multi-agent DQN model. The agents are CRs, whose action is
discrete, representing the possible transmit powers. The state space is
discrete, reflecting whether the limits for DSA are being met or not,
depending on the relative throughput change at all the primary network links.
The reward is also discrete, which is a function of the throughput of the
links and the environment’s state. Experimental results reveal that their
proposed scheme finds policies that yield performance within 3% of an
exhaustive search solution, and it finds the optimal policy in nearly 70% of
cases.

Industrial IoT (IIoT) has emerged recently as an innovative networking
ecosystem that facilitates data collection and exchange in order to improve
network efficiency, productivity, and other economic benefits [139]. RRAM in
such a sophisticated ecosystem is also a challenge that needs more
investigation. In this context, recently in [140], the authors propose a
solution for spectrum resource management for IIoT networks, with the goal of
enabling spectrum sharing between different kinds of UEs. In particular, a
single-agent DQN algorithm is proposed in which the agent is the BS. The
action space is discrete, which corresponds to the allocation of spectrum
resources for all UEs. The observation space is a hybrid (continuous and
discrete) consisting of four elements; the current action (i.e., the
allocation of spectrum resources), the data priority of type I UEs, the buffer
length of type II UE, and the communication status of the first type of UEs.
The reward function is continuous, defined to address their optimization
problem. It is divided into four objectives; 1) maximizing the spectrum
resource utilization; 2) quickly transmitting the high-priority data; 3)
meeting the threshold rate requirement of the first type of UEs; 4) ensuring
that the second type of UEs completes the transmission in time. Using
simulation results, it is demonstrated that their proposed algorithm achieves
better network performance with a faster convergence rate compared with other
algorithms.

In [141], the authors propose a multi-agent Double DQN-based DRL model to
address the problem of DSA in distributed wireless networks. In particular,
they design a channel access scheme to maximize channel throughput regarding
fair channel access. The agents in their scheme are the users. The action
space is discrete, which is ”0” if the user does not attempt to transmit
packets during the current time slot, and it is ”1” if it has attempted to
transmit. The state space is discrete, consisting of four main elements; each
user action taken on the current time slot, channel capacity (which could be
negative, positive, or zero), a binary acknowledgment signal showing if the
user transmits successfully or not, and a parameter that enables each user to
estimate other users’ situations. The reward is a discrete binary function
that takes the value of ”1” if the user transmits successfully; otherwise, it
is ”0” meaning that the user transmitted with collision. It is shown using
simulation results that their scheme can maximize the total throughput while
trying to make fair resource allocation among users. Also, it is shown that
their proposed scheme outperforms the conventional Slotted-Aloha scheme in
terms of sum-throughput.

Wang et al. [142] address the problem of DSA in VANETs, by proposing an
interesting scheme that combines multi-hop forwarding via vehicles and DSA.
The optimal DSA policy is defined to be the joint maximization of channel
utilization and minimization of the packet loss rate. A multi-agent DRL
network structure is presented that combines RNN and DQN for learning the
time-varying process of the system. In their scheme, each user acts as an
agent whose action space is discrete, corresponding to choosing a channel for
transmission at time slot $t$. The state space is discrete, composed of three
components; a binary transmission condition $\eta$, which is ”1” if the
transmission is successful and ”0” otherwise, the channel selection action,
and the channel status indicator after each dynamic access process. The reward
is a discrete binary function, which takes a positive value if $\eta=1$,
otherwise it takes the value of ”0”. Simulation results show that their
proposed scheme: 1) reduces the packet loss rate from 0.8 to around 0.1, 2)
outperforms Slotted-Aloha and DQN in terms of reducing collision probability
and channel idle probability by about 60%, and 3) enhances the transmission
success rate by around 20%.

Most recently in [143], the authors propose multi-agent DQN-based DRL schemes
to address the problem of joint cooperative spectrum sensing and channel
access in clustered cognitive UAV (CUAV) communication Networks. Three
algorithms are proposed: 1) a time slot multi-round revisit exhaustive search
based on virtual controller (VC-EXH), 2) a $Q$-learning based on independent
learner (IL-Q) and 3) a DQN based on independent learner (IL-DQN). The agents
are the CUAVs in the network. The action space of any CUAV agent is a discrete
function defined by the steps that this agent moves clockwise in time slot $t$
relative to the channel selected in time slot $t-1$ on the PU channel ring.
The state space is a discrete set consisting of two main elements: 1) the
number of CUAVs agents that have selected a particular channel to sense and
access in the previous time slot and 2) a binary indicator function that shows
the occupancy status of a particular channel in the previous time slot. The
reward is a discrete function defined in terms of spectrum sensing, channel
access, utility, and cost. Experimental results show that all the three
algorithms proposed show efficient results in terms of convergence speed and
the enhancement of utilization of idle spectrum resources.

The authors in [144] propose a multi-agent deep recurrent Q-network-based DRL
model to address the problem of DSA in dynamic heterogeneous environments with
partial observations. In their work, the authors consider a case-study with
multiple heterogeneous PUs sharing multiple independent radio channels. The
agents are the SUs, whose action space is discrete, corresponding to deciding
whether to transmit in a particular band or wait during the next time slot.
The state space is discrete, representing whether the channels are occupied,
idle, or unknown. The reward function is discrete, represented by two values;
100 per channel for successful transmission and -50 per channel for collision.
Using simulation results, the authors show that their proposed algorithm
handles various dynamic communication environments, and its performance
outperforms the myopic conventional methods and very close to the
optimization-based approaches.

####  IV-B3 In Satellite Networks

In the following paragraphs, we review works that employ DRL algorithms to
address the spectrum and access control problem in satellite networks and
emerging satellite IoT systems.

The work in [145] proposes a single-agent DQN-based DRL algorithm that
considers the problem of channel assignment in multi-beam satellite systems.
In their scheme, the agent is the satellite, whose action is discrete,
including an index that indicates the channel that the newly arrived user has
occupied. The agent’s reward is discrete, which contains a positive value if
the service is satisfied, and a negative value if the service is not satisfied
or blocked. The state space is also discrete, which comprises three elements;
the current users, the current channel assignment matrix, and a list of the
new user arrivals. Experimental results demonstrate that their proposed scheme
decreases the blocking probability and improves the carried traffic up to
24.4% as well as enhances the spectrum efficiency compared to the conventional
fixed channel assignment approach.

The authors in [146] propose a single-agent DQN-based DRL algorithm to address
the problem of dynamic channel allocation in multi-beam satellite systems. In
particular, an image-like tensor formulation on the system environments is
considered in order to extract traffic spatial and temporal features. The
agent in their model is the satellite, whose action space is discrete,
corresponding to determining the resource allocation schemes. The state space
is continuous, consisting of two elements; the system resource allocation
state and the users’ service request state. The reward function is discrete,
which is defined in terms of the optimization objective function.

Recently, the authors in [147] propose a single-agent DQN-based DRL approach
for energy-efficient channel allocation in SIoT. The agent in their model is
the LEO satellite, whose action space is discrete, corresponding to mapping
from newly coming node tasks to channels to be allocated. The state space is
discrete, including information about user tasks, such as the size and
location of tasks. The reward is continuous, which is divided into two
normalized reward function components; the power efficiency reward and the
normalized value of the service blocking rate. Both of these reward components
are functions of power set up by the agent, the optimal power decided by the
location of the beam, and the number of served nodes. Experimental results
demonstrate that their proposed algorithm saves energy consumption of around
67.86% compared to some conventional approaches.

In [148], the authors propose a centralized single-agent DQN-based scheme to
address the problem of dynamic channel allocation in SIoT. The agent in their
model is the satellite, whose action is discrete, corresponding to selecting
which sensors to allocate channels to. The state space is discrete, comprised
of three parts; the number of tasks in each time step, the bandwidth that a
sensor node requires, and the duration of a new task. The reward is
continuous, which is a function of the duration of data transmission for the
sensor. Using simulation results, it is shown that their proposed algorithm
both provides higher transmission success rates and reduces data transmission
latency by at least 87.4% compared to the conventional channel allocation
algorithms.

Zheng et al. [149] propose a single-agent $Q$-learning-based RL model to
address the problem of combination allocation of fixed channel pre-allocation
and dynamic channel scheduling in a network architecture of LEO satellites
that utilizes a centralized resource pool. In their model, the satellite
serves as an agent whose action is discrete, corresponding to assigning
channels to users. The state space is discrete, defined by the channel
assignment of users in each beam. The reward is continuous, which is a
function of the user’s supply-demand ratio. Experimental results demonstrate
that their proposed approach enhances system supply-demand ratio by 14% and
18% compared to the static channel allocation the Lagrange algorithm channel
allocation method.

####  IV-B4 In Multi-RAT HetNets

In the following paragraphs, we review works that employ DRL algorithms to
address the spectrum and access control problem in multi-RAT HetNets. This
includes the coexistence of various variants of the wireless networks shown in
Fig. 2.

In [150], a multi-agent DQN-based model that jointly tackles the dynamic
channel selection and interference management in Small Base Stations (SBSs)
cellular networks that share a set of unlicensed channels in Long Term
Evolution (LTE) networks. In the proposed scheme, the SBSs are the agents who
choose one of the available channels for transmitting packets in each time
slot. The agent’s action is channel access and channel selection probability.
The DQL input includes the channels’ traffic history of both the SBSs and
Wireless Local Area Networks (WLAN), while the output is the agent’s predicted
action vectors. Simulation results reveal that their proposed DQL strategy
enhances the average data rate by up to 28% when compared to the conventional
$Q$-learning scheme.

In [151], a single-agent DQN-based model is proposed to tackle the dynamic
spectrum allocation for multiple users that share a set of $K$ channels in the
same network settings of [150]. In their scheme, the agent is the user whose
action is either choosing a channel with a particular attempt probability or
selecting not to transmit. The agent’s state includes the history of the
actions of the agent and its current observations. The DQL model input is the
previous actions along with their observations, while the output is the
Q-values corresponding to the actions. Simulation results demonstrate that
their proposed DQL strategy achieves a double data rate compared to the state-
of-the-art Slotted-Aloha scheme.

Wang et al. [152] propose a single-agent prediction-DDPG-based DRL algorithm
to study the problem of the dynamic multichannel access (MCA) for the hybrid
long-term evolution and wireless local area network (LTE-WLAN) aggregation in
dynamic HetNets. The agent is the central BS controller, whose state space is
continuous, consisting of both the channels’ service rates and the users’
requirement rates. The action space, on the other hand, is discrete,
representing the users’ index. Two reward functions are provided; online
traffic real reward and online traffic prediction reward, each of which are
functions of users’ requirements, channels’ supplies, degree of system
fluctuation, the relative resource utilization, and the quality of user
experience. Using simulation results, the authors demonstrate the efficiency
of the proposed prediction-DDPG model in solving the dynamic MCA problem
compared to conventional methods.

Other interesting work in [153], the authors investigate the joint allocation
of the spectrum, computing, and storing resources in multi-access edge
computing (MEC)-based vehicular networks. In particular, the authors propose
multi-agent DDPG-based DRL algorithms to address the problem in a hierarchical
fashion considering a network comprised of macro eNodeB (MeNB) and Wi-Fi APs.
The agents are the controller installed at MEC servers. The agents’ action
space is discrete including the spectrum slicing ratio set, spectrum
allocation fraction sets for the MeNB and for each Wi-Fi AP, computing
resource allocation fraction, and storing resource allocation fraction. The
state space is discrete representing information of the vehicles within the
coverage area of the MEC server, including vehicles’ number, x-y coordinates,
moving state, position, and task information. The reward function is discrete,
defined in terms of the delay requirement, and requested storing resources
required to guarantee the QoS demands of an offloaded task. Provided
experimental results reveal that their proposed schemes achieve high QoS
satisfaction ratios compared with the random assignment techniques.

Recently in [154], the authors propose a single-agent DQN algorithm based on
Monte Carlo Tree Search (MCTS) to address the problem of dynamic spectrum
sharing between 4G LTE and 5G NR systems. In particular, the authors used the
MuZero algorithm to enable a proactive BW split between 4G LTE and 5G NR. The
agent is a controller located at the network core, whose action space is
discrete, corresponding to a horizontal line splitting the BW to both 4G LTE
and 5G NR. The state space is discrete, defined by five elements: 1) an
indicator if the user is an NR user or not, 2) the number of bits in the
user’s buffer, 3) an indicator of whether the user is configured with
multimedia broadcast single frequency network (MBSFN) or not, 4) the number of
bits that can be transmitted for the user in a given subframe, and 5) the
number of bits that will arrive for each user in the future subframes. The
reward function is a continuous function defined as a summation of the
exponential of the delayed packet per user. Experimental results show that
their proposed scheme provides comparable performance to the state-of-the-art
optimal solutions.

##### Synthesis and Reflections

This section reviews the applications of DRL for dynamic spectrum allocation
and access control in modern wireless networks. These types of radio resources
are inherently coupled with user association, network/RAT selection, dynamic
multi-channel access, and DSA. Table VI summarizes the reviewed papers in this
section. In general, the application of DRL for spectrum allocation and access
control problems has received considerable attention lately. We observe that
most DRL algorithms, when deployed for non-IoT networks, are implemented in
centralized fashions at network controllers, such as BSs, RSUs, and satellites
[116, 132, 145]. This is done to utilize the controllers’ powerful and
advanced hardware capabilities in collecting network information and designing
cross-layer policies. Hence, we observe that DRL models are deployed as a
single-agent at the network controllers [140]. On the contrary, DRL provides a
flexible tool in diversified IoT networks and systems, conventionally
involving dynamic system modeling and multi-agent interactions, such as CRNs
and distributed systems.

In addition, the management of such types of radio resources falls in the
discrete action space. Therefore, the value-based algorithms are utilized more
than the policy-based ones. We also observe from Table VI that the use of DRL
techniques for IoT and emerging wireless networks receives more attention than
other networks, especially for the cognitive radio-based systems as in [143].

The exponential increase in smart IoT devices mandates to take autonomous
decisions locally, especially for delay-sensitive IoT applications and
services. In this context, we anticipate that the research on spectrum
allocation and access control using distributed multi-agent DRL algorithms for
future IoT networks will attract more attention as in [130, 132, 142, 143]

TABLE VI: A Summary List of Papers Related to DRL for Spectrum Allocation and
Access Control.

|  |  | Learning Algorithm
---|---|---|---
Network Type | Ref. | Radio Resource (or Issues Addressed) | Mode | Algorithm
Cellular & HomNets | UAV-assisted LTE | Chen et al.[113] | Joint user association, spectrum allocation, & content caching | Multi-agent | DQN
Macro & femto BS | Zhao et al. [115, 114] | Joint BS & channel selections | Multi-agent | Dueling DDQN
Cellular IoT | Zhang et al.[116] | User association | Single-agent | DQN
IAB cellular | Lei et al.[117] | Dynamic spectrum allocation | Single-agent | DDQN & actor-critic
CV2X | Li et al.[118] | User association | Multi-agent |  $Q$-learning
5G HetNets | Perez et al.[119] | User association | Multi-agent |  $Q$-learning
Hybrid NOMA-based 5G | Zheng et al.[120] | Dynamic spectrum allocation | Single-agent | actor-critic
Wireless DSA | Song et al.[121] | Dynamic spectrum allocation | Multi-agent | DQN
IoT & Other Emerging Wireless Networks | IoT sensor networks | Wang et al.[122, 123, 124] | Dynamic multi-channel access | Single-agent | DQN
Energy harvesting-based IoT sensors | Chu et al.[126] | Dynamic spectrum allocation | Single-agent | DQN
CRNs | Zhang et al.[127] | Dynamic multi-channel access | Multi-agent | DQN
Wireless networks | Xu et al.[128] | DSA | Single-agent | DQN & DDQN
V2X | Liang et al.[130, 129] | Dynamic spectrum sharing | Single-& multi-agent | DQN
Multi-sensor scenario | Zhu et al.[131] | Joint channel access & packet forwarding | Single-agent | DQN
mmWave V2X | Khan et al.[132] | User association | Multi-agent | A3C
CRNs | Yang et al.[133] | DSA | Multi-agent | DQN
NTNs | Cao et al.[134] | Dynamic multi-channel access | Multi-agent | DQN
CRNs | Yang et al.[135] | Joint dynamic spectrum access & mode selection | Multi-agent | DQN
Multi-channel wireless networks | Tomovic et al.[136] | DSA | Single-agent | RNN-base DDQN
Wireless networks | Zhong et al.[137] | Dynamic multi-channel access | Single-& multi-agent | actor-critic
CRNs | Tondwalkar et al.[138] | DSA | Multi-agent | DQN
IIoT networks | Shi et al.[140] | Dynamic spectrum sharing | Single-agent | DQN
Distributed wireless networks | Janiar et al.[141] | DSA | Multi-agent | DDQN
VANETs | Wang et al.[142] | DSA | Multi-agent | RNN-based DQN
CUAV | Jiang et al.[143] | Joint spectrum sensing & channel access | Multi-agent | DQN
Heterogeneous environments | Xu et al.[144] | DSA | Multi-agent | RNN-based DQN
Satellite Nets | Multi-beam satellite systems | Liu et al.[145] | Dynamic spectrum allocation | Single-agent | DQN
Multi-beam satellite systems | Hu et al.[146] | Dynamic spectrum allocation | Single-agent | DQN
SIoT | Zhao et al.[147] | Dynamic spectrum allocation | Single-agent | DQN
SIoT | Liu et al.[148] | Dynamic spectrum allocation | Single-agent | DQN
| LEO satellites | Zheng et al.[149] | Joint channel pre-allocation & dynamic channel scheduling | Single-agent |  $Q$-learning
Multi-RAT | Small BSs cellular | Challita et al.[150] | Joint dynamic channel selection & interference management | Multi-agent | DQN
Small BSs cellular | Naparstek et al.[151] | Dynamic spectrum allocation | Single-agent | DQN
LTE-WLAN HetNets | Wang et al.[152] | Dynamic multi-channel access | Single-agent | prediction-DDPG
MEC-based V2X | Peng et al.[153] | Joint allocation of spectrum, computing, & storing | Multi-agent | DDPG
4G LTE and 5G NR systems | Challita et al.[154] | Dynamic spectrum sharing | Single-agent | DQN

###  IV-C DRL for Rate Control

This refers to the adaptive rate control in the uplink and downlink of
wireless networks. With the explosive increase in the number of user devices
and the emergence of massive types of data-hungry applications [3], it becomes
essential to keep high network KPIs in terms of data rates and users’ QoE.
Adaptive rate control schemes must ensure satisfactory QoS in highly dynamic
and unpredictable wireless environments. In this context, DRL techniques can
be efficiently deployed to solve adaptive rate control problems instead of
conventional approaches that possess high complexity and heavily rely on
accurate network information and instantaneous CSI.

In the following paragraphs, we review works that employ DRL algorithms to
address the rate control issue in cellular and HomNets.

Liu et al. [155] address the problem of network resource allocation, in terms
of rate, for 5G network slices. The problem is decomposed into a master-slave,
and a multi-agent DDPG-based DRL scheme is then proposed to solve it. The
agents are located in every network slice, whose action space is continuous,
defining the resource allocation to users in the network slice. The state
space is continuous and has two main parts; the first one shows how much
utility the user obtained compared to its minimum utility requirement, while
the second part shows the auxiliary and dual variables from the master
problem. The reward is a continuous function defined in terms of utility,
utility requirements, and auxiliary and dual variables. Simulation results
demonstrate that their proposed algorithm outperforms the baseline approaches
and gives a near-optimal solution.

In [156], the authors propose a single-agent DQN-based DRL algorithm to
address the problem of dynamic uplink/downlink radio resources allocation in
terms of network capacity in high mobility 5G HetNets. Their proposed
algorithm is based on Time Division Duplex (TDD )configuration in which the
agent is the BS, whose action space is discrete, corresponding to the
configurations of TDD sub-frame allocation at the BS. The sate space is
discrete, comprised of different kinds of features of the BS, including
uplink/downlink occupancy, buffer occupancy, and channel condition of all
uplinks/downlinks to the BS. The reward is discrete, defined as a function of
the uplink and downlink channel utility, which mainly depends on the channel
occupancy with chosen TDD configuration. Using experimental results, the
authors show that their proposed algorithm achieves performance improvement in
terms of both network throughput and packet loss rate, compared to some
conventional TDD resource allocation algorithms.

##### Synthesis and Reflections

This section reviews the use of DRL techniques for adaptive rate control in
modern wireless networks. In general, there is limited research that is solely
dedicated to addressing the rate radio resource. Most of the research is
dedicated to video streaming applications, as summarized in [15]. However, as
we discussed in the previous sections, the data rate control issue is
typically addressed via controlling other radio resources such as power and
spectrum. In addition, the adaptive rate control is typically addressed as a
joint optimization with other radio resources, as we will elaborate in the
next section, e.g., as in [157, 158].

We also observe that DRL approaches for cellular and HomNets receive more
attention than other wireless networks, and there is a lack of research on
adaptive rate control for IoT and satellite networks. This also deserves more
in-depth investigation and analysis.

###  IV-D DRL for Joint RRAM

Due to the massive complexity and large-scale nature of modern wireless
networks, it becomes necessary to design efficient schemes that account for
the joint radio resources. In many scenarios, the design problem in wireless
networks might end up with competing objectives. For example, in UDNs,
increasing the power level is beneficial in combating path loss and enhancing
the received signal quality. However, this might cause serious interference to
the neighboring user devices and BSs. Hence, the joint design of power level
control and interference management becomes mandatory. Conventional approaches
for solving joint RRAM problems require complete and instantaneous knowledge
about network statistics, such as traffic load, channel quality, and radio
resources availability. However, obtaining such information is not possible in
such large-scale networks. In this context, DRL techniques can be adopted to
learn system dynamics and communication context to overcome the limited
knowledge of wireless parameters.

This section intensively reviews the works that implement DRL algorithms for
the problem of joint RRAM in modern wireless networks. Particularly, we
present related works that jointly optimize the radio resources shown in Fig.
2, such as power allocation, spectrum resources, user association, dynamic
channel access, cell selection, etc.

####  IV-D1 In Cellular and HomNets

In the following paragraphs, we review works that employ DRL algorithms to
address the joint RRAM issue in cellular and HomNets shown in Fig. 2.

The work in [159] presents a single-agent DDPG-based DRL model to study the
joint optimization of user mode selection, bandwidth allocation, power
control, and channel selection with the aim for maximizing the EE in
D2D-enabled cellular HetNets. The agent is a controller in the macrocell,
whose state space is continuous, representing the user’s QoS satisfaction
degree. The action space is continuous, which is to select power, channel, and
bandwidth allocation factor, while the reward is a continuous function defined
by the system EE. Experimental results demonstrate the efficiency of their
proposed algorithm in terms of both the convergence speed and improving the EE
in a D2D-enabled HetNets compared with other benchmark schemes.

In another interesting work in [160], the authors study the problem of joint
optimization of transmission mode selection and resource allocation for CV2X.
They propose single-agent settings in which DQN and federated learning (FL)
models are integrated to improve the model’s robustness. The agent in their
model is each V2V pair. The action space is discrete, representing the
resource block (RB) allocation, communication mode selection, and transmit
power level of the V2V transmitter. The state space is a hybrid (continuous
and discrete) consisting of five main parts; the received interference power
at the V2V receiver and the BS on each RB at the previous subframe, the number
of selected neighbors on each RB at the previous subframe, the large-scale
channel gains from the V2V transmitter to its corresponding V2V receiver and
the BS, current load, and remaining time to meet the latency threshold. The
reward is a continuous function defined in terms of the sum-capacity of
vehicular UEs as well as the QoS requirements of both vehicular UEs and V2V
pairs. Using experimental results, the authors show that their proposed two-
timescale federated DRL algorithm outperforms other decentralized baselines.

Jang et al. [161] propose a multi-agent DQN-based algorithm to address the
problem of sum-rate maximization via a joint optimization of resource
allocation and power control in small cell wireless networks. The agents in
their proposed model are the small cell BSs, whose action space is discrete,
corresponding to selecting the resource allocation and power control of small
BS on RB. The state space is continuous, including all the CSI that the small
BS collects on RB, such as local CSI, local CSI at the transmitter, etc. The
reward is a continuous function expressed by the average sum-rate of its own
serving users and the other small BSs. Experimental results show that their
proposed approach both outperforms the conventional algorithms under the same
CSI assumptions and provides a flexible tradeoff between the amount of CSI and
the achievable sum-rate.

In another interesting work in [162], the authors propose a model-driven
multi-agent Double DQN-based framework for resource allocation in UDNs. In
particular, They first develop a DNN-based optimization framework comprised of
a series of ADMM iterative procedures that uses the CSI as the learned
weights. Then, a channel information absent $Q$-learning resource allocation
algorithm is presented to train the DNN-based optimization scheme without
massive labeling data, where the EE, SE, and fairness are jointly optimized.
The agents are each D2D transmitter, whose action space is discrete,
corresponding to selecting a subcarrier and corresponding transmission power.
The state space is a hybrid (continuous and discrete) consisting of two parts;
user association information and interference power. The reward function is
continuous, comprised of two components; the network EE and the fairness of
service quality, which is expressed by the variance of throughput between
authorized users. Using experimental results, it is demonstrated that their
proposed algorithm has a rapid convergence speed, well characterizes the
extent of optimization objective with partial CSI, and outperforms other
existing resource allocation algorithms.

Liu et al. [163] propose a single-agent Dueling DQN-based DRL model to address
the problem of joint energy efficiency (EE) and spectral efficiency (SE) in 5G
ultra-dense networks (UDNs). In their approach, the agent is a macro gNodeB
(MgNB) whose state space is continuous, comprised of the number and throughput
of all small cells as well as the allocation of all resource blocks (RBs) in
the network. The agent’s action space is to define which RBs are reused by the
small cell in the network, while the reward is a function of the tradeoff
between SE and EE. Experimental results demonstrate that their proposed
algorithm performs better than both the conventional $Q$-learning and DQN.

The authors in [164] propose a multi-agent Double DQN-based scheme to address
the problem of joint subcarrier assignment and power allocation in D2D
underlying 5G cellular networks. The agents in their model are the D2D pairs,
whose action space is discrete, corresponding to determining the transmit
power allocation on the available subcarriers. The state space is a hybrid
(continuous and discrete), comprised of four components: 1) local information
(including the previous transmit power, previous SE achieved by transmitters,
channel gain, and SINR), 2) the interference that each agent causes at the BS
side, 3) the interference received from agent’s interfering neighbors and the
SE achieved by agent’s neighbors, and 4) the interference that each agent
causes to its neighbors. The reward is a continuous function comprised of
three elements: 1) the SE achieved by each agent, 2) the SE degradation of the
agent’s interfered neighbors, and 3) the penalty due to the interference
generated at the BS. Experimental results show that their proposed algorithm
outperforms both the exhaustive and random subcarrier and even power (RSEP)
assignment methods in terms of SE of D2D pairs.

In previous work in [165], the authors propose a multi-agent $Q$-learning-
based scheme to address the problem of joint channel and power level selection
in autonomous D2D-based heterogeneous cellular networks. Two spectrum usage
scenarios are considered; when the D2D pairs transmit over the dedicated
spectrum bands and when they shared cellular/D2D channels. The agents are the
D2D pairs whose action space is discrete, defined as the set of their possible
channel and power level decisions. The state space is also discrete, which
takes the value of ”1” if the SINR is greater than some predefined threshold;
otherwise, the value of the state is ”0”. The agent’s reward is a continuous
function, defined by the current agent’s state and its partially observable
actions. Experimental results show that their proposed algorithm has
relatively fast convergence and near-optimal performance after a few number of
iterations.

Most recently in [166], the authors propose a multi-agent DQN-based DRL scheme
to address the problem of spectrum allocation and power control for Mission-
critical communication (MCC) in 5G networks. In MCC, multiple D2D users reuse
non-orthogonal wireless resources of cellular users without BS in order to
enhance the network’s reliability. The paper aims to help the D2D users
autonomously select the channel and allocate power to maximize system capacity
and SE while minimizing interference to cellular users. The agents are the D2D
transmitters whose action space is discrete, corresponding to channel and
power level selection. The state space is discrete, defined in a three-
dimensional matrix, which includes information on the channel state of uses,
the state of power level, and the number of the D2D pairs. The reward function
is discrete, defined in terms of the total system capacity and constraints.
Simulation results show that their proposed learning approach significantly
improves spectrum allocation and power control compared to traditional
methods.

The authors in [167] propose a multi-agent DQN-based model to address the
problem of joint user association and power control in OFDMA-based wireless
HetNet. The agents are the UEs, whose action space is discrete, corresponding
to jointly associate with the BS and determine the transmit power. The state
space is discrete, which is defined by the situation of all UEs association
with BS and power control. The reward function is continuous, which is defined
in terms of the sum-EE of all UEs. Using simulation results, it is shown that
their proposed method outperforms the $Q$-learning method in terms of
convergence and EE.

The authors in [168] propose a decentralized multi-agent DQN-based scheme to
address the problem of joint channel resource and transmit power in D2D
underlay cellular networks. In their scheme, the agents are each D2D
transmitter. The action space is discrete, corresponding to the selection of
each D2D’s channel resource and transmit power for data transmission. The
state space is discrete consisting of six elements; the transmit power level
of the cellular users, the maximum transmit power of each D2D transmitter, the
distances between the cellular users and the D2D receiver devices, distances
between D2D transmit devices and D2D receiver devices, selected channel of
other agents, and selected transmit power level of other agents. The reward is
a continuous function defined by the total effective throughput of D2D pairs
in the network. Simulation results show that their proposed algorithm achieves
better performance in terms of the total effective data rate than the random
resource allocation method.

In [169], a single-agent DQN-based DRL model is proposed to address the
problem of joint optimization of user association, resource allocation, and
power allocation in HetNets. The agent is the BS, whose action is discrete,
corresponding to power allocation to users. The state space is discrete,
defined by the channel gain matrix and the set of users association. The
reward function is continuous, defined by the utility function of users’
achieved data rate. Using simulation results, the authors show that their
proposed algorithm outperforms some of the existing methods in terms of SE and
convergence speed.

Huang et al. [170] propose a single-agent DQN model to address the problem of
delay minimization via joint spectrum and power resource allocation in mmWave
mobile hybrid access network. The agent is located in the roadside BS, whose
action space is discrete, corresponding to allocating spectrum and power
resources for data. The state space is discrete, consisting of information
about the current power and spectrum of the resource pool, required spectrum
and power, and the number of spectrum and power levels. The reward signal is a
continuous function defined in terms of queueing delay and the resource length
required for each data. Using simulation results, it is shown that their
proposed model guarantees the URLLC delay constraint when the load does not
exceed 130%, which outperforms other conventional methods such as random and
greedy algorithms.

####  IV-D2 In IoT and Other Emerging Wireless Networks

In the following paragraphs, we review works that employ DRL algorithms to
address the joint RRAM issue in IoT and emerging wireless networks depicted in
Fig. 2.

For the same system settings in [102], the authors in [64] extended their work
and propose a multi-agent DDPG-based DRL framework to address the problem of
the joint spectrum and power allocation in wireless networks. Two DRL-based
algorithms are proposed, which are executed and trained simultaneously in two
layers in order to jointly optimize the discrete subband selection and
continuous power allocation. The agent in their approach is each transmitter.
In the top layer, the action space of all agents is discrete, representing the
discrete subband selection, while the bottom layer has a continuous action
space corresponding to the transmit power allocation. The state space is a
hybrid (continuous and discrete), containing information on achieved SE,
transmit power, sub-band selection, rank, and downlink channel gain. The
reward is shared by both layers, which is a continuous function defined in
terms of the externality of agents to interference and the spectral
efficiency. Using experimental results, the authors show that their proposed
framework outperforms the conventional fractional programming algorithm.

Based on their initial work in [171], the authors in [172] extended their work
and propose a distributed multi-agent DQN-based DRL scheme to address the
problem of joint channel selection and power control in D2D networks. The
agents in their model are the D2D pairs, whose action space is discrete,
corresponding to selecting a channel and a transmit power. The state space of
each agent is a hybrid (continuous and discrete) which contains three sets of
information; local information, non-local information from the agent’s
receiver-neighbor set, and non-local information from the agent’s transmitter-
neighbor set. The reward function of each agent is continuous, which is
decomposed into the following elements; its own received signal power, its own
total received SINR, its interference caused to transmitter-neighbors, the
received signal power, and the total received SINR of transmitter-neighbors.
Using simulation results, it is shown that the performance of their scheme
closely approaches that of the FP-based algorithm even without knowing the
instantaneous global CSI.

In [173], the authors extended their previous work in [174, 175] and present a
distributed multi-agent DQN-based model to address the problem of joint sub-
band selection and power level control in V2V communication networks. Their
proposed model is applicable to both unicast and broadcast scenarios. The
agents are the V2V link or vehicles whose action space is discrete,
corresponding to the selection of the frequency band and transmission power
level that generate small interference to all V2V and V2I links while ensuring
enough resources to meet latency constraints. The state space is continuous,
containing the following information; the CSI of the V2I link, the received
interference signal strength in the previous time slot, the channel indices
selected by neighbors in the previous time slot, the remaining load for
transmission, and the time left before violating the latency constraint. The
reward function is continuous, consisting of three components; the capacity of
V2I links, the capacity of V2V links, and the latency constraint. Using
experimental results, it is shown that agents learn to satisfy the latency
constraints on V2V links while minimizing the interference to V2I
communications.

The authors in [176] propose a single-agent Double DQN-based DRL to address
the problem of joint channel selection and power allocation with network
slicing in CRNs. The aim of their study is to provide high SE and QoS for
cognitive users. The agent is the overall CRN, whose action space is discrete,
corresponding to the channel selection and power allocation of SUs. The state
space is continuous, defined by the SINR of the PU. The reward function is
continuous, which is a function of the system SE, user QoS, interference
temperature, and the interference temperature threshold. Experimental results
show that their proposed algorithm improves the SE and QoS and provides faster
convergence and more stable performance compared to the $Q$-learning and DQN
algorithms.

TABLE VII: Summary of the Related Works that Address the Joint RRAM.

|  |  | Learning Algorithm
---|---|---|---
Network Type | Ref. | Types of Joint Radio Resources (or Issues Addressed) | Mode | Algorithm
Cellular & HomNets | D2D-enabled cellular | Zhang et al.[159] | User mode selection, bandwidth allocation, power control, & channel selection | Single-agent | DDPG
CV2X | Zhang et al.[160] | Transmission mode selection & resource allocation | Single-agent | DQN
Small cell networks | Jang et al.[161] | Resource allocation & power control | Multi-agent | DQN
UDNs | Liao et al.[162] | Subcarrier selection & transmission power | Multi-agent | DDQN
5G UDNs | Liu et al.[163] | EE and SE | Single-agent | Dueling DQN
D2D underlying 5G cellular | Zhang et al.[164] | Subcarrier assignment & power allocation | Multi-agent | DDQN
D2D-based heterogeneous cellular | Asheralieva et al.[165] | Channel & power level selection | Multi-agent |  $Q$-learning
Mission-critical in 5G | Wang et al.[166] | Spectrum allocation & power control | Multi-agent | DQN
OFDMA-based networks | Ding et al.[167] | User association & power control | Multi-agent | DQN
D2D underlay cellular | Yu et al.[168] | Channel resource & transmit power | Multi-agent | DQN
HetNets Cellular | Zhang et al.[169] | User association, resource allocation, & power allocation | Single-agent | DQN
mmWave mobile hybrid access | Huang et al.[170] | Spectrum & power resource allocation | Single-agent | DQN
IoT & Emerging Nets | Wireless networks | Nasir et al.[64] | Spectrum & power allocation | Multi-agent | DDPG
D2D networks | Tan et al.[172, 171] | Channel selection & power control | Multi-agent | DQN
V2V networks | Ye et al.[173, 174, 175] | Sub-band selection & power level control | Multi-agent | DQN
CRNs | Yuan et al.[176] | Channel selection & power allocation | Single-agent | DDQN
5G-based NOMA systems | Zhang et al.[177] | Subcarrier assignment & power allocation | Multi-agent | DQN & DDPG
Multi-carrier NOMA-based systems | He et al.[178] | Channel assignment & power allocation | Single-agent | DQN
Wireless networks | Jiang et al.[179] | Channel selection & power allocation | Multi-agent | DQN
NOMA-based V2X networks | Xu et al.[180] | Spectrum & power allocation | Multi-agent | DDPG
UAV-assisted IoT networks | Munaye et al.[158] | Bandwidth, throughput, & power | Multi-agent | DQN
Multi-RAT | Hybrid RF/VLC systems | Shrivastava et al.[181] | Bandwidth, power, & user association | Multi-agent | DQN
| Multi-RAT HetNets | Alwarafy et al.[39] | RAT selection & power control | Single& multi-agent | DQN & DDPG
| Heterogeneous health systems | Chkirbene et al.[157] | RAT selection, data split control, & compression ratio control | Single-agent | DDPG

The problem of joint subcarrier assignment and power allocation in an uplink
multi-user 5G-based NOMA systems is addressed in [177]. A multi-agent two-step
DRL algorithm is proposed; the first step employs the DQN algorithm to output
the optimum subcarrier assignment policy, while the second step employs the
DDPG algorithm to dynamically allocate the transmit power for the network’s
users. The agent is a controller located at the BS, whose action space is a
hybrid (discrete and continuous), corresponding to the subcarrier assignment
decisions and power allocation decisions. The state space is continuous, which
is defined by the users’ channel gains at each subcarrier. The reward function
is defined as the sum EE of the NOMA system. Experimental results show that
their proposed algorithm provides better EE than the fixed and DQN-based power
allocation schemes.

In [178], the authors propose a single-agent DQN-based algorithm to address
the problem of joint channel assignment and power allocation in multi-carrier
NOMA-based systems. The authors used an attention-based neural network (ANN)
to perform the channel assignment process. The agent in their approach is the
BS, whose action is discrete, representing the selection of a channel for a
user. The state space is discrete, characterized by the channel information
that is defined by user-channel assignment pairs. The reward function is
continuous, defined by the user’s achieved data rate. Using simulation
results, the authors show that their proposed algorithm achieves better system
performance in terms of sum-rate, compared to the state-of-the-art exhaustive
search approaches and the method reported in [182].

In another work [179], the authors address the problem of dynamic optimization
of channel selection and power allocation in wireless networks based on
distributed multi-agent DQN algorithm. The authors adopted a partially
distributed framework with the aim of minimizing the long-term average data
backlog of the network’s users. In their model, the agents are the users whose
action space is discrete and finite, representing the users’ transmit power
allocation for data blocks in the data buffer. The state space is also
discrete and finite comprised of two parts; the users’ data size and data
block remaining lifetime. The reward function is continuous, defined in terms
of the size of the remaining data in the data block and the transmitted data
taken from the data block. Experimental results demonstrate that their
proposed solution outperforms the conventional exhaustive search and Block
Coordinate Descent-based optimization algorithms in terms of EE and data
backlog.

The authors in [180] propose a multi-agent DDPG-based model to address the
problem of joint power and spectrum allocation in NOMA-based V2X networks. In
particular, the authors are looking to maximize the sum-rate of V2I
communications. The agents are the V2V communication links. The state space is
discrete, defined by a set of actions performed by V2I and V2V communication
links. The set includes the transmit power of both V2I and V2V links as well
as the spectrum multiplexing factor of both V2V and V2V links. The state space
is continuous, defined by five parts; the local channel gain information of
each V2V link, interference channels from other V2V communication links,
interference channel from each link’s own transmitter to the BS, interference
channel from all V2I transmitters, and the state of queue length in the buffer
of each V2V transmitter. The reward function is continuous, defined by the
achieved sum-rate of V2I communication links and the delivery probability of
V2V communication links. Compared with both the DQN algorithm and random
resource allocation scheme, simulation results show that their proposed
algorithm outperforms both of them in terms of maximizing the sum-rate of V2I
communication links while meeting the latency and reliability requirements of
V2V communications.

Munaye et al. [158] propose a multi-agent DQN-based DRL model to address the
problem of joint radio resources of bandwidth, throughput, and power in UAV-
assisted IoT networks. The agents are the UAVs, whose action space is
discrete, corresponding to jointly selecting channel allocation of bandwidth,
throughput, and power. The state space is discrete, comprising three
components; the air-to-ground channel used by users, the rate of power
consumption, and the interference. The reward is a discrete function, defined
in terms of throughput, power allocation, bandwidth, and SINR levels.
Simulation results show that their proposed algorithm outperforms other
algorithms in terms of accuracy, convergence speed, and error.

####  IV-D3 In Multi-RAT HetNets

In the following paragraphs, we review works that employ DRL algorithms to
address the joint RRAM problem in multi-RAT HetNets. This includes the
coexistence of various variants of the wireless networks as illustrated in
Fig. 2.

Most recently, the authors in [181] present a multi-agent DQN-based algorithm
to address the problem of joint optimization of bandwidth, power, and user
association in hybrid RF/VLC systems. The APs are the agents whose action is
discrete, representing the bandwidth, association function, and power level.
The state space is discrete, which is a function of the problem constraints
such as system bandwidth, association function, and power levels. The reward
is discrete, which is a function of the rates delivered by the APs.
Experimental results show that their algorithms improve the achievable sum-
rate and number of iterations for convergence by 10% and 54% compared to that
obtained using conventional optimization approaches.

Another interesting work is proposed by Alwarafy et al. in [39]. In that work,
the authors propose a hierarchical multi-agent DQN and DDPG-based algorithm to
address the problem of sum-rate maximization in multi-RAT multi-connectivity
wireless HetNets. The authors addressed the problem of multi-RATs assignment
and continuous power allocation that maximize the network sum rate. In their
model, single and multi-agents are proposed. The edge server acts as a single
agent employed by DQN, while RATs APs behave as multi-agents employed by DDPG.
For the single-agent DQN model, the action space is discrete, corresponding to
the RATs-EDs assigning process. The state space of the DQN is continuous,
comprised of the link gains and the required data rates of users. The reward
function of the DQN agent is continuous, defined by the difference between the
achieved rate and the required rate by users. For the multi-agent DDPG models,
the action space is continuous, representing the power allocation of each RAT
AP agent. The state space is a hybrid (continuous and discrete) consisting of
four elements: the RATs-EDs assignment process performed by the DQN agent, the
minimum data rate of users, the gains of the links, and the achieved data
rate. Experimental results show that their algorithm’s performance is
approximately 98.1% and 95.6% compared to the conventional CVXPY solver that
assumes full knowledge of the wireless environment.

An interesting work is presented in [157], in which the authors address the
problem of network selection with the aim of optimizing medical data delivery
over heterogeneous health systems. In particular, an optimization problem is
formulated in which the network selection problem is integrated with adaptive
compression to minimize network energy consumption and latency while meeting
applications’ QoS requirements. A single-agent DDPG-based DRL model is
proposed to solve it. The agent is a centralized entity that can access all
radio access networks (RANs) information and Patient Edge Node (PEN) data
running in the core network. The action space is discrete, corresponding to
the joint selection of data split over the existing RANs and the adequate
compression ratio. The state space is a hybrid (continuous and discrete)
defined by two elements: the fraction of time that the PENs should use over a
particular RAN and the PEN investigated in the current timestamp. The reward
is a continuous function, which is defined in terms of: the fraction of data
of PEN that will be transferred through RAN, the energy consumed by PEN to
transfer bits over RAN, distortion, expected latency of RANs, the monetary
cost of PENs to use RANs, the resource share, the fraction of time that the
PENs should use over a particular, and the data rate. Simulation results
demonstrate that their proposed scheme outperforms the greedy techniques in
terms of minimizing energy consumption and latency while satisfying different
PENs requirements.

##### Synthesis and Reflections

This section reviews the use of DRL methods for joint radio resources. Table
VII summarizes the reviewed papers in this section. We observe that DRL tools
can be efficiently deployed to address different types of joint radio
resources for diversified network scenarios. The results obtained using DRL
models are better than the heuristic methods [158, 180] and comparable to the
state-of-the-art optimization approaches [172].

We also observe that multi-agent DRL deployment based on value-based
algorithms receives more attention than policy-based algorithms. The reason is
that users tend to have more control over their channel selection, data
control, and transmission mode selection, and hence we find a more popular
implementation of DRL agents at local IoT devices. In addition, the
integration of value-based and policy-based algorithms for joint RRAM is also
an interesting concept as presented in [177, 39], which requires more
investigation, especially for multi-agent deployment scenarios.

We also observe that DRL methods for cellular and HomNets as well as IoT
wireless networks gain more attention than multi-RAT networks, particularly
for D2D communications. In addition, there is a lack of research on
applications of DRL for emerging IoT applications, such as healthcare systems
as investigated recently in [157], which is also a promising field that
requires more attention. Furthermore, we observe a lack of research on DRL
applications for joint RRAM in satellite networks, which also deserves more
in-depth investigation.

##  V Open Challenges and Future Research Directions

Throughout the previous section, we have demonstrated the superiority of DRL
algorithms over traditional methods in addressing complex RRAM problems for
modern wireless networks. However, there are still several challenges and open
issues that either not explored yet or need further exploration. This section
provides highlights these open challenges and provides insights on future
research directions in the context of DRL-based RRAM for next generation
wireless networks.

####  V-1 Open Challenges

#####  V-1a Centralized vs. Decentralized RRAM Techniques

Future wireless networks are characterized mainly by their massive
heterogeneity in wireless RANs, the number of user devices, and types of
applications. Centralized DRL-based RRAM schemes are efficient in guaranteeing
enhanced network QoS and fairness in allocating radio resources. They also
ensure that RRAM optimization problems will not get stuck in local minima due
to their holistic view of the system. However, formulating and solving RRAM
optimization problems become tough tasks in such large-scale HetNets. Hence,
centralized DRL-based RRAM solutions are typically unscalable. This motivates
distributed multi-agent DRL-based algorithms that enable edge devices to make
resource allocation decisions locally. Stochastic Game-based DRL algorithms
are one promising research direction in this context [22]. However, the rapid
increase in the number of edge devices (players) makes information exchange in
such networks unmanageable. Also, the partial observability of agents might
lead to suboptimal RRAM policies. Therefore, it is an open challenge to
develop DRL-assisted algorithms that optimally balance between the
centralization and distribution issue in RRAM.

#####  V-1b Dimensionality of State Space in HetNets

In modern wireless HetNets, service requirements and network conditions are
rapidly changing. Hence, single-agent DRL algorithms must be designed to
capture and respond to these fast network changes. To this end, it is required
to reduce the state space and action space during the learning process, which
inevitably degrades the quality of the learned policies. The existence of
multi-agents and their interactions will also complicate the agents’
environment and prohibitively increase the dimensionality of state space,
which will slow down the learning algorithms.

#####  V-1c Reliability of Training Dataset

Although the DRL-based solutions for RRAM we reviewed previously demonstrate
efficient performance results, almost all the models are developed based on
simulated training and testing datasets. The simulated dataset is typically
produced based on some stochastic models, which provide simplified versions of
practical systems and greatly ignore hidden system patterns. This methodology
greatly weakens the reliability of the developed policies as their performance
on practical networks would be skeptical. Hence, it is imperative to develop
more effective and reliable approaches that generate precise simulation
datasets and capture practical system settings as much as possible. This
ensures high reliability and confidence during the training and testing modes
of the developed RRAM policies. Developing such approaches is still a
challenge due to the large-scale nature and rapid variations of future
wireless environments.

On the other hand, the DRL models are sensitive to any change in the input
data. Any minor changes in the input data will cause considerable change in
the models’ output. This mainly deteriorates the reliability of DRL
algorithms, especially when deployed for modern IoT applications that require
ultra-reliability, such as remote surgery or any other mission-critical IoT
applications. Hence, ensuring high reliability for DRL models is a challenging
issue.

#####  V-1d Engineering of DRL Models for RRAM

Since DRL employs DNNs as function approximators for the reward functions, DRL
models will inherit some of the challenges that exist in the DNN world. For
example, it is still challenging to optimize the DNN hyperparameters, such as
the type of DNNs used (e.g., convolutional, fully connected, or RNN), the
number of hidden layers, the number of neurons per hidden layer, the learning
rate, etc. This challenge is even exacerbated in multi-agent settings as all
agents share the same radio resources and must converge simultaneously to some
policies.

On the other hand, the engineering of DRL parameters such as state space and
reward function is challenging for RRAM. The state space must be engineered to
capture useful and representative information about the wireless environment,
such as the available radio resources, users’ QoS requirements, channel
quality, etc. Such information is crucial and heavily defines the learning and
convergence behaviors of DRL agents. Again, the presence of multi-agents will
even make it more challenging, as discussed in [22]. Also, since DRL models
are reward-driven learning algorithms, the design of the reward function is
also essential to guide the agent during the policy-learning stage.
Formulating reward functions that capture the network objective and account
for the available radio resources is also challenging.

#####  V-1e System Dependency of DRL models

DRL models are system-dependent as they are trained and tested for specific
wireless environments and networks. Therefore, they provide effective results
when employed to solve specific types of problems for which they are trained.
However, if there would be a significant change in the characteristics of the
wireless environment or the nature of the RRAM problem, such as network
topology and available radio resources, the DRL model must be retrained as the
old model is no longer reflecting the new training experiences. In modern
wireless HetNets, such cases are frequently encountered, especially with real-
time applications or in highly dynamic environments. In such a case, it
becomes quite challenging for DRL agents to update and retrain their DNNs with
rapidly changing input information from the HetNet environment [1].

#####  V-1f Continuous Training of DRL Models

DRL algorithms require big datasets to train their models, which is typically
associated with a high cost [15]. The network system pays this cost during the
information collection process due to, e.g., the high delays, extra overhead,
and energy consumption. The emergence of a large number of real-time
applications and services has even increased this training cost. In this
context, DRL models require to be continuously retrained with fresh data
collected from the wireless environment to be up-to-date and to ensure
accurate and long-term control decisions. It is not practical to conduct
manual retraining of the models in such large-scale HetNets settings.
Therefore, continuous retraining is preferred, in which a dedicated autonomous
system can be employed to continuously assess and retrain old DRL models.
However, it would be quite challenging for the new autonomous system to
promptly perform this operation due to the environment’s rapid variations.
Also, monitoring and updating DRL models in multi-agent scenarios becomes an
expensive task.

#####  V-1g Context of RRAM

The implementation of DRL algorithms basically depends on the use-cases. The
context and deployment scenarios in which RRAM is required must be considered
during the development of DRL models. For example, RRAM in health-sector IoT
applications is different from the environmental IoT applications
counterparts. Due to the high sensitivity of data in the health-sector
applications, extra data pre-processing must be performed, including data
compression and encryption [157]. This will directly affect the number of
radio resources to be allocated for such applications. Hence, DRL models must
be aware of the context aspect of applications, which is considered another
challenge.

#####  V-1h Competing Objective Design of DRL Models

Next generation wireless networks are expected to provide enhanced system QoS
in terms of high data rate, high EE/SE, and reduced latency in order to
support the emerging IoT vital applications. Depending on the deployment
scenario, formulating multi-objective RRAM optimization problems usually ends
with many competing objectives and/or constraints. For instance, in cellular
UDNs, high resource utilization of, e.g., power allocation or channel may
cause severe interference. Also, for IoT applications such as vehicular
communications, we require to ensure ultra-reliable and low-latency
communication links, which are usually competing objectives. Therefore,
developing multi-objective DRL-based RRAM models that accommodate these
competing requirements is still a persisting challenge.

####  V-2 Future Research Directions

#####  V-2a DRL with Explainable AI (XDRL) for RRAM

Explainable AI (XAI) has recently emerged as an efficient technology to
improve the performance of DRL models. It is mainly envisioned to unlock the
”black-box” nature of conventional ML approaches and provide interpretability
for DRL models [183]. In particular, XAI explains the reasons behind certain
predictions made by DRL models (or ML models in general) by fully
understanding the precise working principle of these models. Hence, ensuring
trust, reliability, and transparency in the DRL algorithms’ policy devlopment
and decision-making processes. The research on XAI technologies in wireless
communication is still at its initial stages, and there are still some key
issues for future research in the context of RRAM for next generation wireless
networks. For example, DRL models can get stuck easily into local optimal
solutions when utilized to solve complex RRAM problems. This issue can be
significantly avoided with the help of XAI. Fortunately, the heterogeneity of
information in modern wireless HetNets will significantly help to achieve the
interpretation for DRL algorithms. In this context, developing RRAM schemes
for wireless HetNets, through entity recognition, entity-relationship
extraction, and representation learning, makes the DRL models’ interpretation
more reliable, accurate, and intuitive, which is a promising research
direction.

#####  V-2b Integrating DRL and Blockchain Techniques

Blockchain-based RRAM has emerged recently as one of the promising enabling
technologies for future wireless HetNets [3]. It has gained considerable
momentum lately due to its ability to provide intelligent, secure, and highly
efficient distributed resource sharing and management. The integration of DRL
with Blockchain is also an interesting research direction, as in [184, 185,
186]. For example, DRL algorithms can be distributively deployed within
participants or within the centralized spectrum-access systems to facilitate
spectrum auctions and transactions [186]. Also, many of the auction’s winner-
determination problems in future wireless HetNets are expected to be extremely
complex and intractable due to the massive increase in the number of
participants, e.g., bidders and sellers. Hence, DRL algorithms are efficient
tools that can be utilized to solve such types of problems.

#####  V-2c Federated DRL (FDRL)-Based RRAM

Federated learning (FL) framework is envisioned mainly to preserve data
privacy in ML algorithms [187, 188]. In FL, ML algorithms are locally
distributed at the wireless network edge, and the data is processed locally
and not shared globally. The local ML models are then utilized for training a
centralized global model. In this context, the federated DRL learning (FDRL)
scheme can be leveraged when many wireless user devices require making
autonomous local decisions. In such a case, DRL agents do not exchange their
local observations, and also, not necessarily all agents receive reward
signals [189].

TABLE VIII: Summary of Challenges and Future Research Directions in the Context of using DRL for RRAM in Future Wireless Networks. |  Developing DRL-based algorithms that optimally balance the centralization and distribution issue of RRAM in future large-scale massive HetNets.
---|---
|  Reducing the dimensionality of state space in distributed MADRL algorithms
during the learning process without slowing down or degrading the quality of
learned RRAM policies.
|  Developing more effective and reliable training approaches that generate
accurate simulation datasets and capture practical system settings.
|  Optimizing DRL models’ hyperparameters, especially in MADRL scenarios, and
engineering the state space and reward functions to capture representative
information about system dynamics.
|  Designing agile DRL algorithms that can quickly update and retrain the DNNs
in response to the rapid change of input information from the highly dynamic
HetNet environment.
|  Performing continuous retraining for the DRL models, especially MADRL, with
fresh data in future large-scale and rapidly changing wireless environments.
Open Challenges |  Developing DRL models that are aware of the context aspect and use cases of various emerging applications.
|  Developing DRL algorithms that accommodate competing multi-objectives
relevant to emerging applications.
|  Developing efficient and reliable DRL algorithms for RRAM in next-
generation HetNets based on the XAI concept through, e.g., entity recognition,
entity-relationship extraction, and representation learning.
|  Developing DRL-based blockchain techniques to address the problem of
distributed resource sharing and management for future large-scale HetNets,
e.g., to facilitate spectrum auctions and transactions, solving the problem of
auction’s winner-determination, etc.
|  Developing FDRL algorithms that ensure global solutions for complex RRAM
optimization problems while guaranteeing data and models privacy during
information sharing and models updating.
|  Developing DRL models to achieve intelligent load balancing in future self-
sustaining (or self-organization) HetNets.
|  Developing light-weighted and agile networked MADRL algorithms that enable
cooperation between agents with different heterogeneous reward functions and
adapt to environments with rapid mobility.
|  Developing ultra-reliable RRAM schemes by integrating DRL algorithms and
GANs techniques to support emerging IoT applications with high-reliability
demands.
Future Research Directions |  Developing end-to-end DRL-based algorithms that jointly optimize the configuration of RIS systems, i.e., elements’ phases and amplitudes, and radio resources of networks, e.g., downlink transmit power.

Developing fine-grained policies in DRL becomes challenging when the state
space is small and the training dataset is very limited [190]. In FDRL, the
direct exchange of data between agents is not possible as this will preach the
privacy promise of FL scheme. Instead, local DRL models can be developed and
trained for agents with the help of other agents while preserving users’ data
privacy, as in [191]. Hence, developing algorithms and schemes that guarantee
data and models privacy during both information sharing and models updating is
an interesting research direction.

FDRL framework can also be exploited in the RRAM of modern wireless networks.
For example, it can be deployed for solving complex wireless network
optimization problems, such as power control in cellular UDNs. In this
context, FDRL can ensure a global solution for complex network optimization
problems without sharing information between BSs; each BS solves its
optimization problem locally and shares the results with neighboring BSs.

#####  V-2d DRL-Based Load Balancing for Self-Sustaining Networks

Load balancing in modern wireless UDNs is another promising research
direction. The objective is to balance the wireless networks by moving some
users from the heavily congested BSs to uncongested ones, thus improving BSs
utilization and providing enhanced QoS provisioning. Although the load
balancing field has been heavily investigated in the literature using
conventional resource management approaches, as in [192, 193, 194], there
still a research gap in applying DRL for such a field. In this context, DRL
can be adopted to realize the self-sustaining (or self-organization) vision of
next generation wireless networks [3]. Hence, developing single/multi-agent
DRL models to achieve intelligent load balancing in future HetNets, is a
possible research direction.

#####  V-2e MADRL Algorithms in Support of Massive Heterogeneity and Mobility

In modern wireless networks, massive heterogeneity is one of the main enabling
technologies to provide ubiquitous coverage and enhance system reliability.
However, some mathematical frameworks, such as the Markov game, are basically
developed for homogeneous systems. To overcome this challenge, there are
proposals on adopting networked MADRL algorithms that enable the cooperation
between agents with different reward functions [22], which is an interesting
research direction. Also, RRAM for networks with high mobility, such as
vehicular and railway communications, is a persistent challenge. In this
context, developing light-weighted and agile MADRL algorithms that account for
and adapt to rapid network mobility is another interesting research direction.

#####  V-2f DRL-Based RRAM with Generative Adversarial Networks (GANs) for
RRAM

Ensuring the reliability of DRL algorithms is one of the major challenges and
objectives in DRL-based RRAM methods. In many real-life scenarios, we may
require to deploy DRL models to allocate resources in vital systems requiring
ultra-reliability, such as IoT healthcare applications [157]. In this context,
there are proposals on Generative Adversarial Networks (GANs), which have
emerged recently as an effective technique to enhance the reliability of DRL
algorithms [195].

In practice, the shortage of realistic training datasets that are required to
train DRL models and learn optimal policies is a challenging issue. To
overcome this, GANs are utilized, which generate large amounts of realistic
datasets synthetically by expanding the available limited amounts of real-time
datasets. From a DRL perspective, GANs-generated synthetic data is more
effective and reliable than traditional augmentation methods [67]. This is
because DRL agents will be exposed to various extreme challenging and
practical situations by merging the realistic and synthetic data, enabling DRL
models to be trained on unpredicted and rare events. Another advantage of GAN
over traditional data augmentation methods is that it eliminates dataset
biases in the synthetic data, which greatly enhances the quality of the
generated data and leads to more reliability in DRL models’ training and
learning processes.

In general, the research in the GANs-based DRL methods for RRAM is still in
its early stages, and we believe that this research direction will take
further pace in the future.

#####  V-2g DRL for RRAM in RIS-Assisted Wireless Networks

Reconfigurable Intelligent Surfaces (RIS) have emerged recently as an
innovative technology to enhance the QoS of future wireless networks [196,
197]. RIS can be deployed in cellular networks as passive reflecting elements
to provide near line-of-sight communication links to users, hence enhancing
communication reliability and reducing latency [198, 199]. Deploying RIS to
assist cellular communication, however, requires judicious RRAM schemes to
optimize network performance. This research field is still nascent, and there
is much to do for future research and investigation, especially in the context
of DRL-based RRAM techniques. Towards this, it is required to develop end-to-
end DRL-based algorithms that jointly optimize the configuration of the RIS
system, i.e., elements’ phases and amplitudes, and radio resources of BSs. For
instance, designing DRL models that intelligently and optimally allocate the
downlink BSs’ transmit power from one side and the amplitude and phase shifts
of the RIS elements on the other side is a promising research direction, as in
[41]. We also believe that the currently ongoing research in RIS-assisted
wireless networks, e.g., [41, 200, 201, 202] will be cornerstones.

Table VIII summarizes the open challenges and future research directions
provided in this section.

##  VI Conclusion

This paper presented a comprehensive survey on the applications of DRL
techniques in RRAM for next generation wireless HetNets. We thoroughly
reviewed the conventional approaches for RRAM, including their types,
advantages, and limitations. We have then illustrated how the emerging DRL
approaches can overcome these shortcomings to enable DRL-based RRAM. After
that, we illustrated how the RRAM optimization problems can be formulated as
an MDP before solving them using DRL techniques. Furthermore, we conducted an
extensive overview of the most efficient DRL algorithms that are widely
leveraged in addressing RRAM problems, including the value- and policy-based
algorithms. The advantages, limitations, and use-cases for each algorithm are
provided. We then conducted a comprehensive and in-depth literature review and
classified the existing related works based on both the radio resources they
are addressing and the type of wireless networks they are considering. To this
end, the types of DRL models developed in these related works and their main
elements are carefully identified. Finally, we outlined important open
challenges and provided insights into future research directions in the
context of DRL-based RRAM.

## Acknowledgment

This publication was made possible by NPRP-Standard (NPRP-S) Thirteen
($13^{\text{th}}$) Cycle grant # NPRP13S-0201-200219 from the Qatar National
Research Fund (a member of Qatar Foundation). The findings herein reflect the
work, and are solely the responsibility, of the authors.

## References

  * [1] F. Hussain, S. A. Hassan, R. Hussain, and E. Hossain, “Machine learning for resource management in cellular and IoT networks: Potentials, current solutions, and open challenges,” _IEEE Communications Surveys & Tutorials_, vol. 22, no. 2, pp. 1251–1275, 2020.
  * [2] K. B. Letaief, W. Chen, Y. Shi, J. Zhang, and Y.-J. A. Zhang, “The roadmap to 6G: AI empowered wireless networks,” _IEEE Communications Magazine_ , vol. 57, no. 8, pp. 84–90, 2019.
  * [3] W. Saad, M. Bennis, and M. Chen, “A vision of 6G wireless systems: Applications, trends, technologies, and open research problems,” _IEEE network_ , 2019.
  * [4] Z. Zhang, Y. Xiao, Z. Ma, M. Xiao, Z. Ding, X. Lei, G. K. Karagiannidis, and P. Fan, “6G wireless networks: Vision, requirements, architecture, and key technologies,” _IEEE Vehicular Technology Magazine_ , vol. 14, no. 3, pp. 28–41, 2019.
  * [5] “6G summit connecting the unconnected,” https://6gsummit.org, accessed: .
  * [6] G. Forecast, “Cisco visual networking index: global mobile data traffic forecast update, 2017–2022,” _Update_ , vol. 2017, p. 2022, 2019.
  * [7] I. Tomkos, D. Klonidis, E. Pikasis, and S. Theodoridis, “Toward the 6G network era: Opportunities and challenges,” _IT Professional_ , vol. 22, no. 1, pp. 34–38, 2020.
  * [8] P. Yang, Y. Xiao, M. Xiao, and S. Li, “6G wireless communications: Vision and potential techniques,” _IEEE Network_ , vol. 33, no. 4, pp. 70–75, 2019\.
  * [9] K. David and H. Berndt, “6G vision and requirements: Is there any need for beyond 5G?” _IEEE Vehicular Technology Magazine_ , vol. 13, no. 3, pp. 72–80, 2018.
  * [10] S. Elmeadawy and R. M. Shubair, “6G wireless communications: Future technologies and research challenges,” in _2019 International Conference on Electrical and Computing Technologies and Applications (ICECTA)_. IEEE, Conference Proceedings, pp. 1–5.
  * [11] T. Huang, W. Yang, J. Wu, J. Ma, X. Zhang, and D. Zhang, “A survey on green 6G network: Architecture and technologies,” _IEEE Access_ , vol. 7, pp. 175 758–175 768, 2019.
  * [12] L. Liang, H. Ye, G. Yu, and G. Y. Li, “Deep-learning-based wireless resource allocation with application to vehicular networks,” _Proceedings of the IEEE_ , vol. 108, no. 2, pp. 341–356, 2020.
  * [13] K. K. Nguyen, T. Q. Duong, N. A. Vien, N.-A. Le-Khac, and M.-N. Nguyen, “Non-cooperative energy efficient power allocation game in D2D communication: A multi-agent deep reinforcement learning approach,” _IEEE Access_ , vol. 7, pp. 100 480–100 490, 2019.
  * [14] R. S. Sutton and A. G. Barto, _Reinforcement learning: An introduction_. MIT press, 2018.
  * [15] N. C. Luong, D. T. Hoang, S. Gong, D. Niyato, P. Wang, Y.-C. Liang, and D. I. Kim, “Applications of deep reinforcement learning in communications and networking: A survey,” _IEEE Communications Surveys & Tutorials_, 2019\.
  * [16] Y. L. Lee and D. Qin, “A survey on applications of deep reinforcement learning in resource management for 5G heterogeneous networks,” in _2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_. IEEE, 2019, pp. 1856–1862.
  * [17] F. Obite, A. D. Usman, and E. Okafor, “An overview of deep reinforcement learning for spectrum sensing in cognitive radio networks,” _Digital Signal Processing_ , p. 103014, 2021.
  * [18] S. Gupta, G. Singal, and D. Garg, “Deep reinforcement learning techniques in diversified domains: A survey,” _Archives of Computational Methods in Engineering_ , pp. 1–40, 2021.
  * [19] Z. Du, Y. Deng, W. Guo, A. Nallanathan, and Q. Wu, “Green deep reinforcement learning for radio resource management: Architecture, algorithm compression, and challenges,” _IEEE Vehicular Technology Magazine_ , vol. 16, no. 1, pp. 29–39, 2021.
  * [20] Y. Qian, J. Wu, R. Wang, F. Zhu, and W. Zhang, “Survey on reinforcement learning applications in communication networks,” _Journal of Communications and Information Networks_ , vol. 4, no. 2, pp. 30–39, 2019.
  * [21] Z. Xiong, Y. Zhang, D. Niyato, R. Deng, P. Wang, and L.-C. Wang, “Deep reinforcement learning for mobile 5G and beyond: Fundamentals, applications, and challenges,” _IEEE Vehicular Technology Magazine_ , vol. 14, no. 2, pp. 44–52, 2019.
  * [22] A. Feriani and E. Hossain, “Single and multi-agent deep reinforcement learning for AI-enabled wireless networks: A tutorial,” _IEEE Communications Surveys & Tutorials_, pp. 1–1, 2021.
  * [23] M. Chen, U. Challita, W. Saad, C. Yin, and M. Debbah, “Artificial neural networks-based machine learning for wireless networks: A tutorial,” _IEEE Communications Surveys & Tutorials_, vol. 21, no. 4, pp. 3039–3071, 2019.
  * [24] A. Zappone, M. Di Renzo, and M. Debbah, “Wireless networks design in the era of deep learning: Model-based, AI-based, or both?” _IEEE Transactions on Communications_ , vol. 67, no. 10, pp. 7331–7376, 2019.
  * [25] H. Khorasgani, H. Wang, and C. Gupta, “Challenges of applying deep reinforcement learning in dynamic dispatching,” _arXiv preprint arXiv:2011.05570_ , 2020.
  * [26] G. S. Rahman, T. Dang, and M. Ahmed, “Deep reinforcement learning based computation offloading and resource allocation for low-latency fog radio access networks,” _Intelligent and Converged Networks_ , vol. 1, no. 3, pp. 243–257, 2020.
  * [27] A. Mohammed, H. Nahom, A. Tewodros, Y. Habtamu, and G. Hayelom, “Deep reinforcement learning for computation offloading and resource allocation in blockchain-based multi-UAV-enabled mobile edge computing,” in _2020 17th International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP)_. IEEE, 2020, pp. 295–299.
  * [28] S. Sheng, P. Chen, Z. Chen, L. Wu, and Y. Yao, “Deep reinforcement learning-based task scheduling in IoT edge computing,” _Sensors_ , vol. 21, no. 5, p. 1666, 2021.
  * [29] X. Chen and G. Liu, “Energy-efficient task offloading and resource allocation via deep reinforcement learning for augmented reality in mobile edge networks,” _IEEE Internet of Things Journal_ , 2021.
  * [30] Q. Liu, T. Han, and E. Moges, “Edgeslice: Slicing wireless edge computing network with decentralized deep reinforcement learning,” _arXiv preprint arXiv:2003.12911_ , 2020.
  * [31] M. Lin and Y. Zhao, “Artificial intelligence-empowered resource management for future wireless communications: A survey,” _China Communications_ , vol. 17, no. 3, pp. 58–77, 2020.
  * [32] Q. T. A. Pham, K. Piamrat, and C. Viho, “Resource management in wireless access networks: A layer-based classification-version 1.0,” 2014.
  * [33] E. Almazrouei, G. Gianini, N. Almoosa, and E. Damiani, “What can machine learning do for radio spectrum management?” in _Proceedings of the 16th ACM Symposium on QoS and Security for Wireless and Mobile Networks_ , 2020, pp. 15–21.
  * [34] S. Dhilipkumar, C. Arunachalaperumal, and K. Thanigaivelu, “A comparative study of resource allocation schemes for D2D networks underlay cellular networks,” _Wireless Personal Communications_ , vol. 106, no. 3, pp. 1075–1087, 2019.
  * [35] K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath, “Deep reinforcement learning: A brief survey,” _IEEE Signal Processing Magazine_ , vol. 34, no. 6, pp. 26–38, 2017.
  * [36] Z. Zhang, D. Zhang, and R. C. Qiu, “Deep reinforcement learning for power system applications: An overview,” _CSEE Journal of Power and Energy Systems_ , vol. 6, no. 1, pp. 213–225, 2019.
  * [37] T. S. Rappaport, Y. Xing, O. Kanhere, S. Ju, A. Madanayake, S. Mandal, A. Alkhateeb, and G. C. Trichopoulos, “Wireless communications and applications above 100 ghz: Opportunities and challenges for 6G and beyond,” _IEEE Access_ , vol. 7, pp. 78 729–78 757, 2019.
  * [38] H. Tataria, M. Shafi, A. F. Molisch, M. Dohler, H. Sjöland, and F. Tufvesson, “6G wireless systems: Vision, requirements, challenges, insights, and opportunities,” _arXiv preprint arXiv:2008.03213_ , 2020.
  * [39] A. Alwarafy, B. S. Ciftler, M. Abdallah, and M. Hamdi, “DeepRAT: A DRL-based framework for multi-RAT assignment and power allocation in hetnets,” in _2021 IEEE International Conference on Communications (ICC)_. IEEE, 2021, pp. 1–6.
  * [40] J. Kong, Z.-Y. Wu, M. Ismail, E. Serpedin, and K. A. Qaraqe, “Q-learning based two-timescale power allocation for multi-homing hybrid RF/VLC networks,” _IEEE Wireless Communications Letters_ , 2019.
  * [41] G. Lee, M. Jung, A. T. Z. Kasgari, W. Saad, and M. Bennis, “Deep reinforcement learning for energy-efficient networking with reconfigurable intelligent surfaces,” in _ICC 2020-2020 IEEE International Conference on Communications (ICC)_. IEEE, 2020, pp. 1–6.
  * [42] B. S. Ciftler, A. Alwarafy, M. Abdallah, and M. Hamdi, “Dqn-based multi-user power allocation for hybrid RF/VLC networks,” _arXiv preprint arXiv:2102.01884_ , 2021.
  * [43] A. Ahmad, S. Ahmad, M. H. Rehmani, and N. U. Hassan, “A survey on radio resource allocation in cognitive radio sensor networks,” _IEEE Communications Surveys & Tutorials_, vol. 17, no. 2, pp. 888–917, 2015.
  * [44] M. El Tanab and W. Hamouda, “Resource allocation for underlay cognitive radio networks: A survey,” _IEEE Communications Surveys & Tutorials_, vol. 19, no. 2, pp. 1249–1276, 2017.
  * [45] M. Naeem, A. Anpalagan, M. Jaseemuddin, and D. C. Lee, “Resource allocation techniques in cooperative cognitive radio networks,” _IEEE Communications Surveys & Tutorials_, vol. 16, no. 2, pp. 729–744, 2014.
  * [46] S. Manap, K. Dimyati, M. N. Hindia, M. S. Abu Talip, and R. Tafazolli, “Survey of radio resource management in 5G heterogeneous networks,” _IEEE Access_ , vol. 8, pp. 131 202–131 223, 2020.
  * [47] M. Peng, C. Wang, J. Li, H. Xiang, and V. Lau, “Recent advances in underlay heterogeneous networks: Interference control, resource allocation, and self-organization,” _IEEE Communications Surveys & Tutorials_, vol. 17, no. 2, pp. 700–729, 2015.
  * [48] Y. Teng, M. Liu, F. R. Yu, V. C. M. Leung, M. Song, and Y. Zhang, “Resource allocation for ultra-dense networks: A survey, some research issues and challenges,” _IEEE Communications Surveys & Tutorials_, vol. 21, no. 3, pp. 2134–2168, 2019.
  * [49] K. Piamrat, A. Ksentini, J.-M. Bonnin, and C. Viho, “Radio resource management in emerging heterogeneous wireless networks,” _Computer Communications_ , vol. 34, no. 9, pp. 1066–1076, 2011.
  * [50] N. Xia, H. Chen, and C. Yang, “Radio resource management in machine-to-machine communications—a survey,” _IEEE Communications Surveys & Tutorials_, vol. 20, no. 1, pp. 791–828, 2018.
  * [51] S. Sadr, A. Anpalagan, and K. Raahemifar, “Radio resource allocation algorithms for the downlink of multiuser ofdm communication systems,” _IEEE Communications Surveys & Tutorials_, vol. 11, no. 3, pp. 92–106, 2009\.
  * [52] E. Yaacoub and Z. Dawy, “A survey on uplink resource allocation in ofdma wireless networks,” _IEEE Communications Surveys & Tutorials_, vol. 14, no. 2, pp. 322–337, 2012.
  * [53] R. O. Afolabi, A. Dadlani, and K. Kim, “Multicast scheduling and resource allocation algorithms for ofdma-based systems: A survey,” _IEEE Communications Surveys & Tutorials_, vol. 15, no. 1, pp. 240–254, 2013.
  * [54] S. Chieochan and E. Hossain, “Adaptive radio resource allocation in ofdma systems: a survey of the state-of-the-art approaches,” _Wireless Communications and Mobile Computing_ , vol. 9, no. 4, pp. 513–527, 2009.
  * [55] D. Niyato and E. Hossain, “Radio resource management in MIMO-OFDM- mesh networks: Issues and approaches,” _IEEE Communications Magazine_ , vol. 45, no. 11, pp. 100–107, 2007.
  * [56] W. Zhao and S. Wang, “Resource sharing scheme for device-to-device communication underlaying cellular networks,” _IEEE Transactions on Communications_ , vol. 63, no. 12, pp. 4838–4848, 2015.
  * [57] L. Song, D. Niyato, Z. Han, and E. Hossain, “Game-theoretic resource allocation methods for device-to-device communication,” _IEEE Wireless Communications_ , vol. 21, no. 3, pp. 136–144, 2014.
  * [58] Y. Kawamoto, H. Nishiyama, N. Kato, F. Ono, and R. Miura, “Toward future unmanned aerial vehicle networks: Architecture, resource allocation and field experiments,” _IEEE Wireless Communications_ , vol. 26, no. 1, pp. 94–99, 2019.
  * [59] A. Masmoudi, K. Mnif, and F. Zarai, “A survey on radio resource allocation for V2X communication,” _Wireless Communications and Mobile Computing_ , vol. 2019, 2019.
  * [60] M. Allouch, S. Kallel, A. Soua, O. Shagdar, and S. Tohme, “Survey on radio resource allocation in long-term evolution-vehicle,” _Concurrency and Computation: Practice and Experience_ , p. e6228, 2021.
  * [61] S. Xu, G. Zhu, B. Ai, and Z. Zhong, “A survey on high-speed railway communications: A radio resource management perspective,” _Computer Communications_ , vol. 86, pp. 12–28, 2016.
  * [62] J. Clausen, “Branch and bound algorithms-principles and examples,” _Department of Computer Science, University of Copenhagen_ , pp. 1–30, 1999\.
  * [63] D. S. Nau, V. Kumar, and L. Kanal, “General branch and bound, and its relation to ${A}^{\ast}$ and ${AO}^{\ast}$,” _Artificial Intelligence_ , vol. 23, no. 1, pp. 29–58, 1984.
  * [64] Y. S. Nasir and D. Guo, “Deep reinforcement learning for joint spectrum and power allocation in cellular networks,” _arXiv preprint arXiv:2012.10682_ , 2020.
  * [65] K. Shen and W. Yu, “Fractional programming for communication systems—part i: Power control and beamforming,” _IEEE Transactions on Signal Processing_ , vol. 66, no. 10, pp. 2616–2630, 2018.
  * [66] M. Kim and I.-Y. Ko, “An efficient resource allocation approach based on a genetic algorithm for composite services in IoT environments,” in _2015 IEEE International Conference on Web Services_. IEEE, 2015, pp. 543–550.
  * [67] M. Naeem, S. T. H. Rizvi, and A. Coronato, “A gentle introduction to reinforcement learning and its application in different fields,” _IEEE Access_ , 2020.
  * [68] H. Sun, X. Chen, Q. Shi, M. Hong, X. Fu, and N. D. Sidiropoulos, “Learning to optimize: Training deep neural networks for interference management,” _IEEE Transactions on Signal Processing_ , vol. 66, no. 20, pp. 5438–5453, 2018.
  * [69] H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 30, no. 1, 2016.
  * [70] Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas, “Dueling network architectures for deep reinforcement learning,” in _International conference on machine learning_. PMLR, 2016, pp. 1995–2003.
  * [71] R. SUTTON, “Policy gradient methods for reinforcement learning with function approximation,” _Advances in Neural information Processing Systems_ , vol. 12, pp. 1057–1063, 2000.
  * [72] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep reinforcement learning,” in _International conference on machine learning_. PMLR, 2016, pp. 1928–1937.
  * [73] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” 2015.
  * [74] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller, “Deterministic policy gradient algorithms,” in _Proceedings of the 31st International Conference on Machine Learning_ , vol. 32, no. 1. PMLR, 2014, pp. 387–395. [Online]. Available: http://proceedings.mlr.press/v32/silver14.html
  * [75] T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized experience replay,” _arXiv preprint arXiv:1511.05952_ , 2015.
  * [76] D. Horgan, J. Quan, D. Budden, G. Barth-Maron, M. Hessel, H. Van Hasselt, and D. Silver, “Distributed prioritized experience replay,” _arXiv preprint arXiv:1803.00933_ , 2018.
  * [77] M. G. Bellemare, W. Dabney, and R. Munos, “A distributional perspective on reinforcement learning,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 449–458.
  * [78] M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver, “Rainbow: Combining improvements in deep reinforcement learning,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 32, no. 1, 2018.
  * [79] M. Hausknecht and P. Stone, “Deep recurrent Q-learning for partially observable mdps,” 2015.
  * [80] S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 1587–1596.
  * [81] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement.”
  * [82] G. Barth-Maron, M. W. Hoffman, D. Budden, W. Dabney, D. Horgan, D. Tb, A. Muldal, N. Heess, and T. Lillicrap, “Distributed distributional deterministic policy gradients,” _arXiv preprint arXiv:1804.08617_ , 2018\.
  * [83] T. T. Nguyen, N. D. Nguyen, and S. Nahavandi, “Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications,” _IEEE transactions on cybernetics_ , vol. 50, no. 9, pp. 3826–3839, 2020\.
  * [84] K. Zhang, Z. Yang, and T. Başar, “Multi-agent reinforcement learning: A selective overview of theories and algorithms,” _arXiv preprint arXiv:1911.10635_ , 2019.
  * [85] I. Althamary, C.-W. Huang, and P. Lin, “A survey on multi-agent reinforcement learning methods for vehicular networks,” in _2019 15th International Wireless Communications & Mobile Computing Conference (IWCMC)_. IEEE, 2019, pp. 1154–1159.
  * [86] D. Lee, N. He, P. Kamalaruban, and V. Cevher, “Optimization for reinforcement learning: From a single agent to cooperative agents,” _IEEE Signal Processing Magazine_ , vol. 37, no. 3, pp. 123–135, 2020.
  * [87] Q. Shi, M. Razaviyayn, Z.-Q. Luo, and C. He, “An iteratively weighted MMSE approach to distributed sum-utility maximization for a MIMO interfering broadcast channel,” _IEEE Transactions on Signal Processing_ , vol. 59, no. 9, pp. 4331–4340, 2011.
  * [88] E. Ghadimi, F. D. Calabrese, G. Peters, and P. Soldati, “A reinforcement learning approach to power control and rate adaptation in cellular networks,” in _2017 IEEE International Conference on Communications (ICC)_. IEEE, 2017, pp. 1–7.
  * [89] X. Hu and Y. Sun, “A deep reinforcement learning-based power resource management for fuel cell powered data centers,” _Electronics_ , vol. 9, no. 12, p. 2054, 2020.
  * [90] A. A. Khan and R. S. Adve, “Centralized and distributed deep reinforcement learning methods for downlink sum-rate optimization,” _IEEE Transactions on Wireless Communications_ , vol. 19, no. 12, pp. 8410–8426, 2020\.
  * [91] Y. S. Nasir and D. Guo, “Deep actor-critic learning for distributed power control in wireless mobile networks,” _arXiv preprint arXiv:2009.06681_ , 2020.
  * [92] Z. Bi and W. Zhou, “Deep reinforcement learning based power allocation for D2D network,” in _2020 IEEE 91st Vehicular Technology Conference (VTC2020-Spring)_. IEEE, 2020, pp. 1–5.
  * [93] S. Saeidian, S. Tayamon, and E. Ghadimi, “Downlink power control in dense 5G radio access networks through deep reinforcement learning,” in _ICC 2020-2020 IEEE International Conference on Communications (ICC)_. IEEE, 2020, pp. 1–6.
  * [94] Z. Zhang, H. Qu, J. Zhao, and W. Wang, “Deep reinforcement learning method for energy efficient resource allocation in next generation wireless networks,” in _Proceedings of the 2020 International Conference on Computing, Networks and Internet of Things_ , 2020, pp. 18–24.
  * [95] F. Meng, P. Chen, L. Wu, and J. Cheng, “Power allocation in multi-user cellular networks: Deep reinforcement learning approaches,” _IEEE Transactions on Wireless Communications_ , vol. 19, no. 10, pp. 6255–6267, 2020\.
  * [96] F. Meng, P. Chen, and L. Wu, “Power allocation in multi-user cellular networks with deep Q learning approach,” in _ICC 2019-2019 IEEE International Conference on Communications (ICC)_. IEEE, 2019, pp. 1–6.
  * [97] L. Zhang and Y.-C. Liang, “Deep reinforcement learning for multi-agent power control in heterogeneous networks,” _IEEE Transactions on Wireless Communications_ , 2020.
  * [98] M. Mohsenivatani, M. Darabi, S. Parsaeefard, M. Ardebilipour, and B. Maham, “Throughput maximization in c-ran enabled virtualized wireless networks via multi-agent deep reinforcement learning,” in _2020 IEEE 31st Annual International Symposium on Personal, Indoor and Mobile Radio Communications_. IEEE, pp. 1–6.
  * [99] X. Li, J. Fang, W. Cheng, H. Duan, Z. Chen, and H. Li, “Intelligent power control for spectrum sharing in cognitive radios: A deep reinforcement learning approach,” _IEEE access_ , vol. 6, pp. 25 463–25 473, 2018.
  * [100] Y. Zhang, C. Kang, T. Ma, Y. Teng, and D. Guo, “Power allocation in multi-cell networks using deep reinforcement learning,” in _2018 IEEE 88th Vehicular Technology Conference (VTC-Fall)_. IEEE, 2018, pp. 1–6.
  * [101] L. Li, Q. Cheng, K. Xue, C. Yang, and Z. Han, “Downlink transmit power control in ultra-dense UAV network based on mean field game and deep reinforcement learning,” _IEEE Transactions on Vehicular Technology_ , 2020.
  * [102] Y. S. Nasir and D. Guo, “Multi-agent deep reinforcement learning for dynamic power allocation in wireless networks,” _IEEE Journal on Selected Areas in Communications_ , vol. 37, no. 10, pp. 2239–2250, 2019.
  * [103] W. Wang, “Smoothing deep reinforcement learning for power control for spectrum sharing in cognitive radios.”
  * [104] N. Zhao, Z. Liu, and Y. Cheng, “Multi-agent deep reinforcement learning for trajectory design and power allocation in multi-UAV networks,” _IEEE Access_ , vol. 8, pp. 139 670–139 679, 2020.
  * [105] J. Xu and B. Ai, “Experience-driven power allocation using multi-agent deep reinforcement learning for millimeter-wave high-speed railway systems,” _IEEE Transactions on Intelligent Transportation Systems_ , 2021.
  * [106] M. Yan, B. Chen, G. Feng, and S. Qin, “Federated cooperation and augmentation for power allocation in decentralized wireless networks,” _IEEE Access_ , vol. 8, pp. 48 088–48 100, 2020.
  * [107] J. G. Luis, M. Guerster, I. del Portillo, E. Crawley, and B. Cameron, “Deep reinforcement learning architecture for continuous power allocation in high throughput satellites,” _arXiv preprint arXiv:1906.00571_ , 2019.
  * [108] J. J. G. Luis, M. Guerster, I. del Portillo, E. Crawley, and B. Cameron, “Deep reinforcement learning for continuous power allocation in flexible high throughput satellites,” in _2019 IEEE Cognitive Communications for Aerospace Applications Workshop (CCAAW)_. IEEE, 2019, pp. 1–4.
  * [109] O. Maraqa, A. S. Rajasekaran, S. Al-Ahmadi, H. Yanikomeroglu, and S. M. Sait, “A survey of rate-optimal power domain NOMA with enabling technologies of future wireless networks,” _IEEE Communications Surveys & Tutorials_, vol. 22, no. 4, pp. 2192–2235, 2020.
  * [110] X. Yan, K. An, Q. Zhang, G. Zheng, S. Chatzinotas, and J. Han, “Delay constrained resource allocation for noma enabled satellite internet of things with deep reinforcement learning,” _IEEE Internet of Things Journal_ , 2020\.
  * [111] A. Alwarafy, M. Alresheedi, A. F. Abas, and A. Alsanie, “Performance evaluation of space time coding techniques for indoor visible light communication systems,” in _2018 International Conference on Optical Network Design and Modeling (ONDM)_ , 2018, pp. 88–93.
  * [112] A. Memedi and F. Dressler, “Vehicular visible light communications: A survey,” _IEEE Communications Surveys & Tutorials_, vol. 23, no. 1, pp. 161–181, 2021.
  * [113] M. Chen, W. Saad, and C. Yin, “Liquid state machine learning for resource allocation in a network of cache-enabled LTE-U UAVs,” in _GLOBECOM 2017-2017 IEEE Global Communications Conference_. IEEE, Conference Proceedings, pp. 1–6.
  * [114] N. Zhao, Y.-C. Liang, D. Niyato, Y. Pei, and Y. Jiang, “Deep reinforcement learning for user association and resource allocation in heterogeneous networks,” in _2018 IEEE Global Communications Conference (GLOBECOM)_. IEEE, 2018, pp. 1–6.
  * [115] N. Zhao, Y.-C. Liang, D. Niyato, Y. Pei, M. Wu, and Y. Jiang, “Deep reinforcement learning for user association and resource allocation in heterogeneous cellular networks,” _IEEE Transactions on Wireless Communications_ , vol. 18, no. 11, pp. 5141–5152, 2019.
  * [116] Q. Zhang, Y.-C. Liang, and H. V. Poor, “Intelligent user association for symbiotic radio networks using deep reinforcement learning,” _IEEE Transactions on Wireless Communications_ , 2020.
  * [117] W. Lei, Y. Ye, and M. Xiao, “Deep reinforcement learning based spectrum allocation in integrated access and backhaul networks,” _IEEE Transactions on Cognitive Communications and Networking_ , 2020.
  * [118] Z. Li, C. Wang, and C.-J. Jiang, “User association for load balancing in vehicular networks: An online reinforcement learning approach,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 18, no. 8, pp. 2217–2228, 2017.
  * [119] J. S. Perez, S. K. Jayaweera, and S. Lane, “Machine learning aided cognitive RAT selection for 5G heterogeneous networks,” in _2017 IEEE International Black Sea Conference on Communications and Networking (BlackSeaCom)_. IEEE, 2017, pp. 1–5.
  * [120] J. Zheng, X. Tang, X. Wei, H. Shen, and L. Zhao, “Channel assignment for hybrid noma systems with deep reinforcement learning,” _IEEE Wireless Communications Letters_ , 2021.
  * [121] H. Song, L. Liu, J. Ashdown, and Y. Yi, “A deep reinforcement learning framework for spectrum management in dynamic spectrum access,” _IEEE Internet of Things Journal_ , 2021.
  * [122] S. Wang, H. Liu, P. H. Gomes, and B. Krishnamachari, “Deep reinforcement learning for dynamic multichannel access in wireless networks,” _IEEE Transactions on Cognitive Communications and Networking_ , vol. 4, no. 2, pp. 257–265, 2018.
  * [123] S. Wang, H. Liu, H. Gomes, and B. Krishnamachari, “Deep reinforcement learning for dynamic multichannel access in wireless networks,” 2018.
  * [124] S. Wang, H. Liu, P. Gomes, and B. Krishnamachari, “Deep reinforcement learning for dynamic multichannel access,” in _International Conference on Computing, Networking and Communications (ICNC)_ , 2017, pp. 257–265.
  * [125] S. H. A. Ahmad, M. Liu, T. Javidi, Q. Zhao, and B. Krishnamachari, “Optimality of myopic sensing in multichannel opportunistic access,” _IEEE Transactions on Information Theory_ , vol. 55, no. 9, pp. 4040–4050, 2009.
  * [126] M. Chu, H. Li, X. Liao, and S. Cui, “Reinforcement learning-based multiaccess control and battery prediction with energy harvesting in IoT systems,” _IEEE Internet of Things Journal_ , vol. 6, no. 2, pp. 2009–2020, 2018.
  * [127] Y. Zhang, P. Cai, C. Pan, and S. Zhang, “Multi-agent deep reinforcement learning-based cooperative spectrum sensing with upper confidence bound exploration,” _IEEE Access_ , vol. 7, pp. 118 898–118 906, 2019.
  * [128] Y. Xu, J. Yu, W. C. Headley, and R. M. Buehrer, “Deep reinforcement learning for dynamic spectrum access in wireless networks,” in _MILCOM 2018-2018 IEEE Military Communications Conference (MILCOM)_. IEEE, Conference Proceedings, pp. 207–212.
  * [129] L. Liang, H. Ye, and G. Y. Li, “Multi-agent reinforcement learning for spectrum sharing in vehicular networks,” in _2019 IEEE 20th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC)_. IEEE, 2019, pp. 1–5.
  * [130] ——, “Spectrum sharing in vehicular networks based on multi-agent reinforcement learning,” _IEEE Journal on Selected Areas in Communications_ , vol. 37, no. 10, pp. 2282–2292, 2019.
  * [131] J. Zhu, Y. Song, D. Jiang, and H. Song, “A new deep-q-learning-based transmission scheduling mechanism for the cognitive internet of things,” _IEEE Internet of Things Journal_ , vol. 5, no. 4, pp. 2375–2385, 2017.
  * [132] H. Khan, A. Elgabli, S. Samarakoon, M. Bennis, and C. S. Hong, “Reinforcement learning-based vehicle-cell association algorithm for highly mobile millimeter wave communication,” _IEEE Transactions on Cognitive Communications and Networking_ , vol. 5, no. 4, pp. 1073–1085, 2019.
  * [133] P. Yang, L. Li, J. Yin, H. Zhang, W. Liang, W. Chen, and Z. Han, “Dynamic spectrum access in cognitive radio networks using deep reinforcement learning and evolutionary game,” in _2018 IEEE/CIC International Conference on Communications in China (ICCC)_. IEEE, 2018, pp. 405–409.
  * [134] Y. Cao, S.-Y. Lien, and Y.-C. Liang, “Deep reinforcement learning for multi-user access control in non-terrestrial networks,” _IEEE Transactions on Communications_ , 2020.
  * [135] N. Yang, H. Zhang, and R. Berry, “Partially observable multi-agent deep reinforcement learning for cognitive resource management,” in _GLOBECOM 2020-2020 IEEE Global Communications Conference_. IEEE, 2020, pp. 1–6.
  * [136] S. Tomovic and I. Radusinovic, “A novel deep q-learning method for dynamic spectrum access,” in _2020 28th Telecommunications Forum (TELFOR)_. IEEE, 2020, pp. 1–4.
  * [137] C. Zhong, Z. Lu, M. C. Gursoy, and S. Velipasalar, “A deep actor-critic reinforcement learning framework for dynamic multichannel access,” _IEEE Transactions on Cognitive Communications and Networking_ , vol. 5, no. 4, pp. 1125–1139, 2019.
  * [138] A. Tondwalkar and D. A. Kwasinski, “Deep reinforcement learning for distributed uncoordinated cognitive radios resource allocation,” _arXiv preprint arXiv:1911.03366_ , 2019.
  * [139] K. Al-Gumaei, K. Schuba, A. Friesen, S. Heymann, C. Pieper, F. Pethig, and S. Schriegel, “A survey of internet of things and big data integrated solutions for industrie 4.0,” in _2018 IEEE 23rd International Conference on Emerging Technologies and Factory Automation (ETFA)_ , vol. 1, 2018, pp. 1417–1424.
  * [140] Z. Shi, X. Xie, H. Lu, H. Yang, M. Kadoch, and M. Cheriet, “Deep reinforcement learning based spectrum resource management for industrial internet of things,” _IEEE Internet of Things Journal_ , 2020.
  * [141] S. B. Janiar and V. Pourahmadi, “Deep-reinforcement learning for fair distributed dynamic spectrum access in wireless networks,” in _2021 IEEE 18th Annual Consumer Communications & Networking Conference (CCNC)_. IEEE, 2021, pp. 1–4.
  * [142] Y. Wang, X. Li, P. Wan, and R. Shao, “Intelligent dynamic spectrum access using deep reinforcement learning for vanets,” _IEEE Sensors Journal_ , 2021\.
  * [143] W. Jiang and W. Yu, “Multi-agent reinforcement learning based joint cooperative spectrum sensing and channel access for cognitive UAV networks,” _arXiv preprint arXiv:2103.08181_ , 2021.
  * [144] Y. Xu, J. Yu, and R. M. Buehrer, “The application of deep reinforcement learning to distributed spectrum access in dynamic heterogeneous environments with partial observations,” _IEEE Transactions on Wireless Communications_ , vol. 19, no. 7, pp. 4494–4506, 2020.
  * [145] S. Liu, X. Hu, and W. Wang, “Deep reinforcement learning based dynamic channel allocation algorithm in multibeam satellite systems,” _IEEE Access_ , vol. 6, pp. 15 733–15 742, 2018.
  * [146] X. Hu, S. Liu, R. Chen, W. Wang, and C. Wang, “A deep reinforcement learning-based framework for dynamic resource allocation in multibeam satellite systems,” _IEEE Communications Letters_ , vol. 22, no. 8, pp. 1612–1615, 2018.
  * [147] B. Zhao, J. Liu, Z. Wei, and I. You, “A deep reinforcement learning based approach for energy-efficient channel allocation in satellite internet of things,” _IEEE Access_ , vol. 8, pp. 62 197–62 206, 2020.
  * [148] J. Liu, B. Zhao, Q. Xin, and H. Liu, “Dynamic channel allocation for satellite internet of things via deep reinforcement learning,” in _2020 International Conference on Information Networking (ICOIN)_. IEEE, 2020, pp. 465–470.
  * [149] F. Zheng, Z. Pi, Z. Zhou, and K. Wang, “Leo satellite channel allocation scheme based on reinforcement learning,” _Mobile Information Systems_ , vol. 2020, 2020.
  * [150] U. Challita, L. Dong, and W. Saad, “Proactive resource management in lte-u systems: A deep learning perspective,” _arXiv preprint arXiv:1702.07031_ , 2017.
  * [151] O. Naparstek and K. Cohen, “Deep multi-user reinforcement learning for distributed dynamic spectrum access,” _IEEE Transactions on Wireless Communications_ , vol. 18, no. 1, pp. 310–323, 2018.
  * [152] S. Wang and T. Lv, “Deep reinforcement learning based dynamic multichannel access in hetnets,” in _2019 IEEE Wireless Communications and Networking Conference (WCNC)_. IEEE, 2019, pp. 1–6.
  * [153] H. Peng and X. S. Shen, “Deep reinforcement learning based resource management for multi-access edge computing in vehicular networks,” _IEEE Transactions on Network Science and Engineering_ , 2020.
  * [154] U. Challita and D. Sandberg, “Deep reinforcement learning for dynamic spectrum sharing of lte and nr,” _arXiv preprint arXiv:2102.11176_ , 2021.
  * [155] Q. Liu, T. Han, N. Zhang, and Y. Wang, “Deepslicing: Deep reinforcement learning assisted resource allocation for network slicing,” _arXiv preprint arXiv:2008.07614_ , 2020.
  * [156] F. Tang, Y. Zhou, and N. Kato, “Deep reinforcement learning for dynamic uplink/downlink resource allocation in high mobility 5G hetnet,” _IEEE Journal on Selected Areas in Communications_ , 2020.
  * [157] Z. Chkirbene, A. Awad, A. Mohamed, A. Erbad, and M. Guizani, “Deep reinforcement learning for network selection over heterogeneous health systems,” _IEEE Transactions on Network Science and Engineering_ , 2021.
  * [158] Y. Y. Munaye, R.-T. Juang, H.-P. Lin, G. B. Tarekegn, and D.-B. Lin, “Deep reinforcement learning based resource management in UAV-assisted IoT networks,” _Applied Sciences_ , vol. 11, no. 5, p. 2163, 2021.
  * [159] T. Zhang, K. Zhu, and J. Wang, “Energy-efficient mode selection and resource allocation for D2D-enabled heterogeneous networks: A deep reinforcement learning approach,” _IEEE Transactions on Wireless Communications_ , 2020\.
  * [160] X. Zhang, M. Peng, S. Yan, and Y. Sun, “Deep-reinforcement-learning-based mode selection and resource allocation for cellular V2X communications,” _IEEE Internet of Things Journal_ , vol. 7, no. 7, pp. 6380–6391, 2019.
  * [161] J. Jang and H. J. Yang, “Deep reinforcement learning-based resource allocation and power control in small cells with limited information exchange,” _IEEE Transactions on Vehicular Technology_ , vol. 69, no. 11, pp. 13 768–13 783, 2020.
  * [162] X. Liao, J. Shi, Z. Li, L. Zhang, and B. Xia, “A model-driven deep reinforcement learning heuristic algorithm for resource allocation in ultra-dense cellular networks,” _IEEE Transactions on Vehicular Technology_ , vol. 69, no. 1, pp. 983–997, 2020.
  * [163] Z. Liu, X. Chen, Y. Chen, and Z. Li, “Deep reinforcement learning based dynamic resource allocation in 5G ultra-dense networks,” in _2019 IEEE International Conference on Smart Internet of Things (SmartIoT)_. IEEE, 2019, pp. 168–174.
  * [164] X. Zhang, Z. Lin, B. Ding, B. Gu, and Y. Han, “Deep multi-agent reinforcement learning for resource allocation in D2D communication underlaying cellular networks,” in _2020 21st Asia-Pacific Network Operations and Management Symposium (APNOMS)_. IEEE, 2020, pp. 55–60.
  * [165] A. Asheralieva and Y. Miyanaga, “An autonomous learning-based algorithm for joint channel and power level selection by D2D pairs in heterogeneous cellular networks,” _IEEE transactions on communications_ , vol. 64, no. 9, pp. 3996–4012, 2016.
  * [166] D. Wang, H. Qin, B. Song, K. Xu, X. Du, and M. Guizani, “Joint resource allocation and power control for d2d communication with deep reinforcement learning in mcc,” _Physical Communication_ , vol. 45, p. 101262, 2021.
  * [167] H. Ding, F. Zhao, J. Tian, D. Li, and H. J. A. H. N. Zhang, “A deep reinforcement learning for user association and power control in heterogeneous networks,” vol. 102, p. 102069, 2020.
  * [168] S. Yu, Y. J. Jeong, and J. W. Lee, “Resource allocation scheme based on deep reinforcement learning for device-to-device communications,” in _2021 International Conference on Information Networking (ICOIN)_. IEEE, 2021, pp. 712–714.
  * [169] Y. Zhang, C. Kang, Y. Teng, S. Li, W. Zheng, and J. Fang, “Deep reinforcement learning framework for joint resource allocation in heterogeneous networks,” in _2019 IEEE 90th Vehicular Technology Conference (VTC2019-Fall)_. IEEE, 2019, pp. 1–6.
  * [170] Q. Huang, X. Xie, and M. Cheriet, “Reinforcement learning-based hybrid spectrum resource allocation scheme for the high load of URLLC services,” _EURASIP Journal on Wireless Communications and Networking_ , vol. 2020, no. 1, pp. 1–21, 2020.
  * [171] J. Tan, L. Zhang, and Y.-C. Liang, “Deep reinforcement learning for channel selection and power control in D2D networks,” in _2019 IEEE Global Communications Conference (GLOBECOM)_. IEEE, 2019, pp. 1–6.
  * [172] J. Tan, Y.-C. Liang, L. Zhang, and G. Feng, “Deep reinforcement learning for joint channel selection and power control in D2D networks,” _IEEE Transactions on Wireless Communications_ , 2020.
  * [173] H. Ye, G. Y. Li, and B.-H. F. Juang, “Deep reinforcement learning based resource allocation for V2V communications,” _IEEE Transactions on Vehicular Technology_ , vol. 68, no. 4, pp. 3163–3173, 2019.
  * [174] H. Ye and G. Y. Li, “Deep reinforcement learning for resource allocation in V2V communications,” in _2018 IEEE International Conference on Communications (ICC)_. IEEE, 2018, pp. 1–6.
  * [175] H. Ye and G. Li, “Deep reinforcement learning based distributed resource allocation for V2V broadcasting,” in _2018 14th International Wireless Communications & Mobile Computing Conference (IWCMC)_. IEEE, 2018, pp. 440–445.
  * [176] S. Yuan, Y. Zhang, W. Qie, T. Ma, and S. Li, “Deep reinforcement learning for resource allocation with network slicing in cognitive radio network,” _Computer Science and Information Systems_ , no. 00, pp. 55–55, 2020.
  * [177] Y. Zhang, X. Wang, and Y. Xu, “Energy-efficient resource allocation in uplink noma systems with deep reinforcement learning,” in _2019 11th International Conference on Wireless Communications and Signal Processing (WCSP)_. IEEE, 2019, pp. 1–6.
  * [178] C. He, Y. Hu, Y. Chen, and B. Zeng, “Joint power allocation and channel assignment for noma with deep reinforcement learning,” _IEEE Journal on Selected Areas in Communications_ , vol. 37, no. 10, pp. 2200–2210, 2019.
  * [179] Z. Jiang, C. Hao, Y. Huang, Q. Wu, and F. Zhou, “Partially distributed channel and power management based on reinforcement learning,” _Journal of Communications and Information Networks_ , vol. 5, no. 4, pp. 423–437, 2020.
  * [180] Y.-H. Xu, C.-C. Yang, M. Hua, and W. Zhou, “Deep deterministic policy gradient (ddpg)-based resource allocation scheme for noma vehicular communications,” _IEEE Access_ , vol. 8, pp. 18 797–18 807, 2020.
  * [181] S. Shrivastava, B. Chen, C. Chen, H. Wang, and M. Dai, “Deep q-network learning based downlink resource allocation for hybrid RF/VLC systems,” _IEEE Access_ , vol. 8, pp. 149 412–149 434, 2020.
  * [182] J. Zhu, J. Wang, Y. Huang, S. He, X. You, and L. Yang, “On optimal power allocation for downlink non-orthogonal multiple access systems,” _IEEE Journal on Selected Areas in Communications_ , vol. 35, no. 12, pp. 2744–2757, 2017.
  * [183] A. Adadi and M. Berrada, “Peeking inside the black-box: a survey on explainable artificial intelligence (XAI),” _IEEE access_ , vol. 6, pp. 52 138–52 160, 2018.
  * [184] Y. He, Y. Wang, C. Qiu, Q. Lin, J. Li, and Z. Ming, “Blockchain-based edge computing resource allocation in IoT: A deep reinforcement learning approach,” _IEEE Internet of Things Journal_ , vol. 8, no. 4, pp. 2226–2237, 2021.
  * [185] F. Guo, F. R. Yu, H. Zhang, H. Ji, M. Liu, and V. C. M. Leung, “Adaptive resource allocation in future wireless networks with blockchain and mobile edge computing,” _IEEE Transactions on Wireless Communications_ , vol. 19, no. 3, pp. 1689–1703, 2020.
  * [186] S. Hu, Y. C. Liang, Z. Xiong, and D. Niyato, “Blockchain and artificial intelligence for dynamic resource sharing in 6G and beyond,” _IEEE Wireless Communications_ , pp. 1–7, 2021.
  * [187] O. A. Wahab, A. Mourad, H. Otrok, and T. Taleb, “Federated machine learning: Survey, multi-level classification, desirable criteria and future directions in communication and networking systems,” _IEEE Communications Surveys & Tutorials_, pp. 1–1, 2021.
  * [188] N. H. Tran, W. Bao, A. Zomaya, M. N. Nguyen, and C. S. Hong, “Federated learning over wireless networks: Optimization model design and analysis,” in _IEEE INFOCOM 2019-IEEE Conference on Computer Communications_. IEEE, 2019, pp. 1387–1395.
  * [189] S. Wang, T. Tuor, T. Salonidis, K. K. Leung, C. Makaya, T. He, and K. Chan, “Adaptive federated learning in resource constrained edge computing systems,” _IEEE Journal on Selected Areas in Communications_ , vol. 37, no. 6, pp. 1205–1221, 2019.
  * [190] X. Wang, C. Wang, X. Li, V. C. Leung, and T. Taleb, “Federated deep reinforcement learning for internet of things with decentralized cooperative edge caching,” _IEEE Internet of Things Journal_ , vol. 7, no. 10, pp. 9441–9455, 2020.
  * [191] H. H. Zhuo, W. Feng, Q. Xu, Q. Yang, and Y. Lin, “Federated reinforcement learning,” _arXiv preprint arXiv:1901.08277_ , 2019.
  * [192] B. Das and S. Roy, “Load balancing techniques for wireless mesh networks: A survey,” in _2013 International Symposium on Computational and Business Intelligence_ , 2013, pp. 247–253.
  * [193] L. Zhu, W. Shen, S. Pan, R. Li, and Z. Li, “A dynamic load balancing method for spatial data network service,” in _2009 5th International Conference on Wireless Communications, Networking and Mobile Computing_ , 2009, pp. 1–3.
  * [194] H. Desai and R. Oza, “A study of dynamic load balancing in grid environment,” in _2016 International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET)_ , 2016, pp. 128–132.
  * [195] A. T. Z. Kasgari, W. Saad, M. Mozaffari, and H. V. Poor, “Experienced deep reinforcement learning with generative adversarial networks (gans) for model-free ultra reliable low latency communication,” _IEEE Transactions on Communications_ , vol. 69, no. 2, pp. 884–899, 2021.
  * [196] R. Alghamdi, R. Alhadrami, D. Alhothali, H. Almorad, A. Faisal, S. Helal, R. Shalabi, R. Asfour, N. Hammad, A. Shams, N. Saeed, H. Dahrouj, T. Y. Al-Naffouri, and M. S. Alouini, “Intelligent surfaces for 6G wireless networks: A survey of optimization and performance analysis techniques,” _IEEE Access_ , vol. 8, pp. 202 795–202 818, 2020\.
  * [197] C. Huang, S. Hu, G. C. Alexandropoulos, A. Zappone, C. Yuen, R. Zhang, M. Di Renzo, and M. Debbah, “Holographic MIMO surfaces for 6G wireless networks: Opportunities, challenges, and trends,” _IEEE Wireless Communications_ , vol. 27, no. 5, pp. 118–125, 2020.
  * [198] Q. Wu and R. Zhang, “Towards smart and reconfigurable environment: Intelligent reflecting surface aided wireless network,” _IEEE Communications Magazine_ , vol. 58, no. 1, pp. 106–112, 2020.
  * [199] C. Pan, H. Ren, K. Wang, M. Elkashlan, M. Chen, M. Di Renzo, Y. Hao, J. Wang, A. L. Swindlehurst, X. You _et al._ , “Reconfigurable intelligent surface for 6g and beyond: Motivations, principles, applications, and research directions,” _arXiv preprint arXiv:2011.04300_ , 2020.
  * [200] A. Taha, Y. Zhang, F. B. Mismar, and A. Alkhateeb, “Deep reinforcement learning for intelligent reflecting surfaces: Towards standalone operation,” in _2020 IEEE 21st International Workshop on Signal Processing Advances in Wireless Communications (SPAWC)_ , 2020, pp. 1–5.
  * [201] C. Huang, R. Mo, and C. Yuen, “Reconfigurable intelligent surface assisted multiuser miso systems exploiting deep reinforcement learning,” _IEEE Journal on Selected Areas in Communications_ , vol. 38, no. 8, pp. 1839–1850, 2020.
  * [202] Z. Yang, Y. Liu, Y. Chen, and J. T. Zhou, “Deep reinforcement learning for ris-aided non-orthogonal multiple access downlink networks,” in _GLOBECOM 2020 - 2020 IEEE Global Communications Conference_ , 2020, pp. 1–6.