# Mobility-Aware Resource Allocation for mmWave IAB Networks: A Multi-Agent RL
Approach

Bibo Zhang and Ilario Filippini Authors are with Dipartimento di Elettronica,
Informazione e Biongegneria, Politecnico di Milano, Milan, Italy, 20133.
E-mail<EMAIL_ADDRESS>
###### Abstract

MmWaves have been envisioned as a promising direction to provide Gbps wireless
access. However, they are susceptible to high path losses and blockages, which
directional antennas can only partially mitigate. That makes mmWave networks
coverage-limited, thus requiring dense deployments. Integrated access and
backhaul (IAB) architectures have emerged as a cost-effective solution for
network densification. Resource allocation in mmWave IAB networks must face
big challenges to cope with heavy temporal dynamics, such as intermittent
links caused by user mobility and blockages from moving obstacles. This makes
it extremely difficult to find optimal and adaptive solutions. In this
article, exploiting the distributed structure of the problem, we propose a
Multi-Agent Reinforcement Learning (MARL) framework to optimize user
throughput via flow routing and link scheduling in mmWave IAB networks
characterized by user mobility and link outages generated by moving obstacles.
The proposed approach implicitly captures the environment dynamics,
coordinates the interference, and manages the buffer levels of IAB relay
nodes. We design different MARL components, considering full-duplex and half-
duplex IAB-nodes. In addition, we provide a communication and coordination
scheme for RL agents in an online training framework, addressing the
feasibility issues of practical systems. Numerical results show the
effectiveness of the proposed approach.

###### Index Terms:

mmWave networks, integrated access and backhaul (IAB), resource allocation,
user mobility, obstacle blockages, multi-agent reinforcement learning (MARL).

##  1 Introduction

The availability of millimeter-wave (mmWave) bands for radio access networks
(RANs) in the 5G standardization has been considered by the 3rd Generation
Partnership Project (3GPP) as one of the main reliefs to the explosive
increase of the global mobile traffic, which is now posing big challenges to
the access capacity provided by sub-6GHz communications. The large bandwidths
(several hundreds of MHz) and mainly-underutilized spectrum portions available
at those high frequencies are the key enablers of a Gbps access throughput.

However, this potential comes at a cost of facing a harsh propagation
environment characterized by very high path losses and no propagation through
obstacles, including not only static buildings but also moving vehicles and
pedestrians. While current antenna design technologies have shown to be
effective in mitigating these path losses through extremely directional
arrays, there is very little they can do against random blockages. In
practice, mmWave networks typically exhibit a coverage-limited behavior due to
the presence of obstacles. Therefore, to guarantee a high-quality coverage, 5G
mmWave access networks require base stations to be much more densely deployed
than in traditional radio access networks. This may translate into high
installation budgets for operators, which are mainly driven by costs to deploy
wired (e.g., fiber) backhaul connections.

Aiming to provide a dense network deployment at minimal costs, Release 16 of
3GPP specifications has introduced a new multi-hop wireless access
architecture, named Integrated Access and Backhaul (IAB)[1]. The rationale is
to place relay nodes, called IAB-nodes, in the service area of a mmWave base
station (BS), called IAB-donor, and form a wireless multi-hop backhaul to
forward data packets between the IAB-donor and user equipments (UEs). An
example of such an IAB network scenario can be found in Fig. 1. Self-
backhauling is a peculiar aspect of this architecture, where both radio access
and backhaul links share the same radio resources and interfaces. Therefore, a
proper radio resource allocation is essential to efficiently operate this
network. In particular, since the adopted multiple access scheme is based on
time-division multiple access (TDMA), the resource optimization must deal with
routing paths and scheduling of directional transmissions along established
links.

Studies on routing and scheduling in wireless multi-hop networks have appeared
in the literature since early 2000s, mainly resorting to optimization
techniques that assume always-available links and static users [2, 3].
However, it is hard for these techniques to provide mobile mmWave IAB networks
with practical solutions that simultaneously deal with the harsh propagation
environment, the strong impact of obstacles on link availability, and the
users’ mobility. Indeed, the optimal solution provided under ideal link
conditions typically underperforms when facing the stochastic on-off link
behavior caused by sudden obstacles and the dynamics of mobile users, which
can even eliminate all the advantages of a careful optimization. We could in
principle re-optimize the network periodically or every time it undergoes a
change, however, this will lead to huge computational costs and, most likely,
it would not be practical. Therefore, flexible and adaptive solutions are
required to schedule real-time operations so as to tackle random obstacles and
users’ mobility.

Given the above context, we believe that Reinforcement Learning (RL)
techniques can play an important role due to their intrinsic ability to adapt
to the behavior of the environment. Indeed, an RL agent can be trained by
playing against the environment and eventually discover a good strategy, even
when the environment’s reply is stochastic. RL agents can automatically grasp
relevant environment statistics during the training and apply the strategy
that provides the best long-term reward in front of any instance of the random
environment. Moreover, routing and scheduling in wireless multi-hop networks
have been traditionally considered as hard problems due to interference
constraints. They become even harder when user mobility and random obstacles
are introduced. Links’ availability is intermittent, which makes it impossible
to harness traditional approaches that are based on sets of potential
concurrent transmissions (i.e., compatible links) because the sets will change
as users arbitrarily move around or links get randomly blocked.

Random mobility patterns characterizing both UEs and obstacles typically
generate local areas with local statistics that may differ from the others
across the network. To leverage such a diverse structure of the problem,
Multi-Agent Reinforcement Learning (MARL) techniques can be used to exploit
multiple RL agents able to both make decisions based on local observations and
coordinate with other agents. This allows to split a single, complex, and
high-dimensional problem – which would otherwise be intractable – into
several, cooperative, and low-dimensional tasks.

In this article, we address the joint flow routing and link scheduling problem
in mmWave IAB networks, coordinating both access and backhaul transmissions to
maximize the downlink throughput perceived by UEs. We provide the first MARL
framework for such networks, which is able to perform real-time operations,
taking into account the IAB-nodes’ buffer levels to avoid multi-hop
starvation, the stochastic behavior of mmWave communications to rely on the
best links, and the UEs’ mobility to dynamically adapt to changing layouts.
The result is a system that performs a smart interference coordination via the
cooperation of MARL agents, adjusting transmission beams to follow moving UEs
and adapting to intermittent blockages generated by random obstacles. To the
best of our knowledge, no other works in literature deal with adaptive joint
flow routing and link scheduling, via RL, in mmWave IAB networks whose
dynamics are caused by both random link obstructions and UE mobility.

Our contributions are summarized as follows:

  * •

We investigate the problem of joint flow routing and link scheduling in mmWave
IAB networks to maximize the user throughput, taking into account intermittent
links due to user mobility and mobile 3D obstacles.

  * •

We exploit the distributed nature of the problem and propose an MARL-based
approach that splits a combinatorial resource allocation problem into multiple
small problems managed by cooperative agents. This allows to execute resource
allocation operations that adapt in real-time to network and environment
conditions.

  * •

We propose an approach that coordinates the interference among concurrent
backhaul and access links and monitors buffer levels at IAB-nodes to avoid the
starvation of access transmissions due to empty relay buffers.

  * •

We design different MARL settings for the cases where IAB-nodes respectively
operate in full-duplex (FD) or half-duplex (HD) modes.

  * •

We provide a communication and coordination scheme for RL agents, which
enables the practical implementations of online training procedures in real
systems.

  * •

We conduct extensive numerical experiments to evaluate the proposed approaches
and show how they outperform traditional resource allocation approaches.

The rest of the paper is organized as follows. We first discuss related works
in Section 2, then we provide a system overview in Section 3. In Sections 4
and 5, the proposed approaches are detailed, while implementation issues are
discussed in Section 6. The results of the numerical evaluation are showcased
and discussed in Section 7. Finally, Section 8 concludes the article with some
final remarks.

##  2 Related Work

Resource management in multi-hop mmWave networks has been largely investigated
in the recent literature, mainly facing the new challenges brought in by
highly directional transmissions. Among all these works, the traffic routing
and transmission scheduling problem has been carefully investigated [4, 5, 6].
However, only a few of them do consider the link status (i.e., line-of-sight
(LOS), non-LOS (NLOS), outages, etc.). The works [7, 8] find routing paths to
bypass the links interrupted by obstacles. The work in [9] performs a slot-by-
slot link scheduling to maximize the instantaneous throughput considering link
blockage probabilities described as a discrete-time Markov chain. Authors in
[10] deal with the relay selection and link scheduling problem to maximize the
end-to-end throughput, using 3D models of buildings as primary blockage
sources. In [11], heuristic algorithms for user scheduling and power
allocation are proposed to reduce outage occurrences.

Most of the studies on mmWave communications consider users’ mobility only in
handover / user-cell association problems [12, 13]. Only a few works cope with
user mobility when addressing resource allocation problems, but none of them
deals with multi-hop networks. The work in [14] proposes a contextual multi-
armed bandit algorithm to schedule transmissions to users with unknown
positions. In [15], Deep Q-Network (DQN) is exploited to allocate capacity for
up/down link transmissions in a 5G heterogeneous network with high-mobility.
Authors in [16] select routing paths based on the mobility and traffic
conditions, then they perform a link-resource time sharing according to flow
occupations. A particle swarm optimization (PSO) algorithm is described in
[17] to properly manage unmanned aerial vehicle (UAV) in mmWave aerial access
and backhaul networks to serve mobile users. In [18], a band and beam
allocation scheme for mmWave networks is presented, which considers massive-
MIMO systems and user trajectories.

Recent years have witnessed a widespread utilization of Reinforcement Learning
(RL) techniques in mmWave networks. A large part of the works in the
literature deal with throughput maximization. The work in [19] defines a
spectrum allocation for IAB networks that maximizes the sum of log-rates
through double DQN and actor critic techniques. Authors in [20] resort to
regret RL and successive convex approximation to perform route selection and
rate allocation, respectively. Risk-sensitive RL is adopted in [21] to control
transmitter beamwidth and power so as to maximize data rate. In [22], the
authors propose a resource allocation framework based on advantage actor
critic and column generation to maximize the throughput of mmWave IAB
networks.

Some of the other works investigate latency performances. Link scheduling
approaches based on deep deterministic policy gradient (DDPG) [23] and multi-
armed bandit [24] are proposed to minimize the end-to-end latency in mmWave
backhaul networks. Interference management and capacity issues have been faced
as well. The work in [25] allocates capacity between the core network and
mmWave BSs to users subject to blockages. Authors in [26] mitigate the inter-
beam inter-cell interference through joint user-cell association and selection
of number of beams. Finally, MARL emerges as a promising approach to cope with
traffic signal control [27] and resource allocation [28] in vehicular
networks, user association [29] and handover management [30] in mmWave
networks.

The above works show several good examples of the interesting problems arising
in mmWave networks as well as the potential of RL techniques in dynamic
scenarios. However, none of them deals with the complexity and the entire set
of features of the problem of adaptive flow allocation and link scheduling in
mmWave IAB networks, characterized by access and backhaul transmissions, whose
dynamics are caused by both intermittent links and UE mobility. To provide
high user throughput, we propose a multi-agent RL approach able to properly
coordinate interference among links concurrently activated and, at the same
time, take into account the status of IAB-node queues. In this article, we
extend the work in [31], which considers simplified sector-based blockage
model and star-topology backhaul, by introducing a realistic link obstruction
model relying on 3D mobile obstacles, a general tree-topology backhaul, and
both full-duplex and half-duplex working modes at IAB-nodes.

##  3 System Model

We consider a mmWave IAB network that consists of a mmWave base station (BS),
IAB-donor, connected to the core network with a wired connection, and a set of
small mmWave BSs, IAB-nodes, wirelessly backhauled to the IAB-donor using
mmWave frequencies. A mobile user equipment (UE) gets access to the network
via either a direct mmWave link connected with the IAB-donor or multiple hops
through IAB-nodes. A typical example of the IAB scenario is depicted in Fig.
1. Backhaul links are established either between the IAB-donor and an IAB-node
or between two IAB-nodes, while access links connect IAB-donor / IAB-nodes to
UEs. Both backhaul and access communications share the same mmWave frequency
band (i.e., in-band backhaul).

Figure 1: An IAB network scenario with the trajectories of some mobile UEs.

This scenario can be represented as a graph
$\mathcal{G(\mathcal{V},\mathcal{E})}$ where the node set $\mathcal{V}$ is
composed of an IAB-donor, IAB-nodes and UEs, and the edge set $\mathcal{E}$
comprises all the potential links among the nodes. $\mathcal{V}$ can be
further divided into the set of IAB-nodes $\mathcal{R}$ and the set of UEs
$\mathcal{U}$. If not differently specified, the IAB-donor is identified as a
special IAB-node. We consider a tree topology for the backhaul, where IAB-
nodes are connected to the IAB-donor either directly or via multiple hops, as
indicated in 3GPP specifications for IAB network simulations [32], which
assume no more than 10 IAB-nodes organized in simple topologies.

###  3.1 Channel Access

The time domain $\mathcal{T}$ is divided into frames, each of which consists
of $T$ slots with equal length $\delta$. The system follows a space-division
multiple access (SDMA) scheme to take advantage of the high directivity of
mmWave antennas, allowing multiple concurrent transmissions involving both
backhaul and access links to be carried out in each slot. Simultaneous
activation of several links requires the network to satisfy several physical
requirements, such as channel conditions (e.g., interference levels, antenna
patterns), duplex communication modes (i.e., full-duplex, half-duplex),
hardware and radio frequency (RF) chain limitations. We study the network
performance regarding the downlink traffic transfer from the IAB-donor to UEs
when all the IAB-nodes operate either in half-duplex (HD) mode (i.e., either
receiving or transmitting in the same slot) or in full-duplex (FD) mode (i.e.,
able to simultaneously receiving and transmitting data). In HD mode, each
antenna consists of side-by-side array panels where transmission (Tx) and
reception (Rx) share the same panel and each panel is equipped with a single
RF chain. In FD mode, each antenna consists of side-by-side pairs of separate
Tx and Rx array panels [33], each of which is installed with a single Tx or Rx
RF chain. Interference between incoming and outgoing links of an IAB-node
operating in FD mode is assumed to be negligible thanks to ideal self-
interference cancellation and isolation techniques111Since self-interference
cancellation technique is out of scope of this paper, we just make this
assumption in the experiments to obtain numerical results. However, our work
can be applied even if self-interference in any form is included.. Finally, a
UE can receive from at most one IAB-node / IAB-donor in a slot.

A signal-to-interference-plus-noise ratio (SINR) model is used to establish a
successful link: a connection is created only if the SINR at the receiver is
larger than the threshold required by the selected modulation and coding
scheme (MCS). The interference that one link receives from other transmitters
is the sum of the power it receives from all the non-intended transmitters
simultaneously activated. We consider rate adaptation where transmission rates
depend on selected MCSs, which in turn depend on achievable SINR values.

We adopt common path loss and antenna pattern models [34, 35] for mmWave
communications. The path loss model for line-of-sight (LOS) transmission is
defined as:

$PL_{dB}=\alpha+k\cdot 10\cdot\text{Log}\left(\frac{d}{d_{0}}\right),$ (1)

where $\alpha=82.02dB$ and $d_{0}=5m$. $d$ is the path length. The propagation
factor $k$ is 2.36 if $d>d_{0}$ and 2 otherwise. The antenna gain is modeled
with a Gaussian main lobe profile:

$\displaystyle G_{dB}(\phi,\theta)$
$\displaystyle=10\cdot\text{Log}(G_{0})-12\cdot\frac{\phi^{2}}{\phi^{2}_{-3dB}}-12\cdot\frac{\beta^{2}}{\beta^{2}_{-3dB}},$
(2) $\displaystyle G_{0}$
$\displaystyle=\frac{16\pi}{6.76\cdot\phi_{-3dB}\cdot\beta_{-3dB}}.$ (3)

$\phi_{-3dB}$ and $\beta_{-3dB}$ are respectively the elevation and azimuth
half power beamwidths (HPBWs). The $\phi$ and $\beta$ are the elevation and
azimuth angle offsets between the main lobe direction and the direction to the
considered transmitter/receiver. We assume omni-directional reception at UEs
due to the typically limited hardware capabilities.

We would like to remark that the above models are just reasonable choices to
obtain realistic scenarios, and thus, meaningful numerical results. Indeed,
the approach we propose can be applied to any specific channel and antenna
pattern model.

###  3.2 Blockage and Mobility Model

In this work, we consider UEs that move within the service area and links that
can be suddenly blocked by unexpected obstacles. We adopt the well-known
Random Waypoint model [36] for UE mobility, while obstacle blockages are
modeled as follows.

Since mmWave signals can be strongly affected by obstacles, which generate
link interruptions, and UEs mobility requires to consider dynamically
established links, we have developed a dynamic and realistic blockage model.
IAB-nodes are expected to be installed at relatively high places (e.g., lamp
posts, roof tops, etc.) to improve visibility and avoid tampering, thus we
deem that backhaul links are less likely to be blocked. In contrast, access
links are exposed to more recurrent blockages caused by nomadic obstacles
(e.g., pedestrian, transportation traffic, etc.). Based on the above
observation, we realistically apply blockages only to access links222Note that
this is not a limitation of our scenario, but rather an effort to make it more
realistic. Indeed, backhaul link blockages can be straightforwardly included
in the approach if a specific use case needs it..

We consider 3D mobile obstacles (blockers) [37] that, like UEs, move around in
the simulation field according to a Random Waypoint model. Fig. 2(a)
illustrates how the blockage occurs. The obstacle is modeled as a cylinder
standing in the LOS path between the transmitter and the receiver. Fig. 2(b)
shows the corresponding top view where the intersection points between the LOS
path and the blocking cylinder are identified as $A$, $B$ and $C$, having
respectively heights $h_{A}$, $h_{B}$ and $h_{C}$. To determine whether a link
blockage occurs, the following cases, represented in Fig. 2(b), are examined:

  * •

Case 1: when LOS path is neither a secant nor a tangent of the blocker’s
cross-section, there is no blockage in the transmission.

  * •

Case 2: when LOS path is a tangent of the blocker’s cross-section, if
$h_{A}\leq h_{block}$, the blockage occurs; otherwise, no blockage occurs.

  * •

Case 3: when the LOS is a secant of the blocker’s cross-section. A blockage
occurs only if $h_{B}\leq h_{block}$ or $h_{C}\leq h_{block}$.

Every time an obstacle interrupts an access transmission, the UE cannot
receive any bits from the IAB-node, while the path loss model in Eq. (1) is
applied if a link is not affected by obstacles.

Figure 2: 3D blockage model.

UEs can move in the scenario with arbitrary directions and speeds, and they
may undergo link blockages caused by random mobile blockers. This makes access
links short-lived and unstable, thus posing a big challenge to resource
allocation problems, especially to those based on traditional optimization
techniques [22]. Indeed, these techniques rely on static assumptions and
usually require a non-negligible amount of time to provide an optimal solution
for a single network snapshot. Therefore, they cannot be applied to dynamic
scenarios.

Thanks to the capability of learning dynamic statistics, RL is an appropriate
tool to provide a practical and adaptive scheduling approach in such a complex
and dynamic network environment. In particular, MARL techniques will allow us
to split the problem into simpler tasks that multiple agents can perform by
relying on local observations of the environment.

Figure 3: An example of IAB scenario. (a) shows an IAB network with 1 IAB-
donor and 4 IAB-nodes. Backhaul links (dashed lines) form a tree topology and
the number of covered UEs is reported in each sector. A sector is shadowed if
at least a mobile blocker has been detected in the previous slot. (b) is the
top view of an IAB-node equipped with 4 Tx/Rx array panels, each of which
manages 4 sectors (a, b, c, d). (c) includes a table reporting the binary
information expressing UE presence and blockage conditions in the layout shown
in (a).

###  3.3 Network Operations

Access Links: UEs connecting to mmWave IAB networks are expected to work in
dual-connectivity mode [38]. Indeed, user devices will be equipped with both
legacy (i.e., 3GPP FR1) and mmWave (3GPP FR2) interfaces. It is arguable that
control-plane information will be exchanged through the legacy FR1 interfaces
of such architecture, in order to provide better coverage and signal
propagation conditions. Instead, high-throughput user-plane channels can be
established, when available, through mmWave links. In addition, the control-
plane FR1 interface can be used to send UE context information to enable
better network access point selection and configuration. Under these
conditions, we can reasonably assume that each IAB-node is informed about
associated UEs in real time, and in particular, about their positions, channel
conditions, collision occurrences, etc.

Considering realistic nodes as shown in Fig. 3(b), each Tx panel covers a
$180^{\circ}$ area, which is divided into $N_{s}$ sectors that indicate the
possible beam directions. Every time a beam is pointed to a specific sector, a
UE, if any, is served. If more than one UE is present, one of them is randomly
selected as the intended receiver. A collision occurs if a UE is selected as
the receiver from more than one panel in the same time slot. Whenever it
occurs, no bit can be delivered to the UE. Finally, if an access link is
blocked by any obstacle, the data transmission fails. Fig. 3(b) provides a top
view of a node. The node is equipped with 4 Tx array panels (0, 1, 2, 3) and
each panel (agent) manages 4 angular sectors (a, b, c, d). Fig. 3(a) shows an
IAB wireless backhaul where 4 IAB-nodes are connected to an IAB-donor in a
tree topology. Each sector of each panel is marked with the number of UEs
under its coverage and shadowed if any access link in it was obstructed in the
previous slot.

Backhaul Links: As for the backhaul transmissions, we assume they are carried
out through pairs of IAB-donor–IAB-node or IAB-node–IAB-node panels. The two
endpoints of each link focus each other using the reciprocal sector and panel
whose pointing direction is the closest to the one of the LOS segment, as
shown in Fig. 3(a) with dashed lines. Therefore, each child IAB-node of the
tree topology is associated with a sector of a panel at the IAB-donor or at
its parent IAB-node.

Node Buffers: Each IAB-node has a queue (buffer)333We assume that buffer sizes
do not limit the performance of the system. However, queue limits can be
easily added to the RL environment and RL agents can be trained accordingly.
storing bits received via backhaul links from the IAB-donor or its parent IAB-
node. The buffer is shared among all the panels of the IAB-node. Therefore,
the throughput eventually delivered to other IAB-nodes or UEs depends not only
on the link capacities defined by SINR values, but also on the number of
available bits in the buffers. This requires to monitor buffer levels at IAB-
nodes, which must be timely refilled to avoid any impact on downstream
transmissions.

In our approach, each IAB-node informs its parent node about its buffer level
at each time slot. Based on the buffer levels of its child IAB-nodes, a parent
IAB-node/IAB-donor – more specifically, the connected RL agent – will have to
choose to send data bits to an IAB-node via a backhaul link, to refill its
buffer, or directly to a UE via an access link, to improve its throughput.
When IAB-nodes operate in HD mode, if an IAB-donor/IAB-node selects to refill
a child IAB-node’s buffer in a specific slot, the child IAB-node cannot carry
out any transmission in the same slot. Vice versa, if IAB-nodes operate in FD
mode, they can always transmit regardless they are receiving or not from their
parent nodes. Finally, in both FD and HD modes, an IAB-node can receive from
at most one transmitter in each time slot, by using any of its Rx panels.

##  4 Adaptive Routing and Link Scheduling via MARL

RL was born as a tool to optimize decision-making and control through the
experience accumulated by an agent during sequential interactions with an
environment, following a trial-and-error strategy. Specifically, at step $t$,
conditionally to the state $s_{t}\in\mathcal{S}$ of the environment, the agent
selects an action $a_{t}\in\mathcal{A}$ according to its current policy $\pi$
and executes it in the environment. At step $t+1$, based on its reaction to
$a_{t}$, the environment switches to state $s_{t+1}$ and gives a reward
$r_{t}$ back to the agent. When states are partially-observable, an agent can
only collect an observation $o_{t}\in\mathcal{O}$, which contains partial
information of the global state $s_{t}$. Therefore, its policy $\pi$ will
select an action $a_{t}$ conditional to current observation $o_{t}$. During
the interactions, the agent adjusts the policy $\pi$ so as to maximize the
long-term cumulative reward, namely the _expected return_
$\mathbb{E}_{\pi}[G_{t}]=\mathbb{E}[\sum^{\infty}_{k=t+1}\gamma^{k-t-1}r_{k}]$,
where $\gamma$ is a discount factor controlling the importance of a future
reward to the current utility.

A first approach to apply RL to IAB networks is to consider a central network
controller that acts as a single RL agent. However, this requires the agent to
know the global state of the whole network and manage all the transmissions,
resulting in combinatorially large numbers of different states and possible
actions, which will increase exponentially with the size of the network. This
would strongly limit the scalability and flexibility of the approach, which
can be even aggravated by the system dynamics due to the UE mobility. These
reasons motivated us to resort to a Multi-Agent RL (MARL) approach in our
framework, which allows to split the overall complexity into several smaller
problems managed by cooperative agents. Cooperation is fundamental, as
directly applying independent Single-Agent RL (SARL) algorithms to each
individual agent to maximize its own return leads to poor results. Indeed,
interpreting other agents’ decisions as part of the environment would make it
non-stationary as other agents’ policies constantly update. This produces
ineffective learning.

In our MARL approach, we consider multiple RL agents and assign one of them to
each of the $N_{p}$ panels installed at an IAB-node (or at the IAB-donor),
such that each agent can control the beamforming of the panel at each time
slot. Each panel (RL agent) cooperates with other panels to learn the dynamics
of the environment and understand the impact of the other panels’ policies.

The collective goal of the agents is to maximize the throughput (namely, the
number of bits per frame) delivered to UEs. This requires a proper management
of the backhaul and access link transmissions during a frame. How to achieve
high UE throughput without significantly reducing fairness depends on how RL
agents (antenna panels) point their beams, which IAB-node buffer is selected
to be refilled, and how data bits flow through the network, crossing IAB-
nodes. One time slot of the frame corresponds to one step of RL interactions,
that is, each agent executes an action at each slot, according to the policy
$\pi$ available at the beginning of the slot. Two types of actions are
possible for each agent: either transmitting over a backhaul link towards an
associated (covered) IAB-node or activating a specific sector to serve a UE
through an access link. As long as an action is selected, the total number of
bits transmitted in a slot by an IAB-node is limited by either the number of
bits in its buffer or the link capacity achievable with the actual SINR value.

###  4.1 RL Formulation

Considering $|\mathcal{R}|$ IAB-nodes (including the IAB-donor), each equipped
with $N_{p}$ antenna panels, provides a total of $N=|\mathcal{R}|\cdot N_{p}$
agents, indexed by $\mathcal{I}_{p}=\\{1,\dots,N\\}$. Each panel (agent) faces
$N_{s}$ sectors indexed by $\mathcal{I}_{s}=\\{1,\dots,N_{s}\\}$. The
observation space $\mathcal{O}$, action space $\mathcal{A}$, and reward
function of the RL agents are defined as follows, considering IAB-nodes
operating in FD and HD modes.

####  4.1.1 Observation Space

Since IAB-nodes can work in FD and HD modes, we design different observations
accordingly. They contain a mode-specific element and several other common
elements.

Full-duplex Mode: For each agent (panel) $p$, the observation includes the
binary information of:

_1) UE presence_ \- $I^{pres}_{p,s}$, which takes value $1$ for sector $s$ of
agent (panel) $p$ if there are UEs located under the coverage of sector $s$.
This information can be easily estimated by using signaling and context
information (e.g., position, received power, etc.) sent by UEs.

_2) Sector blockage_ \- $I^{block}_{p,s}$, which takes value $1$ if sector $s$
of agent (panel) $p$ encountered a blockage in the previous slot (i.e., the
previous transmission carried out in sector $s$ failed). Blockers moving at
realistic speed can lead to sector (beam) obstructions that last hundreds, or
even thousands, of slots and suddenly disappear in few slots with the
departure of the blockers [39]. A similar behavior repeats in the opposite
transition from blocked to free sectors. Therefore, the blockage status in the
previous slot is the optimal predictor for all the slots but those
experiencing a transition, which are limited in number and whose occurrence
instants are very hardly predictable in any case. An example of the above
binary information vectors is shown in Fig. 3(c).

_3) Child-node buffer level_ \- $L_{n}$, which indicates the number of bits in
the buffer of child IAB-node $n$, reachable through panel $p$. Since we are
interested in defining a policy that maximizes the UE throughput, we assume
full-buffer conditions for the IAB-donor. As for IAB-nodes, instead, their
buffer-level information is essential for parent IAB-donor/IAB-node agents to
plan their buffer-refilling strategies.

The above information is organized in concatenated sub-vectors to form an
observation vector for agent $p\in\mathcal{I}_{p}$ that is collected at each
RL iteration. Namely, we have:

$o_{p}={[[I^{pres}_{p,s}]_{s\in\mathcal{I}_{s}},[I^{block}_{p,s}]_{s\in\mathcal{I}_{s}},[L_{n}]_{n\in\mathcal{R}_{p}}]},$
(4)

where $\mathcal{R}_{p}$ is the set of child IAB-nodes reachable via panel $p$.

Half-duplex Mode: All the previous three sub-vectors also appear in the
observation vector of the HD mode. However, differently from FD mode,
operating in HD mode introduces synchronization issues between parent and
child nodes: if a parent node transmits to a child node, the receiving child
node cannot simultaneously transmit. This may hinder the data delivery in the
tree topology where data bits require multiple backhaul hops to reach UEs.
This poses a big challenge to the cooperation of RL agents, which have to
coordinate several concurrent and dynamic factors (i.e., interference
reduction, buffer refill and collision avoidance).

Therefore, we add a fourth sub-vector, $[B_{n}]_{n\in\mathcal{R}_{p}}$, to the
observation vector, which indicates the number of bits transmitted downstream
in the current slot by every child IAB-node $n$ reachable through the panel
$p$ ($n\in\mathcal{R}_{p}$). This information, together with $L_{n}$, allows
the parent agent to balance between the buffer level and the transmission
opportunity of each child node in order to avoid empty buffers and provide a
good downstream throughput. As a result, the observation vector of the agent
$p\in\mathcal{I}_{p}$ working in HD mode is:

$o_{p}={[[I^{pres}_{p,s}]_{s\in\mathcal{I}_{s}},[I^{block}_{p,s}]_{s\in\mathcal{I}_{s}},[L_{n}]_{n\in\mathcal{R}_{p}},[B_{n}]_{n\in\mathcal{R}_{p}}]}$
(5)

####  4.1.2 Action Space

Each agent $p\in\mathcal{I}_{p}$, working in the HD or FD mode, can choose to
(1) activate one of its $N_{s}$ sectors to transmit to a covered UE ($ACC$
action), to (2) transmit to one of the reachable IAB-nodes in set
$\mathcal{R}_{p}$ ($BH$ action), or to (3) stay silent ($SIL$ action).

Using sector-based transmissions in (1) allows to reduce the impact of the UE
location, which changes over time, on action policies, making them more stable
and robust against mobility. In addition, this permits the same action space
to remain widely applicable even if the number of UEs in the service area is
not constant. Indeed, once a sector is selected, only one UE, if any, will be
randomly selected to be served.

In each slot, every agent tries to establish a link according to the selected
action: activating a sector or connecting to an IAB-node. Concurrent links
interfere with each other and can transfer a number of bits which depends on
the actual SINR value. Whether or not activated links can truly deliver that
number of bits finally depends on whether IAB nodes have enough bits buffered
or an obstacle produces a blockage. If buffer level is the limiting factor,
fairness is pursued by equally sharing buffered bits among the links
simultaneously activated. If an obstacle is present, no bit can be delivered.
The coordination among agents aims at maximizing the number of bits
transferred to UEs, which is obtained by finely tuning the following rewards.

####  4.1.3 Reward Function

We have designed two reward functions for FD and HD, respectively. Similarly
to the definition of the observation space, the reward for the HD case shares
some common elements with the one for the FD case and has an additional
element to prevent IAB-node starvation.

Full-duplex Mode: Each agent (panel) can either (1) activate an access link,
or (2) transmit along a backhaul link, or (3) stay silent. According to these
actions, three types of reward must be designed.

If agent $p$ plays (1) and succeeds in serving a UE, it gets a positive reward
equal to the number of bits $B_{p}$ sent by the panel $p$, multiplied by
$(h(p)+1)$, where $h(p)$ is the number of hops separating the agent $p$’s IAB-
node from the IAB-donor, and normalized by $c_{min}$, the minimum capacity
(minimum MCS) available in the whole network.

Note that the factor $h(p)$ leads to higher rewards to transmissions serving
UEs farther away from the IAB-donor. This allows to efficiently use the
resources of the backhaul network rather than relying on the myopic delivery
of traffic only to nearby UEs, directly connected to the IAB-donor.

If agent $p$ plays (2) and successfully transmits data via a backhaul link,
similarly to the case of an access transmission, the reward is computed as the
number of sent bits $B_{p}$ normalized by $c_{min}$ and multiplied by
$(h(p)+1)$. In addition, the fraction is further multiplied by a weight
$\rho_{BH}\in(0,1)$ that is used to counterbalance the fact that the backhaul
link capacity, and thus the number of transferred bits, is usually remarkably
larger than access link capacity. Without $\rho_{BH}$, agents would largely
prefer to activate backhaul links, accumulating data bits in IAB-nodes’
buffers.

When agent $p$ plays (3), there are two types of outcomes. 1) The buffer of
the IAB-node where $p$ is installed is empty, which is an external limitation
not directly ascribable to the agent’s policy (namely, the panel $p$ belongs
to the set of panels with empty buffer, $p\in\mathcal{I}_{p}^{EB}$).
Therefore, the reward is set to 0 to prevent the agent’s training process from
being biased by its empty queue. 2) If agent $p$, despite having bits in the
buffer, selects to stay silent, it gets a penalty $-\zeta$ (empirically,
$\zeta=1$) in order to incentivize policies that increase the throughput.
Similarly, if agent $p$ has bits and its transmission experiments a blockage
or a collision, it will get a penalty $-\zeta$ for its neglecting the
obstacles or not cooperating well with the other partner agents.

Therefore, the reward of agent $p$ working in FD mode is:

$r_{p}=\left\\{\begin{array}[]{lll}\frac{(h(p)+1)\cdot
B_{p}}{c_{min}},&\text{if }B_{p}>0,p\notin\mathcal{I}^{EB}_{p},ACC\mbox{
act.},\\\ \rho_{BH}\cdot\frac{(h(p)+1)\cdot B_{p}}{c_{min}},&\text{if
}B_{p}>0,p\notin\mathcal{I}^{EB}_{p},BH\mbox{ act.},\\\ 0,&\text{if
}p\in\mathcal{I}^{EB}_{p},\\\ -\zeta,&\text{otherwise}.\end{array}\right.$ (6)

Half-duplex Mode: The reward function of an agent $p$ working in HD mode is
the same as the one of a FD agent, except for two aspects.

For a backhaul transmission to an IAB-node $n$, the agent $p$ gets a reward
computed in the same way as in FD case, but additionally scaled by the number
of bits $L_{n}$ in $n$’s buffer after the reception. This additional scaling
factor gives smaller rewards to the transmissions to nodes with more bits in
the buffer. This helps to avoid both parent-node transmissions that prevent
child nodes from transmitting downstream and large deviations in the buffer
levels of different IAB-nodes.

When agent $p$ does not transmit any bit, the reward is set to $0$ not only if
the IAB-node’s buffer is empty ($p\in\mathcal{I}_{p}^{EB}$), but also if the
IAB-node is receiving data from a parent node in the same slot (
$p\in\mathcal{I}_{p}^{RX}$). To simplify the notation we define
$\mathcal{I}_{p}^{ER}=\mathcal{I}_{p}^{EB}\cup\mathcal{I}_{p}^{RX}$.
Therefore, the reward of agent $p$ working in HD mode is written as follows.

$r_{p}=\left\\{\begin{array}[]{lll}\frac{(h(p)+1)\cdot
B_{p}}{c_{min}},&\text{if }B_{p}>0,p\notin\mathcal{I}^{ER}_{p},ACC\mbox{
act.},\\\ \rho_{BH}\cdot\frac{(h(p)+1)\cdot B_{p}}{c_{min}\cdot
L_{n}},&\text{if }B_{p}>0,p\notin\mathcal{I}^{ER}_{p},BH\mbox{ act.},\\\
0,&\text{if }p\in\mathcal{I}^{ER}_{p},\\\
-\zeta,&\text{otherwise}.\end{array}\right.$ (7)

##  5 Learning Approach

In this section, we detail the learning process for RL agents, whose
components have been described in the previous section. We first introduce
some preliminaries about the single-agent RL techniques, then we present our
multi-agent learning approach, followed by a discussion on communication and
coordination aspects.

###  5.1 Actor and Critic

The goal of an RL agent is to adjust its policy $\pi$ so as to maximize its
_expected return_
$\mathbb{E}_{\pi}[G_{t}]=\mathbb{E}[\sum^{\infty}_{k=t+1}\gamma^{k-t-1}r_{k}]$.

A first type of learning approaches resort to an action-value function
$Q(s_{t},a_{t})=\mathbb{E}_{\pi}[G_{t}|s_{t},a_{t}]$ that corresponds to the
expected return from state $s_{t}$, taking action $a_{t}$ and following policy
$\pi$ afterwards. An RL agent iteratively estimates this action-value function
and selects at each step the action with the maximum function value in the
current state. This is the fundamental idea of value-based RL approaches.
$Q^{\psi}(s_{t},a_{t})$ can be approximated as a $\psi$-parametric function of
state and action, which can take the form of a deep neural network (DNN) with
weights $\psi$. Parameters $\psi$ can be estimated via temporal-difference
approaches by minimizing the regression loss
$\mathcal{L}_{Q}(\psi)=\mathbb{E}_{(s_{t},a_{t},r_{t},s_{t+1})\sim
H}[(Q^{\psi}(s_{t},a_{t})-y)^{2}]$, where
$y=r_{t}+\gamma\mathbb{E}_{a_{t+1}\sim\pi(s_{t+1})}[Q^{\bar{\psi}}(s_{t+1},a_{t+1})]$
is the updated return, $Q^{\bar{\psi}}$ is a moving average of the past Q
functions, and $H$ is a replay buffer storing past agent-environment
interaction data tuples, including states, actions, and rewards.

A second family of learning techniques solve the problem from a different
perspective, which are called policy-gradient RL approaches. They see a policy
as a function indicating the probability of selecting action $a_{t}$ in state
$s_{t}$, parameterized with vector $\theta$,
$\pi_{\theta}(a_{t}|s_{t})=\Pr_{\theta}\\{a_{t}|s_{t}\\}$, which can be
represented by a DNN as well, with $\theta$ as connection weights. Parameters
$\theta$ are updated by applying approximate gradient ascent to
$\mathbb{E}[G_{t}]$, thus considering $\nabla_{\theta}\mathbb{E}[G_{t}]$,
whose unbiased estimate is
$\nabla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})G_{t}$. In addition, $G_{t}$ can
be approximated by its expectation $\mathbb{E}_{\pi}[G_{t}|s_{t},a_{t}]$,
which corresponds to the action-value function $Q^{\psi}(s_{t},a_{t})$. To
reduce the estimate variance during updates, a state-dependent baseline
$b(s_{t})$ value is often subtracted from the unbiased estimate:
$\nabla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})(Q^{\psi}(s_{t},a_{t})-b(s_{t}))$,
where $Q^{\psi}(s_{t},a_{t})-b(s_{t})$ is the advantage of selecting action
$a_{t}$ over the other actions at state $s_{t}$.

The previous two paragraphs briefly outlined the basics of Actor-Critic
technique [40], which is emerging as one of the best-performing RL approaches
as it is derived from a policy-gradient approach, but incorporates the
strength of a value-based approach. In particular, the _critic_ part estimates
the action-value function based on past interactions, thus generating the
values of $Q^{\psi}(s_{t},a_{t})$, while the _actor_ part updates the policy
$\pi_{\theta}(a_{t}|s_{t})$ according to the gradient direction, which in turn
depends on the action-value function generated by the critic444Following the
convention in Actor-Critic techniques, we will interchangeably refer to,
respectively, critic or action-value function $Q$ and actor or policy function
$\pi$ in the remainder of the article.. Note that when considering partially-
observable situations, the policy becomes $\pi_{\theta}(a_{t}|o_{t})$, which
maps partial observation $o_{t}$ into a probability distribution over the
action set. Similarly, the action value function becomes
$Q^{\psi}(o_{t},a_{t})$.

Figure 4: Basic principle of the MARL framework. (a) Architecture overview of
our approach, together with (b) the communication and coordination scheme in
the training framework .

###  5.2 Multi-Agent Learning

The approach we propose in this article is based on the Multi-Actor-Attention-
Critic (MAAC) [41] technique that allows to train multiple agents to pursue a
collective goodness. MAAC has two remarkable advantages: (1) its critic part
allows each agent to automatically consider observations and actions only from
relevant agents (based on the idea of attention), thus filtering out
information not correlated to a performance improvement; (2) it trains
decentralized policies with centrally-computed critics, which allows agents to
use individual policies and independently apply them, once training is
completed. Therefore, we have adapted this technique to our scenario.

The centralized critics can be computed at a central entity located at the
IAB-donor, while actors are installed at IAB-nodes and associated to antenna
panels. The distributed architecture relies on message exchanges between the
IAB-donor and IAB-nodes through direct control-plane links, which can be
established at FR1 frequencies. Message exchanges are required only during the
training phase, leaving the operation phase with fully distributed policies.
Note that, as we will show in the following, only a limited amount of
information has to be exchanged during training to achieve good results.
Considering our MARL-based approach, its architecture can be seen in Fig.
4(a).

Denoting observations, actions, rewards and policy parameters of all the $N$
agents as respectively $o=(o_{1},\dots,o_{N})$, $a=(a_{1},\dots,a_{N})$,
$r=(r_{1},\dots,r_{N})$ and $\theta=(\theta_{1},\dots,\theta_{N})$555To
simplify the notation when dealing with multiple agents, we implicitly assume
that observations, actions and rewards occur at time step $t$, f.i., $o\equiv
o_{t}$. Instead, we use subscript $i$ in $o_{i}$ to indicate the observation
at time $t$ of the $i$-th agent. Observations occurring at time $t+1$ are
indicated as $o^{\prime}$., the key idea of MAAC is to train each agent $i$’s
individual policy $\pi_{\theta_{i}}(a_{i}|o_{i})$ by means of its action-value
$Q^{\psi}_{i}(o,a)$. The value of $Q^{\psi}_{i}(o,a)$ is centrally computed by
using information from other relevant agents, according to the attention
mechanism. Therefore, agent $i$’s $Q_{i}^{\psi}(o,a)$ depends not only on its
own observation $o_{i}$ and action $a_{i}$, but also on those of the other
agents. It takes the following form:

$\displaystyle Q^{\psi}_{i}(o,a)=f_{i}(e_{i}(o_{i},a_{i}),x_{\setminus i}),$
(8) $\displaystyle x_{\setminus i}=\sum\nolimits_{j\neq i}\omega_{j}\cdot
g(e_{j}(o_{j},a_{j})),$ (9)

where $f_{i}$ consists of one fully connected layer with leaky ReLU non-
linearity and one linear layer, while $e_{i}$ is an embedding function
implemented as a fully connected layer with leaky ReLU. The contribution
$x_{\setminus i}$ to agent $i$ from the other agents is a weighted sum of
functions of the other agents’ embedding functions. In particular, $g$ is a
fully connected layer with leaky ReLU. Weight $\omega_{j}$ is the attention
weight associated to the information provided by agent $j$, which is optimized
during the training phase, together with the other weights of the neural
network. The value of $\omega_{j}$ is computed according to a similarity value
between $e_{i}$ and $e_{j}$, which is computed using a matching approach
between “query” based on $e_{i}$ and “key” based on $e_{j}$ (See [41] for
further details). In Eqs. (8) and (9), $f_{i}$ and $e_{i}$ are _unique parts_
to each agent, while $g$ and $\omega_{j}$ are shared parts among all the
agents. They can be visualized in Fig. 4(a). The centralized action-value
function $Q^{\psi}_{i}(o,a)$ of agent $i$, encircled with a black dot line,
includes the black shared critic NN and an orange black unique critic NN.

As mentioned before, functions and parameters in Eqs. (8) and (9) are
implemented as DNNs, whose weight vector $\psi$ is updated considering a
replay buffer $H$ that stores history trajectories in the form of
$(o,a,r,o^{\prime})$ entries ($o^{\prime}$ is the next observation, resulting
from the application of action $a$), which summarize the interactions occurred
in the previous steps. In particular, based on a set of entries randomly
sampled from the replay buffer, we update $\psi$ to minimize the following
joint regression loss function:

$\displaystyle\mathcal{L}_{Q}(\psi)=\sum^{N}_{i=1}\mathbb{E}_{(o,a,r,o^{\prime})\sim
H}[(Q^{\psi}_{i}(o,a)-y_{i})^{2}],$ (10) $\displaystyle
y_{i}=r_{i}+\gamma\mathbb{E}_{\bar{a}^{\prime}\sim\pi_{\bar{\theta}}(o^{\prime})}[Q_{i}^{\bar{\psi}}(o^{\prime},\bar{a}^{\prime})-\tau\log(\pi_{\bar{\theta}_{i}}(\bar{a}^{\prime}_{i}|o^{\prime}_{i}))],$
(11)

where $Q^{\bar{\psi}}$ and $\pi_{\bar{\theta}}$, respectively called target
action-value functions and target policies, are moving averages of the past
action-value and policy functions (used to stabilize the training), while
$\bar{a}^{\prime}_{i}$ is the next action that agent $i$ would select by
applying the target policy $\pi_{\bar{\theta}_{i}}$ to the next observation
$o^{\prime}_{i}$. The logarithmic term in Eq. (11) is called policy entropy.
It encourages the exploration by promoting random selections, thus reducing
the probability to converge to deterministic policies with poor local optima.
Parameter $\tau$ is the trade-off weight used to balance the importance of the
reward maximization over the random exploration. Finally, note that the
evaluation of this loss function requires a joint optimization of $\psi$
across all the individual action-value functions $Q_{i}^{\psi}(o,a)$,
therefore it is performed at the central entity.

Once $Q^{\psi}_{i}(o,a)$ is updated using Eq. (10), the individual policy
$\pi_{\theta_{i}}(a_{i}|o_{i})$ of each agent $i$ (shown as an orange actor NN
in Fig. 4(a)) can be updated as well, according to a gradient ascent approach
over DNN weights $\theta_{i}$:

$\displaystyle\nabla_{\theta_{i}}J(\pi_{\theta})=\mathbb{E}_{o\sim
H,\hat{a}\sim\pi_{\theta}}\left[\vphantom{Q^{\psi}_{i}}\nabla_{\theta_{i}}\log(\pi_{\theta_{i}}(\hat{a}_{i}|o_{i}))\cdot\right.$
$\displaystyle\left.\left(-\tau\log(\pi_{\theta_{i}}(\hat{a}_{i}|o_{i}))+Q^{\psi}_{i}(o,\hat{a})-b(o,\hat{a}_{\backslash
i})\right)\right].$ (12)

Each policy network with $\theta_{i}$ is composed of three linear layers and a
final leaky ReLU non-linearity. The baseline $b\left(o,\hat{a}_{\backslash
i}\right)$ allows to reduce the variance of the gradient and is computed by
averaging $Q^{\psi}_{i}(o,\hat{a})$(i.e.,
$Q^{\psi}_{i}(o,(\hat{a}_{i},\hat{a}_{\backslash i}))$) over all the possible
actions of agent $i$ ($\hat{a}_{i}$) according to the policy distribution
$\pi_{\theta_{i}}$, keeping fixed the actions of the other agents
($\hat{a}_{\backslash i}$). Note that the update of $\theta_{i}$, although
based on a per-agent action-value $Q^{\psi}_{i}(o,\hat{a})$, requires the
knowledge of the other agents’ observations sampled from $H$ and the other
agents’ action probabilities expressed by $\pi_{\theta}$.

##  6 Communication and Coordination Scheme

Our MARL-based approach is characterized by per-agent policy functions
$\pi_{\theta_{i}}(a_{i}|o_{i})$ ($i\in\mathcal{I}_{p}$) based on DNNs that
determine the actions of individual agents according to local observations.
These policy functions must be tuned to the specific environment (i.e., node
locations, blockage behavior, and user mobility), thus an initial training
phase is needed to adjust DNNs’ weights. Once trained, each agent can select
its action in isolation based on its own policy function and local
observation, with no need to exchange data with other agents. Note that the
training procedures can be re-activated at any point in time during the
operation of the IAB system, f.i., periodically or when a substantial
performance decrease is detected. The training cycle is shown in Fig. 4(b).

The training phase is based on the centralized update of the action-value
functions $Q^{\psi}_{i}(o,a)$ ($i\in\mathcal{I}_{p}$) according to Eqs. (10)
and (11). This task requires the central entity to have a centralized replay
buffer $H$ storing past experiences of all the agents.

During network operations, agent $i$ plays action $a_{i}$ at each step $t$,
selected by using its own policy $\pi_{\theta_{i}}$ on the basis of its
observation $o_{i}$. This leads to a reward $r_{i}$ and a new observation
$o^{\prime}_{i}$. Each new tuple $(o_{i},a_{i},r_{i},o^{\prime}_{i})$ is
temporarily and locally accumulated by each agent $i$ which periodically sends
a tuple batch $tup_{i}=\\{(o_{i},a_{i},r_{i},o^{\prime}_{i})\\}_{T_{upd}}$ to
the central entity considering the last $T_{upd}$ steps. The central entity
merges received tuples into vector tuples $\\{(o,a,r,o^{\prime})\\}_{T_{upd}}$
and inserts them into the centralized replay buffer $H$ used by the learning
algorithm.

The batch update allows to perform DNN training in parallel to network
operations. Indeed, when a concurrent training phase takes place, the actions
performed during network operations will be selected according to the current
(sub-optimal) version of the policy functions, which will be updated and
improved after few interactions. Note that only the experience collected after
the policy update will be sent to the central entity for the critic updates,
as illustrated by the orange dashed arrow in Fig. 4(b), while other
interactions are used to keep the IAB network in operation.

In each periodic update, the central entity samples a random mini-batch $h$
from $H$ consisting of a number $h_{size}$ of vector tuples (i.e.,
$h=\\{(o,a,r,o^{\prime})\\}_{h_{size}}$) and uses it in the minimization of
the joint regression loss function (10) to update $Q^{\psi}_{i}(o,a)$
($i\in\mathcal{I}_{p}$). Namely, for each vector tuple in $h$, the central
entity updates $\psi$ according to the following equations:

$\displaystyle\mathcal{L}_{Q}(\psi)=\sum_{i\in\mathcal{I}_{p}}[(Q^{\psi}_{i}(o,a)-y_{i})^{2}],$
(13) $\displaystyle
y_{i}=r_{i}+\gamma[Q_{i}^{\bar{\psi}}(o^{\prime},\bar{a}^{\prime})-\tau\log(\pi_{\bar{\theta}_{i}}(\bar{a}^{\prime}_{i}|o^{\prime}_{i}))].$
(14)

Note that, while mini-batch $h$ is generated at the central entity,
$\bar{a}^{\prime}_{i}$ and the value of
$\log(\pi_{\bar{\theta}_{i}}(\bar{a}^{\prime}_{i}|o^{\prime}_{i}))$ are
received from remote agents. After the update, target action-value functions
$Q_{i}^{\bar{\psi}}$ ($i\in\mathcal{I}_{p}$) are updated using a moving
average with an update rate $\kappa$.

The overall information sent by each IAB-node array panel (remote agent) $i$
to the IAB-donor (centralized entity), shown by blue arrows ① in Fig. 4, is
dominated by the set of the tuples
$tup_{i}=\\{(o_{i},a_{i},r_{i},o^{\prime}_{i})\\}_{T_{upd}}$. The size of a
tuple is mainly determined by the size of an observation $o_{i}$, which
consists of $2N_{s}+N^{ch}_{i}L$ bits for FD case and
$2N_{s}+N^{ch}_{i}\cdot(L+B)$ bits for HD case, where $2N_{s}$ is given by the
UE presence ($I_{i,s}^{pres}$) and sector blockage ($I_{i,s}^{block}$)
bitmaps, $L$ and $B$ are respectively the numbers of bits used to indicate the
buffer level and the amount of the delivered data, and $N^{ch}_{i}$ is the
number of child IAB-nodes connected to $i$, typically small (e.g., $2$-$4$).
Together with tuples, each IAB-node additionally sends for each tuple
$t_{i}=(o_{i},a_{i},r_{i},o^{\prime}_{i})\in tup_{i}$:

  * •

action $\hat{a}_{i}$ that it would play in front of current observation
$o_{i}$ in $t_{i}$, selected according to the current policy function
$\pi_{\theta_{i}}(\hat{a}_{i}|o_{i})$;

  * •

action $\bar{a}^{\prime}_{i}$ that it would play in front of the next
observation $o^{\prime}_{i}$ in $t_{i}$, selected according to its target
policy function $\pi_{\bar{\theta}_{i}}(\bar{a}^{\prime}_{i}|o^{\prime}_{i})$;

  * •

the value of
$\log(\pi_{\bar{\theta}_{i}}(\bar{a}^{\prime}_{i}|o^{\prime}_{i}))$,
conditional to $o^{\prime}_{i}$ in $t_{i}$ and according to the target policy
function $\pi_{\bar{\theta}_{i}}(\bar{a}^{\prime}_{i}|o^{\prime}_{i})$.

Actions $\bar{a}^{\prime}_{i}$ and the values of
$\log(\pi_{\bar{\theta}_{i}}(\bar{a}^{\prime}_{i}|o^{\prime}_{i}))$ are used
in the centralized action-value function updates according to Eqs. (13) and
(14), while $\hat{a}_{i}$ is used at the central entity to compute
$Q^{\psi}_{i}(o,\hat{a})$ and $b(o,\hat{a}_{\backslash i})$ that will be
redistributed to all the agents to perform policy updates.

Once the $Q^{\psi}_{i}(o,a)$ ($i\in\mathcal{I}_{p}$) functions are updated,
the central entity immediately sends to each agent $i$ the needed information
to update its policy (shown by green arrows ② in Fig. 4). In particular, the
information includes:

  * •

observation set $\\{o_{i}\\}_{h_{size}}$ derived from tuples in $h$;

  * •

the values of $Q^{\psi}_{i}(o,\hat{a})$ and $b(o,\hat{a}_{\backslash i})$.

Each agent $i$ performs gradient ascent to update $\theta_{i}$, as in Eq.
(12), and obtains a new $\pi_{\theta_{i}}(a_{i}|o_{i})$, which will be locally
used in the next steps of the learning phase until an updated policy is
generated. In practice, after new data are received from the central entity,
agents’ policy parameters are modified as follows:

$\displaystyle\nabla_{\theta_{i}}$ $\displaystyle
J(\pi_{\theta})=\nabla_{\theta_{i}}\log(\pi_{\theta_{i}}(\hat{a}_{i}|o_{i}))\cdot$
$\displaystyle\left(-\tau\log(\pi_{\theta_{i}}(\hat{a}_{i}|o_{i}))+Q^{\psi}_{i}(o,\hat{a})-b(o,\hat{a}_{\backslash
i})\right).$ (15)

Note that action $\hat{a}_{i}$ is the same as the one sent to the central
entity, thus it is generated at agents before sending information to the
central entity and then stored to be used for computing Eq. 15. As a final
task, each agent updates its target policy with a moving average.

###  6.1 Dealing with the Temporal Dynamics

The communication and coordination scheme discussed above is based on DNN
updates, which require interactions between the central entity at IAB donor
and distributed agents at IAB nodes. The time between two updates directly
impacts on the convergence speed, but it is potentially arbitrary. Indeed, it
can be set according to the length of an arbitrary update episode, the time to
process an update, or the replay-buffer sampling factor to generate a mini-
batch. However, to strike a balance between model training efficiency and
message exchange costs, a proper update interval needs to be set.

Theoretically, new information collected at the agents can be accumulated and
sent to the central entity at any time. Likewise, critic updates can be
performed at any time, when a sufficient number of new tuples have been
inserted in the replay buffer. Policy updates at distributed agents, instead,
must be performed only when new Q-function values are received from the
central entity.

In addition, note that coordination information exchanges may be characterized
by a non-negligible latency (e.g., $T_{l}^{A}$ and $T_{l}^{C}$ in Fig. 5),
which introduces some delay between last-collected interaction tuples and DNN
updates. This may not be a critical issue, as model updates are based on
random samples of the past tuples included in the replay buffer $H$, thus they
can be performed at arbitrary time points instead of waiting for the data
arrival. However, we provide two possible training timelines to show the
latency impact.

Solution 1. Updates can be triggered by the central entity as soon as it
receives new tuples from remote agents, as illustrated in Fig. 5(a). It
updates the central DNNs, then, sends new values to the agents, which update
their policies.

Solution 2. Updates at the central entity and at distributed agents are
performed periodically, based on timers. Central DNN updates are performed
with data sampled from the most recent content of the replay buffer, while
distributed agents’ DNNs are updated with the latest data received from the
central entity. Fig. 5(b) illustrates this idea, where colored DNNs’ updates
are based on the data transmitted through the arrows with the same colors.
Compared with Solution 1, this solution allows to overcome the effect of
transmission latency, thus accelerating the training process.

Finally, note that multiple DNN updates can be performed at each update time
instant, by consecutively sampling $K$ random mini-batches from the replay
buffer and updating DNNs’ weights according to the content of each of them.
Moreover, since the network needs an initial transient period to reach a
realistic steady state (i.e., stationary buffer levels and link behaviors), we
usually need a warm-up period (e.g., the first $T_{wup}$ steps), during which
no update is performed.

Following the approach of Solution 2, we perform a numerical analysis where
the IAB-donor and IAB-nodes’ DNNs are periodically updated assuming no
transmission latency, $K$ consecutive updates, and an initial warm-up period.

Figure 5: An illustrative example of the transmission latency impact on the
model training.

##  7 Numerical Results

In this section, we discuss the results of the performance tests of our MARL-
based resource allocation approach on several instances considering IAB-nodes
working in FD or HD mode, mobile UEs, and random link failures caused by
mobile obstacles. Every value shown in the figures of this section is the
result of an average over 10 random instances.

###  7.1 Scenario Settings

In line with 3GPP NR IAB simulation guidelines [1], we consider a
$300$m$\times 300$m service area where 1 IAB-donor is located at the left-side
midpoint and 4 IAB-nodes are randomly deployed in the area. A set of 30 UEs
move around in the area, with random initial positions and directions. The
considered heights of IAB-donor, IAB-node and UE devices are $25$m, $6$m and
$1.5$m, respectively.

Both the IAB-donor and IAB-nodes are equipped with $N_{p}$=$4$ antenna panels,
each of which manages $N_{s}$=$5$ sectors. The transmission power of each
panel at the IAB-donor and at IAB-nodes is respectively $29.3$ dBm and $20.3$
dBm. The receiver noises at IAB-node and UE are $-84.023$ dBm and $-82.023$
dBm, respectively. The azimuth Half-Power BeamWidths (HPBWs) for the IAB-donor
and IAB-nodes are $\pi/36$ and $\pi/12$, and the elevation HPBW is $\pi/4$ for
both. SINR thresholds and rates for MCSs in access and backhaul links are
those indicated in MCS index table 3 for PDSCH in the 3GPP NR specification
[42]. Finally, one frame consists of $80$ slots, each with a duration
$\delta$=$125$$\mu$s.

User mobility: UEs move in the playground according to a Random Waypoint model
[36]. Specifically, a UE randomly selects a direction in the angular range
$\xi\in[-180^{\circ},180^{\circ}]$ from the current direction. Then it travels
along the selected direction with a constant speed uniformly chosen within the
range $[2,20]$m/s ( $[20,60]$m/s or $[60,100]$m/s in the extended analysis).
Speeds and directions of different UEs are selected independently and
randomly. After moving for $t_{m}\in[2,6]$s, a UE pauses for an interval
$t_{p}\in[0,1]$s before resuming. UEs bounce back when they reach the area
boundary.

Obstacles: We assess the performance of our approach according to two levels
of obstacle densities in the area and refer to them as _low_ and _high_. They
are implemented by dropping in the simulated service area, respectively, $15$
and $60$ cylindrical obstacles with a radius of $2.5$m and a height of $2$m.
They move at a speed of $2$m/s - $20$m/s following the same Random Waypoint
model applied to UEs.

Figure 6: Training curves in the FD and HD cases (UEs’ speeds: 2m/s-20m/s,
high density).

###  7.2 MARL Model Settings

We train the DNN models of our approach for at most $5000$ episodes, each of
which consists of $80$ steps (slots). This corresponds to a training period of
$5000$ frames, thus a total time of $50$s. All DNN models have a hidden
dimension of $128$. We consider $K$=$4$ consecutive DNN updates, a data
collection period for $T_{upd}$=$100$ steps, and a warm-up period of
$T_{wup}$=$10$ episodes. The updates are performed via Adam optimizer with a
learning rate of $0.001$ for both distributed policies and centralized
critics. The weight $\rho_{BH}$ for backhaul transmission in the reward
function is set to $0.8$. The discount factor $\gamma$ is $0.99$ and the
weight $\tau$ of the policy entropy is set to $0.01$. The weight vector
$\bar{\psi}$ of target critic network, similarly to $\bar{\theta}$ of target
actor network, is updated via $\bar{\psi}=\kappa\bar{\psi}+(1-\kappa)\psi$
with update rate $\kappa=0.001$. The replay buffer $H$ has a maximum length of
$10^{6}$ entries, and each update uses mini-batches of $1024$ entries,
randomly sampled from $H$.

Training curves expressing the total traffic volume delivered in a frame from
the IAB-donor to all the UEs (which corresponds to the throughput objective of
the resource optimization problem) are shown in Fig. 6, where IAB-nodes
operate in FD and HD modes, respectively. To better appreciate the learning
behavior, we only show the first $1000$ episodes, which involve the key
performance-improving period before convergence. As we can see, the training
process is immediately effective in increasing the throughput in both FD and
HD. Indeed, blue and orange curves can reach high values just after less than
$100$ episodes, showing the model can learn fast from the experience.
Nevertheless, we train it up to $5000$ steps to let it accumulate sufficient
experience and eventually obtain a stable performance in any situation.

###  7.3 Performance Analysis

(a) Avg. bit volume delivered per frame.

(b) Avg. percent of served UEs per frame.

(c) Avg. traffic volume split per frame.

Figure 7: Performance comparison of the four schemes in the FD case.

(a) Avg. bit volume delivered per frame.

(b) Avg. percent of served UEs per frame.

(c) Avg. traffic volume split per frame.

Figure 8: Performance comparison of the four schemes in the HD case.

(a) UE data rate in the FD case.

(b) UE data rate in the HD case.

Figure 9: Box plots of the average UE data rate per frame. Bottom, middle and
top lines indicate respectively the 25th percentile, median, 75th percentile
of the rate distribution. Green triangles indicate mean data rates.

(a) CDF of average UE data rate.

(b) CDF of per-frame UE data rate.

Figure 10: CDFs of UE data rates achieved at different UE speeds and duplex
modes.

In our previous work [22], we have shown how optimization-based resource
allocation approaches cannot efficiently cope with the dynamics emerging in
mmWave environments. Adaptive RL solutions typically lead to much better
performance. However, to the best of our knowledge, all the RL-based
scheduling approaches proposed so far for mmWave IAB networks do not consider
mobile UEs, therefore they are inapplicable to the scenario we are
considering. Therefore, we compare our approach based on MARL (MARL in the
following) against the following three schemes:

  * •

MARL-NoIAB: Our MARL-based approach where IAB-node relaying is disabled,
forcing all UEs to connect directly to the IAB-donor.

  * •

RR: A Round-Robin scheme where each panel of the IAB-donor and IAB-nodes
iteratively serves slot-by-slot every UE under its coverage in a traditional
round-robin fashion. If the number of bits in a queue of an IAB-node drops
below a threshold, it will be refilled by its parent IAB-node in the next
slot. This is a common scheduling scheme when dealing with wireless access
transmissions.

  * •

RND: A random scheme where each panel randomly picks an action from its
candidate action set.

We begin the analysis for both FD and HD cases by considering typical UE urban
speeds, in the range of $2$m/s - $20$m/s, to assess the impact of different
obstacle densities on the performance, as shown in Figs. 7-9. Then, we extend
our analysis to consider different speed ranges as reported in Fig. 10.

Traffic Volume and Data Rate: Figs. 7(a) and 8(a) show, respectively in the
case of FD and HD, the average overall traffic volume delivered to UEs in one
frame. Our MARL scheme can achieve an improvement of $30\%$ (FD) and $13\%$
(HD) over the common RR scheme, as it can better coordinate interference among
links and adapt its decisions to UEs’ mobility, obstacle obstructions, and
IAB-nodes’ queue levels. The smaller gain in the HD mode is mainly due to the
reduction of the degrees of freedom caused by duplex constraints at IAB-nodes.
Indeed, the performance in the HD mode is strongly driven by backhaul
transmissions to refill IAB-node buffers, which preclude access transmissions.
Therefore, the overall result is less sensitive to the smart scheduling of
access transmissions provided by MARL that is very effective in considering
obstacles and mobility. In practice, these HD limitations prevent smart
resource allocation schemes from fully exploiting all available wireless
resources.

Another evident aspect is that MARL and RR schemes outperform MARL-NoIAB
scheme, demonstrating that IAB nodes can remarkably boost the network
throughput, even in simple tree-topologies as those recommended by 3GPP
specifications. Note that MARL-NoIAB shows the same performance in FD and HD
as it does not rely on IAB-nodes. Finally, compared to RR and RND schemes,
MARL can better take advantage of the flexibility provided by FD operations,
producing a larger improvement over HD than in the other approaches.

The number of bits downloaded by each UE in a frame produces the UE data rate.
A comparison of the data rate averaged over the entire simulation achieved by
the four schemes, under low and high obstacle densities, is reported in Figs.
9(a) and 9(b), respectively, in the cases of FD and HD. The lower and upper
extremes of a bar respectively indicate the 25th percentile and 75th
percentile of the distribution, the middle line is its median, and the green
triangle indicates the mean value. As expected, higher obstacle densities lead
to lower mean and top rates.

Given the direct relationship between frame traffic volume and data rate, we
can observe the same general trends as the ones in Figs. 7(a) and 8(a).
However, analyzing data rate figures we can further notice that the main
advantage of MARL scheme over RR is due to a smarter exploitation of the UEs
in good locations and channel conditions. Indeed, although showing similar
median data rates, MARL allows to achieve a much better mean value than RR.
This proves the ability of our approach to discover the most effective
strategy to increase the overall throughput.

Coverage: Figs. 7(b) and 8(b) indicate the average percentage of served UEs
per frame. We adopt two definitions of served UE: the first considers a UE
served if experimenting an average data rate larger than $0$Mbps, the second
definition increases the minimum data rate to $50$Mbps. Similarly to the
above, the results for low and high obstacle densities are shown. The first
aspect to note is that IAB-nodes have a strong impact on the coverage by
remarkably enlarging the percentage of served UE, as we can see if we compare
the results of MARL scheme with those of MARL-NoIAB scheme.

Some interesting aspects further emerge: considering the minimum data rate of
$0$Mbps, RR and RND schemes show higher percentages of served UEs than MARL
scheme. This is a consequence of the throughput-vs.-fairness trade-off, while
RR and RND schemes reach all UEs with the same probability, thus providing the
best fairness, MARL scheme tends to preferably serve the UEs that provide the
best overall throughput. However, we can also note that the difference
remarkably reduces when considering UEs that experience at least $50$Mbps.
This confirms that the service guaranteed by RR and RND schemes is mainly
driven by UEs with low data rates. This further demonstrates that the adaptive
resource allocation of our MARL scheme can very effectively deal with
intermittent links.

Backhaul Load: Figs. 7(c) and 8(c) provide an insight into the traffic volume
delivered to UEs via the IAB wireless backhaul per frame. The upper
translucent part of each bar represents the average percentage of the traffic
volume received by UEs from IAB-nodes via multi-hop, while the lower opaque
part reports the complementary percentage of the traffic directly received
from the IAB-donor. We can see that MARL scheme heavily resorts to backhaul
IAB-nodes when operating in FD mode, even if the larger panels and the higher
transmission power of the IAB-donor may lead UEs to directly connect to it. In
the HD case, all the schemes reduce the load on the wireless backhaul. Indeed,
HD IAB-nodes are less effective relaying nodes, causing transmission
concurrency problems along the tree-like topology of the wireless backhaul.

UE Speed Sensitivity: Fig. 10 shows the performance of MARL scheme over
different speed ranges in both FD and HD cases. Fig. 10(a) shows the
cumulative distribution function (CDF) of the average data rate that a UE can
experiment in a simulation. We can see that in both FD and HD cases,
increasing speeds lead to a deterioration of the UE data rate. This was
expected as fast moving UEs naturally create beam tracking issues. However,
even at the extremely high speeds of $[20,60]$m/s, which can be rarely seen in
the urban scenarios where such networks are envisioned, the impact on the
performance is limited.

Fig. 10(b) shows the CDF of the data rate a UE achieves in a single frame.
Despite different speed ranges, the curves considering the same duplex mode
almost overlap. This implies that the effect of different UE speeds has in
practice a negligible impact on the distribution of per-frame data rates
across different UEs.

##  8 Conclusion

In this article, we have investigated the resource allocation problem in
mmWave 5G IAB networks where user mobility and random obstructions caused by
mobile obstacles produce strong network dynamics. Indeed, they generate short-
lived access links and link-failure statistics that vary across different
regions of the same service area. Leveraging these local network behaviors, we
have proposed an MARL-based approach that splits a combinatorial monolithic
SARL problem, characterized by huge network state and action spaces, into
smaller problems managed by different MARL agents located at IAB-nodes.

Through the cooperation among these MARL agents, we have developed a resource
allocation framework that can coordinate link interference, monitor IAB-node
buffer levels, and capture network dynamics. We have designed different MARL
setups for full-duplex and half-duplex node operations. Moreover, we have
provided a communication and coordination scheme for the training procedure
that can solve potential feasibility issues (e.g., temporal dynamics) of real
systems. The results of our numerical tests have shown that our MARL-based
scheme can achieve good throughput performance without significantly harming
the network fairness.

## Acknowledgments

This work was partially supported by China Scholarship Council (CSC) Grant No.
201806470077.

## References

  * [1] 3GPP  “Study on integrated access and backhaul, TR 38.874”
  * [2] Dingwen Yuan, Hsuan-Yin Lin, Jörg Widmer and Matthias Hollick  “Optimal joint routing and scheduling in millimeter-wave cellular networks”  In _IEEE Conference on Computer Communications (INFOCOM)_ , 2018, pp. 1205–1213
  * [3] Antonio Capone, Ilario Filippini, Stefano Gualandi and D Yuan  “Resource optimization in multi-radio multi-channel wireless mesh networks”  In _Mobile Ad Hoc Networking: Cutting Edge Directions, Second Edition_ Wiley Online Library, 2013, pp. 239–274
  * [4] Michele Polese et al.  “Distributed path selection strategies for integrated access and backhaul at mmWaves”  In _IEEE Global Communications Conference (GLOBECOM)_ , 2018, pp. 1–7
  * [5] Jorma Kilpi et al.  “Link scheduling for mmWave WMN backhaul”  In _IEEE International Conference on Communications (ICC)_ , 2017, pp. 1–7  IEEE
  * [6] Mohamed Saad and Saeed Abdallah  “On millimeter wave 5G backhaul link scheduling”  In _IEEE Access_ 7 IEEE, 2019, pp. 76448–76457
  * [7] Juan García-Rois et al.  “Delay-aware optimization framework for proportional flow delay differentiation in millimeter-wave backhaul cellular networks”  In _IEEE Transactions on Communications_ 66.5 IEEE, 2018, pp. 2037–2051
  * [8] Yong Niu et al.  “Relay-assisted and QoS aware scheduling to overcome blockage in mmWave backhaul networks”  In _IEEE Transactions on Vehicular Technology_ 68.2, 2019, pp. 1733–1744
  * [9] Zhifeng He, Shiwen Mao, Sastry Kompella and Ananthram Swami  “Minimum time length scheduling under blockage and interference in multi-hop mmWave networks”  In _IEEE Global Communications Conference (GLOBECOM)_ , 2015, pp. 1–7
  * [10] Qiang Hu and Douglas M Blough  “Relay selection and scheduling for millimeter wave backhaul in urban environments”  In _IEEE 14th International Conference on Mobile Ad Hoc and Sensor Systems (MASS)_ , 2017, pp. 206–214
  * [11] Chun-Han Yao, Yin-Yi Chen, Biswa PS Sahoo and Hung-Yu Wei  “Outage reduction with joint scheduling and power allocation in 5G mmWave cellular networks”  In _2017 IEEE 28th Annual International Symposium on Personal, Indoor, and Mobile Radio Communications (PIMRC)_ , 2017, pp. 1–6  IEEE
  * [12] Hamza Khan et al.  “Reinforcement learning-based vehicle-cell association algorithm for highly mobile millimeter wave communication”  In _IEEE Transactions on Cognitive Communications and Networking_ 5.4 IEEE, 2019, pp. 1073–1085
  * [13] Sara Khosravi, Hossein Shokri-Ghadikolaei and Marina Petrova  “Learning-based handover in mobile millimeter-wave networks”  In _IEEE Transactions on Cognitive Communications and Networking_ 7.2 IEEE, 2020, pp. 663–674
  * [14] Ryangsoo Kim et al.  “Online learning-based downlink transmission coordination in ultra-dense millimeter wave heterogeneous networks”  In _IEEE Transactions on Wireless Communications_ 18.4, 2019, pp. 2200–2214
  * [15] Fengxiao Tang, Yibo Zhou and Nei Kato  “Deep reinforcement learning for dynamic uplink/downlink resource allocation in high mobility 5G HetNet”  In _IEEE Journal on Selected Areas in Communications_ 38.12, 2020, pp. 2773–2782
  * [16] Emmanouil Pateromichelakis and Konstantinos Samdanis  “Context-aware joint routing & scheduling for mm-Wave backhaul/access networks”  In _IEEE Global Communications Conference (GLOBECOM)_ , 2018, pp. 1–6
  * [17] Nikita Tafintsev et al.  “Aerial access and backhaul in mmWave B5G systems: Performance dynamics and optimization”  In _IEEE Communications Magazine_ 58.2 IEEE, 2020, pp. 93–99
  * [18] Li-Hsiang Shen and Kai-Ten Feng  “Mobility-aware subband and beam resource allocation schemes for millimeter wave wireless networks”  In _IEEE Transactions on Vehicular Technology_ 69.10 IEEE, 2020, pp. 11893–11908
  * [19] Wanlu Lei, Yu Ye and Ming Xiao  “Deep reinforcement learning based spectrum allocation in integrated access and backhaul networks”  In _IEEE Transactions on Cognitive Communications and Networking_ , 2020
  * [20] Trung Kien Vu et al.  “Path selection and rate allocation in self-backhauled mmWave networks”  In _IEEE Wireless Communications and Networking Conference (WCNC)_ , 2018, pp. 1–6
  * [21] Trung Kien Vu et al.  “Ultra-reliable communication in 5G mmWave networks: a risk-sensitive approach”  In _IEEE Communications Letters_ 22.4, 2018, pp. 708–711
  * [22] Bibo Zhang, Francesco Devoti, Ilario Filippini and Danilo De Donno  “Resource allocation in mmWave 5G IAB networks: A reinforcement learning approach based on column generation”  In _Computer Networks_ Elsevier, 2021, pp. 108248
  * [23] Manan Gupta et al.  “Learning link schedules in self-backhauled millimeter wave cellular networks”  In _IEEE Transactions on Wireless Communications_ 19.12 IEEE, 2020, pp. 8024–8038
  * [24] Andrea Ortiz et al.  “SCAROS: A scalable and robust self-backhauling solution for highly dynamic millimeter-wave networks”  In _IEEE Journal on Selected Areas in Communications_ 37.12 IEEE, 2019, pp. 2685–2698
  * [25] Mingjie Feng and Shiwen Mao  “Dealing with limited backhaul capacity in millimeter-wave systems: a deep reinforcement learning approach”  In _IEEE Communications Magazine_ 57.3, 2019, pp. 50–55
  * [26] Medhat Elsayed, Melike Erol-Kantarci and Halim Yanikomeroglu  “Transfer Reinforcement Learning for 5G New Radio mmWave Networks”  In _IEEE Transactions on Wireless Communications_ 20.5 IEEE, 2020, pp. 2838–2849
  * [27] Tianshu Chu, Jie Wang, Lara Codecà and Zhaojian Li  “Multi-agent deep reinforcement learning for large-scale traffic signal control”  In _IEEE Transactions on Intelligent Transportation Systems_ 21.3 IEEE, 2019, pp. 1086–1095
  * [28] Dohyun Kwon and Joongheon Kim  “Multi-agent deep reinforcement learning for cooperative connected vehicles”  In _IEEE Global Communications Conference (GLOBECOM)_ , 2019, pp. 1–6
  * [29] Mohamed Sana et al.  “Multi-agent reinforcement learning for adaptive user association in dynamic mmWave networks”  In _IEEE Transactions on Wireless Communications_ 19.10, 2020, pp. 6520–6534
  * [30] Delin Guo, Lan Tang, Xinggan Zhang and Ying-Chang Liang  “Joint optimization of handover control and power allocation based on multi-agent deep reinforcement learning”  In _IEEE Transactions on Vehicular Technology_ 69.11, 2020, pp. 13124–13138
  * [31] Bibo Zhang and Ilario Filippini  “Mobility-Aware Resource Allocation for mmWave IAB Networks via Multi-Agent RL”  In _2021 IEEE 18th International Conference on Mobile Ad Hoc and Smart Systems (MASS)_ , 2021, pp. 17–26  IEEE
  * [32] 3GPP  “Study on new radio access technology physical layer aspects, TR 38.802”
  * [33] Zhenyu Xiao, Pengfei Xia and Xiang-Gen Xia  “Full-duplex millimeter-wave communication”  In _IEEE Wireless Communications_ 24.6 IEEE, 2017, pp. 136–143
  * [34] Ilario Filippini, Vincenzo Sciancalepore, Francesco Devoti and Antonio Capone  “Fast cell discovery in mm-wave 5G networks with context information”  In _IEEE Transactions on Mobile Computing_ 17.7, 2017, pp. 1538–1552
  * [35] A Maltsev  “D5. 1-Channel modeling and characterization”  In _MiWEBA Project (FP7-ICT-608637), Public Deliverable_ , 2014
  * [36] Fan Bai and Ahmed Helmy  “A survey of mobility models”  In _Wireless Ad-hoc Networks. University of Southern California_ 206, 2004, pp. 147
  * [37] Margarita Gapeyenko et al.  “On the temporal effects of mobile blockers in urban millimeter-wave cellular scenarios”  In _IEEE Transactions on Vehicular Technology_ 66.11 IEEE, 2017, pp. 10124–10138
  * [38] Michele Polese et al.  “Improved handover through dual connectivity in 5G mmWave mobile networks”  In _IEEE Journal on Selected Areas in Communications_ 35.9 IEEE, 2017, pp. 2069–2084
  * [39] George R MacCartney, Theodore S Rappaport and Sundeep Rangan  “Rapid fading due to human blockage in pedestrian crowds at 5G millimeter-wave frequencies”  In _IEEE Global Communications Conference (GLOBECOM)_ , 2017, pp. 1–7
  * [40] Volodymyr Mnih et al.  “Asynchronous methods for deep reinforcement learning”  In _International Conference on Machine Learning (ICML)_ , 2016, pp. 1928–1937  PMLR
  * [41] Shariq Iqbal and Fei Sha  “Actor-attention-critic for multi-agent reinforcement learning”  In _International Conference on Machine Learning_ , 2019, pp. 2961–2970  PMLR
  * [42] 3GPP  “Physical layer procedures for data, TS 38.214”

|  Bibo Zhang Bibo Zhang received the B.S. degree in information engineering
and the M.S. degree in electronics and communication engineering from Beijing
University of Posts and Telecommunications, China, in 2015 and 2018. She is
currently pursuing the Ph.D. degree in information technology, from
Politecnico di Milano, Italy. Her current research interests include resource
management in 5G mmWave networks.
---|---
|  Ilario Filippini Ilario Filippini received B.S. and M.S. degrees in
Telecommunication Engineering and a Ph.D in Information Engineering from the
Politecnico di Milano, in 2003, 2005, and 2009, respectively. He is currently
an Associate Professor with the Dipartimento di Elettronica, Informazione e
Bioingegneria, Politecnico di Milano. His research interests include planning,
optimization, and game theoretical approaches applied to wired and wireless
networks, performance evaluation and resource management in wireless access
networks, and traffic management in software defined networks. On these
topics, he has published over 60 peer-reviewed articles. He serves in the TPC
of major conferences in networking and as an Associate Editor of IEEE
Transactions on Mobile Computing and Elsevier Computer Networks.
---|---