# Self-Organizing mmWave MIMO Cell-Free Networks With Hybrid Beamforming: A
Hierarchical DRL-Based Design

Yasser Al-Eryani, Student Member, IEEE and Ekram Hossain, Fellow, IEEE Y. Al-
Eryani and E. Hossain are with the Department of Electrical and Computer
Engineering at the University of Manitoba, Canada (emails:
<EMAIL_ADDRESS>Ekram.Hossain@umanitoba.ca). The work was supported
by a Discovery Grant form the Natural Sciences and Engineering Research
Council of Canada (NSERC).

###### Abstract

In a cell-free wireless network, distributed access points (APs) jointly serve
all user equipments (UEs) within the their coverage area by using the same
time/frequency resources. In this paper, we develop a novel downlink cell-free
multiple-input multiple-output (MIMO) millimeter wave (mmWave) network
architecture that enables all APs and UEs to dynamically self-partition into a
set of independent cell-free subnetworks in a time-slot basis. For this, we
propose several network partitioning algorithms based on deep reinforcement
learning (DRL). Furthermore, to mitigate interference between different cell-
free subnetworks, we develop a novel hybrid analog beamsteering-digital
beamforming model that zero-forces interference among cell-free subnetworks
and at the same time maximizes the instantaneous sum-rate of all UEs within
each subnetwork. Specifically, the hybrid beamforming model is implemented by
using a novel mixed DRL-convex optimization method in which analog
beamsteering between APs and UEs is conducted based on DRL while digital
beamforming is modeled and solved as a convex optimization problem. The DRL
models for network clustering and hybrid beamsteering are combined into a
single hierarchical DRL design that enables exchange of DRL agents’
experiences during both network training and operation. We also benchmark the
performance of DRL models for clustering and beamsteering in terms of network
performance, convergence rate, and computational complexity. Results show a
significant rate enhancement and complexity reduction of the proposed hybrid
beamforming scheme compared to its conventional all-digital counterpart. This
performance enhancement becomes more significant as the number of network
partitions increases. For DRL-based network clustering, the policy gradient
(PG) algorithm offers the best possible performance in terms of stability and
convergence rate while the state-action-reward-state-action (SARSA) algorithm
suffers from significant variance, slower convergence, and slightly inferior
performance than other algorithms. For DRL-based beamsteering, the soft actor-
critic (SAC) algorithm with continuous action space shows the best
performance. Also, online training of the agents with varying channel state
information (CSI) is observed to increase the variance of the Q-values and
decrease the convergence rate, with no significant effect on the average
reward.

###### Index Terms:

Cell-free MIMO networks, mmWave MIMO systems, downlink hybrid beamforming,
deep reinforcement learning (DRL), DRL-based clustering and beamforming,
convex optimization.

##  I Introduction

###  I-A Background and Related Work

Cell-free (or cell-less) network architectures are envisioned to increased
coverage and transmission rates in future generation wireless systems [1]. In
a cell-free wireless network, a large number of user equipments (UEs) in a
geographical area will be served simultaneously by a large number of
distributed access points (APs) based on non-orthogonal multiple access. The
distributed APs in a cell-free system coordinate/cooperate with each other
through a centralized processing pool [2] for estimating channel state
information (CSI) [3, 4], uplink (downlink) decoding (beamforming) [5, 6, 7],
and improving transmission performance [8, 9, 2]. The majority of the works on
cell-free systems in the literature tackle the following major technical
challenges: i) pilot contamination, ii) high computational and hardware
complexity of centralized processing, and iii) traffic/signaling overhead. For
instance, in [4], the authors designed a joint uplink/downlink pilot training
scheme that uses orthogonal subsets of pilots in the downlink instead of using
channel reciprocity concept. In [10], the authors developed a semi-blind
channel estimation of uplink cell-free massive MIMO network utilizing an
enhanced $K$-means clustering algorithm. In [5], the authors proposed a
downlink conjugate beamforming and zero-forcing (ZF) precoding scheme for a
fully centralized downlink cell-free network. It was shown that the ZF
technique outperforms the conjugate beamforming method at the expense of
increased computational complexity. However, when the number of UEs and/or APs
increases, the complexity of using ZF beamforming increases significantly due
to the requirement of matrix inversion. Accordingly, in [7], a modified
conjugate beamforming technique was proposed that uses CSI coordination among
the distributed APs. Different machine learning (ML) techniques were used for
CSI estimation [3] and beamforming [2] in cell-free networks. For instance,
the authors in [3] developed a channel estimation technique for millimeter
wave (mmWave)-enabled massive cell-free network using a supervised learning-
based denoising convolutional neural network. The authors in [2] formulated
and solved a joint problem for AP clustering and uplink beamforming in a
massive cell-free network using deep reinforcement learning (DRL) techniques.

To reduce the complexity of centralized data processing in a cell-free
network, the authors in [8] proposed a user-centric partitioning method111We
use the terms clustering and partitioning interchangeably.. The proposed
method also uses multi-level successive interference cancellation (SIC) at
each receiver. Another low-complexity design for cell-free networks was
proposed in [2]. The core idea is to reduce the dimensionality of beamforming
matrices by dynamic clustering of APs. Each cluster then represents a single
multi-antenna AP. All of the aforementioned low-complexity designs, however,
sacrifice the performance gain of centralized processing. The complexity of
solving the beamforming problem in a centralized manner (e.g. to obtain the
beamforming vectors at a centralized processing unit) can however be reduced
by using a distributed learning and/or processing approach while the detection
of the transmitted data is still performed at the central unit. Such a
solution has been recently investigated in [11, 12]. Specifically, in [12],
the authors utilized supervised learning to solve the beamforming problem in a
cell-free network by using a neural network optimizer in each AP.

Along with cell-free network architectures, mmWave transmissions can be used
to improve network capacity [13, 14]. Interestingly, cell-free networks were
found to provide an efficient solution for the poor scattering and high path-
loss problem of mmWave transmissions [13]. Due to the propagation issues
related to mmWave channels, massive multiple-input multiple-output (MIMO)
systems are usually used for mmWave-supported UEs [15]. A mmWave and MIMO-
based cell-free network, however, requires very significant amount of
computational capabilities at the central unit, especially when the number of
UEs within the network increases.

###  I-B Motivation and Contributions

For a cell-free network, the complexity of solving the beamforming problem in
a centralized manner can be reduced by partitioning the network into a group
of cell-free subnetworks, each with independent set of APs and UEs. However,
fixed partitioning will not be performance-efficient under fast-varying
channel conditions and varying number of UEs per unit area. Therefore, dynamic
partitioning into subnetworks based on current network and channel status will
be desirable, and for practical feasibility, low-complexity solutions will be
required. This motivates us to design a novel mmWave MIMO cell-free network
architecture based on dynamic partitioning (or clustering) along with a hybrid
analog-digital downlink beamforming method by using DRL techniques. The
proposed design provides us with efficient and implementation-friendly
solutions.

The main contributions of this paper can be summarized as follows:

  * •

For a mmWave MIMO cell-free network, we design a self-organizing network
architecture that dynamically partitions the network into a group of
subnetworks, each acting as an independent cell-free architecture.

  * •

To simultaneously mitigate inter-subnetwork interference (ISNI) and intra-UE
interference (IUI) while maximizing the per-UE transmission rate, we develop
an innovative hybrid analog beamsteering-digital beamforming method for the
proposed mmWave MIMO cell-free network.

  * •

The problem of joint network partitioning, analog beamsteering, and digital
beamforming is solved through a novel DRL-cum-convex optimization model.
Specifically, the model consists of two interacting networks: i) one DRL model
with discrete-action subspace for UE and AP clustering, ii) and another DRL
model with continuous-action subspace used for analog beamsteering, the first
step of the proposed hybrid beamforming method. The second step of digital
beamforming is formulated and solved as a convex optimization problem within
the environment of the DRL agent for analog beamsteering.

  * •

For network partitioning and beamforming, we benchmark several DRL algorithms
and evaluate their performances under different system parameters.

Table I presents the definitions of major system model parameters. The rest of
the paper is organized as follows. Sec. II presents the proposed dynamic
mmWave MIMO cell-free network architecture and the related modeling
assumptions. Also, the general problem formulation to design the proposed
system (e.g. clustering and beamforming design) is presented in this section.
Sec. III presents the beamforming scheme developed for the proposed system.
The hierarchical DRL model for joint clustering and beamforming is presented
in Sec. IV. In Sec. V, the complexity analysis of the proposed models and
algorithms is presented. Sec. VI present the simulation results. Sec. VII
concludes the paper.

TABLE I: Definitions of major system model parameters Parameter | Definition | Parameter | Definition
---|---|---|---
eAP | Enhanced Access Point | $\mathcal{N}$ | Number of possible subnetwork (or cluster) configurations
ECP | Edge Cloud Processor | $\mathcal{A}_{m_{n}}$ |  Beamsteering matrix of eAP $m_{n}$
ISNI | Inter-Subnetwork Interference | $\mathcal{D}_{n,j}^{\text{A}}$ |  Number of eAPs in the $n$-th subnetwork
IUI | Intra-UE Interference | $\mathcal{D}_{n,j}^{\text{U}}$ |  Number of UEs in the $n$-th Subnetwork
NCC | Network Cloud Controller | $\bm{w}_{k_{n}m_{n}}$ |  Beamforming vector for the link $m_{n}\rightarrow k_{n}$
$\bm{B}^{*}$ |  Hermitian transpose of a matrix $\bm{B}$ | $m_{n}$ |  $m$-th eAP in the $n$-th subnetwork
$\bm{B}^{T}$ |  Transpose of a matrix $\bm{B}$ | $k_{n}$ |  $k$-th UE in the $n$-th subnetwork
$\bm{H}_{k_{n}m_{n}}$ |  CSI for the link $m_{n}\rightarrow k_{n}$ | $\bm{\mathcal{H}}_{k_{n}m_{n}}$ |  Equivalent CSI for the $m_{n}\rightarrow k_{n}$ link
M | Number of eAPs | $\mathcal{S}$ | State space of a DRL model
$a$ | Number of antennas per eAP | $\bm{s}$ |  State vector at time $t$
K | Number of UEs | $\bm{s}^{\prime}$ |  State vector at time $t+1$
$u$ | Number of antennas per UE | $r$ | Immediate reward of a DRL agent
$N$ | Number of cell-free subnetworks | $\bm{A}$ | Action space of DRL model
$\mathcal{L}$ | Number of mmWave paths | $\bm{a}$ |  Action vector at time $t$
$\bm{\mathcal{C}}_{j}$ |  $j$-th clustering configuration | $\bm{a}^{\prime}$ |  Action vector at time $t+1$

##  II System Model and Assumptions

###  II-A Network Architecture

We consider a downlink network with $M$ APs and $K$ UEs (Fig. 1). Each of the
APs and UEs is assumed to be equipped with $a$ and $u$ antennas, respectively.
To enable multiuser transmission, each AP is assumed to be equipped with $L$
RF chains.

Figure 1: An example scenario for the proposed network architecture
($M=19,K=21$, and $N=4$).

All of the APs are connected to each other through fronthaul/backhaul links to
form a cell-free network architecture [16]. This enables the distributed APs
to collaborate to simultaneously serve all UEs within the network coverage
area. Specifically, the APs collaborate through a network cloud controller
(NCC). Each of the APs is assumed to be equipped with a baseband processor
that is capable of performing operations related to uplink channel training
and downlink beamforming of signals transmitted to different UEs. Such an AP
is referred to as an “enhanced-AP” (eAP) to distinguish it from conventional
APs with passband transmission/reception functionalities. In the proposed
network architecture, it is assumed that the all of the eAPs and UEs are
partitioned into a set of $N$, where $1\leq N\leq M$, non-overlapping clusters
(i.e. cell-free subnetworks) on a time-slot basis. All of the UEs of a certain
subnetwork are served by all of the eAPs of that subnetwork using the same
time-frequency resources. Accordingly, the number of RF chains required at
each eAP will be equal to the maximum allowable number of UEs per cell-free
subnetwork, i.e. $L=K-N+1$222The maximum number of UEs per subnetwork may be
defined based on the hardware cost/complexity of the eAPs. This will have a
direct impact on the number of RF chains required per eAP and the average
amount of energy consumption [17].. The group of all baseband processors of
eAPs within each cluster can be coordinated to form an virtual edge cloud
processor (ECP) unit that is responsible for performing multiuser downlink
beamforming within each subnetwork, considering signals from other clusters as
Inter-subnetwork interference (ISNI) components. Each eAP may act as an ECP
for its subnetwork, or all eAPs of a single subnetwork may form a virtual ECP.
Furthermore, the clustering of the cell-free network into a group of non-
overlapping cell-free subnetworks is assumed to be performed centrally at the
NCC. These two operations of network clustering and subnetwork beamforming are
performed either at each time-slot or every several time-slots, based on
current CSI and time-varying propagation characteristics of the network. Note
that, when $N=1$, all eAPs and UEs of the network will belong to the same
subnetwork which will form a fully centralized cell-free network. On the other
hand, when $N=M$, the overall architecture will act as a conventional wireless
cellular network with a reuse factor of $1$.

Let us denote by
$\bm{\mathcal{C}}=\\{\\{\bm{\mathcal{C}}^{\text{A}}_{1},\bm{\mathcal{C}}^{\text{U}}_{1}\\}\dots\\{\bm{\mathcal{C}}^{\text{A}}_{j},\bm{\mathcal{C}}^{\text{U}}_{j}\\}\dots\\{\bm{\mathcal{C}}^{\text{A}}_{\mathcal{N}},\bm{\mathcal{C}}^{\text{U}}_{\mathcal{N}}\\}\\}$
the set of all possible AP-UE clustering configurations such that every
cluster contains at least one AP and one UE. $\mathcal{N}$ is the total number
of possible clustering configurations which is a function of $M$, $K$, and
$N$, i.e. $\mathcal{N}=\Theta(M,K,N)$ (to be defined in subsequent sections).
As an example, with $M=4$, $K=3$, and $N=2$, one possible set is

$\bm{\mathcal{C}}_{j}=\\{\underbrace{\\{\\{\text{AP}_{1},\text{AP}_{3},\text{AP}_{4}\\}}_{\bm{\mathcal{C}}_{1,j}^{\text{A}}},\underbrace{\\{\text{UE}_{2}\\}}_{\bm{\mathcal{C}}_{1,j}^{\text{U}}}\\},\\{\underbrace{\\{\text{AP}_{2}\\}}_{\bm{\mathcal{C}}_{2,j}^{\text{A}}},\underbrace{\\{\text{UE}_{1},\text{UE}_{3}\\}\\}}_{\bm{\mathcal{C}}_{2,j}^{\text{U}}}\\}.$

Let $\mathcal{D}_{n,j}^{\text{A}}$ and $\mathcal{D}_{n,j}^{\text{U}}$
represent, respectively, the number of eAPs and UEs at the $n$-th subnetwork
of the $j$-th possible configuration, where
$\mathcal{D}_{n,j}^{\text{A}}=\text{Cardinality}\left\\{\bm{\mathcal{C}}_{n,j}^{\text{A}}\right\\}$
and
$\mathcal{D}_{n,j}^{\text{U}}=\text{Cardinality}\left\\{\bm{\mathcal{C}}_{n,j}^{\text{U}}\right\\},n=1,\dots,N$
and $j=1,\dots,\Theta\left(M,K,N\right)$. For this model, we assume that the
$m_{n}$-th eAP sends a weighted sum of signals of all UEs within the $n$-th
subnetwork. Accordingly, for a given cell-free network clustering
configuration, $\bm{\mathcal{C}}_{j}$, the antennas of the $m$-th eAP at the
$n$-th cluster (denoted by $m_{n}$) will have at least
$\mathcal{D}_{n,j}^{\text{U}}$ streams333This assumption will require that the
number of RF chains at each eAP does not fall below
$\mathcal{D}_{n,j}^{\text{U}}$ [18].. For simplicity, we assume that at each
time instant, the $m_{n}$-th eAP will use only
$\mathcal{D}_{n,j}^{\text{U}}\leq L$ RF chains at a time.

The NCC and the ECP will be responsible for the entire communication process.
At the beginning of each time slot, the NCC will first estimate the CSI values
for the UEs with respect to all serving eAPs. Then the processes of eAP
clustering and per-subnetwork downlink beamforming will be performed jointly
by the NCC and the virtual ECP.

###  II-B Channel and Antenna Model

The communications between the eAPs and the distributed UEs occur in the 24-39
GHz mmWave bands in which transmissions suffer from limited scattering and
spatial selectivity. The asymptotic orthogonality assumption among different
mmWave channels does not apply to highly correlated mmWave MIMO channels [19].
Accordingly, we adopt the well-known three-dimensional clustered model [20].
We consider a uniform planner array (UPA)444UPA is suitable for mmWave
beamforming due to smaller array dimensions, ability to perform 3D beamforming
(at the elevation domain), and possibility of packing many antenna elements in
a small space[21, 22]. at the $m_{n}$-th eAP and $k_{n}$-th UE with
$a=L_{m,1}L_{m,2}$ and $u=L_{k,1}L_{k,2}$ for which $L_{m,1}(L_{k,1})$ and
$L_{m,2}(L_{k,2})$ represent the number of columns and rows of antenna
elements, respectively. The downlink channel gain matrix for the
$m_{n}\rightarrow k_{n}$ link (denoted by
$\bm{H}_{k_{n}m_{n}}\in\mathbb{C}^{u\times a}$) can be then expressed as [20,
23]

$\bm{H}_{k_{n}m_{n}}=\sum_{l=1}^{\mathcal{L}}h_{k_{n}m_{n},l}\bm{b}_{\text{U}}\left(\vartheta_{k_{n}m_{n},l},\varphi_{k_{n}m_{n},l}\right)\bm{b}^{*}_{\text{A}}\left(\theta_{k_{n}m_{n},l},\phi_{k_{n}m_{n},l}\right)=\sum_{l=1}^{\mathcal{L}}h_{k_{n}m_{n},l}\mathcal{B}\left(\bm{\theta}_{k_{n}m_{n},l},\bm{\phi}_{k_{n}m_{n},l}\right),$
(1)

where $\mathcal{L}$ is the number of paths for the $m_{n}\rightarrow k_{n}$
link,
$h_{k_{n}m_{n},l}=\sqrt{\frac{1}{\kappa+\mathcal{L}-1}}\alpha_{k_{n}m_{n},l}$
is the complex channel gain at the $l$-th path in the $m_{n}\rightarrow k_{n}$
link with
$\alpha_{k_{n}m_{n},l}\thicksim\mathcal{CN}\left(0,\sigma_{k_{n}m_{n},l}\right)$,
in which $\sigma_{k_{n}m_{n},1}=\kappa$ (the ratio of the line-of-sight [LoS]
path power to non-line-of-sight (NLoS) path power), and
$\sigma_{k_{n}m_{n},l}=1,l=2,\dots,\mathcal{L}$. Also,
$\mathcal{B}\left(.\right)=\bm{b}_{\text{U}}\left(.\right)\bm{b}^{*}_{\text{A}}\left(.\right)$
with
$\bm{\theta}_{k_{n}m_{n}}=\left[\vartheta_{k_{n}m_{n},l},\theta_{k_{n}m_{n},l},\right]$
and
$\bm{\phi}_{k_{n}m_{n}}=\left[\varphi_{k_{n}m_{n},l},\phi_{k_{n}m_{n},l}\right]$.
In (1),
$\bm{b}_{\text{U}}\left(\vartheta_{k_{n}m_{n},l},\varphi_{k_{n}m_{n},l}\right)\in\mathbb{C}^{u\times
1}$ and
$\bm{b}_{\text{A}}\left(\theta_{k_{n}m_{n},l},\phi_{k_{n}m_{n},l}\right)\in\mathbb{C}^{a\times
1}$ are the antenna array responses at the $k_{n}$-th UE and the $m_{n}$-th
eAP, respectively. The antenna array response at the $m_{n}$-th eAP and
$k_{n}$-th UE, respectively, can be defined as

$\bm{b}_{\text{A}}\left(\theta_{k_{n}m_{n},l},\phi_{k_{n}m_{n},l}\right)=\left[e^{j2\pi\frac{d\left(0\sin\theta_{k_{n}m_{n},l}\cos\phi_{k_{n}m_{n},l}+0\sin\phi_{k_{n}m_{n},l}\right)}{\lambda}},\dots,e^{j2\pi\frac{d\left(w\sin\theta_{k_{n}m_{n},l}\cos\phi_{k_{n}m_{n},l}+z\sin\phi_{k_{n}m_{n},l}\right)}{\lambda}},\\\
\dots,e^{j2\pi\frac{d\left(\left(L_{m,1}-1\right)\sin\theta_{k_{n}m_{n},l}\cos\phi_{k_{n}m_{n},l}+\left(L_{m,2}-1\right)\sin\phi_{k_{n}m_{n},l}\right)}{\lambda}}\right]^{T},$
(2)
$\bm{b}_{\text{U}}\left(\vartheta_{k_{n}m_{n},l},\varphi_{k_{n}m_{n},l}\right)=\left[e^{j2\pi\frac{d\left(0\sin\vartheta_{k_{n}m_{n},l}\cos\varphi_{k_{n}m_{n},l}+0\sin\varphi_{k_{n}m_{n},l}\right)}{\lambda}},\dots,e^{j2\pi\frac{d\left(w\sin\vartheta_{k_{n}m_{n},l}\cos\varphi_{k_{n}m_{n},l}+z\sin\varphi_{k_{n}m_{n},l}\right)}{\lambda}},\\\
\dots,e^{j2\pi\frac{d\left(\left(L_{k,1}-1\right)\sin\vartheta_{k_{n}m_{n},l}\cos\varphi_{k_{n}m_{n},l}+\left(L_{k,2}-1\right)\sin\varphi_{k_{n}m_{n},l}\right)}{\lambda}}\right]^{T},$
(3)

where $\theta_{k_{n}m_{n},l}$ and $\vartheta_{k_{n}m_{n},l}$ are the elevation
angles at the $m_{n}$-th eAP and $k_{n}$-th UE, respectively,
$\phi_{k_{n}m_{n},l}$ and $\varphi_{k_{n}m_{n},l}$ are the azimuth angles at
the $m_{n}$-th eAP and $k_{n}$-th UE related to the $l$-th path in the
$m_{n}\rightarrow k_{n}$ link, respectively, $d$ is the antenna spacing of
eAPs and UEs, and $\lambda$ is the carrier wavelength.

###  II-C Downlink Data Transmission

We assume that downlink transmission is performed based on two types of
beamforming schemes, namely, analog RF beamsteering and baseband digital
beamforming. Assuming that a certain cell-free network partitioning
configuration (say $\bm{\mathcal{C}}_{j}$) is selected by the NCC, the
received combined signal at the $i_{n}$-th UE can be expressed as

$y_{i_{n}}=\bm{\delta}_{i_{n}}^{\text{T}}\sum_{l=1}^{N}\sum_{m_{l}=1}^{\mathcal{D}_{l,j}^{\text{A}}}\bm{H}_{i_{n}m_{l}}\sum_{k_{l}=1}^{\mathcal{D}_{n,j}^{\text{U}}}\bm{\mathcal{A}}_{m_{l}}\bm{w}_{k_{l}m_{l}}x_{k_{l}}+\bm{\delta}_{i_{n}}^{\text{T}}\bm{\eta}_{i_{n}}\\\
=\underbrace{\sum_{m_{n}=1}^{\mathcal{D}_{n,j}^{\text{A}}}\bm{\delta}_{i_{n}}^{\text{T}}\bm{H}_{i_{n}m_{n}}\bm{\mathcal{A}}_{m_{n}}\bm{w}_{i_{n}m_{n}}x_{i_{n}}}_{\text{Desired
Signal}}+\underbrace{\sum_{m_{n}=1}^{\mathcal{D}_{n,j}^{\text{A}}}\bm{\delta}_{i_{n}}^{\text{T}}\bm{H}_{i_{n}m_{n}}\bm{\mathcal{A}}_{m_{n}}\bm{w}_{k_{n}m_{n}}\sum_{k_{n}=1,k_{n}\neq
i_{n}}^{\mathcal{D}_{n,j}^{\text{U}}}x_{k_{n}}}_{\text{IUI}}+\underbrace{\sum_{l=1,l\neq
n}^{N}\sum_{m_{l}=1}^{\mathcal{D}_{l,j}^{\text{A}}}\bm{\delta}_{i_{n}}^{\text{T}}\bm{H}_{i_{n}m_{l}}\sum_{k_{l}=1}^{\mathcal{D}_{l,j}^{\text{U}}}\bm{\mathcal{A}}_{m_{l}}\bm{w}_{k_{l}m_{l}}x_{k_{l}}}_{\text{ISNI}}+\underbrace{\bm{\delta}_{i_{n}}^{\text{T}}\bm{\eta}_{i_{n}}}_{\text{AWGN}},$
(4)

where $\bm{H}_{i_{n}m_{l}}\in\mathbb{C}^{u\times a}$ is the channel gain
matrix for the $m_{l}\rightarrow i_{n}$ link,
$\bm{\mathcal{A}}_{m_{l}}\in\mathbb{C}^{a\times\mathcal{D}_{l,j}^{\text{U}}}$
is the analog RF beamsteering matrix at the $m_{l}$-th eAP,
$\bm{w}_{k_{l}m_{l}}\in\mathbb{R}^{\mathcal{D}_{l,j}^{\text{U}}\times 1}$ is
the digital baseband beamforming vector related to the $m_{l}\rightarrow
k_{l}$ link, $\bm{\delta}_{m_{l}}\in\mathbb{C}^{u\times 1}$ is the analog
beamsteering/combining vector at the $k_{l}$-th UE, $x_{k_{l}}$ is the
transmitted symbol related to the $k_{l}$-th UE such that
$\mathbb{E}\left[|x_{k_{l}}|^{2}\right]=P/\mathcal{D}_{l,j}^{\text{U}}$ where
$P$ is the transmission power budget at each eAP, and $\bm{\eta}_{i_{n}}$ is
the additive white Gaussian noise (AWGN) vector at the input of the $i_{n}$-th
UE where
$\bm{\eta}_{i_{n}}\thicksim\mathcal{N}\left(\bm{0},\sigma_{i_{n}}\bm{I}\right),\forall
i_{n}=1,2,\dots,\mathcal{D}_{n,j}^{\text{U}},n=1,\dots,N$. The instantaneous
signal-to-interference-plus-noise ratio (SINR) at the input of the $i_{n}$-th
UE under clustering configuration $\bm{\mathcal{C}}_{j}$ can be expressed as

$\gamma_{i_{n}}^{\left\\{\bm{\mathcal{C}}_{j}\right\\}}=\frac{\sum_{m_{n}=1}^{\mathcal{D}_{n,j}^{\text{A}}}|\bm{\delta}_{i_{n}}^{\text{T}}\bm{H}_{i_{n}m_{n}}\bm{\mathcal{A}}_{m_{n}}\bm{w}_{i_{n}m_{n}}|^{2}}{\begin{pmatrix}\sum_{m_{n}=1}^{\mathcal{D}_{n,j}^{\text{A}}}\sum_{k_{n}=1,k_{n}\neq
i_{n}}^{\mathcal{D}_{n,j}^{\text{U}}}|\bm{\delta}_{i_{n}}^{\text{T}}\bm{H}_{i_{n}m_{n}}\bm{\mathcal{A}}_{m_{n}}\bm{w}_{k_{n}m_{n}}|^{2}+\tilde{\sigma}_{i_{n}}\sum_{j=1}^{u}\delta_{i_{n}j}^{2}\\\
+\sum_{l=1,l\neq
n}^{N}\left(\frac{\mathcal{D}_{n,j}^{\text{U}}}{\mathcal{D}_{l,j}^{\text{U}}}\right)^{2}\sum_{m_{l}=1}^{\mathcal{D}_{l,j}^{\text{A}}}\sum_{k_{l}=1}^{\mathcal{D}_{l,j}^{\text{U}}}|\bm{\delta}_{i_{n}}^{\text{T}}\bm{H}_{i_{n}m_{l}}\bm{\mathcal{A}}_{m_{l}}\bm{w}_{k_{l}m_{l}}|^{2}\end{pmatrix}},$
(5)

where
$\tilde{\sigma}_{i_{n}}=\left(\frac{\sigma_{i_{n}}\mathcal{D}_{n,j}^{\text{U}}}{{2P}}\right)^{2}$.
Note that equation (5) is derived based on the assumption that both
transmitter and receiver have a full knowledge of CSI of the corresponding
link.

###  II-D General Problem Formulation

To achieve the best performance with the proposed cell-free architecture, the
operations of network partitioning, analog beamsteering, and digital
beamforming must be jointly optimized (e.g. by solving an optimization problem
globally). The objective of this problem can be, for example, maximization of
the sum-rate of all users (i.e. max-sum objective), or maximization of the
minimum rate of the users (i.e. max-min objective to achieve fairness). The
general problem formulation can be stated as follows:

$\displaystyle~{}\textbf{{P}}_{1}:\underset{j,\left\\{\bm{\mathcal{A}}_{m_{n}},\bm{\Delta}_{m_{n}},\bm{W}_{m_{n}}\right\\}^{n=1,\dots,N}_{m_{n}=1,\dots,\mathcal{D}_{n,j}^{\text{A}}}}{\text{max}}~{}$
$\displaystyle\text{}f\left(\left\\{\gamma_{i_{n}}^{\\{\bm{\mathcal{C}}_{j}\\}}\right\\}^{n=1,\dots,N}_{i_{n}=1,\dots,\mathcal{D}_{n,j}^{\text{U}}}\right)$
(6) Subject to: $\displaystyle~{}\textbf{
{C}}_{1}:|\bm{\mathcal{A}}_{m_{n}}\left(q,z\right)|^{2}=1,\forall~{}~{}q=1,\dots,a,\;\mbox{and}\;z=1,\dots,\mathcal{D}_{n,j}^{\text{U}},$
$\displaystyle~{}\textbf{
{C}}_{2}:|\bm{\delta}_{k_{n}}\left(q\right)|^{2}=1,\forall~{}~{}k_{n}=1,\dots,\mathcal{D}_{n,j}^{\text{U}},\;\mbox{and}\;q=1,\dots,u,$
$\displaystyle~{}\textbf{
{C}}_{3}:||\bm{W}_{m_{n}}\left(\left[1\dots\mathcal{D}_{n,j}^{\text{U}}\right],z\right)||^{2}\leq
1,\forall~{}~{}z=1,\dots,\mathcal{D}_{n,j}^{\text{U}},$

where
$\bm{\mathcal{A}}_{m_{n}}\in\mathbb{C}^{a\times\mathcal{D}_{n,j}^{\text{U}}}$
is the analog beamsteering matrix at the $m_{n}$-th eAP,
$\bm{W}_{m_{n}}\in\mathbb{R}^{\mathcal{D}_{n,j}^{\text{U}}\times\mathcal{D}_{n,j}^{\text{U}}}$
is the digital beamforming matrix at the $m_{n}$-th eAP with
$\bm{W}_{m_{n}}=\left[\bm{w}_{1_{n}m_{n}},\dots,\bm{w}_{\mathcal{D}_{n,j}^{\text{U}}m_{n}}\right]$,
where
$\bm{w}_{k_{n}m_{n}}=\left[\sqrt{w_{k_{n}m_{n},1}},\dots,\sqrt{w_{k_{n}m_{n},\mathcal{D}_{n,j}^{\text{U}}}}\right]^{\text{T}}$
is the digital baseband beamforming vector related to the $m_{n}\rightarrow
k_{n}$ link,
$\bm{\Delta}_{m_{n}}\in\mathbb{R}^{u\times\mathcal{D}_{n,j}^{\text{U}}}$ is
the analog beamsteering/combining matrix related to all UEs of the $m_{n}$-th
subnetwork, in which
$\bm{\Delta}_{m_{n}}=\left[\bm{\delta}_{1_{n}},\dots,\bm{\delta}_{\mathcal{D}_{n,j}^{\text{U}}}\right]$,
where
$\bm{\delta}_{k_{n}}=\left[\delta_{k_{n}1},\dots,\delta_{k_{n}u}\right]^{\text{T}}$
is the analog beamsteering/combining vector at the $k_{n}$-th UE. Furthermore,
the index $j\in\left[1,\dots,\Theta(M,K,N)\right]$ refers to the selected
cell-free partitioning configuration.

$\textbf{{P}}_{1}$ is a combinatorial optimization problem which is
characterized by: i) non-convexity of the objective function $f\left(.\right)$
(discrete $j$), ii) discrete nature of optimization variable $j$, iii) non-
affine nature of the constraints $\textbf{ {C}}_{1}$ and $\textbf{ {C}}_{2}$.
To solve $\textbf{{P}}_{1}$ optimally, a simultaneous optimization for $j$,
$\bm{\mathcal{A}}_{m_{n}}$, $\bm{\Delta}_{m_{n}}$, and
$\bm{W}_{m_{n}},~{}\forall~{}n=1,\dots,N$,
$m_{n}=1,\dots,\mathcal{D}_{n,j}^{\text{A}}$, and
$k_{n}=1,\dots,\mathcal{D}_{n,j}^{\text{U}}$ will be required. This is
achieved by going through every possible clustering configuration of the cell-
free network $(\bm{\mathcal{C}}_{j},j=1,\dots,\Theta\left(M,K,N\right))$, and
for each $\bm{\mathcal{C}}_{j}$, we will need to find the corresponding
optimal analog beamsteering and digital beamforming matrices (i.e.
$\bm{\mathcal{A}}_{m_{n}}$, $\bm{\Delta}_{m_{n}}$, and $\bm{W}_{m_{n}}$). The
globally optimal solution is then the one that gives the best performance
among all possible clustering configurations and the corresponding matrices
$\bm{\mathcal{A}}_{m_{n}}$, $\bm{\Delta}_{m_{n}}$, and $\bm{W}_{m_{n}}$. The
solution will have a combinatorial computational complexity in terms of the
network parameters such as $M,K,N,a_{m_{n}}$, and $u_{k_{n}}$ (see Section V
for further discussions).

##  III mmWave Hybrid Beamforming Design

As has been mentioned in the last section, the problem $\textbf{{P}}_{1}$ in
(6) is a combinatorial optimization problem with four overlapping feasible
spaces (spaces of $j$, $\bm{\mathcal{A}}_{m_{n}}$, $\bm{\Delta}_{m_{n}}$, and
$\bm{W}_{m_{n}}$). Such a problem can be solved by global optimization
techniques such as deterministic methods (e.g. inner and outer approximation
and cutting-plane methods), stochastic methods (e.g. direct Monte-Carlo
sampling and stochastic tunneling), and heuristic methods (e.g. genetic
algorithms and swarm-based optimization algorithms) [24]. However, generating
an efficient solution of $\textbf{{P}}_{1}$ with reasonable computational
complexity and short computing time becomes very challenging as the number of
eAPs and/or UEs increases. In this section, we develop an efficient low-
complexity mixed DRL-cum-convex optimization-based solution of
$\textbf{{P}}_{1}$. In the following, we will first discuss the problem of
downlink beamforming at each cell-free subnetwork and then develop a
hierarchical DRL-based scheme that jointly performs network clustering and
per-subnetwork hybrid beamforming.

###  III-A Hybrid Beamforming for Cell-Free MIMO

Partitioning the overall cell-free network architecture (i.e. eAPs and UEs)
into a set of computationally independent cell-free subnetworks introduces
ISNI to the received signal. Furthermore, simultaneous in-band transmission
will cause IUI to all UEs that belong to the same subnetwork. We develop a
novel hybrid analog-digital beamforming scheme that efficiently mitigates the
effects of ISNI and IUI. Specifically, in the proposed method, downlink
beamforming at the multi-antenna eAPs within each cluster is performed in two
consecutive stages (Fig. 2(a)).

Figure 2: Hybrid Beamforming: (a) Example scenario, (b) Block diagram.

Under a certain network partitioning configuration (e.g.
$\bm{\mathcal{C}}_{j}$ for some $j$), each cell-free subnetwork first performs
an analog beamsteering for all eAPs such that the ISNI from the nearby
clusters is minimized. This is achieved by directing the main beams of eAPs
(i.e. the main lobes of antennas at each eAP) to the UEs belonging to the same
cluster and setting the beam directions of annihilated side lobes to the UEs
located outside the intended subnetwork (ISNI minimization). Once the beams of
different subnetworks are steered to their desired coverage areas, digital
beamforming (Fig. 2(b)) is performed at each eAP to maximize the overall
performance and mitigate IUI for the UEs that are located within the intended
cell-free subnetwork. Digital beamforming is performed using the overall
effective channel after applying the analog beamsteering phase matrices to the
original CSI matrices.

For the proposed beamforming scheme, each eAP is assumed to be equipped with
$L$ RF chains. Each UE within each subnetwork is assigned to one communication
stream by each eAP in that subnetwork555A single baseband communication stream
is handled by a single RF chain.. To achieve this, the number of UEs at each
cell-free subnetwork must not exceed the number of RF chains at each eAP (i.e.
$\mathcal{D}_{n,j}^{\text{U}}\leq L$). Fig. 2(b) illustrates the functional
block diagram of the $m_{n}$-th eAP transmitter/beamformer and the $k_{n}$-th
UE. At the UE side, we assume that signal from different antennas are combined
through a low-complexity analog beamsteering/combining scheme using the analog
combining vector $\bm{\delta}_{k_{n}}$. After analog beamsteering at all eAPs
and UEs, digital beamforming takes place considering the effective CSI
obtained after applying analog beamsteering at both eAPs and UEs (i.e. after
applying $\bm{\mathcal{A}}_{m_{n}}$ and
$\bm{\delta}_{k_{n}},~{}\forall~{}n,m,k$).

###  III-B Analog Beamsteering Subsystem: ISNI Mitigation

The main beam (also referred to as the main lobe) of an antenna element
contains the largest portion of the field strength (either radiated or
absorbed). The direction of the main beam of a single antenna can be adjusted
to match the direction of arrival of the transmitted signal (elevation and
azimuth angles) [25]. We propose an analog beamsteering technique to be used
in the first-stage of downlink signal transmission at each cell-free
subnetwork. This is achieved by utilizing the spatial signatures between UEs
of the overall cell-free network and those of eAPs of the intended subnetwork.
Specifically, analog beamsteering in a cell-free subnetwork is used to
minimize the ISNI caused to the UEs from outside the intended subnetwork. This
is achieved by directing the main lobes of the eAPs of each subnetwork toward
the UEs belonging to the same subnetwork and setting the directions of the
weakest beam side lobes to those outside the intended subnetwork.
Designing the beamsteering objective function: In order to mitigate the ISNI
components, analog beamsteering matrices
$\bm{\mathcal{A}}_{m_{n}},\forall~{}m_{n}=1,\dots,\mathcal{D}_{n,j}^{\text{U}}$
and $n=1,\dots,N$ at the $m_{n}$-th eAP have to be designed such that they
“zero-force” the communication links between all eAPs of the $n$-th subnetwork
with UEs outside the intended subnetwork. At the same time, the communication
links between all eAPs of the $n$-th subnetwork and UEs inside the intended
subnetwork are optimized. To achieve this, let us first define the “null
space” of an arbitrary $m_{x}\rightarrow k_{y}$ MIMO link using the following
axiom.

######  Axiom 1.

Let $\bm{H}_{k_{y}m_{x}}\in\mathbb{C}^{u\times a}$ be an arbitrary mmwave MIMO
channel matrix. If the singular value decomposition (SVD) of
$\bm{H}_{k_{y}m_{x}}$ is given by

$\bm{H}_{k_{y}m_{x}}=\bm{U}_{k_{y}m_{x}}\bm{\Sigma}_{k_{y}m_{x}}\bm{V}_{k_{y}m_{x}}^{*}=\left[\bm{U}_{k_{y}m_{x}}^{(1)}\bm{U}_{k_{y}m_{x}}^{(0)}\right]\bm{\Sigma}_{k_{y}m_{x}}\left[\bm{V}_{k_{y}m_{x}}^{(1)},\bm{V}_{k_{y}m_{x}}^{(0)}\right]^{*},$
(7)

then the left null space of $\bm{H}_{k_{y}m_{x}}$ is given by

$\text{Null}_{\textit{L}}\left(\bm{H}_{k_{y}m_{x}}\right)=\bm{U}_{k_{y}m_{x}}^{(0)}.$
(8)

Furthermore, the right null space of $\bm{H_{k_{y}m_{x}}}$ is given by

$\text{Null}_{\textit{R}}\left(\bm{H}_{k_{y}m_{x}}\right)=\bm{V}_{k_{y}m_{x}}^{(0)},$
(9)

where $\bm{U}_{k_{y}m_{x}}\in\mathbb{C}^{u\times u}$ and
$\bm{V}_{k_{y}m_{y}}\in\mathbb{C}^{a\times a}$ are unitary matrices, and
$\bm{\Sigma}_{k_{y}m_{x}}\in\mathbb{R}^{u\times a}$ is a diagonal matrix
containing the eigenvalues of $\bm{H}_{k_{y}m_{x}}$.
$\bm{U}_{k_{y}m_{x}}^{(1)}\in\mathbb{C}^{u\times r}$ and
$\bm{V}_{k_{y}m_{x}}^{(1)}\in\mathbb{C}^{a\times r}$ are the matrices with
columns from $\bm{U}_{k_{y}m_{x}}$ and $\bm{V}_{k_{y}m_{x}}$, respectively,
corresponding to the non-zero eigenvalues of $\bm{H}_{k_{y}m_{x}}$, and
$\bm{U}_{k_{y}m_{x}}^{(0)}\in\mathbb{C}^{u\times\left(u-r\right)}$
$\bm{V}_{k_{y}m_{x}}^{(0)}\in\mathbb{C}^{a\times\left(a-r\right)}$ with
columns from $\bm{U}_{k_{y}m_{x}}$ and $\bm{V}_{k_{y}m_{x}}$, respectively,
corresponding to the zero eigenvalues of $\bm{H}_{k_{y}m_{x}}$, where
$r=\text{rank}\left(\bm{H}_{k_{y}m_{x}}\right)$.

Remark: To guarantee the existence of a null space for any arbitrary
$\bm{H}_{k_{y}m_{x}}$, the number of antennas at each eAP must exceed that of
UEs served by that eAP. This condition complies with the fact that mmWave
networks use massive MIMO systems at all distributed eAPs. Given the left and
right null spaces of $\bm{H}_{k_{y}m_{x}}$, the projection of a complex vector
$\bm{\delta}_{k_{y}}$ into $\bm{U}_{k_{y}m_{x}}^{(0)}$ can be given by

$\bm{\delta}_{k_{y}m_{x}}^{\perp}=\bm{\delta}_{ky}^{\text{T}}\bm{U}_{k_{y}m_{x}}^{(0)}\left(\bm{U}_{k_{y}m_{x}}^{(0)}\right)^{*}.$
(10)

Furthermore, the projection of a matrix $\bm{\mathcal{A}}_{m_{x}}$ into
$\bm{V}_{k_{y}m_{x}}^{(0)}$ can be given by

$\bm{\mathcal{A}}_{k_{y}m_{x}}^{\perp}=\bm{V}_{k_{y}m_{x}}^{(0)}\left(\bm{V}_{k_{y}m_{x}}^{(0)}\right)^{*}\bm{\mathcal{A}}_{m_{x}}.$
(11)

(10) and (11) above can be derived using the fact that
$\left(\bm{\delta}_{k_{y}}-\bm{\delta}_{k_{y}m_{x}}^{\perp}\right)^{\text{T}}\bm{U}_{k_{y}m_{x}}^{(0)}=\bm{0}$,
$\left(\bm{V}_{k_{y}m_{x}}^{(0)}\right)^{*}\left(\bm{\mathcal{A}}_{m_{x}}-\bm{\mathcal{A}}_{k_{y}m_{x}}^{\perp}\right)=\bm{0}$
and
$\bm{U}_{k_{y}m_{x}}^{(0)}\left(\bm{U}_{k_{y}m_{x}}^{(0)}\right)^{*}=\bm{I}_{u}$,
$\bm{V}_{k_{y}m_{x}}^{(0)}\left(\bm{V}_{k_{y}m_{x}}^{(0)}\right)^{*}=\bm{I}_{a}$,
where $\bm{I}_{u}\in\mathbb{R}^{u\times u}$, $\bm{I}_{a}\in\mathbb{R}^{a\times
a}$ are identity matrices. Using a similar procedure, the projection of
$\bm{\delta}_{k_{y}}$ and $\bm{\mathcal{A}}_{m_{x}}$ on the left and right
“non-annihilating” subspaces of $\bm{H}_{k_{y}m_{x}}$ can be given,
respectively, by 666Here, the non-annihilating subspace refers to the subspace
of $\bm{H}_{k_{y}m_{x}}$ after it’s null subspace has been removed.

$\bm{\delta}_{k_{y}m_{x}}^{\not\perp}=\bm{\delta}_{k_{y}}^{\text{T}}\bm{U}_{k_{y}m_{x}}^{(1)}\left(\bm{U}_{k_{y}m_{x}}^{(1)}\right)^{*},\;\mbox{and}$
(12)
$\bm{\mathcal{A}}_{k_{y}m_{x}}^{\not\perp}=\bm{V}_{k_{y}m_{x}}^{(1)}\left(\bm{V}_{k_{y}m_{x}}^{(1)}\right)^{*}\bm{\mathcal{A}}_{m_{x}}.$
(13)

The objective of analog beamsteering is to reduce ISNI within each cell-free
subnetwork. However, focusing only on ZF technique to remove the interfering
beams (i.e. ISNI) between adjacent subnetworks may result in misalignment of
the main beams of the eAP antennas with those of the UEs within the same
subnetwork, and hence reduced/inappropriate antenna directivity. Therefore, we
propose a novel analog beamsteering scheme based on maximizing the so-called
“secrecy sum power gains” at each subnetwork. This is done at the $n$-th
subnetwork by maximizing the sum of powers of two channel projections: i) the
non-annihilating projections of $\bm{\delta}_{k_{n}}$ and
$\bm{\mathcal{A}}_{m_{n}}$ on $\bm{H}_{k_{n}m_{n}}$ (i.e,
$\bm{\delta}_{k_{n}m_{n}}^{\not\perp}$ and
$\bm{\mathcal{A}}_{k_{n}m_{n}}^{\not\perp},~{}\forall~{}k_{n}=1,\dots,\mathcal{D}_{n,j}^{\text{U}}~{}\&~{}~{}m_{n}=1,\dots,\mathcal{D}_{n,j}^{\text{A}}$),
and ii) the annihilating projections of $\bm{\mathcal{A}}_{m_{n}}$ on
$\bm{H}_{k_{l}m_{n}}$ (i.e.
$\bm{\mathcal{A}}_{k_{l}m_{n}}^{\perp},~{}\forall~{}l\neq n$).

We are now ready to formulate the beamsteering optimization problem
$\textbf{{P}}_{2}$ to mitigate ISNI, which will need to be solved for the
$n$-th cell-free subnetwork, as follows:

$\displaystyle~{}\textbf{{P}}_{2}:\underset{\left\\{\bm{\delta}_{k_{n}},\bm{\mathcal{A}}_{m_{n}}\right\\}^{m_{n}=1,\dots,\mathcal{D}_{n,j}^{\text{A}}}_{k_{n}=1,\dots,\mathcal{D}_{n,j}^{\text{U}}}}{\text{max}}~{}$
$\displaystyle\text{}\sum_{m_{n}=1}^{\mathcal{D}_{n,j}^{\text{A}}}\left(\sum_{k_{n}=1}^{\mathcal{D}_{n,j}^{\text{U}}}||\bm{\delta}^{\not\perp}_{k_{n}m_{n}}\bm{\Sigma}_{k_{n}m_{n}}\bm{\mathcal{A}}_{k_{n}m_{n}}^{\not\perp}||^{2}\right.$
(14) $\displaystyle\text{\hskip 128.0374pt}\left.+\sum_{l=1,l\neq
n}^{N}\sum_{k_{l}=1}^{\mathcal{D}_{l,j}^{\text{U}}}||\bm{\delta}^{\perp}_{k_{l}m_{n}}\left(t-1\right)\bm{\Sigma}_{k_{l}m_{n}}\bm{\mathcal{A}}_{k_{l}m_{n}}^{\perp}||^{2}\right),$
Subject to:
$\displaystyle~{}\textbf{{C}}_{1}:|\bm{\mathcal{A}}_{m_{n}}\left(q,z\right)|^{2}=1,\forall~{}q=1,\dots,a\;\mbox{and}\;z=1,\dots,u,$
$\displaystyle~{}\textbf{{C}}_{2}:|\bm{\bm{\delta}}_{k_{n}}\left(q\right)|^{2}=1,\forall~{}~{}q=1,\dots,u.$

In the objective function of $\textbf{{P}}_{2}$ in (14), the variable matrices
$\bm{\mathcal{A}}_{k_{n}m_{n}}^{\not\perp}$,
$\bm{\mathcal{A}}_{k_{n}m_{n}}^{\not\perp}$, and
$\bm{\delta}^{\not\perp}_{k_{n}m_{n}}$ are non-linear functions of
$\bm{\delta}_{k_{y}}$ and $\bm{\mathcal{A}}_{m_{n}}$. This relationship can be
inferred from (12) and (13) as a non-linear truncation of unitary matrices of
the SVD of $\bm{\delta}_{k_{y}}$ and $\bm{\mathcal{A}}_{m_{n}}$. Accordingly,
$\textbf{{P}}_{2}$ is a non-convex combinatorial optimization problem.

###  III-C Digital Beamforming Subsystem: Transmission Rate Maximization

After the analog beamsteering at all MIMO transmitters (eAPs) and receivers
(UEs) has been performed, the actual CSI matrices (i.e.
$\bm{H}_{k_{n}m_{n}},~{}\forall~{}n,m_{n}$, and $k_{n}$) will be multiplied by
the beamsteering matrices (from the right side) and the analog
combining/beamsteering vectors (from the left side). Accordingly, the
effective channel gain at the $m_{n}\rightarrow k_{n}$ link (denoted by
$\bm{\mathcal{H}}_{k_{n}m_{n}}\in\mathbb{C}^{1\times\mathcal{D}_{n,j}^{\text{U}}}$)
will be given by

$\bm{\mathcal{H}}_{k_{n}m_{n}}=\bm{\delta}_{k_{n}}^{\text{T}}\bm{H}_{k_{n}m_{n}}\bm{\mathcal{A}}_{m_{n}}.$
(15)

Note that, the $i_{n}$-th element in $\bm{\mathcal{H}}_{k_{n}m_{n}}$
($i_{n}=1_{n},\dots,\mathcal{D}_{n,j}^{\text{U}}$) corresponds to the signal
radiated from the beam steered at UE $i_{n}$ within the $m_{n}$-th subnetwork.
However, each element in $\bm{\mathcal{H}}_{k_{n}m_{n}}$ will contain portions
of signals send to all UEs of the $n$-th subnetwork.
UE ordering and SIC decoding: In downlink multiuser single-input single-output
(SISO) wireless networks, SIC-based UEs (usually referred to as non-orthogonal
multiple access [NOMA] UEs) are first ordered based on their instantaneous
channel gains. Then, the UEs with lower link gains are allocated higher
transmission power compared to those with better communication link. At the
receiver side, multi-level SIC operations are conducted such that the
interfering signals related to the UEs with lower channel gains are decoded
and then subtracted [26]. When the UEs of a downlink NOMA system are served by
a single eAP with multiple antennas or by multiple single-antenna APs, the UEs
can be ordered based on the norm of their channel quality vector [8]. Such a
channel quality metric is denoted as the “effective channel gain”. In our
considered network model, however, all the eAPs and the UEs in each cluster
are assumed to be equipped with multiple antennas. Therefore, the channel gain
between each eAP and any arbitrary NOMA UE is represented by a complex matrix.
We use the squared norm of the effective channel gain vectors (i.e.
$\bm{\mathcal{H}}_{k_{n}m_{n}}$) as the NOMA effective channel gain.
Accordingly, we assume that the UEs within the $n$-th cluster are arranged in
an ascending order as follows:

$\frac{\sum_{m_{n}=1}^{\mathcal{D}_{n,j}^{\text{A}}}|\bm{\mathcal{H}}_{1_{n}m_{n}}|^{2}}{\sum_{l=1,l\neq
n}^{N}\sum_{m_{l}=1}^{\mathcal{D}_{l}^{\text{A}}}\sum_{k_{l}=1}^{\mathcal{D}_{l}^{\text{U}}}|\bm{\mathcal{H}}_{k_{l}m_{l}}|^{2}}\leq\dots\leq\frac{\sum_{m_{n}=1}^{\mathcal{D}_{n,j}^{\text{A}}}|\bm{\mathcal{H}}_{i_{n}m_{n}}|^{2}}{\sum_{l=1,l\neq
n}^{N}\sum_{m_{l}=1}^{\mathcal{D}_{l}^{\text{A}}}\sum_{k_{l}=1}^{\mathcal{D}_{l}^{\text{U}}}|\bm{\mathcal{H}}_{k_{l}m_{l}}|^{2}}\leq\dots\leq\frac{\sum_{m_{n}=1}^{\mathcal{D}_{n,j}^{\text{A}}}|\bm{\mathcal{H}}_{\mathcal{D}_{n,j}^{\text{U}}m_{n}}|^{2}}{\sum_{l=1,l\neq
n}^{N}\sum_{m_{l}=1}^{\mathcal{D}_{l}^{\text{A}}}\sum_{k_{l}=1}^{\mathcal{D}_{l}^{\text{U}}}|\bm{\mathcal{H}}_{k_{l}m_{l}}|^{2}}$.

Note that, for the UE ordering, we divide the effective channel metric of each
UE by the corresponding ISNI assuming the availability of channel gain
matrices of the overall cell-free network at each eAP. With proper power
allocation for the UEs, at the receiver side, the $i_{n}$-th UE located in the
$n$-th cluster will be able to remove interference components from $i_{n}-1$
UEs with higher overall power gain. Accordingly,
$\gamma_{i_{n}}^{\left\\{\bm{\mathcal{C}}_{j}\right\\}}$ can be rewritten as

$\gamma_{i_{n}}^{\left\\{\bm{\mathcal{C}}_{j}\right\\}}=\frac{\sum_{m_{n}=1}^{\mathcal{D}_{n,j}^{\text{A}}}|\bm{\mathcal{H}}_{i_{n}m_{n}}\bm{w}_{i_{n}m_{n}}|^{2}}{\begin{pmatrix}\sum_{m_{n}=1}^{\mathcal{D}_{n,j}^{\text{A}}}\sum_{k_{n}=i_{n}+1}^{\mathcal{D}_{n,j}^{\text{U}}}|\bm{\mathcal{H}}_{i_{n}m_{n}}\bm{w}_{k_{n}m_{n}}|^{2}+\tilde{\sigma}_{i_{n}}\sum_{j=1}^{u}\delta_{i_{n}j}^{2}\\\
+\sum_{l=1,l\neq
n}^{N}\left(\frac{\mathcal{D}_{n,j}^{\text{U}}}{\mathcal{D}_{l,j}^{\text{U}}}\right)^{2}\sum_{m_{l}=1}^{\mathcal{D}_{l,j}^{\text{A}}}\sum_{k_{l}=1}^{\mathcal{D}_{l,j}^{\text{U}}}|\bm{\mathcal{H}}_{i_{n}m_{l}}\bm{w}_{k_{l}m_{l}}|^{2}\end{pmatrix}},$
(16)

where
$\tilde{\sigma}_{i_{n}}=\left(\frac{\sigma_{i_{n}}\mathcal{D}_{n,j}^{\text{U}}}{{2P}}\right)^{2}$.

We propose a beamforming (i.e. precoding) scheme that maximizes the sum-rate
of the UEs. The digital beamforming problem at the $n$-th cell-free subnetwork
under a certain cell-free network partitioning configuration
($\bm{\mathcal{C}}_{j}$) can be formulated as

$\displaystyle~{}\textbf{{P}}_{3}:\underset{\left\\{\bm{w}_{k_{n}m_{n}}\right\\}_{k_{n}=1,\dots,\mathcal{D}_{n,j}^{\text{U}}}^{m_{n}=1,\dots,\mathcal{D}_{n,j}^{\text{A}}}}{\text{max}}~{}$
$\displaystyle\text{}\sum_{n=1}^{N}\sum_{i_{n}=1}^{\mathcal{D}_{n,j}^{\text{U}}}\log_{2}\left(1+\gamma_{i_{n}}^{\\{\bm{\mathcal{C}}_{j}\\}}\right)$
(17) Subject to:
$\displaystyle~{}\textbf{{C}}_{1}:\sum_{m_{n}=1}^{\mathcal{D}_{n,j}^{\text{A}}}\left(||\bm{\mathcal{H}}_{i_{n}m_{n}}\bm{w}_{\delta_{i_{n}}m_{n}}||^{2}-\sum_{w=\delta_{i_{n}}+1}^{i_{n}}||\bm{\mathcal{H}}_{i_{n}m_{n}}\bm{w}_{wm_{n}}||^{2}\right)\geq\epsilon,$
$\displaystyle~{}\textbf{{C}}_{2}:||\bm{w}_{k_{n}m_{n}}||^{2}\leq 1,$
$\displaystyle\text{\hskip
28.45274pt}\forall~{}\delta_{i_{n}}=1,\dots,{i_{n}}-1,l=2,\dots,\mathcal{D}_{n,j}^{\text{U}},m_{n}=1,\dots,\mathcal{D}_{n,j}^{\text{A}},$
$\displaystyle\text{\hskip
28.45274pt}k_{n}=1,\dots,\mathcal{D}_{n,j}^{\text{U}},\;\mbox{and}\;n=1,\dots,N.$

In [8, Appendix C], it was shown that problem $\textbf{{P}}_{3}$ in (17)
represents a convex optimization problem under the assumption of UE ordering
and SIC-based decoding. Specifically, it was shown that the objective function
of $\textbf{{P}}_{3}$ in (17) can be decomposed into a sum of convex and
concave functions with the convex function having a more increasing rate than
that of the concave one. Furthermore, it is easy to confirm that the
constraints $\textbf{{C}}_{1}$ and $\textbf{{C}}_{2}$ represent affine
relations of $\bm{w}_{k_{n}m_{n}}$777This can be easily confirmed by rewriting
the vector form of $\textbf{{C}}_{1}$ and $\textbf{{C}}_{2}$ in a sum of
products format, rather than vector format.. This convex problem can be easily
solved by using the Karush-Kuhn-Tucker (KKT) conditions and utilizing some
numerical methods for calculating the first and second differentiation of the
Lagrangian function888Despite the existence of the second derivative of the
objective function, a closed-form expression is difficult to derive due to the
multi-dimensional nature of the optimization variables..

##  IV Hierarchical DRL Design: Joint Network Partitioning and Hybrid
Beamforming

###  IV-A DRL Techniques for Solving Optimization Problems

DRL techniques have been used to solve optimization problems in wireless
communications systems (e.g. for optimization of downlink power control in a
multi-cell system [27], beamforming optimization in a cell-free network [2]).
In these cases, a DRL agent (e.g. a network entity) aims at learning the
“optimal” mapping between a system state $\bm{s}$ and the action $\bm{a}$
(e.g. a policy function or a value function) in order to maximize its reward
discounted reward over a time horizon. Depending on the agent’s objective, DRL
techniques are commonly classified into three categories:

  * •

Value-based methods such as deep Q-learning (DQL) and
state–action–reward–state–action (SARSA) learn the value function $V(\bm{s})$
or the state-value function $Q(\bm{s},\bm{a})$ to find a policy.

  * •

Policy-based methods obtain the mapping between the system state and the
action (i.e. policy) directly. These methods generally suffer from noisy
gradients and high variance [28].

  * •

Actor-critic methods are a hybrid of the value-based and policy-based methods.
Value-based methods are used to reduce the variance of the policy-based
methods by estimating the value function or the action-value function (a.k.a.
the critic) to improve the performance of the policy (a.k.a. the actor).

###  IV-B Hierarchical DRL Architecture

The proposed solution consists of two-levels of interacting DRL models. The
first-level of the proposed system is responsible for network partitioning
(i.e. clustering) and it consists of a single DRL model. The agent of the
first-level DRL model is located at the NCC and is mainly responsible for
partitioning the overall cell-free network into a set of non-overlapping cell-
free subnetworks. The second-level of the proposed hierarchical architecture
consists of $N$ independent DRL subsystems. Each DRL subsystem is responsible
for conducting the hybrid analog beamsteering-digital beamforming process in a
single cell-free subnetwork. This is achieved by training the DRL subsystem
agent to optimize the analog beamsteering vectors of all eAPs and UEs of the
corresponding cell-free subnetwork while the digital beamforming problem for
the same subnetwork is modeled and solved as a convex optimization problem
inside the environment of the DRL subsystem for analog beamsteering. All of
the DRL subsystems are within the environment of the first-level DRL
clustering system (Fig. 3(a)).

Figure 3: Block diagram of the hierarchical DRL clustering system.

In terms of the time-scale of operation, the overall cell-free network is
assumed to cluster (partition) every $\tau\geq 1$ time instants999The value of
$\tau$ may be considered as a design parameter that can depend, for example,
on the time-varying nature of the propagation environment.. On the other hand,
the hybrid beamforming process is assumed to take place at each time slot.
Further details about the DRL model action spaces, rewards, and observed
states are described in the following sections.

###  IV-C Learn to “Cluster”

Our objective is to design a self-organizing cell-free network that has the
ability to self-partition (self-cluster) into a group of cell-free
subnetworks, in a time-slot basis, based on the instantaneous CSI. For the
proposed dynamic cell-free network with $M$ eAPs, $K$ UEs, and $N$
subnetworks, there will be
$\Theta\left(M,N\right)=\left(\frac{N!}{\sqrt{2}}\right)^{2}C\left(M,N\right)C\left(K,N\right)$
possible configurations for cell-free subnetworks, where $C\left(M,N\right)$
is the Stirling number, which can be calculated as [29]

$C\left(M,N\right)=\left\\{\begin{array}[]{ll}M\\\
N\end{array}\right\\}=\frac{1}{N!}\sum_{i=0}^{N}\left(-1\right)^{i}\binom{N}{i}\left(N-i\right)^{M}.$
(18)

Optimally updating the cell-free network configuration on a time-slot basis
requires going through all possible configurations (as discussed in Section
II-D) which will be practically infeasible for a massive cell-free network
with large numbers of eAPs and UEs101010As an example, for $M=100$, $K=50$,
and $N=10$, there will be approximately $1.28962122\times 10^{138}$ possible
cell-free subnetwork configurations..

In this section, we design several low-complexity DRL-based methods to
efficiently perform network clustering on a time-slot basis. Each of these
methods accepts certain network information (e.g. instantaneous CSI values
across the entire network) and outputs a certain network partitioning
configuration that maximizes a predefined network performance metric. Table II
summarizes the environment design for the DRL models in terms of the problem
parameters.

TABLE II: DRL model for network partitioning Clustering Environment Variables |  Network Partitioning System Equivalence .
---|---
State ${\bm{s}_{\text{c}}}=\left\\{{s}_{\text{c},1},\dots,{s}_{\text{c},N}\right\\}$ | $\left\\{\prod_{t=1}^{\tau}\prod_{i_{1}=1}^{\mathcal{D}_{1,j}^{\text{U}}}\gamma_{i_{1}}^{\\{\mathcal{C}_{j}\\}}\left(t\right),\dots,\prod_{t=1}^{\tau}\prod_{i_{N}=1}^{\mathcal{D}_{N,j}^{\text{U}}}\gamma_{i_{N}}^{\\{\mathcal{C}_{j}\\}}\left(t\right)\right\\}$
Reward $r_{\text{c}}$ | $\prod_{t=1}^{\tau}\left(\prod_{n=1}^{N}\left(\sum_{i_{n}=1}^{\mathcal{D}_{n,j}^{\text{U}}}\;\log\left(1+\gamma_{i_{n}}^{\\{\mathcal{C}_{j}\\}}\left(t\right)\right)\right)\right)$
Action $\bm{a}_{\text{c}}$ | $\bm{\mathcal{C}}_{j}=\left\\{\left\\{\bm{\mathcal{C}}_{1,j}^{\text{A}},\bm{\mathcal{C}}_{1,j}^{\text{U}}\right\\},\dots,\left\\{\bm{\mathcal{C}}_{N,j}^{\text{A}},\bm{\mathcal{C}}_{N,j}^{\text{U}}\right\\}\right\\}$

Note that, in Table II, the DRL system state vector ($\bm{s}_{\text{c}}$)
corresponds to a clustering configuration with $N$ clusters (i.e. an
$N$-element vector) and the value of each element is the product of the SINR
values of all the UEs in the corresponding cluster (or partition). The
immediate reward value for a state is given by the product of the sum of the
rates of the users in each cluster, under the corresponding clustering
configuration. Among the different DRL methods, we investigate (i) value-based
DRL methods, namely, the deep double Q-network DDQN [30] and
State–action–reward–state–action (SARSA) [31], (ii) a policy-based DRL method,
namely, the policy gradient (PG) [32] method, and (iii) the actor-critic (AC)
[33] method. The performance, complexity, and convergence rate of these
methods are then evaluated and compared.
Value-based DRL methods (e.g. DDQN and SARSA): In value-based clustering, each
network partitioning configuration is assigned a certain value through a
state-value function $V^{\pi}(\bm{\Gamma}_{N})$, also known as the expected
return function when starting at a certain state $\bm{\Gamma}_{M}$, where
$\bm{\Gamma}_{N}=(\gamma_{1}^{\text{P}},\dots,\gamma_{N}^{\text{P}})$ and
$\gamma_{n}^{\text{P}}=\prod_{t=1}^{\tau}\prod_{i_{n}=1}^{\mathcal{D}_{n,j}^{\text{U}}}\gamma_{i_{n}}^{\\{\bm{\mathcal{C}}_{j}\\}}\left(t\right)$.
The state-value function is defined as
$V^{\pi}\left(\bm{\Gamma}_{N}\right)=\mathbb{E}\left[r|\bm{\Gamma}_{N},\pi\right],$
where $r$ is the immediate reward, $\pi$ is the followed policy which can be
found such that
$V^{*}\left(\bm{\Gamma}_{N}\right)=\underset{\pi}{{\text{max}}}~{}V^{\pi}\left(\bm{\Gamma}_{N}\right),~{}\forall~{}\gamma_{n}^{\text{P}}\in\mathbb{R},~{}n=1,\dots,N$.
Given $V^{*}\left(\bm{\Gamma}_{N}\right)$, the optimal policy $\pi^{*}$ is
found by selecting the best cell-free network partitioning configuration that
maximizes
$\mathbb{E}_{\bm{\Gamma}_{N}^{\prime}\thicksim\mathcal{T}\left(\bm{\Gamma}_{N}^{\prime}|\bm{\Gamma}_{N},\bm{\mathcal{C}}_{j}\right)}\left[V^{*}\left(\bm{\Gamma}_{N}^{\prime}\right)\right]$,
where
$\mathcal{T}\left(\bm{\Gamma}_{N}^{\prime}|\bm{\Gamma}_{N},\bm{\mathcal{C}}_{j}\right)$
is the transition dynamics that is usually unavailable. Hence, the value
function is replaced by a quality state-action-value function
$Q^{\pi}\left(\bm{\Gamma}_{N},\bm{\mathcal{C}}_{j}\right)$, which is different
from $V^{\pi}$ due to the fact that a random cell-free network partitioning
configuration action $\bm{\mathcal{C}}_{0}$ is provided and the policy $\pi$
is only counted from the succeeding state, i.e.
$Q^{\pi}\left(\bm{\Gamma}_{N},\bm{\mathcal{C}}_{j}\right)=\mathbb{E}\left[r|\bm{\Gamma}_{N},\bm{\mathcal{C}}_{j},\pi\right]$.
The learning of the $Q^{\pi}$ network is performed by using the Bellman
equation with the recursive form
$Q^{\pi}\left(\bm{\Gamma}_{N},\bm{\mathcal{C}}_{j}\right)=\mathbb{E}_{\bm{\Gamma}_{N}^{\prime}}\left[r^{\prime}+\zeta
Q^{\pi}\left(\bm{\Gamma}_{N}^{\prime},\pi\left(\bm{\Gamma}_{N}^{\prime}\right)\right)\right]$
[34]. This means that the quality function can be improved by bootstrapping
(i.e. using current values of $Q^{\pi}$ to improve our estimate). This
modeling is the basis of Q-learning [30] and SARSA [31] algorithms that is
defined as $Q^{\pi}\left(\bm{\Gamma}_{N},\bm{\mathcal{C}}_{j}\right)\leftarrow
Q^{\pi}\left(\bm{\Gamma}_{N},\bm{\mathcal{C}}_{j}\right)+\alpha\delta,$ where
$\alpha$ is the learning rate and
$\delta=Y-Q^{\pi}\left(\bm{\Gamma}_{N},\bm{\mathcal{C}}_{j}\right)$ is the
temporal difference error with $Y$ representing a target (as in standard
regression problems). Using the Q-learning cell-free network partitioning
agent, the target $Y$ directly approximates $Q^{*}$ by setting
$Y=r+\zeta\underset{j}{\text{max}}Q^{\pi}\left(\bm{\Gamma}_{N}^{\prime},\bm{\mathcal{C}}_{j}\right)$
(off-policy agent), where $\zeta$ is the discount factor. On the other hand,
the SARSA algorithm improves the estimate of $Q^{\pi}$ by deriving a
behavioural policy from $Q^{\pi}$. This is achieved by setting $Y=r+\zeta
Q^{\pi}\left(\bm{\Gamma}_{N}^{\prime},\bm{\mathcal{C}}_{j}^{\prime}\right)$
(on-policy agent).
Policy-based DRL method for clustering: In policy gradient (PG) algorithms,
the modeling and optimization of a certain policy is conducted directly
through a parameterized function,
$\mu_{\theta}\left(\bm{\mathcal{C}}_{j},\bm{\Gamma}_{N}\right)$. The value of
the objective function (the reward) directly depends on the policy. The value
of the reward function for PG-based clustering is given by

$\displaystyle
J(\theta)=\sum_{\bm{s}_{\text{c}}\in\mathbb{R}^{N}}d^{\mu}\left(\bm{\Gamma}_{N}\right)V^{\mu}\left(\bm{\Gamma}_{N}\right)=\sum_{\bm{s}_{\text{c}}\in\mathbb{R}^{N}}d^{\mu}\left(\bm{\Gamma}_{N}\right)\sum_{j\in\mathbb{Z}}\mu_{\theta}\left(\bm{\mathcal{C}}_{j}|\bm{\Gamma}_{N}\right)Q^{\mu}\left(\bm{\Gamma}_{N},\bm{\mathcal{C}}_{j}\right),$
(19)

where $d^{\mu}\left(\bm{\Gamma}_{N}\right)$ is the stationary state
distribution of Markov chain. Note that the gradient of $J\left(\theta\right)$
(denoted by $\nabla_{\theta}J\left(\theta\right)$) depends both on the
selected actions $\bm{a}_{\text{c}}$ and the stationary distribution
$d^{\mu}\left(\bm{\Gamma}_{N}\right)$. We also use a PG algorithm that
simplifies the computation of the gradient by removing the dependence of
$J\left(\theta\right)$ on $d^{\mu}\left(\bm{\Gamma}_{N}\right)$ as follows
[28, Sec. 13.2]:

$\nabla_{\theta}J\left(\theta\right)=\nabla_{\theta}\sum_{\bm{s}_{\text{c}}\in\mathbb{R}^{N}}d^{\mu}\left(\bm{\Gamma}_{N}\right)\sum_{j\in\mathbb{Z}}\mu_{\theta}\left(\bm{\mathcal{C}}_{j}|\bm{\Gamma}_{N}\right)Q^{\mu}\left(\bm{\Gamma}_{N},\bm{\mathcal{C}}_{j}\right)\\\
\propto\sum_{\bm{s}_{\text{c}}\in\mathbb{R}^{N}}d^{\mu}\left(\bm{\Gamma}_{N}\right)\sum_{j\in\mathbb{Z}}\mu_{\theta}\left(\bm{\mathcal{C}}_{j}|\bm{\Gamma}_{N}\right)\nabla_{\theta}Q^{\mu}\left(\bm{\Gamma}_{N},\bm{\mathcal{C}}_{j}\right).$
(20)

The general policy gradient method has a high variance. Accordingly, the
following general form is used as a foundation of different PG algorithms:

$\nabla_{\theta}J\left(\theta\right)=\mathbb{E}_{\mu_{\theta}}\left[\sum_{t=0}^{T-1}G_{t}\nabla_{\theta}\log\mu_{\theta}\left(\bm{\mathcal{C}}_{j}|\bm{\Gamma}_{N}\right)\right].$
(21)

The PG-based DRL model for network partitioning can be then trained through
the following steps:

  * i.

Initialize the actor $\mu\left(\bm{\Gamma}_{N}\right)$ with random weights
$\theta_{\mu}$.

  * ii.

For each training episode (i.e. every $T$ training steps), generate the
experiences by following $\mu\left(\bm{\Gamma}_{N}\right)$: the actor
generates the probability values for each possible cell-free partitioning,
then the DRL agent randomly selects an action based on a certain probability
distribution. This process continues for $T$ steps.

  * iii.

At each step of a certain episode, calculate the return value
$G_{t}=\sum_{l=t}^{T}\zeta^{l-1}r_{l}$.

  * iv.

Find the cumulative sum of the actor network gradients during one entire
learning episode as

$d\theta_{\mu}=\sum_{t=1}^{T}G_{t}\nabla_{\theta_{\mu}}\ln\mu\left(\bm{\Gamma}_{N}|\theta_{\mu}\right)$
(22)

  * v.

Update the actor network using: $\theta_{\mu}\leftarrow\theta_{\mu}+\alpha
d\theta_{\mu}$, where $\alpha$ is the learning rate.

(Actor-critic)-based DRL method: In the PG-based partitioning algorithm, the
value function $G_{t}$ is estimated based on a preassigned policy. However,
the estimation of $G_{t}$ for a predefined policy introduces a relatively high
variance of the policy gradient which in turn reduces the quality of cell-free
network partitioning action. In order to tackle the high variance problem, a
second DNN can be used that can accurately learn the value of $G_{t}$[35].

###  IV-D Learn to “Beamsteer”

As discussed in Sec. III-A, the downlink beamforming is performed through two
consecutive stages, namely, analog beamsteering and digital beamforming. In
this section, we develop a mixed DRL-cum-convex optimization subsystem that
performs the two-stage beamforming operation for each cell-free subnetwork
(Fig. 3(b)). In the proposed system, the non-convex analog beamsteering
(problem $\textbf{{P}}_{2}$, which is a non-convex combinatorial optimization
problem) is solved by training a DRL agent that “interacts” with the
propagation medium (i.e. DRL environment) on a time-slot basis. The convex
digital beamforming problem $\textbf{{P}}_{3}$, which is a strictly convex
optimization problem, on the other hand, is solved within the DRL environment
by using conventional convex optimization methods (e.g. Newton and Broyden
methods111111The Lagrangian function of $\textbf{{P}}_{3}$ is twice
differentiable w.r.t all optimization variables [36].). This process of hybrid
analog beamsteering-digital beamforming is performed independently at each
subnetwork on a time-slot basis.

Table III shows the main design parameters of the DRL model.

TABLE III: DRL model for hybrid beamforming in subnetwork $n$ Beamforming Environment Variables |  Network Beamforming System Equivalence .
---|---
State $\bm{s}_{\text{b}}=\left\\{{s}_{\text{b},1_{n}},\dots,{s}_{\text{b},\mathcal{D}_{n,j}^{\text{U}}}\right\\}$ |  $\left\\{\gamma_{1}^{\\{\mathcal{C}_{j}\\}},\dots,\gamma_{\mathcal{D}_{n,j}^{\text{U}}}^{\\{\mathcal{C}_{j}\\}}\right\\}$ (Involving Digital Beamforming)
Reward $r_{\text{b}}$ | $\sum_{m_{n}=1}^{\mathcal{D}_{n,j}^{\text{A}}}\left(\sum_{k_{n}=1}^{\mathcal{D}_{n,j}^{\text{U}}}||\bm{\delta}^{\not\perp}_{k_{n}}\bm{\Sigma}_{k_{n}m_{n}}\bm{\mathcal{A}}_{m_{n}}^{\not\perp}||^{2}+\sum_{l=1,l\neq n}^{N}\sum_{k_{l}=1}^{\mathcal{D}_{l,j}^{\text{U}}}||\bm{\delta}^{\perp}_{k_{l}}\left(t-1\right)\bm{\Sigma}_{k_{l}m_{n}}\bm{\mathcal{A}}_{m_{n}}^{\perp}||^{2}\right)$
Action $\bm{a}_{\text{b}}$ | $\left\\{\bm{\delta}_{k_{n}},\bm{\mathcal{A}}_{m_{n}}\right\\}^{m_{n}=1,\dots,\mathcal{D}_{n,j}^{\text{A}}}_{k_{n}=1,\dots,\mathcal{D}_{n,j}^{\text{U}}}$

Note that the operation of digital beamforming is performed as a part of the
DRL environment computations that produce the observed states and reward for
the analog beamsteering agent. The proposed subsystem can be implemented by
using several DRL algorithms. In this paper, we implement and benchmark two
DRL algorithms with continuous action space, namely, the PG algorithm and the
Soft Actor-Critic Agents (SAC) algorithm.
Policy-based beamsteering: The PG algorithm used previously for cell-free
network partitioning can be also implemented to learn the best beamsteering
vectors by solving problem $\textbf{{P}}_{2}$. This can be achieved by
optimizing over the discrete action space and then estimating the best
continuous beamsteering action. Such an approximation process is relatively
slow/inefficient. However, working directly with policies that emit
probability distributions can increase the estimation speed of the continuous
action space since sampling a well-known distribution is easier than sampling
from value functions.
Soft actor-critic beamsteering: On-policy actor-critic algorithms improve the
stability of the network by allowing random exploration of experience from
actions replay buffers [28]. However, this on-policy training results in a
poor sample complexity. On the other hand, off-policy algorithms have been
developed to improve the sampling efficiency while maintaining robustness by
developing more advanced variance reduction techniques and at the same time
incorporating the off-policy samples (e.g. the DDPG family of algorithms)
[37]. However, the interaction between the off-policy DDQN value estimator and
the deterministic actor setting makes DDPG extremely difficult to stabilize
and adjust the hyper-parameter settings. This issue becomes more severe as the
size of the cell-free network increases. We propose to utilize the SAC
algorithm to solve the beamsteering problem at each cell-free subnetwork [38,
Algorithm 1]. The SAC algorithm enables off-policy actor-critic training with
a stochastic actor. The main difference between the SAC algorithm and the GP
and AC ones is that the SAC algorithm uses a general objective that maximizes
entropy along with the cumulative reward [39]. The addition of policy entropy
encourages the actor network to explore new experiences. Accordingly, the
expected reward in Eq. (21) can be modified to [39]

$\nabla_{\theta_{\mu}}J\left(\theta_{\mu}\right)=\nabla_{\theta_{\mu}}\sum_{t=0}^{T-1}\mathbb{E}_{\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}},\bm{\mathcal{F}}_{n}\right)\thicksim\rho_{\theta_{\mu}}}\left[r\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}},\bm{\mathcal{F}}_{n}\right)+\alpha\mathcal{H}\left(\mu\left(\bm{\mathcal{F}}_{n}|\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}\right)\right)\right],$
(23)

where
$\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}=\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}$,
$\bm{\mathcal{F}}_{n}=\bm{\mathcal{F}}_{n}$, and $\mathcal{H}\left(.\right)$
is the entropy measure of the policy
$\mu\left(\bm{\mathcal{F}}_{n}|\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}\right)$,
$\alpha$ is a temperature factor that determines the relative importance of
the policy entropy against the reward
$r\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}},\bm{\mathcal{F}}_{n}\right)$,
and $\rho_{\mu}\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}\right)$ and
$\rho_{\mu}\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}},\bm{\mathcal{F}}_{n}\right)$
are the state and state-action of the trajectory distribution introduced by
$\mu\left(\bm{\mathcal{F}}_{n}|\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}\right)$.
The soft state-value function of SAC algorithm is given by [38]

$V\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}\right)=\mathbb{E}_{\bm{a}_{\text{b}}\thicksim\bm{\mu}}\left[Q\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}},\bm{\mathcal{F}}_{n}\right)-\log\mu\left(\bm{\mathcal{F}}_{n}|\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}\right)\right].$
(24)

Accordingly, the soft Q-value will be defined as [38]

$Q\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}},\bm{\mathcal{F}}_{n}\right)=r\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}},\bm{\mathcal{F}}_{n}\right)+\gamma\mathbb{E}_{{\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}}^{\prime}\thicksim\rho_{\mu}\left({\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}}^{\prime}\right)}\left[Q\left({\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}}^{\prime},\bm{\mathcal{F}}_{n}^{\prime}\right)-\log\mu\left(\bm{\mathcal{F}}_{n}^{\prime}|{\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}}^{\prime}\right)\right].$
(25)

The SAC algorithm aims to learn three functions, namely, i) a policy function
with parameters $\theta$ and $\pi_{\theta}$, ii) a soft Q-value function
approximated (parameterized) by $w$ and $Q_{w}$, and iii) a soft state value
function parameterized by $\psi$ and $V_{\psi}$. The soft state value is
trained to minimize the mean square error with gradient function given as
follows [38]:

$\nabla_{\psi}J_{V}\left(\psi\right)=\nabla_{\psi}\mathbb{E}_{\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}\thicksim\mathcal{R}}\left[\frac{1}{2}\left(V_{\psi}\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}\right)-\mathbb{E}\left[Q_{w}\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}},\bm{\mathcal{F}}_{n}\right)-\log\pi_{\theta}\left(\bm{\mathcal{F}}_{n}|\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}\right)\right]\right)^{2}\right]\\\
\approx\nabla_{\psi}V_{\psi}\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}\right)\left(V_{\psi}\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}\right)-Q_{w}\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}},\bm{\mathcal{F}}_{n}\right)+\log\pi_{\theta}\left(\bm{\mathcal{F}}_{n}|\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}\right)\right),$
(26)

where $\mathcal{R}$ is the distribution of previously sampled actions and
states (in the replay buffer). Furthermore, the soft Q function is trained to
minimize the soft Bellman residual with gradient function given as

$\nabla_{w}J_{Q}\left(w\right)=\nabla_{w}\mathbb{E}_{\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}},\bm{\mathcal{F}}_{n}\right)\thicksim\mathcal{R}}\left[\frac{1}{2}\left(Q_{w}\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}},\bm{\mathcal{F}}_{n}\right)-\left(r\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}},\bm{\mathcal{F}}_{n}\right)\\\
+\zeta\mathbb{E}_{{\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}}^{\prime}\thicksim\rho_{\pi}\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}\right)}\left[V_{\bar{\psi}}\left({\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}}^{\prime}\right)\right]\right)\right)^{2}\right]\\\
\approx\nabla_{w}Q\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}},\bm{\mathcal{F}}_{n}\right)\left(Q_{w}\left(\bm{\mathcal{F}}_{n},\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}\right)-r\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}},\bm{\mathcal{F}}_{n}\right)-\zeta
V_{\bar{\psi}}\left({\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}}^{\prime}\right)\right),$
(27)

where $\bar{\psi}$ is an exponentially moving average target function. The
desired policy is then trained using the information projection that is
defined in terms of Kullback-Leibler (KL)-divergence [40]. Accordingly, the
policy is updated according to

$\pi_{\text{new}}=\operatorname*{argmax}_{\pi^{\prime}\in\Pi}D_{\text{KL}}\left(\pi^{\prime}\left(.|\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}\right)||\frac{\exp\left\\{Q^{\pi_{\text{old}}}\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}},.\right)\right\\}}{Z^{\pi^{\text{old}}}\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}\right)}\right),$
(28)

where $\Pi$ denotes a set of potential policies that $\pi$ must restricted to.
$Z^{\pi^{\text{old}}}\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}\right)$ is
a partitioning function that is used for normalizing the distribution. The
objective update function of the policy $\pi_{\theta}$ is [38]

$\nabla_{\theta}J_{\pi}\left(\theta\right)=D_{\text{KL}}\left(\pi_{\theta}\left(.|\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}\right)||\exp\left\\{Q_{w}\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}},\bm{\mathcal{F}}_{n}\right)-\log
Z_{w}\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}\right)\right\\}\right)\\\
=\mathbb{E}_{\bm{\mathcal{F}}_{n}\thicksim\pi}\left[-\log\left(\frac{\exp\left\\{Q_{w}\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}},\bm{\mathcal{F}}_{n}\right)-\log
Z_{w}\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}\right)\right\\}}{\pi_{\theta}\left(\bm{\mathcal{F}}_{n}|\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}\right)}\right)\right]\\\
=\mathbb{E}_{\bm{\mathcal{F}}_{n}\thicksim\pi}\left[\log\pi_{\theta}\left(\bm{\mathcal{F}}_{n}|\bm{s}_{\text{b}}\right)-Q_{w}\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}},\bm{\mathcal{F}}_{n}\right)+\log
Z_{w}\left(\bm{\Gamma}_{n}^{\\{\mathcal{C}_{j}\\}}\right)\right].$ (29)

We use the SAC algorithm developed in [38, Algorithm 1].

Algorithm 1 shows the sequence of processes performed during network
operation, where $E_{\text{c}}$ and $E_{\text{b}}$ are the number of episodes
for the DRL models used for clustering and beamsteering, respectively,
$T_{\text{c}}$ and $T_{\text{b}}$ are the number of training steps in each
episode for the DRL models used for clustering and beamsteering, respectively.

Algorithm 1 Training for network partitioning and beamforming

1:Input: $\bm{H_{k_{n}m_{n}}}$, $M$, $K$, $a$, $u$, and $N$.

2:Initialize target networks of main DRL system and subsystems in Figs. 3(a)
and (b)121212The type of network will depend on the utilized clustering and
beamsteering techniques (Sec. V.B and Sec. V.C)..

3:Initialize replay buffers of the main DRL system and the subsystems in Figs.
3(a) and (b).

4:Initialize $\bm{\mathcal{C}}_{j}$, $\bm{\mathcal{A}}_{m_{n}}$,
$\bm{\delta}_{k_{n}}$ and $\bm{w}_{k_{n}m_{n}}$, $\forall~{}n,m_{n}$, and
$k_{n}$.

5:for $\text{Episodes}_{\text{Clustering}}=1\;\mbox{to}\;E_{\text{c}}$ do

6: for $t_{\text{c}}=1\;\mbox{to}\;T_{\text{c}}$ do $\triangleright$ Training
steps for network partitioning

7: Update weights of target networks using Algorithms in Sec. V.B

8: parfor $n=1\;\mbox{to}\;N$ do$\triangleright$ Simultaneous beamforming in
all subnetworks

9: for $\text{Episodes}_{\text{Beamsteering}}=1\;\mbox{to}\;E_{\text{b}}$ do

10: for $t_{b}=1\;\mbox{to}\;T_{b}$ do

11: Update weights of target networks using the Algorithms in Sec. V.C.

12: Compute
$\bm{\mathcal{H}}_{k_{n}m_{n}},~{}\forall~{}k_{n}=1,\dots,\mathcal{D}_{n,j}^{\text{U}}$
and $m_{n}=1,\dots,\mathcal{D}_{n,j}^{\text{A}}$.

13: Solve problem $\textbf{{P}}_{3}$ using a convex optimizer.

14: Update $\bm{\mathcal{A}}_{m_{n}}$, $\bm{\delta}_{k_{n}}$, and
$\bm{w}_{k_{n}m_{n}}$, $\forall~{}n,m_{n}$, and $k_{n}$.

15: end for

16: end for

17: end parfor

18: Update $\bm{\mathcal{C}}_{j}$.

19: end for

20:end for

.

##  V Complexity Analysis and Signaling Overhead

To solve problem $\textbf{{P}}_{1}$ in (6), the following subproblems will
need to be solved: the combinatorial problem related to selecting the best
network partitioning configuration, the non-convex problem related to finding
the best beamsteering matrices (problem $\textbf{{P}}_{2}$ in (14)), and a
convex optimization problem related to finding the optimal digital beamforming
at each eAP.

In Sec. IV-C, we discussed how the complexity of finding best network
partitions grows exponentially with increasing values of $M$, $K$, and $N$, as
can be seen from (18). Furthermore, solving $\textbf{{P}}_{2}$ in (14) through
an exhaustive search with a step size $\Delta$ will have a complexity of order
$O\left(\prod_{n=1}^{N}\left(\frac{1}{\Delta}\right)^{(a\times\mathcal{D}_{n,j}^{\text{A}})\times(u\times\mathcal{D}_{n,j}^{\text{U}})}\right)$.
Since the problem $\textbf{{P}}_{3}$ in (17) is strictly convex, the solution
for this problem has a computational complexity of
$O\left(\left(\sum_{n=1}^{N}\mathcal{D}_{n,j}^{\text{U}}\times\mathcal{D}_{n,j}^{\text{A}}\right)^{3}\right)$.

To evaluate the time-complexity of a deep neural network used in a DRL model,
the conventional measure is the floating-point operations per second (FLOPs).
For any fully connected layer $\mathbb{L}_{i}$ of input size $I_{i}$ and
output size $O_{i}$, the number of FLOPs is given by
$\text{FLOPS}(\mathbb{L}_{i})=2\,I_{i}\,O_{i}$. The policy network has two
hidden layers of size 256 and 128. Thus, for the DRL models, the total number
of FLOPS during the inference is

$\displaystyle\text{FLOPs}_{\text{DRL}}$
$\displaystyle=\sum_{i=1}^{3}\,\text{FLOPs}(\mathbb{L}_{i})$
$\displaystyle=2\cdot\Big{(}256\cdot\mathcal{|S|}+128\cdot\bm{|A|}+32768\Big{)},$
(30)

where $\mathcal{|S|}$ and $|\bm{A}|$ are the dimensions of the state space and
action space, respectively. Table IV compares the FLOPS for inference as well
as the convergence rate for the DRL algorithms used in this paper.

TABLE IV: Complexity of different DRL models for clustering DRL Agent | Inference FLOPS | Convergence
---|---|---
Conventional solution . |  $\Theta\left(M,K,N\right)$ |  Linear convergence
SARSA | $32768+256\cdot{K}+128$ | Slow
DDQN | $32768+256\cdot{K}+128$ | Geometric
PG | $32768+256\cdot{K}+128$ | Sub-linear
Actor-Critic | $2\left(32768+256\cdot{K}+128\right)$ | Fast

Note that for network clustering, the dimensionality of the action space is
$|\bm{A}|=1$. Similarly, the complexity and convergence properties for the
considered beamsteering agents are summarized in Table V131313The ellipsoid
method requires a total of
$O\left(\left[\left((\mathcal{D}_{n,j}^{\text{A}}\times\mathcal{D}_{n,j}^{\text{U}}\right)\times\left(a\times
u\right)\right]^{4}q\right)$ operations, where $q$ is the length of binary
coding of the input..

TABLE V: Complexity of DRL models for beamsteering in the $n$-th subnetwork DRL Agent | Inference FLOPS | Convergence
---|---|---
Conventional solution . |  $O\left(\left[\left(\mathcal{D}_{n,j}^{\text{A}}\times\mathcal{D}_{n,j}^{\text{U}}\right)\times\left(a\times u\right)\right]^{2}q\right)$ |  Linear convergence
PG | $32768+256\cdot{K}+128\cdot{\mathcal{M}_{n}}$ | Sub-linear
DDPG | $32768+256\cdot{K}+128\cdot{\mathcal{M}_{n}}$ | Unknown
SAC | $32768+256\cdot{K}+128\cdot{\mathcal{M}_{n}}$ | Unknown

$\mathcal{M}_{n}$ in Table V represents the dimensionality of the analog
beamsteering problem and is given by
$\mathcal{M}_{n}=\left(\mathcal{D}_{n,j}^{\text{A}}\times\mathcal{D}_{n,j}^{\text{U}}\right)\times\left(a\times
u\right)$.

In terms of signaling overhead, in the proposed methods, the NCC first will
have to collect the estimated CSI matrices from distributed eAPs and send full
CSI to the ECP of each cell-free subnetwork. Next, the NCC will collect the
performance metric (e.g. sum-rate) from the ECP of each subnetwork and use it
to decide on the new network partitioning configuration.

##  VI Numerical Results

###  VI-A Parameters and Assumptions

Table VI presents the values of different parameters used in generating the
simulation results.

TABLE VI: Simulation parameters Parameter | Value
---|---
AWGN PSD at UE |  $-169$ dBm/Hz
Path-loss exponent | 2 (outdoor)
mmWave carrier frequency, $\frac{3\times 10^{8}}{\lambda}$ |  $24$ GHz (unless specified otherwise)
mmWave paths, $\mathcal{L}$ |  $3$ (unless specified otherwise )
SIC sensitivity, $P_{s}$ |  $1$ dBm
$\\#$ of training episodes | $\\{2000,4000\\}$
$\\#$ of training steps/episode | $200$
Discount factor, $\zeta$ | $0.01$
Learning rate, $\alpha$ | $0.001$

All the results for the conventional methods are produced using Mont-Carlo
simulations each with $10^{6}$ runs. Additionally, we assume that all channel
small-scale fading gains $h_{k_{n}m_{n}}$ are drawn from a set of i.i.d random
variables. We assume that all APs and UEs are uniformly distributed over a
disc of radius 18 m (corresponding to a network total coverage area of
$1\text{km}^{2}$).

###  VI-B Hybrid Beamforming Scheme

We start this section by evaluating the performance of the proposed network
architecture under the designed hybrid beamforming system (Fig. 4(a)).

Figure 4: Hybrid vs. conventional beamforming techniques.

It can be noticed from this figure that the designed hybrid analog
beamsteering-digital beamforming scheme significantly outperforms that of
conventional (all digital) beamforming scheme. For example, a gain of $2.3$
bps/Hz and $1.5$ bps/Hz are achieved at $35$ dBm with $N=1$ and $N=2$,
respectively (upper graph in Fig. 4(a)). Interestingly, even without network
partitioning (i.e. for $N=1$), our proposed hybrid beamforming scheme shows
significant increase in performance compared to its conventional beamforming
counterpart. The reason is that our designed objective function for
beamsteering (see Problem $\texttt{P}_{2}$ in (14)) aims to enhance beams of
desired UEs and “zero-null” beams to undesired UEs at the same time.
Accordingly, with $N=1$, beamsteering will focus on optimally directing the
antenna main lobes of APs and UEs to each other. However, this performance
gain is observed to decrease as the network scales up (see lower graph of Fig.
4(a)). In order to study the effect of multiple antennas on the per-UE rate
performance, Fig. 4(b) shows per-UE transmission rate versus different MIMO
layouts. It can be noticed that a significant increase in per-UE rate
performance can be achieved by increasing the number of antennas at the UEs
and eAPs (upper graph of Fig. 4(b)). This rate enhancement, however, decreases
as the values of $a$ and/or $b$ increase (due to increased interference
levels). Furthermore, the system performance enhances better as the antenna
order at the UEs increases more than that at the eAPs (lower graph of Fig.
4(b)).

###  VI-C Evaluation and Benchmarking of Hierarchical DRL Models

The performances of the DRL models for network partitioning and analog
beamsteering are investigated separately. This is done by first training the
different DRL clustering agents while using conventional methods for hybrid
analog beamsteering-digital beamforming operations. On the other hand, the
DRL-based beamforming methods are evaluated while clustering is performed
through the trained DRL agents in the inference mode. This separate evaluation
enables us to extract more insights on the performances of that DRL models
under discrete and continuous action spaces.

We start by evaluating the performances of the DRL-based clustering algorithms
for the proposed self-partitioning cell-free network architecture (Fig. 5(a)).
We use two training modes for each of the studied DRL algorithms. The first
training mode considers a fixed CSI (i.e. constant $\bm{H}$ matrix), while in
the second mode, we use different CSI realizations at every training step of
each episode. In Fig. 5(a), we train four DRL agents using PG, DDQN, SARSA,
and AC algorithms for network partitioning for a single CSI realization.

Figure 5: Performance of clustering agents: (a) fixed CSI, and (b) varying
CSI.

As can be observed, the PG algorithm provides the best clustering performance
in terms of stability and convergence, while the DDQN algorithm comes second,
and the SARSA algorithm comes last with significantly high covariance in the
Q-values per episode and slower convergence rate (upper graph in Fig. 5(a)).
In terms of per-UE rate performance, even though all of the three algorithms
show relatively similar performance levels, however, with a closer look, one
can find out that the PG algorithm provides the highest per-UE transmission
rate.

Fig. 5(b) evaluates the effect of training DRL agents during the real-time
operation of the cell-free network. Specifically, we assume that a training
step is performed during one time slot. This means that, state transitions of
the DRL model will result from both current action $\bm{a}_{c}$ and the
instantaneous CSI $\bm{H}$. It can be observed from Fig. 5(b) that changing
$\bm{H}$ during training of the clustering agents has a negative impact on
both the convergence rate and the per-UE rate performance. This can be
observed clearly by the significant increase in the variance of the Q-values
in the upper graph of Fig. 5(b). It can also be observed that the convergence
of the SARSA-based clustering is the worst. For SARSA, to tackle the weak
stability issue, we double the number of training episodes from 2000 to 4000
episodes and retrain the SARSA agent under varying CSI conditions (Fig. 6(a)).

Figure 6: Performance of different clustering agents with different number of
training episodes.

As can be observed from this figure, increasing the number of episodes
improves the stability of the SARSA algorithm significantly. However, with
more training episodes the per-UE rate performance does not improve (with
average reward of around $1.5408$ bps/Hz). More numerical results on the
performances of the DRL-based clustering schemes are given in Table VII.

TABLE VII: Numerical results on the performances of different clustering
schemes. $(M,K)=(5,3)$ and $(a,u)=(1,1)$ with optimal performance: $2.03786$
pbs/Hz
---
Agent |  Average reward |  Inference mode Fixed $\bm{H}$ |  Inference mode Variable $\bm{H}$ |  Training duration (2000 episodes)
PG trained by fixed $\bm{H}$ |  $1.7607$ bps/Hz |  $1.5626$ bps/Hz |  $1.4841$ bps/Hz |  $15.4072$ Mins
PG trained by varying $\bm{H}$ |  $1.6090$ bps/Hz |  $1.5592$ bps/Hz |  $1.5332$ bps/Hz |  $15.0098$ Mins
DDQN trained by fixed $\bm{H}$ |  $1.7308$ bps/Hz |  $1.5802$ bps/Hz |  $1.5355$ bps/Hz |  $16.5467$ Mins
DDQN trained by varying $\bm{H}$ |  $1.5579$ bps/Hz |  $1.5321$ bps/Hz |  $1.4654$ bps/Hz |  $14.0558$ Mins
SARSA trained by fixed $\bm{H}$ |  $1.4733$ bps/Hz |  $1.4376$ bps/Hz |  $1.5153$ bps/Hz |  $18.2317$ Mins
SARSA trained by varying $\bm{H}$ |  $1.5862$ bps/Hz |  $1.5047$ bps/Hz |  $1.5072$ bps/Hz |  $16.3313$ Mins.
AC trained by fixed $\bm{H}$ |  $1.4186$ bps/Hz |  $1.5876$ bps/Hz |  $1.4991$ bps/Hz |  $15.3618$ Mins
AC trained by varying $\bm{H}$ |  $1.6084$ bps/Hz |  $1.5664$ bps/Hz |  $1.4919$ bps/Hz |  $15.5362$ Mins

It can be observed that the off-policy algorithm (i.e. DDQN) gives the worst
performance under varying CSI conditions. This is due to the fact that the
DDQN agent selects the action related to the highest Q-value in a
deterministic fashion, without any exploration. This action-selection strategy
will prevent the network from learning/sensing the stochastic variations of
the states (or alternatively the CSI matrix $\bm{H}$). On the other hand, all
of the on-policy based algorithms (PG, SARSA, and AC) show good performance
under varying CSI, and we observe a noticeable enhancement on per-UE rate
performance due to the AC algorithm. The reason is that the AC algorithm
allows the DRL agent to learn the stochastic properties of the state (or
alternatively, the CSI matrix $\bm{H}$). Finally, Fig. 6(b) evaluates and
compares the performance of several DRL-based beamsteering methods. For these
simulations, we use the inference mode of the DDQN algorithm to solve the
network partitioning problem. It can be noticed that the AC algorithm shows
the best per-UE rate performance compared to the DDPG and PG algorithms.
Furthermore, when both network clustering and analog beamsteering are
implemented through the DRL agents, the per-UE rate performance of the SAC
algorithm drops to around $70\%$ of the optimal performance, and for the PG
algorithm, it is $51\%$ of the optimal performance.

##  VII Conclusion

A novel self-partitioning MIMO cell-free network architecture has been
proposed in which a cell-free network is partitioned into a set of independent
cell-free subnetworks. To efficiently solve the problem of network
partitioning for a large-scale network, we have proposed, evaluated, and
benchmarked several state-of-the-art DRL methods with discrete action space.
Furthermore, to reduce the interference between adjacent cell-free
subnetworks, we have designed a novel downlink hybrid analog beamsteering-
digital beamforming scheme. We have also evaluated several state-of-the-art
DRL methods with continuous action space to solve the combinatorial problem of
analog beamsteering while the digital beamforming problem has been solved as a
strictly convex optimization problem. Results have showed a significant rate
enhancement and complexity reduction due to the proposed hybrid beamforming
scheme compared to its conventional all-digital counterpart. It has been
observed that online training of different DRL agents is only slightly
affected by changing the CSI in the network. However, changing the CSI can
significantly affect the variance and convergence rate of the DRL algorithms
such as the SARSA algorithm. Furthermore, it has also been noticed that all
DRL methods for network clustering and beamsteering have almost the same per-
UE rate performance with a slight superiority of the PG and AC algorithms when
used for network clustering and analog beamsteering, respectively. A potential
extension of this work is to enable distributed beamforming at each cell-free
subnetwork. This may be achieved by utilizing a multiple agent algorithm with
continuous action space. Another extension of this work is to solve the
problem of pilot assignment using distributed multiple agent DRL modeling.

## References

  * [1] H. Q. Ngo, A. Ashikhmin, H. Yang, E. G. Larsson, and T. L. Marzetta, “Cell-free massive MIMO versus small cells,” _IEEE Trans. on Wireless Commun._ , vol. 16, no. 3, pp. 1834–1850, March 2017.
  * [2] Y. Al-Eryani, M. Akrout, and E. Hossain, “Multiple access in cell-free networks: Outage performance, dynamic clustering, and deep reinforcement learning-based design,” _IEEE J. Select. Areas Commun._ , pp. 1–1, 2020\.
  * [3] Y. Jin, J. Zhang, S. Jin, and B. Ai, “Channel estimation for cell-free mmWave massive MIMO through deep learning,” _IEEE Trans. Veh. Technol._ , vol. 68, no. 10, pp. 10 325–10 329, 2019.
  * [4] G. Interdonato, H. Q. Ngo, P. Frenger, and E. G. Larsson, “Downlink training in cell-free massive MIMO: A blessing in disguise,” _IEEE Trans. on Wireless Commun._ , vol. 18, no. 11, pp. 5153–5169, 2019.
  * [5] E. Nayebi, A. Ashikhmin, T. L. Marzetta, H. Yang, and B. D. Rao, “Precoding and power optimization in cell-free massive MIMO systems,” _IEEE Trans. on Wireless Commun._ , vol. 16, no. 7, pp. 4445–4459, 2017\.
  * [6] M. Alonzo, S. Buzzi, A. Zappone, and C. D’Elia, “Energy-efficient power control in cell-free and user-centric massive mimo at millimeter wave,” _IEEE Trans. on Green Commun. and Networking_ , vol. 3, no. 3, pp. 651–663, 2019.
  * [7] M. Attarifar and et. al., “Modified conjugate beamforming for cell-free massive MIMO,” vol. 8, no. 2, pp. 616–619, April 2019.
  * [8] Y. Al-Eryani, E. Hossain, and D. I. Kim, “Generalized coordinated multipoint (GCoMP)-enabled NOMA: Outage, capacity, and power allocation,” _IEEE Trans. on Commun._ , vol. 67, no. 11, pp. 7923–7936, Nov. 2019.
  * [9] E. Björnson and L. Sanguinetti, “Scalable cell-free massive MIMO systems,” _IEEE Trans. on Commun._ , vol. 68, no. 7, pp. 4247–4261, 2020\.
  * [10] X. Huang, X. Zhu, Y. Jiang, and Y. Liu, “Efficient enhanced k-means clustering for semi-blind channel estimation of cell-free massive MIMO,” in _IEEE Inter’l Conf. on Commun. (ICC’20)_ , 2020, pp. 1–6.
  * [11] F. Fredj, Y. Al-Eryani, S. Maghsudi, M. Akrout, and E. Hossain, “Distributed uplink beamforming in cell-free networks using deep reinforcement learning,” 2020\.
  * [12] S. Chakraborty, E. Björnson, and L. Sanguinetti, “Centralized and distributed power allocation for max-min fairness in cell-free massive MIMO,” in _2019 53’rd Asilomar Conf. on Signals, Systems, and Computers_ , 2019, pp. 576–580.
  * [13] J. García-Morales, G. Femenias, and F. Riera-Palou, “Energy-efficient access-point sleep-mode techniques for cell-free mmWave massive MIMO networks with non-uniform spatial traffic density,” vol. 8, 2020, pp. 137 587–137 605.
  * [14] X. Zhang, J. Wang, and H. V. Poor, “Statistical delay and error-rate bounded QoS provisioning over mmWave cell-free mMIMO and FBC-HARQ-IR based 6G wireless networks,” _IEEE J. on Select. Areas in Commun._ , vol. 38, no. 8, pp. 1661–1677, 2020.
  * [15] R. W. Heath, N. González-Prelcic, S. Rangan, W. Roh, and A. M. Sayeed, “An overview of signal processing techniques for millimeter wave MIMO systems,” _IEEE J. of Select. Topics in Sig. Process._ , vol. 10, no. 3, pp. 436–453, 2016.
  * [16] S. Zhou, M. Zhao, X. Xu, J. Wang, and Y. Yao, “Distributed wireless communication system: A new architecture for future public wireless access,” _IEEE Mag. Commun._ , vol. 41, no. 3, pp. 108–113, March 2003.
  * [17] T. Gong, N. Shlezinger, S. S. Ioushua, M. Namer, Z. Yang, and Y. C. Eldar, “Rf chain reduction for MIMO systems: A hardware prototype,” _IEEE Systems Journal_ , pp. 1–12, 2020.
  * [18] A. Alkhateeb, G. Leus, and R. W. Heath, “Limited feedback hybrid precoding for multi-user millimeter wave systems,” vol. 14, no. 11, pp. 6481–6494, 2015.
  * [19] O. E. Ayach, S. Rajagopal, S. Abu-Surra, Z. Pi, and R. W. Heath, “Spatially sparse precoding in millimeter wave MIMO systems,” _IEEE Trans. on Wireless Commun._ , vol. 13, no. 3, pp. 1499–1513, 2014\.
  * [20] V. Raghavan and A. M. Sayeed, “Sublinear capacity scaling laws for sparse MIMO channels,” _IEEE Trans. on Inf. Theory_ , vol. 57, no. 1, pp. 345–364, 2011.
  * [21] A. Forenza, D. J. Love, and R. W. Heath, “Simplified spatial correlation models for clustered MIMO channels with different array configurations,” _IEEE Trans. Veh. Technol._ , vol. 56, no. 4, pp. 1924–1934, 2007.
  * [22] W. Tan, S. D. Assimonis, M. Matthaiou, Y. Han, X. Li, and S. Jin, “Analysis of different planar antenna arrays for mmwave massive MIMO systems,” in _IEEE Veh. Technol. Conf. (VTC’85 Spring)_ , 2017, pp. 1–5.
  * [23] B. Ai, K. Guan, G. Li, and S. Mumtaz, “Chapter 8–mmwave massive MIMO channel modeling,” in _mmWave Massive MIMO_ , S. Mumtaz, J. Rodriguez, and L. Dai, Eds. Academic Press, 2017, pp. 169–194.
  * [24] S. Boyd and L. Vandenberghe, _Convex Optimization_. Cambridge University Press, 2004.
  * [25] C. A. Balanis, _Antenna theory: Analysis and Design_ , 4th ed. Hoboken, NJ: John Wiley, 2016.
  * [26] M. S. Ali, H. Tabassum, and E. Hossain, “Dynamic user clustering and power allocation for uplink and downlink non-orthogonal multiple access (NOMA) systems,” _IEEE Access_ , vol. 4, pp. 6325–6343, 2016.
  * [27] K. I. Ahmed and E. Hossain, “A deep Q-learning method for downlink power allocation in multi-cell networks,” 2019.
  * [28] R. Sutton and A. Barto, _Reinforcement Learning: An Introduction_. The MIT Press Cambridge, Massachusetts London, England., 2017.
  * [29] O. P. Ronald L. Graham, Donald E. Knuth, _Concrete Mathematics_. Addison-Wesley, Reading MA, 1988.
  * [30] V. Mnih and et. al., “Playing atari with deep reinforcement learning,” 2013.
  * [31] G. Rummery, M. Niranjan, and U. of Cambridge. Engineering Department, _On-line Q-learning Using Connectionist Systems_ , ser. CUED/F-INFENG/TR. University of Cambridge, Department of Engineering, 1994.
  * [32] R. Williams, “Simple statistical gradient-following algorithms for reinforcement learning,” in _Machine Learning_ , 1992, pp. 229–256.
  * [33] V. Mnih and et. al., “Asynchronous methods for deep reinforcement learning,” 2016\.
  * [34] R. Bellman, “On the theory of dynamic programming,” _Proc. of the National Academy of Sciences_ , vol. 38, no. 8, pp. 716–719, 1952.
  * [35] P. Christodoulou, “Soft actor-critic for discrete action settings,” 2019.
  * [36] C. G. Broyden, “A class of methods for solving nonlinear simultaneous equations,” 1965.
  * [37] T. Lillicrap and et al., “Continuous control with deep reinforcement learning,” _arXiv preprint arXiv:1509.02971_ , 2015.
  * [38] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor,” _arXiv e-prints_ , p. arXiv:1801.01290, Jan. 2018.
  * [39] B. Ziebart, “Modeling purposeful adaptive behavior with the principle of maximum causal entropy,” Ph.D. dissertation, USA, 2010.
  * [40] S. Kullback and R. A. Leibler, “On information and sufficiency,” _Ann. Math. Statist._ , vol. 22, no. 1, pp. 79–86, 03 1951.