# Smart Helper-Aided F-RANs:
Improving Delay and Reducing Fronthaul Load

Hesameddin Mokhtarzadeh, Mohammed S. Al-Abiad,
Md Jahangir Hossain, Julian Cheng,
Hesameddin Mokhtarzadeh, M. J. Hossain, and Julian Cheng are with the School
of Engineering, The University of British Columbia, Kelowna, BC V1V 1V7,
Canada (e-mails<EMAIL_ADDRESS><EMAIL_ADDRESS>julian.cheng@ubc.ca). Mohammed S. Al-Abiad is with the Department of
Electrical and Computer Engineering, University of Toronto, Toronto, ON M5S,
Canada (e-mail: mohammed.saif@utoronto.ca).

###### Abstract

In traditional Fog-Radio Access Networks (F-RANs), enhanced remote radio heads
(eRRHs) are connected to a macro base station (MBS) through fronthaul links.
Deploying a massive number of eRRHs is not always feasible due to site
constraints and the cost of fronthaul links. This paper introduces an
innovative concept of using smart helpers (SHs) in F-RANs. These SHs do not
require fronthaul links and listen to the nearby eRRHs’ communications. Then,
they smartly select and cache popular content. This capability enables SHs to
serve users with frequent on-demand service requests potentially. As such,
network operators have the flexibility to easily deploy SHs in various
scenarios, such as dense urban areas and temporary public events, to expand
their F-RANs and improve the quality of service (QoS). To study the
performance of the proposed SH-aided F-RAN, we formulate an optimization
problem of minimizing the average transmission delay that jointly optimizes
cache resources and user scheduling. To tackle the formulated problem, we
develop an innovative multi-stage algorithm that uses a reinforcement learning
(RL) framework. Various performance measures, e.g., the average transmission
delay, fronthaul load, and cache hit rate of the proposed SH-aided F-RAN are
evaluated numerically and compared with those of traditional F-RANs.

###### Index Terms:

F-RANs, low latency communications, reinforcement learning, resource
allocation, smart helpers.

##  I Introduction

In the digital era, there is a proliferation of connected devices and services
that are to be supported by wireless networks [1, 2, 3]. As a result, there is
a pressing need for wireless networks that can provide superior performances,
such as ultra-reliable low latency communications (URRLC). To address this
need, network densification by expanding the network access layer with more
remote radio heads (RRHs) is shown to be a promising solution. Cloud-radio
access networks (C-RANs) were proposed by China Mobile [4] as a potential
network architecture that supports network densification. In C-RANs, the
baseband units (BBUs) of the RRHs are centralized within the macro base
station (MBS), also called the central unit (CU). The RRHs are then connected
to the MBS via the fronthaul links [5].

The prime concern of centralized load within the CU is that it leads to
notable transmission delay and heavy traffic on fronthaul links. More
recently, Fog-RANs (F-RANs) have been introduced to address these issues [6].
In F-RANs, the conventional RRHs are replaced with enhanced RRHs (eRRHs) with
more sophisticated signal processing capabilities. More importantly, eRRHs
have caching capabilities. F-RANs offer two key benefits as mentioned as
follows. First, network functionalities can be divided between eRRHs and the
cloud based on the specific latency and reliability requirements. Furthermore,
it enhances the efficiency and speed of content delivery as the eRRHs can
serve the cached content to the users without frequently requesting these
contents from the MBS [7, 8]. Consequently, caching capability helps to reduce
the load on the fronthaul link by reducing the amount of data that needs to be
retrieved from the MBS [9, 10]. Network densification, by incorporating more
eRRHs, offers improved file caching and delivery capabilities. Nevertheless,
this enhancement comes at the expense of significantly increased hardware and
energy consumption. In addition, deploying a large number of eRRHs can be
challenging in densely populated urban areas, as it may be constrained by site
limitations and the scarcity of available space. Even if a massive deployment
of eRRHs is possible, there remains a significant challenge in managing the
additional fronthaul link setup. As an attempt to further reduce the traffic
load on fronthaul links that the centralized-based F-RAN architectures may
face, cache-enabled device-to-device (CE-D2D) communication systems have been
proposed [11]. In CE-D2D communication systems, devices can cache popular
content and assist in offloading eRRHs by transferring data between each other
[12]. In particular, eRRHs and CE-D2D users transmit their cached content to
interested users using cellular and D2D links, respectively. CE-D2D
communications did not gain popularity for various reasons, including privacy
and power consumption concerns of devices.

Optimized cache resources play a crucial role in efficiently serving users
directly at the network edge [13, 14, 15]. One trivial strategy is to predict
the popularity of content requested by users, which helps in making decision
in terms of placing various contents within the network [14]. However, despite
significant advancements in the prediction of content popularity, the dynamic
nature of wireless networks, user mobility, and data access behavior
necessitate a joint optimization of edge caching and wireless resource
allocation [16, 17, 18, 19]. Achieving such joint optimization often requires
exhaustive search methods due to the interdependence between caching decisions
and resource allocation, which are not feasible for implementations [20, 21].
To overcome the limitations of traditional optimization-based resource
allocation algorithms, distributed machine learning (ML) algorithms have been
proposed as potential solutions [22]. For example, reinforcement learning (RL)
frameworks have been extensively explored to optimize caching decisions in an
online manner, without prior knowledge of resource allocation decisions. These
RL frameworks learn the access behavior of users and the mobility of the
network, enabling the joint optimization of caching decisions and resource
allocation within a distributed F-RAN architecture.

###  I-A Related Works

A flourish of works has been done towards maximizing the overall throughput on
F-RANs, thereby improving the end-to-end latency. Existing works mainly
focused on either optimizing radio resources, such as user scheduling and
channel coding [23, 24, 25, 26, 27, 28, 29], or integration of caching with
radio resource optimization strategies [30, 31, 32, 33, 34, 35]. In what
follows, we review the related works on radio resource optimization and joint
caching and resource optimization.

####  I-A1 Radio Resource Optimization

In [23, 24], the authors proposed optimization strategies such as channel
precoding, fronthaul compression, and superposition coding to reduce
transmission latency and enhance the overall quality of service (QoS) in
F-RANs. While these works have improved the performance of the network, they
did not fully leverage the capabilities offered by the architecture of F-RANs.
In addition, some research works have explored the benefits of cooperation
between eRRHs in F-RANs. For example, the work in [25] proposed a distributed
computing and content-sharing mechanism to enhance the file delivery process.
Although their collaborative system improves the QoS, it can also result in
higher traffic on the links connecting these eRRHs together. To address this
issue, the works in [26, 27] explored the capabilities of D2D communication
modes. They demonstrated that incorporating D2D communication in F-RANs has
the potential to ease the transmission burden on the eRRHs and significantly
decrease the bandwidth usage of both RANs and fronthaul links. However, D2D
communication may pose unexamined challenges, such as security risks and
increased power consumption for the users. In addition to enhancing the
transmission mechanism, several research studies have focused on investigating
the placement of eRRHs and addressing the challenges associated with the
dynamic nature of mobile eRRH network topology [28, 29]. In [28], a study
focused on maximizing throughput by dynamically determining optimal fog node
locations using a clustering algorithm. Furthermore, an adaptive radio
resource balancing scheme was proposed in [29] to enhance the QoS in a system
with mobile eRRHs. However, their study did not address the challenges
associated with deploying high-capacity fronthaul links between mobile eRRHs
and the CU.

####  I-A2 Joint Cache and Resource Optimization

In [30], the authors performed an information-theoretic analysis to assess
delivery latency relative to system interference. Their study shed light on
the potential benefits of jointly optimizing caching and resource allocation
on data delivery efficiency within this network architecture. Nevertheless,
the interdependence between caching decisions and resource allocation makes
such joint optimization a complex task that can only be optimally solved via
exhaustive search solutions, which is impractical. In this regard, researchers
have recently taken advantage of RL algorithms to tackle the challenges of
such problems in F-RANs. For example, the authors of [33] presented an RL
framework on double deep Q-network for caching optimization and power
allocation incorporating personalized user request modelling. Similarly, in
[31], a deep RL-based algorithm was developed for joint dynamic cache resource
optimization and power allocation to minimize the total transmission cost.
Additionally, in [32], an optimization problem was presented to minimize the
weighted system cost for the uplink input data transmission, specifically to
access the cached requested services of the fog access points. The authors
adopted a two-timescale approach, combining a game-based resource allocation
on a small timescale with a multi-agent RL (MARL) algorithm for caching
decisions on a larger timescale. However, it should be noted that the problem
of optimal user assignment was explicitly not addressed in these research
works. In [34], a joint RL algorithm and game-theoretic approach for cache
optimization and user assignment was proposed to maximize the sum data rate of
the network, considering fixed transmission power for each fog node. In [35],
a more comprehensive system model was introduced, incorporating D2D and eRRH
communication modes. They proposed utilizing MARL and cross-layer network
coding techniques to maximize the overall system sum rate.

###  I-B Motivation and Contributions

In all previous works, a fronthaul link between the eRRHs and the MBS was
assumed. However, in emerging practical applications for dense networks,
connecting the eRRHs to the MBS using fronthaul links may not be feasible due
to hardware limitations and cost constraints. For example, regarding optical
fibre eRRH-MBS links, installing multiple fibres may be financially unbearable
for many practical applications or even hard due to harsh geographical
terrains. On the other hand, wireless eRRHs-MBS links may cause error
propagation losses at far distances since buildings or trees often act as link
blockers. In view of this, we introduce the concept of using cost-effective
elements, referred to as smart helpers (SHs), to extend the traditional F-RANs
with no fronthaul connections between the SHs with the MBS. The proposed
system is referred to SH-aided F-RAN system. In the SH-aided F-RAN, we
consider that a few number of eRRHs are deployed and directly connected to the
MBS using fronthaul links, while the SHs are not using fronthaul links to the
MBS. Specifically, the SHs can efficiently listen to the communications
between the eRRHs and the users in their vicinity without the need for
fronthaul links. In conventional F-RANs, where only the eRRHs are deployed,
the users’ requests can be delivered from the eRRHs in a collaborative way.
However, this is not enough in the proposed system since the SHs can cache
some popular users’ requested services. Hence, developing caching strategies
for both eRRHs and SHs and prioritizing the association of users with them for
an effective file delivery becomes quite challenging. To this end, our paper
aims to fill this gap by introducing the concept of economical SHs while
tackling the joint optimization of caching strategy at eRRHs and SHs and user-
eRRH/SH priority association. In more detail, the contribution of the paper is
summarized as follows:

  * •

We present the proposed SH-aided F-RAN system, where the SHs do not need
fronthaul link connections to the MBS but can listen to the communications
between nearby eRRHs and users to smartly cache popular content. Then, the SHs
use their cached contents to potentially serve users with frequent on-demand
service requests.

  * •

We develop a priority-based user assignment algorithm to improve the cache hit
rate of our SH-aided F-RAN. We then develop a novel MARL algorithm, where
popular file segments are treated as virtual agents that make binary caching
decisions. The reward function for these virtual agents is designed to
minimize the average transmission delay taking into account the limited
caching capacity of eRRHs and SHs. We also thoroughly analyze the convergence
rate and computational complexity of the proposed algorithm.

  * •

Numerical results of the proposed framework are presented in various scenarios
and compared to several benchmark schemes of conventional F-RANs. Presented
numerical results show that SHs can significantly alleviate the load on the
fronthaul link and considerably reduce the average transmission delay.
Interestingly, our findings also reveal that by tolerating a negligible amount
of transmission delay, each eRRH can be replaced with a number of compact SHs.

The rest of the paper is organized as follows. In Section II, we introduce the
concept of SH-aided F-RAN. Section III formulates the problem of minimizing
average content delivery delay within our proposed SH-aided F-RAN framework.
In Section IV, we offer the solution by presenting algorithms for resource
allocation and cache optimization. Section V presents simulation results to
evaluate the performance of the proposed SH-aided FRAN and the designed
algorithm. Finally, Section VI concludes the paper.

##  II Proposed SH-Aided F-RAN

Figure 1: SH-aided F-RAN architecture.

###  II-A Overall Description

Our proposed SH-aided F-RAN is shown in Fig. 1, where there is a number of $H$
SHs with $\mathcal{H}=\\{1,2,\cdots,H\\}$ denoting the set of all SHs, $S$
eRRHs with $\mathcal{S}=\\{1,2,\cdots,S\\}$ denoting the set of all eRRHs, $U$
users in total with $\mathcal{U}=\\{1,\dots,U\\}$ denoting the set of all
users, and an MBS. We assume that all nodes have a single antenna for
simplicity. Each SH/eRRH has a certain subset of users under their service
area. We represent these subsets by
$\mathcal{U}^{\mathcal{H}}_{h}\subseteq\mathcal{U}$ for the users under the
service area of the $h$th SH and
$\mathcal{U}^{\mathcal{S}}_{s}\subseteq\mathcal{U}$ for those under the
service area of the $s$th eRRH. eRRHs are connected to the MBS using fronthaul
links with a limited capacity of $C_{fh}$. They serve users by utilizing their
cache or obtaining requested content of users from the MBS as needed. SHs are
relatively cheaper transceivers that have caching capability with a certain
amount of caching capacity. We assume that there is no fronthaul link between
SHs and the MBS. They listen to the wireless communications between the eRRHs
and users happening in their vicinity. Then, they smartly select and cache
relevant data and efficiently serve users’ requests based on the cached
information. It is assumed that eRRHs and SHs refine their cache locally to
store more on-demand content and serve users effectively. Furthermore, we
assume that SHs are placed randomly, and SHs are allowed to decode and cache
content. Optimal placement of SHs can lead to improved performance; however,
their location optimization is out of the scope of this paper. Users are
associated with various eRRHs/SHs based on the content availability on the
cache of them by the MBS. The MBS communicates user scheduling decisions to
eRRHs via the fronthaul links. Also, a low-rate wireless control channel is
used to transmit the user scheduling decisions from the MBS to the SHs. We
consider that the eRRHs and SHs use $R$ orthogonal time/frequency radio
resource blocks (RRBs) to serve users. The set of all RRBs is denoted by
$\mathcal{R}=\\{1,2,\dots,R\\}$. In particular, we consider the F-RAN’s
resource settings in [5, 13].

The content delivery process of SH-aided FRAN is shown in Fig. 2. Whenever a
user requests some content, the MBS associates the user with an SH caching the
requested content. In cases where none of the nearby SHs have the requested
content in their cache, the system turns to nearby eRRHs to fulfill the user’s
request. Since the eRRHs are also equipped with caching capabilities, they can
directly serve the requested content from their cache to the user without
requiring back-and-forth communication with the MBS. However, if none of the
nearby SHs or eRRHs have the requested content in their caches, the user will
be associated with one of the nearby eRRHs so the eRRH can obtain the content
from the MBS and serve the user. Users are served with different power levels
based on the availability of power in SHs and eRRHs, ensuring an efficient
content delivery experience.

Figure 2: Content delivery process.

###  II-B Advantages and Application Scenarios

The use of SHs in F-RANs offers numerous advantages for network operators. SHs
are relatively inexpensive compared to eRRHs, and they can be readily deployed
without fronthaul links and significant planning effort. As such, it offers a
scalable and flexible solution to improve the performance of F-RANs. SHs
effectively address concerns related to the privacy and power consumption
issues associated with CE-D2D communications. For example, SHs can be deployed
in dense and crowded urban areas. They can reduce network congestion by
providing localized caching and enhancing user experience. Also, for public
events and other events where people gather for a certain duration, SHs can be
deployed easily to serve users from the cached content. In such scenarios,
installation of additional eRRHs is time-consuming or may not be feasible.
Moreover, in rural and remote areas with limited infrastructure, SHs can store
frequently accessed content locally, reducing the load on the fronthaul link
and improving network performance. In disaster and emergency response
situations, SHs can cache critical information and operate as temporary
network nodes, ensuring uninterrupted communication.

###  II-C Operating Constraints

Let $\mathbf{A}^{\mathcal{S},(t)}$ and $\mathbf{A}^{\mathcal{H},(t)}$ be
respectively three-dimensional matrices representing the assignment of RRBs
and users to eRRHs and SHs during time frame $t$. The matrix
$\mathbf{A}^{\mathcal{S},(t)}$ has dimensions of $U\times R\times S$, where
$[\mathbf{A}^{\mathcal{S},(t)}]_{u,r,s}=a^{\mathcal{S},(t)}_{u,r,s}$ indicates
if the $u$th user is assigned to the $r$th RRB and associated with the $s$th
eRRH. Specifically, $a^{\mathcal{S},(t)}_{u,r,s}=1$, if the $r$th RRB and the
$s$th eRRH are assigned to the $u$th user, and $a^{\mathcal{S},(t)}_{u,r,s}=0$
otherwise. Similarly, the matrix $\mathbf{A}^{\mathcal{H},(t)}$ has a
dimension of $U\times R\times H$, where
$[\mathbf{A}^{\mathcal{H},(t)}]_{u,r,h}=a^{\mathcal{H},(t)}_{u,r,h}$
represents whether the $u$th user is assigned to the $r$th RRB and the $h$th
SH during time frame $t$, with $a^{\mathcal{H},(t)}_{u,r,h}=1$ if assigned and
$a^{\mathcal{H},(t)}_{u,r,h}=0$ otherwise. To avoid interference between
users, each RRB is assigned to only one user. Also, due to the radio resource
limitation and to ensure fairness among the users, each user is assigned to
only one RRB. Hence, the following constraints should be satisfied

$\displaystyle\sum_{r}(\sum_{s}a^{\mathcal{S},(t)}_{u,r,s}+\sum_{h}a^{\mathcal{H},(t)}_{u,r,h})\leq
1,$ (1) $\displaystyle\
\sum_{u}(\sum_{s}a^{\mathcal{S},(t)}_{u,r,s}+\sum_{h}a^{\mathcal{H},(t)}_{u,r,h})\leq
1.$ (2)

Since the number of users is generally higher than the number of available
RRBs, we consider serving a certain subset of users during each time frame.
Particularly, given a total of $R$ available RRBs in the network, a maximum of
$R$ users can be served from the eRRHs and the SHs in a given time frame.

Let $\mathbf{P}^{\mathcal{S},(t)}$ and $\mathbf{P}^{\mathcal{H},(t)}$ be the
power allocation matrices of the eRRHs and SHs during time frame $t$.
Specifically, $[\mathbf{P}^{\mathcal{S},(t)}]_{s,r}=p^{\mathcal{S},(t)}_{s,r}$
and $[\mathbf{P}^{\mathcal{H},(t)}]_{h,r}=p^{\mathcal{H},(t)}_{h,r}$ are the
allocated powers to the $r$th RRB by the $s$th eRRH and the $h$th SH,
respectively. eRRHs can transmit with the maximum allowed power on each RRB,
denoted by $\bar{P}_{\mathcal{S}}$. However, SHs are required to properly
allocate their available transmit power, limited by $\bar{P}_{\mathcal{H}}$,
to the associated users. Hence, it is essential to consider the following
constraints for power allocation

$\displaystyle\sum_{r}p^{\mathcal{H},(t)}_{h,r}\leq\bar{P}_{\mathcal{H}}$
$\displaystyle\forall h=\\{1,2,\dots,H\\},$ (3) $\displaystyle
p^{\mathcal{S},(t)}_{s,r}=\\{0,\bar{P}_{\mathcal{S}}\\}$ $\displaystyle\forall
s=\\{1,2,\dots,S\\}.$ (4)

We consider that the requested content by the users is divided into popular
and unpopular file segments. We indicate $a^{\text{req}}_{u}$ as the request
type parameter of the $u$th user, where $a^{\text{req}}_{u}=1$ if the user
requests a popular file segment, and $a^{\text{req}}_{u}=0$ otherwise. Denote
the set of all popular $F$ file segments by $\mathcal{F}=\\{1,2,\cdots,F\\}$,
and without loss of generality, assume that each file segment has a size of
$B$ bytes. We assume that the popularity of file segments follows well-known
Zipf distribution [35, 32]. According to the Zipf distribution, the popularity
of file segment $f$, denoted by $z_{f}$, is given by
$z_{f}=\frac{1}{f^{\gamma}}/\sum_{i=1}^{F}\frac{1}{i^{\gamma}}$, where
$\gamma$ is the Zipf parameter and governs the skewness of popularity
distribution, and $\mathcal{Z_{\mathcal{F}}}=\\{z_{1},z_{2},\dots,z_{F}\\}$
represents the popularity distribution of the file segments. We also assume
that $\mathcal{F}$ and the popularity of file segments do not change during a
large number of time frames. Let $\mathbf{C}^{\mathcal{S},(t)}$ and
$\mathbf{C}^{\mathcal{H},(t)}$ be two matrices respectively representing the
caching status of popular file segments in the cache of the eRRHs and SHs
during time frame $t$. The dimension of $\mathbf{C}^{\mathcal{S},(t)}$ is
$S\times F$, where
$[\mathbf{C}^{\mathcal{S},(t)}]_{s,f}=c^{\mathcal{S},(t)}_{s,f}$ indicates
whether the $f$th popular file segment is stored in the cache of the $s$th
eRRH. Specifically, $c^{\mathcal{S},(t)}_{s,f}=1$ if the segment is cached and
$c^{\mathcal{S},(t)}_{s,f}=0$ otherwise. Similarly, the matrix
$\mathbf{C}^{\mathcal{H},(t)}$ has a dimension of $H\times F$, where
$[\mathbf{C}^{\mathcal{H},(t)}]_{h,f}=c^{\mathcal{H},(t)}_{h,f}$ denotes
whether the $f$th file segment is cached in the $h$th SH, with
$c^{\mathcal{H},(t)}_{h,f}=1$ if cached and $c^{\mathcal{H},(t)}_{h,f}=0$
otherwise. We assume eRRHs and SHs can cache up to $K_{\text{eRRH}}$ and
$K_{\text{SH}}$ file segments, respectively that lead to following constraints

$\displaystyle\sum_{f=1}^{F}c^{\mathcal{H},(t)}_{h,f}\leq K_{\text{SH}}$
$\displaystyle\forall h=\\{1,2,\dots,H\\},$ (5)
$\displaystyle\sum_{f=1}^{F}c^{\mathcal{S},(t)}_{s,f}\leq K_{\text{eRRH}}$
$\displaystyle\forall s=\\{1,2,\dots,S\\}.$ (6)

We use the matrices $\mathbf{G}^{\mathcal{S},(t)}$ and
$\mathbf{G}^{\mathcal{H},(t)}$ to denote the channel gains at time frame $t$
from users to eRRHs and SHs, respectively. Specifically, we define
$[\mathbf{G}^{\mathcal{S},(t)}]_{u,r,s}=g^{\mathcal{S},(t)}_{u,r,s}$ as the
channel gain from the $s$th eRRH to the $u$th user over the $r$th RRB, and
$[\mathbf{G}^{\mathcal{H},(t)}]_{u,r,h}=g^{\mathcal{H},(t)}_{u,r,h}$ as the
channel gain from the $h$th SH to the same user over the same RRB. We assume
that the channel remains constant while transmitting a single uncoded or coded
file segment but changes between successive transmissions. Thus, the
achievable rate for serving the $u$th user is determined by

$\displaystyle R_{u}=\sum_{r}W\log(1+\Gamma^{(t)}_{u,r})$ (7)

where $W$ is the bandwidth of each RRB, and $\Gamma^{(t)}_{u,r}$ is the SINR
received by the $u$th user over the $r$th RRB through an additive white
Gaussian noise (AWGN) channel during time frame $t$, which is given by

$\displaystyle\Gamma^{(t)}_{u,r}=\frac{\sum_{h}a^{\mathcal{H},(t)}_{u,r,h}g^{\mathcal{H},(t)}_{u,r,h}p^{\mathcal{H},(t)}_{h,r}+\sum_{s}a^{\mathcal{S},(t)}_{u,r,s}g^{\mathcal{S},(t)}_{u,r,s}p^{\mathcal{S},(t)}_{s,r}}{N_{0}}$
(8)

where $N_{0}$ is the noise variance.

###  II-D Notations

Table I summarizes the common notations used throughout the paper.

TABLE I: Summary of Notations $S$, $H$ |  Numbers of eRRHs and SHs
---|---
$U$, $R$ |  Numbers of users and RRBs
$c^{\mathcal{S},(t)}_{s,f}$, $c^{\mathcal{H},(t)}_{h,f}$ |  Availability of popular file segment $f$ in the $s$th eRRH and the $h$th SH during time frame $t$
$a^{\mathcal{S},(t)}_{u,r,s}$, $a^{\mathcal{H},(t)}_{u,r,h}$ |  Assignment of the $r$th RRB and the $u$th user to the $s$th eRRH and the $h$th SH during time frame $t$
$p^{\mathcal{S},(t)}_{s,r}$, $p^{\mathcal{H},(t)}_{h,r}$ |  The allocated power of the $s$th eRRH and the $h$th SH to the $r$th RRB during time frame $t$
$g^{\mathcal{S},(t)}_{u,r,s}$, $g^{\mathcal{H},(t)}_{u,r,h}$ |  Channel gain from the $u$th user to the $s$th eRRH and the $h$th SH on the $r$th RRB during time frame $t$
$a^{(t)}_{f,s}$, $b^{(t)}_{f,h}$ |  Action of agent $f$ in the $s$th eRRH and the $h$th SH at time $t$
$u^{i,(t)}_{f,s}$, $v^{i,(t)}_{f,h}$ |  Estimated utility of the action $i$ for the $f$th file segment in the $s$th eRRH and the $h$th SH at time $t$
$\pi^{i,(t)}_{f,s}$, $\eta^{(t),i}_{f,h}$ |  Preference of the $s$th eRRH and the $h$th SH to take an action $i$ for the $f$th file segment at time $t$
$K_{\text{eRRH}}$, $K_{\text{SH}}$ |  Maximum caching capacity of each eRRH and SH
$\bar{P}_{\mathcal{S}}$ |  Maximum allowed transmission power of eRRH
$\bar{P}_{\mathcal{H}}$ |  Available transmission power of each SH

##  III Delay Minimization in SH-aided FRAN

The transmission delay of serving the $u$th user with a popular file segment
of a size $B$ bytes from an SH, denoted by $D^{\mathcal{H}}_{u}$, or an eRRH,
denoted by $D^{\mathcal{S}}_{u}$, is given by $B/R_{u}$. When the requested
content is unavailable in the SHs or eRRHs, the transmission delay comprises
both the time it takes to fetch the content from the MBS and the time to
transmit it to the user through the connected eRRH. Hence, the transmission
delay for serving the user from the MBS is determined by
$D_{u}^{MBS}=B/R_{u}+B/C_{\text{fh}}$. Accordingly, the average latency for
the $u$th user to download its requested content can be expressed as

$\displaystyle D_{u}=$ $\displaystyle
a^{\text{req}}_{u}[\sum_{s,r}a^{\mathcal{S},(t)}_{u,r,s}c^{\mathcal{S},(t)}_{s,f}D^{\mathcal{S}}_{u}+\sum_{h,r}a^{\mathcal{H},(t)}_{u,r,h}c^{\mathcal{H},(t)}_{h,f}D^{\mathcal{H}}_{u}$
$\displaystyle+\sum_{s,r}a^{\mathcal{S},(t)}_{u,r,s}(1-c^{\mathcal{S},(t)}_{s,f})D^{MBS}_{u}]$
$\displaystyle+(1-a^{\text{req}}_{u})D^{MBS}_{u}.$ (9)

Since the maximum number of served users is equal to the number of the
available RRBs, the average file segments downloading delay of all the users
can be obtained by $D_{\text{ave}}~{}=~{}\frac{1}{R}~{}\sum_{u=1}^{U}D_{u}$.
Now, we formulate an optimization problem that minimizes average content
delivery delay by considering user scheduling and cache optimization subject
to the constraints (1)–(6) as follows.

$\displaystyle\min_{\underset{\mathbf{C}^{\mathcal{H},(t)},\mathbf{A}^{\mathcal{H},(t)},\mathbf{P}^{\mathcal{H},(t)}}{\mathbf{C}^{\mathcal{S},(t)},\mathbf{A}^{\mathcal{S},(t)},\mathbf{P}^{\mathcal{S},(t)}}}\sum_{u=1}^{U}D_{u}$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}\textrm{s.t.}\begin{array}[]{clc}&\textrm{C1:
}\sum_{r}(\sum_{s}a^{\mathcal{S},(t)}_{u,r,s}+\sum_{h}a^{\mathcal{H},(t)}_{u,r,h})\leq
1,{}{}\\\ \vspace{0.1cm}\hfil&\textrm{C2:
}\sum_{u}(\sum_{s}a^{\mathcal{S},(t)}_{u,r,s}+\sum_{h}a^{\mathcal{H},(t)}_{u,r,h})\leq
1,{}{}\\\ \vspace{0.1cm}\hfil&\textrm{C3:
}\sum_{f}c^{\mathcal{H},(t)}_{h,f}\leq K_{\text{SH}},\\\
\vspace{0.1cm}\hfil&\textrm{C4: }\sum_{f}c^{\mathcal{S},(t)}_{s,f}\leq
K_{\text{eRRH}},\\\ \vspace{0.1cm}\hfil&\textrm{C5:
}a^{\mathcal{S},(t)}_{u,r,s},a^{\mathcal{H},(t)}_{u,r,h}\in\\{0,1\\},\\\
&\textrm{C6: }\sum_{r}p^{\mathcal{H},(t)}_{h,r}\leq\bar{P}_{\mathcal{H}},\\\
\vspace{0.1cm}\hfil&\textrm{C7:
}p^{\mathcal{S},(t)}_{s,r}=\\{0,\bar{P}_{\mathcal{S}}\\},\\\ &\textrm{C8:
}c^{\mathcal{S},(t)}_{s,f},c^{\mathcal{H},(t)}_{h,f},\in\\{0,1\\},\end{array}$
(18)

where constraints C1 and C2 ensure a one-to-one mapping between the users and
the RRBs such that each user is assigned to only one RRB, and each RRB is
assigned to only one user. Constraints C3 and C4 limit the caching capacity of
the SHs and the eRRHs to $\Gamma_{SH}$ and $\Gamma_{eRRH}$ file segments,
respectively. Constraint C6 specifies a maximum available transmission power
for the SHs, while constraint C7 sets a limit on the maximum transmission
power on each RRB in the eRRHs. Constraint C5 and C8 consider binary decisions
for variables $a^{\mathcal{S},(t)}_{u,r,s}$, $a^{\mathcal{H},(t)}_{u,r,h}$,
$c^{\mathcal{S},(t)}_{s,f}$, and $c^{\mathcal{H},(t)}_{h,f}$. The optimization
variables are the cache matrices for both SHs and eRRHs, assignment vectors
that indicate which user is assigned to which RRB in each eRRH and SH, and the
power allocation vectors for both eRRHs and SHs.

The derived optimization problem in (III) is a mixed-integer nonlinear program
(MINLP). It is generally difficult to obtain the globally optimal solution of
a MINLP. In fact, due to its inherent computational complexity resulting from
the interdependence of variables $c^{\mathcal{S},(t)}_{s,f}$,
$c^{\mathcal{H},(t)}_{h,f}$, $a^{\mathcal{S},(t)}_{u,r,s}$, and
$a^{\mathcal{H},(t)}_{u,r,h}$, solving this problem is computationally
intractable. To tackle this intractability, we propose to solve (III)
iteratively. Specifically, the optimization problem is first divided into two
subproblems: 1) user assignment and power allocation and 2) cache resource
optimization. Then, each subproblem is separately optimized. In the next
section, each subproblem and the corresponding solution will be discussed in
detail. We also analyze their convergence and computational complexity.

##  IV Proposed solution: Resource Allocation and Cache Optimization

###  IV-A User Assignment and Power Allocation

In this subsection, we effectively schedule users to eRRHs and SHs to minimize
the average transmission delay through the following subproblem

$\displaystyle\min_{\mathbf{A}^{\mathcal{H},(t)},\mathbf{A}^{\mathcal{S},(t)},\mathbf{P}^{\mathcal{H},(t)},\mathbf{P}^{\mathcal{S},(t)}}~{}~{}~{}\sum_{u=1}^{U}D_{u},~{}~{}~{}\textrm{s.t.~{}C1,~{}C2,~{}C5
--C8}.$ (19)

We construct a conflict graph that includes all the possible user assignment
states to consider constraints C1 and C2. Then, we present an algorithm to
associate the users with SHs and eRRHs by finding the graph’s minimum weight
independent set (MWIS). Additionally, this algorithm optimizes the allocation
of transmission power between users. The obtained user association and power
allocation solution minimizes the average transmission delay of the network.

Consider
$\mathcal{G}=(\mathcal{V}^{\mathcal{H}},\mathcal{V}^{\mathcal{S}},\mathcal{E},\mathcal{W})$
as a weighted undirected conflict graph, for which $\mathcal{V}^{\mathcal{H}}$
and $\mathcal{V}^{\mathcal{S}}$ are the sets of vertices of the SHs and eRRHs,
respectively. The graph has edges represented by $\mathcal{E}$, and weights
assigned to the vertices are given by $\mathcal{W}$. Let
$v^{\mathcal{H}}=(h,u,r)$ represent the vertices belonging to
$\mathcal{V}^{\mathcal{H}}$, where each vertex comprises an SH
$h\in\mathcal{H}$, a user $u\in\mathcal{U}^{\mathcal{H}}_{h}$, and an RRB
$r\in\mathcal{R}$. Similarly, $v^{\mathcal{S}}=(u,r,s)$ represents vertices of
$\mathcal{V}^{\mathcal{S}}$, where each vertex includes an eRRH
$s\in\mathcal{S}$, a user $u\in\mathcal{U}^{\mathcal{S}}_{s}$, and an RRB
$r\in\mathcal{R}$. Every two vertices have a connecting conflict edge if they
include the same user or the same RRB. Hence, the vertices of
$\mathcal{V}^{\mathcal{H}}$ and $\mathcal{V}^{\mathcal{S}}$ are also linked to
each other if they contain the same user or RRB. For instance, consider two
sample vertices: $v^{\mathcal{H}}=(u,r,h)$ and
$v^{\mathcal{S}}=(u^{\prime},r^{\prime},s)$. These two vertices are connected
to each other if either $r=r^{\prime}$, $u=u^{\prime}$, or both conditions are
met. The weight of each vertex is the transmission delay of serving the
associated user of that vertex by the corresponding RRB and SH/eRRH. We
prioritize SHs for serving users. For users requesting a specific file segment
not present in SHs, we designate eRRHs to serve to them. In this way, we
reduce the average transmission delay by maximizing the utilization of SHs’
cached resources to serve a larger number of users.

0:
$\mathbf{C}^{\mathcal{S},(t)},\mathbf{C}^{\mathcal{H},(t)},\mathbf{G}^{\mathcal{S},(t)},\mathbf{G}^{\mathcal{H},(t)}$

0:
$\mathbf{A}^{\mathcal{H},(t)},\mathbf{A}^{\mathcal{S},(t)},\mathbf{P}^{\mathcal{H},(t)},\mathbf{P}^{\mathcal{S},(t)}$

1: Graph Formation:

2: for $r=1:R$ do

3: for $h=1:H$ do

4: for $u=1:\mathcal{U}^{\mathcal{H}}_{h}$ do

5: initiate a vertex $v^{\mathcal{H}}=(u,r,h)$

6: $\mathbf{w}(u,h,r)=D_{u}$ using (III)

7: end for

8: end for

9: for $s=1:S$ do

10: for $u=1:\mathcal{U}^{\mathcal{S}}_{s}$ do

11: initiate a vertex $v^{\mathcal{S}}=(u,r,s)$

12: $\mathbf{w}(u,s,r)=D_{u}$ using (III)

13: end for

14: end for

15: end for

16: Establish connecting edges between vertices sharing the same RRB and/or
user.

17: User Association:

18: MWIS($\mathcal{V}^{\mathcal{H}}$)

19: MWIS($\mathcal{V}^{\mathcal{S}}$)

20: Power Allocation:

21: for $s=1:S$ do

22: for $r=1:R$ do

23:
$p^{\mathcal{S},(t)}_{s,r}=\bar{P}_{\mathcal{S}}\sum_{u}a^{\mathcal{S},(t)}_{u,r,s}$

24: end for

25: end for

26: for $h=1:H$ do

27: Power allocation of the $h$th SH using WF algorithm

28: end for

Algorithm 1 User Assignment and Power Allocation Algorithm

We propose the following procedure for user scheduling. First, we construct
the graph by generating all the possible vertices of the eRRHs and the SHs,
calculating their corresponding weights, and making the above-mentioned
conflict links. Second, we need to find the MWIS of the whole graph such that
we assign the users first to the SHs, and the remaining users need to be
assigned to the eRRHs. Since this process is intractable, we use a heuristics
method that was used in [5] to find a suboptimal user scheduling solution. In
this fashion, first, we find the vertex of the SHs with the minimum weight,
assign the user of that vertex to its RRB and SH, and remove all of the
vertices linked to that vertex. This process is repeated till there are no
more SH vertices in the graph. Then, the same procedure will be applied to the
remaining vertices of $\mathcal{V}^{\mathcal{S}}$, so that all of the RRBs are
assigned to the users. Third, to allocate the power resources among the eRRHs
effectively, since each RRB is assigned to only one user and the RRBs are
orthogonal, we grant $\bar{P}_{\mathcal{S}}$ to the RRBs used by an eRRH.
However, due to the limited amount of power in each SH during a time slot, a
Water-Filling (WF) algorithm is utilized at each SH to allocate the power
between users properly. Such a power allocation procedure takes into account
the number of users served by each SH and their respective power requirements
to ensure that each user receives an appropriate amount of power. This ensures
that the power is utilized effectively and that each user is able to receive
its requested file segment with a possible minimum delay. To summarize, the
MBS takes the following three steps to schedule the users: (i) graph
construction, (ii) user association, and (iii) power allocation, which are
presented in Algorithm 1.

###  IV-B Cache Optimization

We consider the following subproblem to optimize the cache state of the eRRHs
and SHs

$\displaystyle\min_{\mathbf{C}^{\mathcal{S},(t)},\mathbf{C}^{\mathcal{H},(t)}}~{}~{}~{}\sum_{u=1}^{U}D_{u},~{}~{}~{}\textrm{s.t.~{}C3,~{}C4,~{}C8}.$
(20)

For each eRRH and SH, there exist permutations of $\Gamma_{eRRH}$ and
$\Gamma_{SH}$, respectively, out of a total of $F$ possible states. This
results in a number of potential solutions for the problem, specifically
$[P(F,\Gamma_{eRRH})]^{S}\times[P(F,\Gamma_{SH})]^{H}$, where
$P(n,r)=n!/(n-r)!$ represents the permutation of selecting $r$ options out of
$n$ possible states. Given the considerable computational complexity involved,
we develop two separate RL algorithms to update the cache resources of eRRHs
and the SHs.

####  IV-B1 Cache Updates in the eRRHs

eRRHs update their cache resources by fetching their desired file segments
from the MBS. The more file segments saved in each eRRH, the more the chance
of serving users from the cache of their assigned eRRH. However, the cache
capacity of each eRRH is limited by $K_{\text{eRRH}}$. In this regard, we
develop an actor-critic MARL algorithm to optimize the cache resources of the
eRRHs. Using the actor-critic algorithm, agents learn both a policy function
and a value function referred to as the actor and critic, respectively. During
each iteration, the critic evaluates and updates the actor’s policy function
by predicting the future rewards. Following this, the actor takes an action
based on its updated policy function and receives a reward. The critic then
updates its value function based on the received reward and the prediction
error [36]. In our proposed algorithm, $F$ virtual agents are created in each
eRRH, with each agent corresponding to one of the popular file segments. These
virtual agents make binary decisions based on each popular file segment’s
learned caching policies.

Since maximizing the cache hit rate at the eRRHs leads to minimizing the
average transmission delay of serving the users, the rewards of the virtual
agents are considered to be related to their cache hit rate. Besides, the
virtual agents should avoid caching the file segments being requested less
because of the limited caching capacity. Accordingly, we consider the reward
of virtual agent $f$ of eRRH $s$ as follows

$\displaystyle\mu^{(t)}_{f,s}=\alpha_{\mu}[(1-2c^{\mathcal{S},(t)}_{s,f})+c_{l}~{}l(f,s)]$
(21)

where $(1-2c^{\mathcal{S},(t)}_{s,f})$ is the cost of caching the file segment
$f$, $c_{l}$ is the impact of the cache hit status, and $l(f,s)$ is the
legitimacy of caching the file segment $f$ and considered as follows. If the
file segment $f$ is requested by a user assigned to the $s$th eRRH,
$l(f,s)=-1$ in case of cache hit failure and $l(f,s)=+1$ if a cache hit
success happens, otherwise $l(f,s)=0$. Denote $b^{\mathcal{S},(t)}_{f,s}$ as
the action of the $f$th virtual agent in the $s$th eRRH at time frame $t$, and
parameterize its policy function as an exponential softmax distribution

$\displaystyle\beta^{(t),i}_{f,s}(u^{i,(t)}_{f,s})=\frac{\exp(\lambda_{p}u^{i,(t)}_{f,s})}{\exp(\lambda_{p}u^{(t),0}_{f,s})+\exp(\lambda_{p}u^{(t),1}_{f,s})}$
(22)

where $u^{i,(t)}_{f,s}$ is the estimated utility of action $i$ at iteration
$t$ and $\lambda_{p}$ is the exploration-exploitation factor. During each
iteration step, every eRRH takes a random action generated from
$\pi^{i,(t)}_{f,s}$ to cache $K_{\text{eRRH}}$ files and obtains its own
reward $\mu^{(t)}_{f,s}$. Obtaining the reward, the policy and the utility
functions are updated as follows

$\displaystyle
u^{(t+1),i}_{f,s}=u^{i,(t)}_{f,s}+\alpha_{u}\left(\mu^{(t)}_{f,s}-u^{(t+1),i}_{f,s}\right)\mathcal{I}\\{b^{\mathcal{S},(t)}_{f,s}=i\\}$
(23)
$\displaystyle\pi^{(t+1),i}_{f,s}=\pi^{i,(t)}_{f,s}+\alpha_{\pi}\left[\beta^{(t),i}_{f,s}(u^{i,(t)}_{f,s})-\pi^{i,(t)}_{f,s}\right].$
(24)

The learning rates $\alpha_{u}$ and $\alpha_{\pi}$ represent the rates at
which the utility and policy functions are updated, respectively. To ensure
the convergence of the algorithm, these learning rates must satisfy specific
conditions [34]:

$\displaystyle\sum_{t\geq 1}\alpha_{u}=+\infty,\sum_{t\geq
1}\alpha_{u}^{2}\leq+\infty,$ (25) $\displaystyle\sum_{t\geq
1}\alpha_{\pi}=+\infty,\sum_{t\geq 1}\alpha_{\pi}^{2}\leq+\infty,$ (26)
$\displaystyle\lim_{t\rightarrow+\infty}\frac{\alpha_{\pi}}{\alpha_{u}}=0.~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}$
(27)

####  IV-B2 Cache Updates in the SHs

Similarly, $F$ virtual agents are created in each SH, where each agent is
dedicated to a specific popular file segment. In contrast to the virtual
agents of eRRHs, these agents make binary caching decisions by considering the
file segments received during the previous time slot or those already stored
in the SH’s cache. However, the critic operates the same as the virtual agents
in eRRHs. Denote the action of the $f$th virtual agent in the $h$th SH and its
reward signal at iteration $t$ as $b^{\mathcal{H},(t)}_{f,s}$ and
$\nu^{(t)}_{f,h}$, respectively. The reward signal of SH virtual agents is
considered as follows

$\displaystyle\nu^{(t)}_{f,h}=\alpha_{\mu}\alpha_{\mu}[(1-2c^{\mathcal{H},(t)}_{h,f})+c_{l}~{}l(h,s)].$
(28)

Then $v^{i,(t)}_{f,h}$ and $\eta^{i,(t)}_{f,s}$ as the utility and policy
functions of the virtual agent are respectively updated based on the decision
of the actor as follows

$\displaystyle
v^{(t+1),i}_{f,h}=v^{i,(t)}_{f,h}+\alpha_{v}\left(\nu^{(t)}_{f,s}-v^{(t+1),i}_{f,s}\right)\mathcal{I}\\{b^{\mathcal{H},(t)}_{f,s}=i\\},$
(29)
$\displaystyle\eta^{(t+1),i}_{f,s}=\eta^{(t),i}_{f,h}+\alpha_{\eta}\left[\beta^{(t),i}_{f,h}(v^{i,(t)}_{f,s})-\eta^{i,(t)}_{f,s}\right],$
(30)

where $\alpha_{v}$ and $\alpha_{\eta}$ are the learning rates of the utility
function and the policy which should satisfy the conditions (25)–(27) for the
convergence of the algorithm.

1: Initialize $t=1$

2: Set $u^{i,(t)}_{f,s}=v^{i,(t)}_{f,h}=0$ and
$\pi^{i,(t)}_{f,s}=\eta^{(t),i}_{f,h}=0.5$ for all agent

3: 1\. Cache Resource Update Stage

4: for $f=1:F$ do

5: for $s=1:S$ do

6: Generate an action $b^{\mathcal{S},(t)}_{f,s}\sim\pi^{i,(t)}_{f,s}$

7: end for

8: for $h=1:H$ do

9: Generate an action $b^{\mathcal{H},(t)}_{f,h}\sim\eta^{(t),i}_{f,h}$

10: end for

11: end for

12: Update the cache of the eRRHs and the SHs by $b^{\mathcal{S},(t)}_{f,s}$
and $b^{\mathcal{H},(t)}_{f,h}$

13: 2\. User Assignment and Power Allocation Stage

14: Assign the users to the eRRHs and the SHs and allocate the power to them
using Algorithm 1

15: 3\. Learning Update Stage (IV-B and IV-B)

16: Obtain the reward of each agent, $\mu^{(t)}_{f,h}$ and $\nu^{(t)}_{f,h}$

17: Update $u^{(t+1),i}_{f,s}$ and $\pi^{(t+1),i}_{f,s}$ using (23)

18: Update $v^{(t+1),i}_{f,h}$ and $\eta^{(t+1),i}_{f,h}$ using (29)

Algorithm 2 User Assignment and Caching Algorithm

###  IV-C Summary Of The Overall Solution

The algorithm proposed to minimize the average transmission delay can be
outlined as follows. At each iteration of the algorithm, the initial step
involves updating the cache resources of the eRRHs, which generates a decision
$b^{\mathcal{S},(t)}_{f,s}$. This decision is created based on the policy
$\pi^{i,(t)}_{f,s}$, and it aims to allocate popular file segments to the
eRRHs’ cache resources. The SHs use their policy, which takes into account the
latest $\eta^{i,(t)}_{f,s}$ and previous received communications, to generate
a decision $b^{\mathcal{H},(t)}_{f,h}$. The decision is based on whether the
file segments have already been cached or were just received in the previous
step. After updating the cache of both eRRHs and SHs, we employ Algorithm 1 to
allocate the RRBs and users to the respective eRRHs and SHs. The eRRHs grant
their assigned RRBs to $\bar{P}_{\mathcal{S}}$, while the SHs use the WF
algorithm to distribute the available power among the assigned RRBs. Each
agent receives a reward based on its cache hit factor, and their policies are
updated using equations (23) and (29) for the next iteration step. The process
of union user scheduling and caching of both the SHs and eRRHs is illustrated
in Algorithm 2.

###  IV-D Convergence and Complexity of the Algorithm

####  IV-D1 Convergence

The process described in Algorithm 2 updates the probability of the agents of
eRRHs and SHs selecting actions at each iteration. When the algorithm
converges, the probabilities become stable and do not change much. This can be
shown by the following theorem.

#### Theorm 1

The policies of binary action selection for each agent, i.e.
$\pi^{i,(t)}_{f,s}$ and $\eta^{i,(t)}_{f,s}$ is converged as
$\lim_{t\to\infty}\pi^{i,(t)}_{f,s}\to\bar{\pi}^{i}_{f,s}$ and
$\lim_{t\to\infty}\eta^{(t),i}_{f,h}\to\bar{\eta}^{i}_{f,h}$, $\forall
i\in\\{1,2\\},f\in\\{1,2,\dots,F\\}$, $s\in\\{1,2,\dots,S\\}$, and
$h\in\\{1,2,\dots,H\\}$. Here

$\displaystyle\bar{\pi}^{i}_{f,s}=\frac{\exp(\lambda_{p}\bar{\mu}_{f,s}^{i})}{\sum_{i=1}^{2}\exp(\lambda_{p}\bar{\mu}_{f,s}^{i})}$
(31)
$\displaystyle\bar{\eta}^{i}_{f,h}=\frac{\exp(\lambda_{p}\bar{\nu}_{f,h}^{i})}{\sum_{i=1}^{2}\exp(\lambda_{p}\bar{\nu}_{f,h}^{i})}$
(32)

where $\bar{\mu}_{f,s}^{i}$ and $\bar{\nu}_{f,h}^{i}$ are expected reward of
taking action $i$ by the $f$th agent of the $s$th eRRH and the $h$th SH,
respectively.

For proof, please refer to the Appendix.

####  IV-D2 Complexity of Algorithm

During each iteration, cache resource update stage requires to generate
$K_{\text{eRRH}}+K_{\text{SH}}$ random values from $\pi^{i,(t)}_{f,s}$ and
$\eta^{(t),i}_{f,h}$. Using the Alias method [37], the computational
complexity of the first stage at most is $\mathcal{O}(F[S+H])$. Over the user
assignment stage, Algorithm 1 at most finds the minimum utility value of
$R(S+H)U$ values for $R$ times. So, the computation complexity of this stage
is generally bounded by $\mathcal{O}(UR^{2}(S+H))$. In the power allocation
stage, the SHs allocate the available power to the assigned users and RRBs
based on WF algorithm. Where the computational complexity of WF for $N$
channels is generally in the range of $\mathcal{O}(N\log N)$ [38]. The number
of users that each SH can serve is bounded to a small number due to the
limited available transmission power. So, the computational complexity order
of this stage is approximately $\mathcal{O}(H)$. As the learning update stage
involves a fixed number of operations, i.e., $2F(S+H)$ exponents,
multiplications and additions, the computational complexity of the last stage
in each learning step is $\mathcal{O}(F(S+H))$. Therefore, the overall worst
case computational complexity of Algorithm 2 for the all learning process
approximately is $\mathcal{O}(T[(UR^{2}+F)(S+H)])$.

##  V Numerical Results and Analysis

We numerically evaluate the performance of our developed scheme of the
envisioned SH-aided F-RAN in terms of average transmission delay, average load
on the fronthaul link, and cache hit rate with benchmark schemes proposed for
conventional F-RANs. In particular, we consider two benchmark schemes: 1)
random caching strategy and 2) caching strategy that caches the most probable
file segments. Moreover, we quantify the performance gain that can be achieved
using SHs in F-RANs.

###  V-A Basic Simulation Settings

We assume the eRRHs, SHs, and users are uniformly located within a circle of
1.5 km radius. The minimum distance between neighbouring eRRHs is set to 300
m. Similarly, the minimum distance between neighbouring SHs is set to 300 m.
The MBS is assumed to have 50 orthogonal RRBs over each time frame to serve
the users. It is assumed that there are 100 popular file segments, each with
the size of $6.25$ MBytes, where their popularity follows the Zipf
distribution with different values of the Zipf parameter. Users are assumed to
request a popular file segment with a probability of 0.3. Each of the eRRHs or
SHs can store/cache up to 35 file segments. The channel gains between users
and eRRHs, $g^{\mathcal{S},(t)}_{u,s,h}$, and the channel gains between users
and SHs, $g^{\mathcal{H},(t)}_{u,r,h}$ are assumed to have a fading model used
in the literature, e.g., [13, 35], that consists of 1) path-loss of
$128.1+37.6\log 10(dis.[km])$, 2) log-normal shadowing with $4dB$ standard
deviation, and 3) Rayleigh fading with zero-mean and unit variance. The
additive white Gaussian (AWGN) noise power is assumed to be
$N_{0}=-174~{}dBm$. The maximum allowed transmission power of each eRRH over
an RRB is $\bar{P}_{S}=4\times 10^{-4}$ mwatt. In addition, the maximum
available power at each SH is assumed to be $\bar{P}^{max}_{H}=15\times
10^{-4}$ mwatt that is allocated to different RRBs using the WF algorithm. The
impact of the cache hit rate on the reward is considered to be $c_{l}=2$. We
evaluate each metric by averaging over 2000 Monte-Carlo simulation run.

###  V-B Evaluation and Comparison

Figure 3: Policy update process of the virtual agents associated with 100
popular file segments with $\Gamma_{eRRH}=35$. Figure 4: Caching probabilities
based on the policies obtained after 1,000 training steps.

Fig. 3 presents the policy update process of virtual agents associated with
100 popular file segments over 10,000 iterations to identify the convergence
point. As seen, the caching policy of the agents associated with both the most
and least probable file segments converges rapidly to 1 and 0, respectively,
shortly after 1,000 iterations. However, the policy of the agents related to
files with popularity orders around the caching capacity of 35 file segments
exhibits continuous fluctuations until the end of the simulation. This
fluctuation results from the competition among these agents to fill the
remaining caching capacity, leading to a more indecisive learning process.
Based on the results, we choose 1,000 iterations as the stopping point for our
learning algorithm to balance learning efficiency and policy stability. By
doing so, we ensure that the algorithm achieves an optimized caching policy
without unnecessary computational overhead and extended training time.

In Fig. 4, we further investigate the caching policy generated by the
algorithm after 1,000 iterations considering various numbers of eRRHs in the
system. Similarly, the results indicate that although a certain caching policy
exists for the most and least popular file segments, the caching policy for
file segments near the caching capacity may not be entirely deterministic. The
optimized policy of the virtual agents of different eRRHs considers both the
experienced request probability of each file segment and its ranking in terms
of popularity among all file segments.

(a) The average cumulative reward of the agents of the eRRHs for $S=~{}1,2,4$.

(b) The average cumulative reward of the agents of the SHs for $H=~{}1,2,4$.

Figure 5: The average cumulative reward of the agents of the eRRHs and the SHs
over 1,000 training iterations.

In Fig. 5, we depict the cumulative rewards of virtual agents associated with
eRRHs and SHs throughout 1,000 iterations. Notably, the rewards are observed
to increase initially and eventually achieve their maximum values before
reaching the 1,000th iteration. This observation validates that 1,000
iterations are indeed appropriate to gain the peak rewards for both eRRHs and
SHs. Furthermore, the figures illustrate that eRRHs outperform SHs in
achieving higher cumulative rewards due to their ability to fetch their
desired file segments based on their policy from the MBS, which grants them
more opportunities to receive rewards.

(a) Cache hit rate of eRRHs and SHs using the proposed algorithm.

(b) Comparing the cache hit rate of the proposed algorithm, most probable
scheme, and random policies.

Figure 6: The total cache hit rate of eRRHs and SHs in different scenarios
with various numbers of the eRRHs and SHs.

(a) Average transmission delay utilizing different numbers of SHs.

(b) Comparing the average transmission delay of the proposed algorithm, most
probable, and random policies

Figure 7: Average transmission delay of different scenarios with varying
number of eRRHs and SHs.

Fig. 6 illustrates the efficiency of different caching schemes in terms of
cache hit rate. In panel (a), we can see how the inclusion of SHs affects the
cache hit rate. Increasing the number of SHs results in a higher cache hit
rate, which in turn enables the serving of more users in a given time frame.
This reduces the load on the fronthaul link and allows more users to be
served. The figure indicates that including SHs in the network is particularly
beneficial when there are a low number of eRRHs to serve the users. As the
number of eRRHs increases, the eRRHs are able to serve more users who request
popular file segments, thereby improving the chances of maximizing the cache
hit rate. However, the inclusion of SHs still remains useful as it reduces the
load on the eRRHs by serving some of the users, leading to a better quality of
service and more users served in each time subframe. In panel (b), the
performance of the proposed algorithm is compared to random and most probable
file segments caching policies. The figure shows that the proposed algorithm
outperforms the other schemes for all values of $\gamma$ from 0 to 1 when the
number of eRRHs is greater than 2. This indicates that the proposed algorithm
is effective in improving the cache hit rate and hence, the overall
performance of the caching strategy in the network.

Fig. 7 presents the average transmission delay for different scenarios with
varying numbers of eRRHs and SHs. Panel (a) shows the average transmission
delay for four different numbers of eRRHs and SHs. As more SHs are added, the
average transmission delay decreases while serving the same or even more
number of users. Additionally, increasing $\gamma$ significantly reduces the
average transmission delay. This is because having a wide range of
probabilities for requesting popular files can help make better caching
decisions, leading to better performance. The figure also shows that SHs can
compensate for the lack of eRRHs in the network. Panel (b) compares the
performance of the proposed algorithm with two different caching schemes. As
seen in the figure, our designed algorithm outperforms the most probable and
random caching schemes for all values of $\gamma$ between 0 to 1 when the
number of eRRHs is greater than two. Similar to the cache hit rate figures,
utilizing SHs is more advantageous when the number of eRRHs is lower. Overall,
the figure demonstrates the effectiveness of the proposed algorithm in
reducing the average transmission delay and improving the overall performance
of the caching strategy in the network.

(a) Average load on the fronthaul link in each transmission subframe.

(b) Comparison on the average load on the fronthaul link during each
transmission time subframe.

Figure 8: Load on the fronthaul link for different schemes varying number of
eRRHs and SHs.

In Fig. 8, the impact of using SHs on the load of the fronthaul link is shown,
along with a comparison of the proposed algorithm with two other caching
schemes. It is evident from the plot that adding up to three SHs to the
network reduces the load on the fronthaul link by around $5\times 10^{8}$
bits/sec. Moreover, the proposed algorithm performs better than the most
probable and random schemes for all $\gamma$ values when the number of eRRHs
is more than 2. This indicates that the proposed algorithm is effective in
reducing the load on the fronthaul link, especially when the network is facing
a high load and excessive average transmission delay.

##  VI Conclusion

In this work, we proposed to expand the resource settings of F-RAN in a cost-
effective manner by incorporating a set of SHs. The SHs act as compact caching
elements that can cache popular file segments and alleviate the load on the
fronthaul link, leading to reduced transmission delay. We developed a
multistage user scheduling and caching algorithm to optimize the network in
terms of average transmission delay and load on the fronthaul link. In
particular, first we developed an MWIS search method to assign the users to
the eRRHs/SHs while proiritizing the SHs to serve the users. Then for caching
strategy at the eRRHs and SHs, we proposed two MARL algorithms to optimize the
caches of eRRHs and SHs. Simulation results showed that the use of SHs
significantly reduces the load on the fronthaul link and transmission delay,
indicating their potential in designing wireless networks facing high loads on
the fronthaul link and transmission delay. Moreover, it is shown that SHs can
be substituted for some eRRHs to further dampen the load on the fronthaul
link.

## Appendix
Proof of Theorem 1

According to [39, Theorm 4], assuming
$\lim_{t\to\infty}\frac{\alpha_{u}}{\alpha_{\pi}}=0$, we can conclude that two
conditions,
$\mathbf{c1:}\lim_{t\to\infty}|u^{i,(t)}_{f,s}-\bar{\mu}_{f,s}^{i}|=0$,
$\forall i,f,s$; and $\mathbf{c2:}$ as $t\to\infty$, are met. Then, the
equation used to update the agent’s action selection probability will converge
to the following ordinary differential equation

$\displaystyle\dot{\pi}^{(t),i}_{f,s}=\beta^{(t),i}_{f,s}(u^{i,(t)}_{f,s})-\pi^{i,(t)}_{f,s}.$
(33)

Considering $\lim_{t\to\infty}\frac{\alpha_{u}}{\alpha_{\pi}}=0$, the update
of action selection probabilities occurs at a slower pace in comparison to the
estimated utility. However, as per condition $\mathbf{c1}$, the estimated
utility will converge as time progresses. By substituting the converged
estimated utility value into (33) and considering the fact that at the
stationary point of (33), $\dot{\pi}^{(t),i}_{f,s}=0$, we can determine the
value of the stationary action selection probability,

$\displaystyle\bar{\pi}^{(t),i}_{f,s}=\beta(\bar{\mu}_{f,s}^{i}).$ (34)

$\displaystyle\\{\bar{\pi}^{(t),1}_{f,s},\bar{\pi}^{(t),2}_{f,s}\\}=\arg\max_{\pi^{1}_{f,s}\geq
0,\pi^{2}_{f,s}\geq
0}\sum_{i=1}^{2}\pi^{i}_{f,s}\left(\bar{\mu}_{f,s}^{i}+\frac{1}{\lambda_{p}}\ln(\frac{1}{\pi^{i}_{f,s}})\right),~{}~{}\text{s.t.}~{}~{}\sum_{i=1}^{2}\pi^{i}_{f,s}=1$
(35)

We proceed to explain that (34) gives the resulting probability for selecting
actions among the agents. Reference [40, eq. (2)] indicates that the most
favourable action selection probability is achieved by maximizing the agents’
expected reward along with the entropy function. Therefore, we derive (35) for
each agent ($s,f$), $f\in\\{1,2,\dots,F\\}$ and $s\in\\{\\{1,2,\dots,S\\}$,
which is shown at the top of the next page. The value $1/\lambda_{p}$ in (35)
determines the relative importance of maximizing the expected reward and
entropy functions. By using the Lagrangian optimization method, we can easily
calculate the optimal values

$\displaystyle\bar{\pi}^{(t),i}_{f,s}=\frac{\exp(\lambda_{p}\bar{\mu}_{f,s}^{i})}{\sum_{i=1}^{2}\exp(\lambda_{p}\bar{\mu}_{f,s}^{i})}\triangleq\beta(\bar{\mu}_{f,s}^{i}).$
(36)

It is apparent that when $t\to\infty$, the likelihood of agents selecting an
action becomes consistent and this consistent likelihood aligns with the ideal
action selection probability for the agents. Hence, as the number of
iterations increases, Algorithm 2 converges.

## References

  * [1] L. Ericsson, “Ericsson mobility report june 2020,” _Ericsson_ , vol. 36, no. 2, pp. 1–36, 2020.
  * [2] M. S. Elbamby, C. Perfecto, M. Bennis, and K. Doppler, “Toward low-latency and ultra-reliable virtual reality,” _IEEE Network_ , vol. 32, no. 2, pp. 78–84, Apr. 2018.
  * [3] R. T. Azuma, “A Survey of Augmented Reality,” _Presence: Teleoperators and Virtual Environments_ , vol. 6, no. 4, pp. 355–385, 08 1997. [Online]. Available: https://doi.org/10.1162/pres.1997.6.4.355
  * [4] China Mobile Research Institute, “the 1st C-RAN International Workshop,” Beijing, China, Apr. 2010.
  * [5] M. S. Al-Abiad, A. Douik, S. Sorour, and M. J. Hossain, “Throughput maximization in cloud-radio access networks using cross-layer network coding,” _IEEE Trans. Mob. Comput._ , vol. 21, no. 2, pp. 696–711, Jul. 2022\.
  * [6] S.-H. Park, O. Simeone, and S. Shamai Shitz, “Joint optimization of cloud and edge processing for fog radio access networks,” _IEEE Trans. Wirel. Commun._ , vol. 15, no. 11, pp. 7621–7632, Aug. 2016.
  * [7] T. G. Rodrigues, K. Suto, H. Nishiyama, and N. Kato, “Hybrid method for minimizing service delay in edge cloud computing through vm migration and transmission power control,” _IEEE Trans. Comput._ , vol. 66, no. 5, pp. 810–819, Oct. 2017.
  * [8] R. Tandon and O. Simeone, “Harnessing cloud and edge synergies: toward an information theory of fog radio access networks,” _IEEE Commun. Mag._ , vol. 54, no. 8, pp. 44–50, Aug. 2016.
  * [9] S. M. S. Tanzil, W. Hoiles, and V. Krishnamurthy, “Adaptive scheme for caching youtube content in a cellular network: Machine learning approach,” _IEEE Access_ , vol. 5, pp. 5870–5881, Mar. 2017.
  * [10] J. Song, M. Sheng, T. Q. S. Quek, C. Xu, and X. Wang, “Learning-based content caching and sharing for wireless networks,” _IEEE Trans. Commun._ , vol. 65, no. 10, pp. 4309–4324, Jun. 2017.
  * [11] H. Xiang, M. Peng, Y. Cheng, and H.-H. Chen, “Joint mode selection and resource allocation for downlink fog radio access networks supported d2d,” in _2015 11th International Conference on Heterogeneous Networking for Quality, Reliability, Security and Robustness (QSHINE)_ , 2015, pp. 177–182.
  * [12] M.-C. Lee and A. F. Molisch, “Caching policy and cooperation distance design for base station-assisted wireless d2d caching networks: Throughput and energy efficiency optimization and tradeoff,” _IEEE Trans. Wirel. Commun._ , vol. 17, no. 11, pp. 7500–7514, Sep. 2018.
  * [13] M. S. Al-Abiad, M. J. Hossain, and S. Sorour, “Cross-layer cloud offloading with quality of service guarantees in fog-rans,” _IEEE Trans. Commun._ , vol. 67, no. 12, pp. 8435–8449, Jun. 2019.
  * [14] M. S. Al-Abiad, A. Douik, and S. Sorour, “Rate aware network codes for cloud radio access networks,” _IEEE Trans. Mob. Comput._ , vol. 18, no. 8, pp. 1898–1910, Sep. 2019.
  * [15] Y. Chen, M. Ding, J. Li, Z. Lin, G. Mao, and L. Hanzo, “Probabilistic small-cell caching: Performance analysis and optimization,” _IEEE Trans. Veh. Technol._ , vol. 66, no. 5, pp. 4341–4354, Sep. 2017.
  * [16] H. Tran-Dang, S. Bhardwaj, T. Rahim, A. Musaddiq, and D.-S. Kim, “Reinforcement learning based resource management for fog computing environment: Literature review, challenges, and open issues,” _Journal of Communications and Networks_ , vol. 24, no. 1, pp. 83–98, Feb. 2022.
  * [17] U. Verma and D. Bhardwaj, “Fog computing paradigm for internet of things: Architectures, issues, challenges, and applications,” in _Advances in Distributed Computing and Machine Learning_ , R. R. Rout, S. K. Ghosh, P. K. Jana, A. K. Tripathy, J. P. Sahoo, and K.-C. Li, Eds. Singapore: Springer Nature Singapore, 2022, pp. 1–11.
  * [18] D. Wang, Z. Liu, X. Wang, and Y. Lan, “Mobility-aware task offloading and migration schemes in fog computing networks,” _IEEE Access_ , vol. 7, pp. 43 356–43 368, Mar. 2019.
  * [19] S. Manzoor, A. N. Mian, A. Zoha, and M. A. Imran, “Federated learning empowered mobility-aware proactive content offloading framework for fog radio access networks,” _Future Generation Computer Systems_ , vol. 133, pp. 307–319, Aug. 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0167739X22001005
  * [20] Y. Sellami, G. Jaber, A. Lounis, H. Lakhlef, and A. Bouabdallah, “A cooperative caching scheme in fog/sensor nodes for ccn,” in _2022 International Wireless Communications and Mobile Computing (IWCMC)_ , 2022, pp. 481–486.
  * [21] Y. Li, H. Ma, L. Wang, S. Mao, and G. Wang, “Optimized content caching and user association for edge computing in densely deployed heterogeneous networks,” _IEEE Trans. Mob. Comput._ , vol. 21, no. 6, pp. 2130–2142, Oct. 2022.
  * [22] T. Zhang, Y. Wang, W. Yi, Y. Liu, C. Feng, and A. Nallanathan, “Two time-scale caching placement and user association in dynamic cellular networks,” _IEEE Trans. Commun._ , vol. 70, no. 4, pp. 2561–2574, Feb. 2022.
  * [23] S.-H. Park, O. Simeone, and S. Shamai, “Joint cloud and edge processing for latency minimization in fog radio access networks,” in _2016 IEEE 17th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC)_ , 2016, pp. 1–5.
  * [24] ——, “Joint optimization of cloud and edge processing for fog radio access networks,” in _2016 IEEE International Symposium on Information Theory (ISIT)_ , 2016, pp. 315–319.
  * [25] G. M. S. Rahman, M. Peng, K. Zhang, and S. Chen, “Radio resource allocation for achieving ultra-low latency in fog radio access networks,” _IEEE Access_ , vol. 6, pp. 17 442–17 454, Feb. 2018.
  * [26] S. Yan, M. Peng, M. A. Abana, and W. Wang, “An evolutionary game for user access mode selection in fog radio access networks,” _IEEE Access_ , vol. 5, pp. 2200–2210, Jan. 2017.
  * [27] X. Wang, S. Leng, and K. Yang, “Social-aware edge caching in fog radio access networks,” _IEEE Access_ , vol. 5, pp. 8492–8501, Apr. 2017.
  * [28] E. Balevi and R. D. Gitlin, “A clustering algorithm that maximizes throughput in 5g heterogeneous f-ran networks,” in _2018 IEEE International Conference on Communications (ICC)_ , 2018, pp. 1–6.
  * [29] N.-N. Dao, J. Lee, D.-N. Vu, J. Paek, J. Kim, S. Cho, K.-S. Chung, and C. Keum, “Adaptive resource balancing for serviceability maximization in fog radio access networks,” _IEEE Access_ , vol. 5, pp. 14 548–14 559, Jun. 2017\.
  * [30] R. Tandon and O. Simeone, “Cloud-aided wireless networks with edge caching: Fundamental latency trade-offs in fog radio access networks,” in _2016 IEEE International Symposium on Information Theory (ISIT)_ , 2016, pp. 2029–2033.
  * [31] G. M. S. Rahman, M. Peng, S. Yan, and T. Dang, “Learning based joint cache and power allocation in fog radio access networks,” _IEEE Trans. Veh. Technol._ , vol. 69, no. 4, pp. 4401–4411, Feb. 2020.
  * [32] Y. Sun, S. Chen, Z. Wang, and S. Mao, “A joint learning and game-theoretic approach to multi-dimensional resource management in fog radio access networks,” _IEEE Trans. Veh. Technol._ , vol. 72, no. 2, pp. 2550–2563, Oct. 2023.
  * [33] J. Yan, Y. Jiang, F. Zheng, F. R. Yu, X. Gao, and X. You, “Distributed edge caching with content recommendation in fog-rans via deep reinforcement learning,” in _2020 IEEE International Conference on Communications Workshops (ICC Workshops)_ , 2020, pp. 1–6.
  * [34] Y. Sun, M. Peng, and S. Mao, “A game-theoretic approach to cache and radio resource management in fog radio access networks,” _IEEE Trans. Veh. Technol._ , vol. 68, no. 10, pp. 10 145–10 159, Aug. 2019.
  * [35] M. S. Al-Abiad, M. Z. Hassan, and M. J. Hossain, “A joint reinforcement-learning enabled caching and cross-layer network code in f-ran with d2d communications,” _IEEE Trans. Commun._ , vol. 70, no. 7, pp. 4400–4416, Apr. 2022.
  * [36] R. S. Sutton and A. G. Barto, _Reinforcement Learning: An Introduction_. Cambridge, MA, USA: A Bradford Book, 2018.
  * [37] A. Walker, _Electronics Letters_ , vol. 10, no. 8, pp. 127–128, Nov. 1974.
  * [38] S. Khakurel, C. Leung, and T. Le-Ngoc, “A generalized water-filling algorithm with linear complexity and finite convergence time,” _IEEE Wirel. Commun. Lett._ , vol. 3, no. 2, pp. 225–228, Feb. 2014.
  * [39] M. Bennis, S. M. Perlaza, P. Blasco, Z. Han, and H. V. Poor, “Self-organization in small cell networks: A reinforcement learning approach,” _IEEE Trans. Wirel. Commun._ , vol. 12, no. 7, pp. 3202–3212, June. 2013.
  * [40] T. Haarnoja, H. Tang, P. Abbeel, and S. Levine, “Reinforcement learning with deep energy-based policies,” in _Proceedings of the 34th International Conference on Machine Learning - Volume 70_ , ser. ICML’17. JMLR.org, 2017, p. 1352–1361.