myequation

$\BODY$

(1)

# Cooperate or not Cooperate: Transfer Learning with Multi-Armed Bandit for
Spatial Reuse in Wi-Fi

Pedro Enrique Iturria-Rivera1, , Marcel Chenier2, Bernard Herscovici2,
Burak Kantarci1, and Melike Erol-Kantarci1, 2NetExperience Inc., Ottawa,
Canada
Emails:{pitur008, burak.kantarci<EMAIL_ADDRESS>{marcel,
<EMAIL_ADDRESS>1School of Electrical Engineering and Computer
Science, University of Ottawa, Ottawa, Canada

###### Abstract

The exponential increase of wireless devices with highly demanding services
such as streaming video, gaming and others has imposed several challenges to
Wireless Local Area Networks (WLANs). In the context of Wi-Fi, IEEE 802.11ax
brings high-data rates in dense user deployments. Additionally, it comes with
new flexible features in the physical layer as dynamic Clear-Channel-
Assessment (CCA) threshold with the goal of improving spatial reuse (SR) in
response to radio spectrum scarcity in dense scenarios. In this paper, we
formulate the Transmission Power (TP) and CCA configuration problem with an
objective of maximizing fairness and minimizing station starvation. We present
four main contributions into distributed SR optimization using Multi-Agent
Multi-Armed Bandits (MA-MABs). First, we propose to reduce the action space
given the large cardinality of action combination of TP and CCA threshold
values per Access Point (AP). Second, we present two deep Multi-Agent
Contextual MABs (MA-CMABs), named Sample Average Uncertainty (SAU)-Coop and
SAU-NonCoop as cooperative and non-cooperative versions to improve SR. In
addition, we present an analysis whether cooperation is beneficial using MA-
MABs solutions based on the $\epsilon$-greedy, Upper Bound Confidence (UCB)
and Thompson techniques. Finally, we propose a deep reinforcement transfer
learning technique to improve adaptability in dynamic environments. Simulation
results show that cooperation via SAU-Coop algorithm contributes to an
improvement of 14.7% in cumulative throughput, and 32.5% improvement of PLR
when compared with no cooperation approaches. Finally, under dynamic
scenarios, transfer learning contributes to mitigation of service drops for at
least 60% of the total of users.

Index Terms — Wi-Fi, 802.11ax, Multi-Agent Multi-Armed Bandits, spatial reuse,
deep transfer reinforcement learning.

##  I Introduction

W ireless connectivity has become an irreplaceable commodity in our modern
society. The exponential trend expected in the wireless technology usage has
lead analysts to predict that by 2023, $71\%$ of the global population will
enjoy some kind of wireless service. In the group of Wireless Local Area
Networks (WLANs), Wireless Fidelity (Wi-Fi) technology presents a growth up to
4-fold over a period of 5 years from 2018 to 2023 [1]. The newest Wi-Fi
standard IEEE-802.11ax [2], also known as Wi-Fi 6 expects to grow 4-fold by
2023 becoming $11\%$ of all the public Wi-Fi hostpots [3].

Spatial reuse (SR) has been of interest for more than 20 years in the wireless
community since it contributes to the reduction of the collisions among
stations and the determination of channel access rights [4]. As the number of
dense WLAN deployments increases, SR becomes more challenging in the context
of Carrier Sense Multiple Access (CSMA) technology as used in Wi-Fi[5]. Wi-Fi
6 comes to address diverse challenges such as increasing number of Wi-Fi
users, dense hotspots deployments and high demanded services such as
Augmented, Mixed and Virtual Reality.

Figure 1: Typical operational scenario: APs adjust their Transmission Power
and CCA threshold towards an efficient spatial reuse.

Moreover, 802.11ax included additional features such as dynamic adjustment of
the Clear Channel Assessment (CCA) threshold and Transmission Power (TP).
Static CCA threshold may not be representative of diverse network topologies,
and cause inefficient channel utilization or concurrent transmissions [6].
Additionally, adjusting TP allows to reduce the interference among the APs and
consequently maximize the network performance [7]. Thus, SR and network
performance can be positively improved by adjusting CCA and TP. Yet, the
complex interactions between CCA and TP, call for intelligent configuration of
both.

To this end, data scarcity and data access are key for any Machine Learning
(ML) method [8]. Recently, AI-based wireless networks have been of remarkable
interest among researchers both in WiFi domain [9], and 5G domain [10] however
the proposed solutions usually require complete availability of the data. In
reality, data access is not always feasible due to privacy restrictions.
Recent wireless network architectures have started to shift to a more open and
flexible design. In 5G networks as well as the O-RAN Alliance architecture
support the utilization of artificial intelligence to orchestrate main network
functions [11]. In the context of Wi-Fi, a novel project named OpenWiFi[12]
released by the Telcom Infra Project intends to disaggregate the Wi-Fi
technology stack by utilizing open source software for the cloud controller
and AP firmware operating system. These paradigm changes allow for the
development of many applications in the area of ML and more specifically in
Reinforcement Learning (RL) applications to become reality.

In this paper111The present work has been submitted to IEEE, we intend to
optimize TP and CCA threshold to improve SR and overall network KPIs.To do so,
we formulate the TP and CCA configuration problem with an objective of
maximizing product network fairness and minimizing station starvation. We
model the SR problem as a distributed multi-agent decision making problem and
use a Multi-Agent Multi-Armed Bandit (MA-MAB) approach to solve it. The
contributions of this work, different from the ones found in the literature,
can be summarized in the following points:

  1. 1.

We propose a solution for reducing the inherent huge action space given the
possible combinations of TP and CCA threshold values per AP. We derive our
solution via worst-case interference analysis.

  2. 2.

We analyse the performance of the network KPIs of well-known distributed MA-
MAB implementations such as $\epsilon$-greedy, UCB and Thompson on the
selection of the TP and CCA values in cooperative and non-cooperative
settings.

  3. 3.

We introduce a contextual MA-MAB (MA-CMAB) named Sample Average Uncertainty-
Sampling (SAU) in cooperative and non-cooperative settings. SAU-MAB is based
on a deep Contextual MAB.

  4. 4.

We propose for the first time, to the best of our knowledge, a deep transfer
learning solution to adapt efficiently TP and CCA parameters in dynamic
scenarios.

With these contributions, our simulation results show that the
$\epsilon$-greedy MAB solution improves the throughput at least 44.4%,
provides improvement of 12.2% in terms of fairness and 94.5% in terms of
Packet Loss Ratio (PLR) over typical configurations when a reduced set of
actions is known. Additionally, we show that the SAU-Coop algorithm improves
the throughput by 14.7% and PLR 32.5% when compared with non cooperative
approaches with full set of actions. Moreover, our proposed transfer learning
based approach reduces the service drops by at least 60%.

The rest of the paper is organized as follows. Section II presents a summary
of recent work that uses Machine Learning to improve SR in Wi-Fi. Section III
covers the basics on Multi-Armed Bandits including deep contextual bandits and
deep transfer reinforcement learning. In IV we present our system model
altogether with an analysis to reduce the action space via worst-case
interference. Section V presents the proposed schemes and the results are
discussed in section VI. Finally, section VII concludes the paper.

##  II Related work

Reinforcement learning-based spatial reuse has been of interest in recent
literature. The studies have focused on distributed solutions with no
cooperation or centralized schemes of multi-armed bandits. These studies are
summarized below.

In [13], the authors present a comparison among well-known MABs as
$\epsilon$-greedy, UCB, Exp3 and Thompson sampling in the context of
decentralized SR via Dynamic Channel Allocation (DCA) and Transmission Power
Control (TPC) in WLANs. The results showed that “selfish learning” in a
sequential matter present better performance than “concurrent learning” among
the agents. Additionally, [14] presents a centralized MAB consisting of an
optimizer based on a modified Thompson Sampling (TS) algorithm and a sampler
based on Gaussian Mixture (GM) algorithm to improve SR in 802.11ax Wi-Fi. More
specifically, the authors propose to deal with the large action space
comprised by TP and Overlapping BSS/Preamble-Detection (OBSS/PD) threshold by
utilizing a MAB variant called Infinitely Many-Armed Bandit (IMAB).
Furthermore, a distributed solution based on Bayesian optimizations of
Gaussian processes to improve SR is proposed in [15].

Other solutions that are not related to reinforcement learning can be found in
the literature with the aim of improving SR in WLANs. For instance, in [16]
the authors propose a distributed algorithm where the APs decide their
Transmission Power based on their RSSI. Moreover, in [17] the authors present
an algorithm to improve SR by utilizing diverse metrics such as SINR,
proximity information, RSSI and BSS color and compare with the legacy existing
algorithms. The ultimate goal of the previous algorithm is the selection of
the channel state (IDLE of BUSY) at the moment of an incoming frame given the
previous metrics. Finally, the authors in [18] presented a supervised
federated learning approach for SR optimization.

In all above works, the authors employ either centralized or decentralized
schemes with no cooperation to address SR optimization in WiFi. In this work,
we propose to address this via a coordination based MA-MAB. In addition, we
tackle some of the issues previously encountered in others works such as the
size of action space due the set of possible values TP and CCA. Finally, to
the best of our knowledge, we propose for the first time to address SR
adaptation in dynamic environments utilizing deep transfer learning.

##  III Background

In this section, we present a background on Multi-Armed Bandits including
$\epsilon$-greedy, Upper Confident Bound, Thompson sampling bandits and an
introduction on contextual MABs with a focus on a neural network-based
contextual bandit. Additionally, we introduce MABs to the multi-agent setting
and we finalize with a background on deep transfer reinforcement learning.

Multi-Armed Bandits (MABs) are a widely used RL approach that tackles the
exploration-exploitation trade-off problem. Their implementation is usually
simpler when compared with full RL off-policy or on-policy algorithms.
However, simplicity often comes with a cost of obtaining suboptimal solutions
[19]. The basic model of MABs corresponds to the stochastic bandit, where the
agent has $K$ possible actions to choose, called arms, and receive certain
reward $R$ as a consequence of pulling the $j^{th}$ arm over $T$ environment
steps. The rewards can be modeled as independent and identically distributed
(i.i.d), adversarial, constrained adversary or random-process rewards [20].
From the four models previously mentioned, two are more commonly found in the
literature: the i.i.d and the adversarial models. In the i.i.d model, each
pulled arm’s reward is drawn independently from a fixed but unknown
distribution $D_{j}$ with an unknown mean $\mu_{j}^{*}$. On the other hand, in
the adversarial model each pulled arm’s reward is randomly sampled from an
adversary or alien to the agent (such as the environment) and not necessarily
sampled from any distribution[21]. The performance of MABs is measured in
terms of cumulative regret $R_{T}$ or total expected regret over the $T$ steps
defined as:  R_T = ∑_t=1^T E[(max_jμ_j^* - μ_j^*)], The utmost goal of the
agent is to minimize $R_{T}$ over the T steps such as the
$\lim_{T\to\infty}R_{T}/T$ = 0 which means the agent will identify the action
with the highest reward in such limit.

###  III-A $\epsilon$-greedy, Upper-Confidence-Bound and Thompson Sampling MAB

The $\epsilon$-greedy MAB is one of the simplest MABs and as the name
suggests, it is based on the $\epsilon$-greedy policy. In this method, the
agent selects greedily the best arm most of time and once a while, with a
predefined small probability ( $\epsilon$), it selects a random arm [22]. The
UCB MAB tackles some the disadvantages of the $\epsilon$-greedy policy at the
moment of selecting non-greedy arms. Instead of drawing randomly an arm, the
UCB policy measures how promising non-greedy arms are close from optimal. In
addition, it takes in to consideration the rewards’ uncertainty in the
selection process. The selected arm is obtained by drawing the action from
$\text{{argmax}}_{a}\left[Q_{t}(a)+c\sqrt{\text{ln }{t}/N_{t}(a)}\right]$,
where $N_{t}(a)$ corresponds to the number of times that action $a$ via the
$j^{th}$ arm has been chosen and $Q_{t}(a)$ the Q-value of action $a$[22, 23].
Finally, Thompson Sampling MAB action selection is based on Thompson Sampling
algorithm as the name indicates. Thompson sampling or posterior sampling is a
Bayesian algorithm that constantly constructs and updates the distribution of
the observed rewards given a previously selected action. This allows the MAB
to select arms based on the probability of how optimal the chosen arm is. The
parameters of the distribution are updated depending on the selection of the
distribution class[24].

###  III-B Deep Contextual Multi-Armed Bandits

Contextual Multi-Armed Bandits (CMABs) are a variant of MABs, that before
selecting an arm, observe a series of features commonly named context[19].
Different from the stateless MAB, a CMAB is expected to relate the observed
context with the feedback or reward gathered from the environment in $T$
episodes and consequently predict the best arm given the received features
[21]. Diverse CMABs have been proposed throughout the literature such as
LinUCB, Neural Bandit, Contextual Thompson Sampling and Active Thompson
Sampling [19]. More recently, a deep neural contextual bandit named SAU-
Sampling has been presented in [25] where the context is related with the
rewards using neural networks. The details of SAU-Sampling will be discussed
in following sections.

###  III-C Multi-Agent Multi-Armed Bandits (MA-MABs)

Multi-agent Multi-Armed Bandits is the multi-agent variant of MABs in which
$N$ agents pull their $j^{th}$ arm and each $m^{th}$ agent will receive a
reward drawn from their distribution $D_{m,j}$ with an unknown mean
$\mu_{m,j}^{*}$ [26]. MA-MABs can be modeled as centralized or distributed. In
centralized settings the agents’ actions are taken by a centralized controller
and in distributed settings each agent will independently choose their own
actions. Distributed decision-making settings scale more effectively [27] and
naturally deals easily with large $K$ set of arms when compared with
centralized settings that suffers of $K$ arms’ cardinality explosion. Finally,
the total regret can be defined as:  R_T = ∑_t=1^T∑_m=1^N E[(max_jμ_m,j^* -
μ_m,j^*)] In this work, we consider two main approaches: distributed non-
cooperative and cooperative MA-MABs with adversarial rewards.

###  III-D Deep Transfer Reinforcement Learning

Transfer learning or knowledge transfer techniques improve learning time
efficiency by utilizing prior knowledge. Typically, this is done by extracting
the knowledge from one or diverse source tasks and then applying such
knowledge in a target task [28]. If the tasks are related in nature and the
target task benefits positively with the acquired knowledge from the source,
then it is called inductive transfer learning [29]. This type of learning is
not uncommon and it is used by the human brain on a daily basis. However, a
phenomena called negative transfer can occur, if after knowledge transfer, the
target task performance is negatively affected [30].

In the realm of transfer learning we can find Deep Transfer Learning (DTL).
DTL is a subset of transfer learning that studies how to utilize knowledge in
deep neural networks. In the context of classification/prediction tasks, large
amount of data is required to properly train the model of interest [31]. In
many practical applications where training time is essential to respond to new
domains [32], retraining using large amount of data is not always feasible and
possibly catastrophic in terms of performance. “What to transfer” corresponds
to one of the main research topics in transfer learning. Specifically, in the
case of deep transfer learning four categories have been broadly identified:
instances-based transfer, where data instances from a source task are
utilized; mapping-based transfer, where a mapping of two tasks is used on a
new target task; network-based transfer, where the network pre-trained model
is transferred to the target task; and adversarial-based transfer, where an
adversarial model is employed to find which features from diverse source tasks
can be transferred to the target task[33].

In this work, we utilize the DTL form called network-based transfer learning
to adapt efficiently TP and CCA parameters in dynamic scenarios. An example of
network-based transfer learning technique is presented in Fig. 2. Such
technique is utilized in deep transfer reinforcement learning as part of a
transfer learning type called policy transfer [34]. In particular, policy
transfer takes a set of source policies $\pi_{S_{1}},...,\pi_{S_{K}}$ that are
trained on a set of source tasks and uses them in a target policy $\pi_{T}$ in
a way that is able to leverage the former knowledge from the source policies
to learn its own. More specifically, the weights and biases that comprise each
of the hidden layers of the source policies are the elements transferred to
the target polices. Note that in practice policies are modeled as neural
networks.

Figure 2: Network-based transfer learning: the neural network source task’s
hidden layers are reutilized in the target network

In this paper, we take advantage of the design of a contextual multi-armed
bandit presented in [25] and apply policy transfer to improve the agent’s SR
adaptability in dynamic environments. The results and observations of applying
DTRL are discussed in section VI-E. In the next section, we will discuss the
details of the system model and present an analysis on reducing the
cardinality of the action space in the proposed SR problem formulation.

##  IV System model and Problem Formulation

TABLE I: Notations Notation |  Definition
---|---
$s$ and $\mathcal{S}$ |  Index and set of stations,
$m$ and $\mathcal{M}$ |  Index and set and the number of [APs] RUs,
$x^{|\mathcal{S}|}$ and $c^{|\mathcal{M}|}$ |  Stations’ positions and AP’s positions
$P_{cs}^{m}$ |  CCA threshold of $m^{th}$ AP,
$P_{tx}^{m}$ |  Transmission Power of $m^{th}$ AP,
$R_{s}^{m}$ |  Throughput of $s^{th}$ STA of $m^{th}$ AP,
$R_{s,A}^{m}$ |  Achievable throughput of $s^{th}$ STA of $m^{th}$ AP,
$D_{s}^{m}$ |  Adaptive data link rate of $s^{th}$ STA of $m^{th}$ AP
$P_{IDLE}^{m}$ |  Probability of a STA is idle in a BSS,
$P_{SUCC,s}^{m}$ |  Probability of succesful transmission by station $s^{th}$ STA to the $m^{th}$ AP,
$\phi_{s}^{m}$ |  Probability of $s^{th}$ STA be transmitting to the $m^{th}$ AP,
$\xi_{CCA}$ |  Binary function, $\xi_{CCA}=1$ if signal is bellow the CCA threshold $P_{cs}$,
$\xi_{ED}$ |  Binary function, $\xi_{ED}=1$ if signal is bellow the Energy Detection (ED) threshold $P_{ed}$,
$\xi_{STA}$ |  Binary function, $\xi_{STA}=1$ if throug is bellow the Energy Detection (ED) threshold $P_{ed}$,
$E(T_{g,s}^{m})$ and $E(I_{g,s}^{m})$ |  Expected length of general time-slot and expected information transmitted by the $s^{th}$ STA of $m^{th}$ AP,
$T_{TXOP}$ and $T_{EDCA}$ |  Packet transmission duration and time required for a successful Enhanced Distributed Channel Access (EDCA) transmission,
$\bar{P}^{fair}$ and $\bar{U}$ |  Average linear product-based network fairness and average station starvation,
$\omega$, $g_{s}^{m}$ and $\sigma^{2}$ |  Fraction of $R_{s,A}^{m}$ in which STAs are consider in starvation, the channel power gain and the power noise.
$P_{tx}^{m}$ and $P_{tx}^{r}$ |  The transmission power at the $m^{th}$ transmitter (AP) and the received signal strength at the $r^{th}$ receiver,
$d_{m,r}$ and $\theta$ |  Distance between the $m^{th}$ transmitter and $r^{th}$ receiver and path loss exponent,
$\mathcal{F}_{m}^{+}$ and $\mathcal{F}_{m}^{-}$ |  Subset of interferers and non-AP interferers,
$\gamma_{m,r}$, $C_{m,r}$ and $C_{T}$ |  Worst-case SINR and Shannon’s maximum capacity of $m^{th}$ transmitter and $r^{th}$ receiver and cumulative maximum network capacity.

In this work, we consider an infrastructure mode Wi-Fi 802.11ax network
$\mathcal{N}$ with $N=|\mathcal{S}|+|\mathcal{M}|$ nodes where $\mathcal{S}$
is the set of stations with
$\\{\bm{x}^{1},\bm{x}^{2},...,\bm{x}^{|\mathcal{S}|}\\}\in\mathbb{R}^{2}$
positions and $\mathcal{M}$ is the set of APs with
$\\{\bm{c}^{1},\bm{c}^{2},...,\bm{c}^{|\mathcal{M}|}\\}\in\mathbb{R}^{2}$
positions. We can assume that $|\mathcal{M}|$ APs positions correspond to
cluster centers and the stations will attach to their closest AP. In addition,
the list of notations utilized in this work can be found in Table I.

In this paper, we improve SR via maximization of the linear product-based
fairness and minimization of the number of stations under starvation by
configuring TP and CCA parameters.

Max $\displaystyle\begin{pmatrix}\text{fairness}\\\ \text{avg. station
starvation complement}\end{pmatrix}$ (2a) s.t. Throughput (2b) var.
Transmission power and CCA threshold selection (2c)

Let’s define the probability of an STA being idle in a BSS as:

$\displaystyle P_{IDLE}^{m}=\prod_{s\in\mathcal{S}}\phi_{s}^{m^{\prime}}$
$\displaystyle\forall m\in\mathcal{M}.$ (3)

where $\phi_{s}^{m}\in[0,1]$ is the probability of an STA transmitting to the
$m^{th}$ AP. In addition, we proceed to define the probability in which an STA
will successfully transmit a packet as:

$\displaystyle
P_{SUCC,s}^{m}=\phi_{s}^{m}\xi_{CCA}^{m}(\cdot)\xi_{ED}^{m}(\cdot)\prod_{s^{\prime}\in\mathcal{S},s^{\prime}\neq
s}^{\mathcal{S}}\phi_{s}^{m^{\prime}}$ $\displaystyle\forall m\in\mathcal{M}.$
(4)

where $\xi_{CCA}(\cdot)=1$ if the sensed signal of a packet sent by the
$s^{th}$ STA is below the CCA threshold ($P_{cs}$), otherwise becomes zero.
Here, $\xi_{ED}(\cdot)=1$ if the sensed signal of packet sent by the $s^{th}$
STA is below the Energy Detection (ED) threshold ($P_{ed}$), otherwise becomes
zero. Additionally, we consider $P_{cs}=P_{ed}$ to simplify our analysis. As
indicated by [35] the expected length of the general time-slot
$\mathbb{E}(T_{g})$ and the expected information transmitted by the $s^{th}$
STA to $m^{th}$ AP $\mathbb{E}(I_{g})$ can be expressed as:

$\displaystyle E(T_{g,s}^{m})=\delta P_{IDLE}^{m}+P_{IDLE}^{m^{\prime}}T$
$\displaystyle\forall m\in\mathcal{M}.$ (5) $\displaystyle
E(I_{g,s}^{m})=P_{SUCC,s}^{m}D_{s}^{m}T_{TXOP}$ $\displaystyle\forall
m\in\mathcal{M},s\in\mathcal{S}.$ (6)

where $D_{s}^{m}$ corresponds to the link data rate, $T_{EDCA}$ corresponds to
the time required for a successful Enhanced Distributed Channel Access (EDCA)
transmission, $T_{TXOP}$ is the transmission duration and $\delta$ the
duration of an idle time slot. The link data rate will adaptively depend on
SNR [36] and mapped based on SNR/BER curves[37]. The received SNR can be
defined as $P_{tx}^{m}g_{s}^{m}\//\sigma^{2}$ where $P_{tx}$ is the
transmission power, $g_{s}^{m}$ the channel power gain and $\sigma^{2}$ the
power noise.

Finally, the throughput of the $s^{th}$ station attached to the $m^{th}$ AP
can be defined as:

$\displaystyle
R_{s}^{m}=\frac{E(I_{g,s}^{m})}{E(T_{g,s}^{m})}=\frac{P_{SUCC,s}^{m}D_{s}^{m}T_{TXOP}}{P_{IDLE}^{m}\delta+P_{IDLE}^{m^{\prime}}T_{EDCA}},$
(7)

Additionally, let’s define the average linear product-based network fairness
and average station starvation in a distributed setting:

$\displaystyle\bar{P}^{fair}(t)=\frac{1}{|\mathcal{M}|}\sum_{m\in\mathcal{M}}\prod_{s\in\mathcal{S}}\frac{R_{s}^{m}}{R_{m,A}^{s}},$
(8)
$\displaystyle\bar{U}(t)=\frac{1}{|\mathcal{M}|}\sum_{m\in\mathcal{M}}\frac{1}{|\mathcal{S}|}\sum_{s\in\mathcal{S}}\xi_{STA}(R_{s}^{m}>\omega
R_{m,A}^{s}),$ (9)

where $R_{m,A}^{s}$ is the achievable throughput of the $s^{th}$ station
attached to the $m^{th}$ AP. Additionally, $\xi_{STA}=1$ if $s^{th}$ station’s
throughput is greater than a fraction $\omega\in(0,1]$ of the achievable
throughput, otherwise becomes zero in which case the station is considered in
starvation. The considered problem is a multi-objective problem and can be
addressed with the weighted sum approach. Thus, in each time step, the problem
can be formulated as follows:

###### Problem 1

$\displaystyle\underset{\mathbf{P_{tx}},\mathbf{P_{cs}}}{\operatorname{max}}\;A_{1}\bar{P}^{fair}(t)+A_{2}(1-\bar{U}(t))$
(10) s.t. $\displaystyle\text{\eqref{thr_eq}},$ (11) $\displaystyle
P_{tx}^{m}\in[P_{tx}^{min},P_{tx}^{max}],P_{cs}^{m}\in[P_{cs}^{min},P_{cs}^{max}]$
$\displaystyle\forall m\in\mathcal{M}$ (12)

Due the dynamic nature of the scenario, the transmission probabilities of the
STAs $\phi_{s}^{m}$ are not directly controllable and require an additional
step to map them to EDCA parameters [35]. Instead, we simplify our analysis by
utilizing a network simulator to model such dynamics and propose to solve the
previous linear programming (LP) problem using a MA-MAB solution as described
in section V.

###  IV-A Optimal action set via worst-case interference

Wi-Fi typical scenarios consist in APs and stations distributed non-uniformly.
Contrary to the analysis presented in [38] we aim obtaining an optimal subset
of TP and CCA threshold values to further reduce the action space size in SR
problems. In this analysis, we only consider the Carrier Sense (CS) threshold
term as form of the CCA threshold.

First, let’s consider the worst-case interference scenario in a $N>2$
arrangement. For the sake of simplicity we use the path-loss radio propagation
model:  P_rx^r = Ptxmdm,rθ, where $P_{tx}^{m}$ and $P_{rx}^{r}$ are the TP at
the $m^{th}$ transmitter (AP) and the received signal strength at the $r^{th}$
receiver, respectively. In addition, $d_{m,r}$ is the distance between the
transmitter and receiver. Finally, $\theta\in[2,4]$ corresponds to the path
loss exponent. Thus, from the perspective of $m^{th}$ AP the worst-case
interference $I_{m}$ is defined as:  I_m = ∑_v ∈F_m^+ PtxvX(m,v)θ \+
P_tx^sta∑_w∈F_m^- 1X(m,w)θ,

where $\mathcal{F}_{m}^{+}$ is the subset of interferers
$|\mathcal{F}_{m}^{+}|=|\mathcal{M}|-1$, corresponding to APs interfering with
the $m^{th}$ AP and $\mathcal{F}_{m}^{-}$ the subset of non-AP interferers
$|\mathcal{F}_{m}^{-}|=|\mathcal{S}|$, corresponding to the stations
interfering with the $m^{th}$ AP. Furthermore, $P_{tx}^{v}$ is the TP of the
$v^{th}$ interferer and $P_{tx}^{sta}$ is a constant corresponding to the
fixed power assigned to all the stations based on the fact that typically
stations are not capable to modify their TP. Additionally, $X^{(m,v)}$ and
$X^{(m,w)}$ corresponds to the distance from the $m^{th}$ AP to the $v^{th}$
AP interferer and $m^{th}$ AP to the $v^{th}$ station interferer,
respectively. $X^{(m,.)}$ is calculated as follows:  X^(m,.) = (D_m+x_m,.)^2 +
d_m,r^2 - 2(D_m+x_m,.)d_m,rcosς_r,., where $(.)$ refers either to the AP or
non-AP interferer, $D_{m}$ is the CCA threshold range of the $m^{th}$ AP,
$\varsigma_{r,.}$ is the distance between the receiver to the interferer $(.)$
and $x_{m,.}$ corresponds to the distance between any $(.)$ interferer and
$D_{m}$.

The corresponding worst-case SINR $\gamma_{m,r}$ at the receiver is defined
as:

$\gamma_{m,r}=\frac{P_{tx}^{m}}{{d_{m,r}}^{\theta}(I_{m}+N_{0})},$ (13)

Let’s assume that $N_{0}<<I_{m}$, thus the equation is reduced to:

$\gamma_{m,r}=\frac{P_{tx}^{m}}{{d_{m,r}}^{\theta}I_{m}},$ (14)

Substituting equations (IV-A) and (IV-A) in (14) we obtain equation (15):

$\displaystyle\gamma_{m,r}=\frac{\frac{P_{tx}^{m}}{{d_{m,r}}^{\theta}}}{\sum_{v\in\mathcal{F}_{m}^{+}}\frac{P_{tx}^{m}}{({\sqrt{(D_{m}+x_{m,v})^{2}+{d_{m,r}}^{2}-2(D_{m}+x_{m,v})d_{m,r}\cos\varsigma_{r,.}}})^{\theta}}+P_{tx}^{sta}\sum_{w\in\mathcal{F}_{m}^{-}}\frac{1}{({\sqrt{(D_{m}+x_{m,w})^{2}+d_{m,r}^{2}-2(D_{m}+x_{m,w})d_{m,r}\cos\varsigma_{r,w}}})^{\theta}}}$
(15)

The aforementioned equation describes $\gamma_{m,r}$ in function of $D_{m}$
and $d_{m,r}$. Additionally, we substitute
$D_{m}=\left(P_{tx}^{m}/T_{cs}^{m}\right)^{1/\theta}$ in equation (15),
obtaining:

γ_m,r=Ptxmdm,rθ∑v∈Fm+PtxmΓm\+ Ptxsta∑w=1Km-ι(m,w),

where,

$\scalebox{0.7}{$\Gamma^{m}=\left({\sqrt{\left[\left(\frac{P_{tx}^{m}}{T_{cs}^{m}}\right)^{\frac{1}{\theta}}+x_{m,v}\right]^{2}+d_{m,r}^{2}-2\left[\left(\frac{P_{tx}^{m}}{T_{cs}^{m}}\right)^{\frac{1}{\theta}}+x_{m,v}\right]d_{m,r}\cos\varsigma_{r,v}}}\right)^{\theta}$},$

$\iota^{(m,w)}=\frac{1}{{(\sqrt{(\Omega_{sta}+x_{m,w})^{2}+d_{m,r}^{2}-2((\Omega_{sta}+x_{m,w})d_{m,r}\cos\varsigma_{r,w}}})^{\theta}}$
and
$\Omega_{sta}=\left(\frac{P_{tx}^{sta}}{T_{cs}^{sta}}\right)^{\frac{1}{\theta}}$.

Now, we proceed to define the maximum channel capacity in terms of TP and
Carrier Sense (CS) threshold ($T_{cs}$). Given a certain value of SINR, the
Shannon maximum capacity is expressed as:  C_m,r = Wlog_2(1 + γ_m,r),

where $W$ is the channel bandwidth in Hz. Then, the cumulative maximum network
capacity can be calculated as:

C_T = ∑_m=1^—M—-1∑_r=1^N C_m,r,

Figure 3: Network capacity as a function of TP and CS threshold.

In figure 3, it is shown a graph of the network maximum capacity as a function
of TP and CS threshold. As observed, the network capacity achieves its higher
values when a combination of high TP and low CS threshold is utilized. Note
that, prior knowledge of the locations are required.

##  V Proposed Multi-Agent Multi-armed bandit algorithms

In this section, we present the action space, context definition and reward
function for the MA-MAB algorithms utilized in this work.

###  V-A Action space

The action space corresponds to the number of combinations of $P_{cs}$ and
$P_{tx}$ which in the context of MABs translates to the number of arms for
each MAB agent. The action space is defined as:  A_cs = {P_cs^min, P_cs^min +
Pcsmax\- PcsminLcs-1,…, P_cs^max },

A_tx = {P_tx^min, P_tx^min + Ptxmax\- PtxminLtx-1,…, P_cs^max}, where
$P_{cs}^{min}$, $P_{cs}^{max}$ and $P_{tx}^{min}$, $P_{tx}^{max}$ are the
minimum and maximum values of CCA threshold and TP values, respectively.
$L_{cs}$ and $L_{tx}$ corresponds to the number of levels to be discretized
the CCA threshold and TP values, respectively. Finally, the number of arms
corresponding to the action space for the $m^{th}$ agent $K_{m}^{AP}$ is
$|A_{cs}^{m}|\cdot|A_{tx}^{m}|$.

###  V-B Reward function in distributed non-cooperative settings

The reward is defined following the optimization problem 1. The reward
resembles the reward presented in [14] which includes a linear product-based
fairness and station’s starvation term [17, 14] but defined in a distributed
manner. A station is considered to be on starvation when its performance is
bellow to a predefined percentage of its theoretical achievable throughput.
The reward is defined as:

$\leavevmode\resizebox{390.25534pt}{}{$r_{m}^{AP}=\frac{|\Psi_{m}^{AP}|\prod_{j\in\Psi_{m}^{AP}}\frac{R_{m}^{s}}{\omega
R_{m,A}^{s}}+|N_{m}^{AP}\setminus\Psi_{m}^{AP}|(N_{m}^{AP}+\prod_{j\in
N_{m}^{AP}\setminus\Psi_{m}^{AP}}\frac{R_{m}^{s}}{R_{m,A}^{s}})}{N_{m}^{AP}(N_{m}^{AP}+1)}$},$
(16)

where $\Psi_{m}^{AP}$ is the set of starving stations attached to the $m^{th}$
AP , $N_{m}^{AP}$ the set of stations attached to the $m^{th}$ AP. We can also
observe, that $r_{m}^{AP}\propto C_{m,r}$ as defined in Eq. IV-A.

In the next subsection, we present the definition of the context considered in
our MA-CMAB solution.

###  V-C Distributed Sample Average Uncertainty-Sampling MA-CMAB

In [25], the authors present an efficient contextual multi-arm bandit based on
a “frequentist approach” to compute the uncertainty instead of using bayesian
solutions as Thompson Sampling. The frequentist approach consist in measuring
the uncertainty of the action-values based on the sample average rewards just
computed instead of relaying on the posterior distribution given the past
rewards. In this work, we present multi-agent cooperative and not cooperative
variants of the previously mentioned RL algorithm.

In our problem, the context is comprised only by the APs’ local observations:

  1. 1.

Number of starving stations, $|\Psi_{m}^{AP}|$ where $m$ corresponds to the
$m^{th}$ AP under $\omega$ fraction of their attainable throughput during the
$t$ episode.

  2. 2.

Average RSSI, $\overline{S}_{m}^{AP}$ where $m$ is the $m^{th}$ AP during the
$t$ episode.

  3. 3.

Average Noise, $\overline{\Upsilon}_{m}^{AP}$ where $m$ denotes the $m^{th}$
AP during the $t$ episode.

Additionally the context is normalized as follows:  ψ_m^AP = —Ψ_m^AP—/N_m^AP,

s_m^AP ={0, -50  dBm ≤¯SmAP≤-60  dBm,0.25, -60  dBm ≤¯SmAP≤-70  dBm,0.5, -70
dBm ≤¯SmAP≤-80  dBm,0.75, -80  dBm ≤¯SmAP≤-90  dBm,1, -90  dBm ≥¯SmAP

^Υ

_m^AP = ¯Υ_m^AP/100,

The multi-agent SAU-Sampling algorithm in its non-cooperative version (SAU-
NonCoop) is described in Algorithm 1.The algorithm starts by initializing
action-value functions $\mu(\bm{x}_{m}|\bm{\hat{\theta}}_{m})$ as a deep
neural networks and the exploration parameters $J_{m,a}^{2}$ and $n_{m,a}$ for
each $m^{th}$ AP. $n_{m,a}$ correspond to the number of times action $a$ was
selected in the $m^{th}$ AP and $J_{m,a}^{2}$ is defined as an exploration
bonus. In each environment step (Algorithm 1, line 2), each agent will observe
their local context and compute the selected arm given the reward prediction.
In (Algorithm 1, line 11) each CMAB agent will update
$\bm{\hat{\theta}}_{m,a}$ using stochastic gradient descent on the loss
between the predicted reward and the real observed reward. Finally, the
exploration parameters are accordingly updated given the the prediction error
as depicted in (Algorithm 1, line 12).

1

2Initialize network $\bm{\hat{\theta}}_{m,a}$, exploration parameters
$J_{m,a}^{2}(t=0)=1$ and $n_{m,a}(t=0)=0$ for all actions $a\in K_{m}$.

3for  _environment step $t\leftarrow 1$ to $T$_ do

4 for  _agent $m$_ do

5 Observe context
${\bm{x}_{m}(t)}=[\psi_{m}^{AP}(t),s_{m}^{AP}(t),\hat{\Upsilon}_{m}^{AP}(t)]$

6 for  _$a=1,...,K_{m}$_ do

7 Calculate reward prediction
$\hat{\mu}_{i,t}(t)=\mu(\bm{x}_{m}|\bm{\hat{\theta}}_{m})$ and
$\tau_{m,a}^{2}(t)=J_{m,a}^{2}/n_{m,a}$

8
$\tilde{\mu}_{m,a}\sim\mathcal{N}(\hat{\mu}_{m,a},n_{m,a}^{1-}\tau_{m,a}^{2})$

9 end for

10 Compute $a_{m}(t)=\text{{argmax}}_{a}(\\{\tilde{\mu}_{m,a}(t)\\}_{a\in
K_{m}}\\})$ if $t>K_{m}$, otherwise $a_{m}(t)\sim\mathcal{U}(0,K)$;

11 Select action $a_{m}(t)$, observe reward $r_{m}^{AP}$;

12 Update $\bm{\hat{\theta}}_{m,a}$ using SGD with gradients $\partial
l_{m}/\partial\theta$ where $l_{m}=0.5(r_{m}^{AP}-\hat{\mu}_{m,a}(t))$ ;

13 Update $J_{m,a}^{2}\leftarrow J_{m,a}^{2}+e_{m}^{2}$ using prediction error
$e_{m}=r_{m}^{AP}(t)-\hat{\mu}_{m,a}(t)$ and $n_{m,a}\leftarrow n_{m,a}+1$;

14 end for

15

16 end for

17

Algorithm 1 SAU-Sampling MA-CMAB

###  V-D Cooperative Sample Average Uncertainty-Sampling MA-CMAB

In this section we present a cooperative version of SAU-Sampling named SAU-
Coop. Different from the non-cooperative version, the total reward $r_{m}^{C}$
considers the network Jain’s fairness index in addition to their local reward
$r_{m}^{AP}$ as:

r_m^C = r_m^AP + r_J,

where $r_{\mathcal{J}}$ as the overall network Jain’s fairness index is
defined as:  r_J = J(R_1,…,R_N) = (∑m=1—M—Rm)2—M—⋅∑m=1—M—Rm2, where
$R_{m}=\sum_{s=1}^{|\mathcal{S}_{m}|}R_{s}^{m}$ is the total throughput of all
the $S_{m}$ stations of the $m^{th}$ AP.

###  V-E Reward-cooperative $\epsilon$-greedy MA-MAB

In addition to the previous cooperative algorithm, we propose a cooperative
approach based on the classical $\epsilon$-greedy strategy [22] that takes
into account in the action’s reward update a percentage of the average reward
of other agents. This algorithm is described in Algorithm 2.

1 Initialize $\epsilon_{m}(t=0)=\epsilon_{0}$, $Q_{m,a}(t=0)\leftarrow 0$,
$N_{m,a}(t=0)\leftarrow 0$ and $\beta$.

2for  _environment step $t\leftarrow 1$ to $T$_ do

3 for  _agent $m$_ do

4 Execute action $a_{m}(t)$:
$a_{m}(t)=\begin{cases}\text{{argmax}}_{k=1,...,K}r_{k,i}(t)&\text{with
probability}1-\epsilon_{m}(t)\\\ k\sim\mathcal{U}(0,K)&\text{o.w}\end{cases}$

5 Calculate reward $r_{m}^{AP}(t)$ based on feedback of the environment

6 Update
$Q_{m,a}(t+1)=Q_{m,a}(t)+\frac{1}{N_{m}(t)}[(r_{m}^{AP}+\beta\cdot\frac{1}{M-1}\sum_{m=1}^{M^{-i}}r_{m}^{AP})-Q_{m,a}(t)]$

7 Update $N_{m}\leftarrow N_{m}(t)+1$;

8 Update $\epsilon_{m}\leftarrow\frac{\epsilon_{m}(t)}{\sqrt{t}}$

9 end for

10

11 end for

12

Algorithm 2 Reward-cooperative $\epsilon$-greedy MA-MAB

Finally, in the next subsection we present the details of the the DTRL scheme
to improve SR adaptation in dynamic environments.

###  V-F Sample Average Uncertainty-Sampling MA-CMAB based Deep Transfer
Reinforcement Learning

Typically, RL agents learn their best policy based on the feedback received
from the environment in a $T$ horizon time. However, in real-world scenarios
the environment conditions can change in $T+1$ and thus, adapting to the
updated environment is necessary[39]. In such cases, the “outdated” agent’s
policy might not be optimal to address the new conditions efficiently. For
instance, a modification on the stations’ distribution over the APs can cause
that the SR-related parameters chosen by the “outdated” agents’ policy affect
the network performance.

Function Detect_Singularity($\mathcal{K}$) ;

// returns True if anomaly is detected in network KPIs data $\mathcal{K}$ at
time $t$, and False otherwise.

1 Let $\mathcal{L}=\\{l|l\in\mathbb{N},l>0\\}$ the set of layers of model
$\bm{\hat{\theta}}_{m,a}^{l}$ and $\mathcal{M}\subset\mathcal{L}$ the subset
of layers to be transferred. Run algorithm SAU-Sampling MA-CMAB (Algorithm 1)
while  _environment step $t<T$_ do

2

3 if  _$\neg$ Detect_Singularity_ then

4 continue;

5 else

6 Reset exploration parameters $S_{m,a}^{2},n_{m,a}$;

7

8 Reinitialize weights $w$ and biases $b$ of the $l^{th}$ layer of
$\bm{\hat{\theta}}_{m,a}^{l\not\in\mathcal{M}}$ via:

9
$\nu_{l}=\left(\sqrt{|\bm{\hat{\theta}}_{m,a}^{l\not\in\mathcal{M}}|}\right)^{-1}$
;

10

11 $\bm{\hat{\theta}}_{m,a}^{l\not\in\mathcal{M}}(w,b)\rightarrow
w_{l}\sim\mathcal{U}(-\nu_{l},\nu_{l}),b_{l}\sim\mathcal{U}(-\nu_{l},\nu_{l})$;

12 Transfer weights and biases via:

13
$\bm{\hat{\theta}}_{m,a}^{l\in\mathcal{M}}(w,b)\rightarrow\bm{\hat{\theta}}_{m,a}^{l\in\mathcal{M}^{\prime}}(w,b)$;

14

15 end if

16

17 end while

18

19

20

Algorithm 3 SAU-Sampling MA-MAB Transfer Learning

To address the previous situation we propose two main solutions: 1. If the
agent detects a change in the environment indicated by a singularity, it will
decide to correct its configuration via forgetting the policy already learnt
(forget) or 2. adapting the agent’s policy to the new conditions via a
transfer learning technique. A singularity is defined as a anomalous behavior
of the KPIs of interest after the policy of the MAB agent has converged. In
this work, we don’t delve into how to detect a singularity and moreover, we
assume the existence of an anomaly detector in our system [40]. In Algorithm
3, we present the transfer learning algorithm depicting the second proposed
solution. At $t=0$ each SAU-Sampling agent will reset their weights and biases
and start learning as part of Algorithm 1. At $t=S1$, where $S1$ corresponds
to the time when an anomaly is detected and the transfer procedure is
activated (Algorithm 3, line 7). In our setup we transfer $l=2$ and reset
$l=1$ (Algorithm 3, line 11) , where $l$ corresponds to the layer of the
neural network utilized in the SAU-Sampling agent. However, as indicated
(Algorithm 3, line 13), the transfer is not constrained to one layer but more
generally to a set of layers. The set of transferred layers is considered as
an hyperparameter to be tuned. The partial transfer of a model avoids negative
transfer by giving the agent room to adapt to the new context since it
mitigates model overfitting.

##  VI Performance Evaluation

###  VI-A Simulation Setting

We consider two scenarios in our simulations. The first one considers
stationary users, meanwhile the second scenario considers mobile users to
model dynamic scenarios (see section VI-E). In addition, stations and APs are
two-antenna devices supporting up to two spatial streams in transmission and
reception. In this work, we assume a frequency of 5 GHz with a 80 MHz channel
bandwidth in a Line of Sight (LOS) setting. The propagation loss model is the
Log Distance propagation loss model with a constant speed propagation delay.
In addition, an adaptive rate data mode is considered with a UDP downlink
traffic. We implement our proposed solutions using ns-3 and also we use OpenAI
Gym to interface between ns-3 and the MA-MAB solution[41]. In Table II and
Table III we present the learning hyperparameters and network settings
parameters, respectively.

TABLE II: Learning hyperparameters

Parameter | Value
---|---
$\epsilon$-greedy MAB |  Annealing $\epsilon$: $\sqrt{T}$
Thompson Sampling MAB | Prior distribution: Beta
Upper Confidence Bound MAB |  Level of exploration, $c=1$
SAU-Sampling |  Number of hidden layers, $N_{h}=2$
|  Number of neurons per hidden layer, $n_{h}=100$
|  Number of inputs, $N_{m}=3$ and number of outputs, $N_{o}=K$
|  Batch size, $B_{s}=64$
| Optimizer : RMSProp (8e-3)
| Weight decay : 5e-4
| Activation function : ReLU
Gym environment step time | 0.05 s

TABLE III: Network settings

Parameter | Value
---|---
Number of APs | 6
Number of Stations | 15
Number of antennas (AP) | 2
Max Supported Tx Spatial Streams | 2
Max Supported Tx Spatial Streams | 2
Channel Number 222We assume all APs are configured to use 1 channel out of the available 11. This is a practical selection to create dense deployment scenarios. | 1
Propagation Loss Model | Log Distance Propagation Loss Model
Wi-Fi standard | 802.11 ax
Frequency | 5 GHz
Channel Bandwidth | 80 MHz
Traffic Model - UDP application |  $[0.011,0.056,0.11\text{\cite[cite]{[\@@bibref{}{WILHELMI201926}{}{}]}},0.16]$ Gbps
Maximum $\&$ minimum Transmission Power |  $P_{tx}^{max}=21.0$ dBm $\&$ $P_{tx}^{min}=1.0$ dBm
Maximum $\&$ minimum CCA threshold |  $P_{cs}^{max}=-62.0$ dbm $\&$ $P_{cs}^{min}=-82.0$ dbm
|  $K_{cs}=1$ and $K_{tx}=1$

###  VI-B Reduced set of actions vs. all actions

In subsection IV-A we presented a mathematical analysis to obtain a reduced
set of optimal actions with the goal of decreasing exploration time and
consequently improving convergence time. As concluded in figure 3, high TP and
low CCA threshold values maximize the network capacity in the simulation
scenario under study. Therefore, we selected a fixed value of CCA threshold
($P_{cs}=-82.0$ dBm) and a reduced set of TP
$P_{tx}\in\\{15,16,17,18,19,20,21\\}$ dBm and observed the performance against
the full set of possible actions described in V-A.

Figure 4: Convergence performance of $\epsilon$-greedy, UCB and Thompson
Sampling MA-MABs under non-cooperative and distributed regimen. The subscript
“all” indicates the usage of the full set of actions.

In figure 4, we present the convergence performance of three MA-MAB algorithms
under UDP traffic of 0.056 Gbps in a non-cooperative and cooperative settings
(indicated with subscripts “non-coop” and “coop”, respectively ). The
algorithms correspond to $\epsilon$-greedy ($MAB^{eg}$), UCB ($MAB^{ucb}$) and
Thompson Sampling ($MAB^{thom}$) MA-MABs. For each algorithm, we plotted three
convergence graphs in terms of fairness, cumulative throughput and station
starvation representing the behavior when a reduced set of actions and the
full action set (indicated with the subscript “all”) are used, respectively.
For the case of the set of optimal actions, we can observe that the
performance is similar with a slight improvement when utilizing MAB-Thompson
Sampling. On the other hand, when utilizing the full action set the behavior
shows a noticeable improvement with MAB $\epsilon$-greedy algorithm with
respect the others. In [43], the authors study the unreasonable behavior of
greedy algorithms when $K$ is sufficiently large. They concluded that when $K$
increases above 27 arms, intelligent algorithms are affected greatly by the
exploration stage. The former results validate ours based on the fact that
$K=|A_{cs}|\cdot|A_{tx}|=21^{2}$. Finally, it can be noted that the impact of
utilizing reduced optimal actions in terms of convergence time and KPI
maximization. The set of optimal tasks allows to reduce the station starvation
when compared with the best performer $MAB_{nocoop_{all}}^{eg}$ by an average
of two starving users. However, in order to obtain such a set it is requires a
prior knowledge of stations and APs geographical locations. In the following
section we compare the results of $\epsilon$-greedy MA-MAB and a default
typical configuration without machine learning.

Figure 5: Performance results: $\epsilon$-greedy MAB w/ optimal set vs.
default configuration with $P_{cs}\in\\{-62.0,-82.0\\}$ dBm.

###  VI-C Distributed $\epsilon$-greedy MA-MAB vs. default configuration
performance results

In this subsection, we present the comparative results and advantages of
utilizing a distributed intelligent solution such as MAB $\epsilon$-greedy
over the default CCA threshold and TP configuration with no ML. In figure 5,
we show the performance under four different UDP data traffic regimes:
$\\{0.011,0.056,0.11,0.16\\}$ Gbps. We considered two typical configurations
of CCA threshold: $-82.0$ dBm and $-62.0$ dBm. In both cases, the AP’s TP is
$16.0$ dBm. It can be observed that MAB $\epsilon$-greedy achieves a
significant improvement over the default configuration ($P_{cs}=-82.0$ dBm)
with an average gain over all the considered traffic of $44.4\%$ in terms of
cumulative throughput, $70.9\%$ in terms of station starvation, $12.2\%$ in
terms of fairness, $138.0\%$ in terms of latency and $94.5\%$ in terms of
packet loss ratio (PLR), respectively. Additionally, a gain over the default
configuration ($P_{cs}=-62.0$ dBm) with an average gain over all the
considered traffics of $53.9\%$ in terms of cumulative throughput, $138.4\%$
in terms of station starvation, $43.0\%$ in terms of fairness, $84.0\%$ in
terms of latency and $105.4\%$ in terms of packet loss ratio (PLR) is shown,
respectively.

Figure 6: Performance results of cooperative algorithms: $\epsilon$-greedy MA-
MAB (Rew-Coop), SAU-Sampling MA-CMAB (SAU-Coop) and non-cooperative versions
of the previous algorithms SAU-NonCoop and Eg-NonCoop under full-set of
actions.

###  VI-D Cooperation vs. non-cooperation performance results

In the two past subsections we have shown the results considering the set of
optimal actions. In this subsection we assume the non-existence of stations
and APs location information and thus, we must rely on the full set of
actions. In consequence, we investigate if cooperation can improve the KPIs of
interest by utilizing the cooperative proposal of the MAB $\epsilon$-greedy
algorithm (Rew-Coop) and the contextual SAU-Sampling algorithm (SAU-Coop).
Additionally, we present two non-cooperative algorithms: SAU-NonCoop which
corresponds to the non-cooperative version of the SAU-Sampling and Eg-NonCoop
that refers to the MAB $\epsilon$-greedy algorithm utilized in the previous
section. As observed in figure 6, simulations show that SAU-Coop improves Eg-
NonCoop over all the data traffic with an average of $14.7\%$ in terms of
cumulative throughput, $21.3\%$ in terms of station starvation, $4.64\%$ in
terms of network fairness, $36.7\%$ in terms of latency and $32.5\%$ in terms
of PLR. Similarly, the distributed version of SAU-Sampling presents a better
performance over Eg-NonCoop, indicating that context is beneficial to solve
the current optimization problem. Additionally, SAU-Coop presents a better
performance over its non-cooperative version, specially when the data rate
increases up to 0.16 Gbps where it is observed a gain of $14.1\%$ in terms of
cumulative throughput, $32.1\%$ in terms of station starvation, $18.2\%$ in
terms of network fairness, $16.5\%$ in terms of latency and $4\%$ in terms of
PLR. To sum up, cooperative approaches contribute positively to the
improvement of SR in WiFi over non-cooperative approaches. In addition, in
cases where cooperation is not possible it is advisable to utilize contextual
multi-armed bandits over stateless multi-armed bandits.

###  VI-E Deep Transfer Learning in Adaptive SR in Dynamic scenarios results

In order to model a dynamic scenario, we design a simulation where the users
move across the simulation area and attach to the AP that offers the best
signal quality. Consequently, the user load in each AP will change and thus,
the dynamics of the environment. We model this scenario with 3 APs and 15
users where the load will change twice throughout the simulation. As depicted
in table IV the user load of the $m^{th}$ AP denoted as $C_{m}$ will change in
two instances in time: 3 and 6 minutes, respectively.

TABLE IV: Dynamic scenario load distribution |  $\bm{t=0}$ min |  $\bm{t=3}$ min |  $\bm{t=6}$ min
---|---|---|---
$C_{1}$ | 8 | 5 | 2
$C_{2}$ | 5 | 5 | 2
$C_{3}$ | 2 | 5 | 11
Figure 7: Network response in terms of fairness and station starvation when
utilizing the forget, full transfer and transfer strategies.

In figure 7 we present the network behavior in terms of fairness and station
starvation under the scenario depicted by Table IV. In addition to the two
methods previously mentioned: forget and transfer, we present the performance
of a third approach called full transfer where the full transfer of the model
is considered. During the first interval ($0-3$min) the performance is similar
in the three methods as expected. However, after the two changes on the
network load, two singularities in each graph are visible in the fairness and
starvation graphs. More specifically, the forget method experiences the worst
behavior, with a $54.3\%$ and $11.7\%$ decrease when compared with the
transfer method in terms of station starvation and fairness, respectively. The
forget method shows some peaks at the moment of the singularities representing
$60\%$ of total of the users with a service drop; this behavior is inherently
related to the agents’ process of start learning again and cannot be avoided.
From the quality of service perspective, a disturbance such as the one
observed is highly non-preferable. Meanwhile, the full transfer method
underperforms the transfer method with $18.7\%$ and $6\%$ decrease in the
previously mentioned KPIs. Interestingly, it can be observed in the second
interval under study ($3-6$min) the forget method is able to overperform at
the end of the period the full transfer method. This is due to a negative
transfer as a result of transferring the whole model. As observed, not only
the partial transfer learning reduces considerably the peaks in performance of
the forget method but also it is able to achieve better adaptation over the
full transfer method. In all methods, the cumulative throughput is similar,
however as observed in figure 7 station starvation and consequently, fairness
are affected.

##  VII Conclusion

In this paper, we propose Machine Learning (ML)-based solutions to the Spatial
Reuse (SR) problem in distributed Wi-Fi 802.11ax scenarios. We presented a
solution to reduce the huge action space given the possible values of
Transmission Power (TP) and Clear-Channel-Assessment (CCA) threshold values
per Access Point (AP) and analysed its impact on diverse well-known
distributed Multi-Agent Multi-Armed Bandit (MA-MAB) implementations. In
distributed scenarios, we showed that $\epsilon$-greedy MA-MAB significantly
improves the performance over typical configurations when the optimal actions
are known. Moreover, the Contextual Multi-Agent Multi-Armed (MA-CMAB) named
SAU-Sampling in the cooperative setting contributes positively to an increase
in throughput and fairness and reduction of PLR when compared with no
cooperation approaches. Under a dynamic scenarios, transfer learning benefits
the SAU-Sampling algorithm to overcome the service drops for at least $60\%$
of the total of users when utilizing the forget method. Additionally, we
obtained that partial transfer learning offers better results than the full
transfer method. To conclude, the utilization of the cooperative version of
the MA-CMAB to improve SR in WiFi scenarios is preferable since it outperforms
the presented ML-based solutions and prevents service drops in dynamic
environments via transfer learning.

##  VIII Acknowledgment

This research is supported by Mitacs Accelerate Program and NetExperience Inc.

## References

  * [1] Cisco Systems Inc., “Cisco Annual Internet Report (2018–2023),” _Whitepaper, Cisco public_ , 2020.
  * [2] “IEEE Standard for Information Technology–Telecommunications and Information Exchange between Systems Local and Metropolitan Area Networks–Specific Requirements Part 11: Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY) Specifications Amendment 1: Enhancements for High-Efficiency WLAN,” _IEEE Std 802.11ax-2021 (Amendment to IEEE Std 802.11-2020)_ , pp. 1–767, 2021.
  * [3] Cisco Systems Inc., “IEEE 802.11ax : The Sixth Generation of Wi-Fi,” _Whitepaper, Cisco public_ , 2020.
  * [4] F. Ye, S. Yi, and B. Sikdar, “Improving Spatial Reuse of IEEE 802.11 Based Ad Hoc Networks,” in _GLOBECOM - IEEE Global Telecommunications Conference_ , 2003.
  * [5] F. Wilhelmi, S. Barrachina-Muñoz, C. Cano, I. Selinis, and B. Bellalta, “Spatial Reuse in IEEE 802.11ax WLANs,” _Computer Communications_ , 2021\.
  * [6] C. Thorpe and L. Murphy, “A survey of adaptive carrier sensing mechanisms for IEEE 802.11 wireless networks,” _IEEE Communications Surveys and Tutorials_ , 2014.
  * [7] T. Huehn and C. Sengul, “Practical power and rate control for WiFi,” _2012 21st International Conference on Computer Communications and Networks_ , 2012.
  * [8] C. Khosla and B. S. Saini, “Enhancing Performance of Deep Learning Models with different Data Augmentation Techniques: A Survey,” in _Proceedings of International Conference on Intelligent Engineering and Management, ICIEM 2020_ , 2020.
  * [9] S. Szott, K. Kosek-Szott, P. Gawłowicz, J. T. Gómez, B. Bellalta, A. Zubow, and F. Dressler, “WiFi Meets ML: A Survey on Improving IEEE 802.11 Performance with Machine Learning,” _arXiv preprint arXiv:2109.04786_ , 2021.
  * [10] M. Elsayed and M. Erol-Kantarci, “AI-Enabled Future Wireless Networks: Challenges, Opportunities, and Open Issues,” _IEEE Vehicular Technology Magazine_ , vol. 14, no. 3, pp. 70–77, 2019.
  * [11] P. E. Iturria-Rivera, H. Zhang, H. Zhou, S. Mollahasani, and M. Erol-Kantarci, “Multi-Agent Team Learning in Virtualized Open Radio Access Networks (O-RAN),” _Sensors_ , vol. 22, no. 14, 2022.
  * [12] TIP, “OpenWiFi Release 2.4 GA,” 2022. [Online]. Available: https://openwifi.tip.build/
  * [13] F. Wilhelmi, C. Cano, G. Neu, B. Bellalta, A. Jonsson, and S. Barrachina-Muñoz, “Collaborative Spatial Reuse in wireless networks via selfish Multi-Armed Bandits,” _Ad Hoc Networks_ , 2019.
  * [14] A. Bardou, T. Begin, and A. Busson, “Improving the Spatial Reuse in IEEE 802.11ax WLANs,” _ACM International Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems (MSWiM’21)_ , 2021.
  * [15] A. Bardou and T. Begin, “Inspire: Distributed bayesian optimization for improving spatial reuse in dense wlans,” _arXiv preprint arXiv:2204.10184_ , 2022.
  * [16] H. Lee, H.-S. Kim, and S. Bahk, “Lsr: Link-aware spatial reuse in ieee 802.11ax wlans,” in _2021 IEEE Wireless Communications and Networking Conference (WCNC)_ , 2021, pp. 1–6.
  * [17] H. Kim and J. So, “Improving Spatial Reuse of Wireless LAN Uplink Using BSS Color and Proximity Information,” _Applied Sciences_ , vol. 11, no. 22, 2021\.
  * [18] F. Wilhelmi, J. Hribar, S. F. Yilmaz, E. Ozfatura, K. Ozfatura, O. Yildiz, D. Gündüz, H. Chen, X. Ye, L. You _et al._ , “Federated spatial reuse optimization in next-generation decentralized ieee 802.11 wlans,” _arXiv preprint arXiv:2203.10472_ , 2022.
  * [19] D. Bouneffouf, I. Rish, and C. Aggarwal, “Survey on Applications of Multi-Armed and Contextual Bandits,” in _2020 IEEE Congress on Evolutionary Computation, CEC 2020 - Conference Proceedings_ , 2020.
  * [20] A. Slivkins, “Introduction to multi-armed bandits,” _Foundations and Trends in Machine Learning_ , 2019.
  * [21] L. Zhou, “A survey on contextual multi-armed bandits,” _arXiv preprint arXiv:1508.03326_ , 2015.
  * [22] R. S. Sutton and A. G. Barto, _Reinforcement learning: An introduction_ , 2nd ed. MIT press, 2018.
  * [23] R. Agrawal, “ Sample mean based index policies by O (log n ) regret for the multi-armed bandit problem ,” _Advances in Applied Probability_ , 1995.
  * [24] D. J. Russo, B. Van Roy, A. Kazerouni, I. Osband, and Z. Wen, “A Tutorial on Thompson Sampling,” 2018.
  * [25] R. Zhu and M. Rigotti, “Deep bandits show-off: Simple and efficient exploration with deep networks,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [26] S. Hossain, E. Micha, and N. Shah, “Fair Algorithms for Multi-Agent Multi-Armed Bandits,” in _Advances in Neural Information Processing Systems_ , vol. 34, 2021, pp. 24 005–24 017.
  * [27] P. C. Landgren, “Distributed Multi-Agent Multi-Armed Bandits,” Ph.D. dissertation, 2019.
  * [28] S. J. Pan and Q. Yang, “A survey on transfer learning,” _IEEE Trans. Knowl. Data Eng._ , vol. 22, no. 10, pp. 1345–1359, 2010.
  * [29] T. Scott, K. Ridgeway, and M. C. Mozer, “Adapted Deep Embeddings: A Synthesis of Methods for k-Shot Inductive Transfer Learning,” in _Advances in Neural Information Processing Systems_ , vol. 31, 2018.
  * [30] F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, and Q. He, “A comprehensive survey on transfer learning,” _Proc. IEEE_ , vol. 109, no. 1, pp. 43–76, 2021.
  * [31] L. Vu, Q. U. Nguyen, D. N. Nguyen, D. T. Hoang, and E. Dutkiewicz, “Deep transfer learning for iot attack detection,” _IEEE Access_ , vol. 8, pp. 107 335–107 344, 2020.
  * [32] M. Elsayed, M. Erol-Kantarci, and H. Yanikomeroglu, “Transfer Reinforcement Learning for 5G New Radio mmWave Networks,” _IEEE Trans. Wirel. Commun._ , 2021.
  * [33] C. Tan, F. Sun, T. Kong, W. Zhang, C. Yang, and C. Liu, “A survey on deep transfer learning,” in _Artificial Neural Networks and Machine Learning – ICANN 2018_ , 2018, pp. 270–279.
  * [34] Z. Zhu, K. Lin, and J. Zhou, “Transfer learning in deep reinforcement learning: A survey,” _arXiv preprint arXiv:2009.07888_ , 2020.
  * [35] M. Derakhshani, X. Wang, D. Tweed, T. Le-Ngoc, and A. Leon-Garcia, “AP-STA association control for throughput maximization in virtualized WiFi networks,” _IEEE Access_ , 2018.
  * [36] G. Holland, N. Vaidya, and P. Bahl, “A rate-adaptive MAC protocol for multi-hop wireless networks,” in _Proceedings of the Annual International Conference on Mobile Computing and Networking, MOBICOM_ , 2001.
  * [37] G. F. Riley and T. R. Henderson, _The ns-3 Network Simulator_. Berlin, Heidelberg: Springer Berlin Heidelberg, 2010, pp. 15–34. [Online]. Available: https://doi.org/10.1007/978-3-642-12331-3_2
  * [38] T. S. Kim, H. Lim, and J. C. Hou, “Improving spatial reuse through tuning transmit power, carrier sense threshold, and data rate in multihop wireless networks,” in _Proceedings of the Annual International Conference on Mobile Computing and Networking, MOBICOM_ , 2006.
  * [39] S. Padakandla, “A Survey of Reinforcement Learning Algorithms for Dynamically Varying Environments,” _ACM Comput. Surv._ , vol. 54, no. 6, pp. 1–25, 2021\.
  * [40] A. Blázquez-García, A. Conde, U. Mori, and J. A. Lozano, “A Review on Outlier/Anomaly Detection in Time Series Data,” _ACM Comput. Surv._ , vol. 54, no. 3, pp. 1–33, 2021.
  * [41] P. Gawłowicz and A. Zubow, “Ns-3 meets OpenAI Gym: The playground for machine learning in networking research,” in _MSWiM 2019 - Proceedings of the 22nd International ACM Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems_ , 2019.
  * [42] F. Wilhelmi, S. Barrachina-Muñoz, B. Bellalta, C. Cano, A. Jonsson, and G. Neu, “Potential and pitfalls of Multi-Armed Bandits for decentralized Spatial Reuse in WLANs,” _Journal of Network and Computer Applications_ , vol. 127, pp. 26–42, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1084804518303655
  * [43] M. Bayati, N. Hamidi, R. Johari, and K. Khosravi, “Unreasonable Effectiveness of Greedy Algorithms in Multi-Armed Bandit with Many Arms,” in _Advances in Neural Information Processing Systems_ , vol. 33, 2020, pp. 1713–1723.