# ReinWiFi: A Reinforcement-Learning-Based Framework for the Application-Layer
QoS Optimization of WiFi Networks

Qianren Li, Bojie Lv, Yuncong Hong, and Rui Wang
Southern University of Science and Technology

###### Abstract

In this paper, a reinforcement-learning-based scheduling framework is proposed
and implemented to optimize the application-layer quality-of-service (QoS) of
a practical wireless local area network (WLAN) suffering from unknown
interference. Particularly, application-layer tasks of file delivery and
delay-sensitive communication, e.g., screen projection, in a WLAN with
enhanced distributed channel access (EDCA) mechanism, are jointly scheduled by
adjusting the contention window sizes and application-layer throughput
limitation, such that their QoS, including the throughput of file delivery and
the round trip time of the delay-sensitive communication, can be optimized.
Due to the unknown interference and vendor-dependent implementation of the
network interface card, the relation between the scheduling policy and the
system QoS is unknown. Hence, a reinforcement learning method is proposed, in
which a novel Q-network is trained to map from the historical scheduling
parameters and QoS observations to the current scheduling action. It is
demonstrated on a testbed that the proposed framework can achieve a
significantly better QoS than the conventional EDCA mechanism.

##  I Introduction

Reinforcement learning (RL) for radio resource management has been attracting
tremendous attention since it is a promising technique to tackle unknown
system statistics and solve the prohibitive policy optimization problem with
tolerable complexity and good performance. Moreover, the RL technique also has
great potential to optimize a wireless system even without accurate or
complete observation of the system state, which might happen in practical
implementations.

There have been a significant amount of works optimizing the throughput, delay
or age-of-information (AoI) of wireless networks via the method of RL. Most of
these works assumed full knowledge of the system state in algorithm design,
which could be applied to the systems where the global system state could be
collected at a centralized controller. On the other hand, RL was also utilized
to optimize the performance of wireless systems with distributive transmission
scheduling, e.g., wireless fidelity (WiFi) systems. For instance, an adaptive
channel contention mechanism was proposed for WiFi systems in [1], where a
local RL agent was deployed at each user equipment (UE). The local agents
adjusted the minimum contention window (MCW) size according to the global
statistics of successful channel contention such that the transmission
fairness among the agents can be ensured. Instead of global statistics, a
distributive RL algorithm with the assistance of federated learning was
proposed in [2] to adapt the channel contention according to the local channel
state, such that the local throughput was optimized. Moreover, a deep multi-
agent RL technique based on the QMIX algorithm [3] was proposed in [4] to
improve network throughput while maintaining user fairness. In this work, the
channel contention decision was made according to the history of the last
transmission duration. In order to resolve the collision issue of the
distributive channel access, deep RL algorithms were proposed in [5] to
determine the timing of doubling the contention window based on the estimated
collision probability. In addition to the adaptive channel contention, a
double deep Q-network (DDQN) [6] based rate adaptation algorithm was proposed
in [7] to improve network throughput, where the agent learned the optimal
transmission rate based on the modulation and coding scheme (MCS) and frame
loss rate. Most of the above literature assumed knowledge of the physical
(PHY) layer and media access control (MAC) layer system states. However, it
might be challenging to obtain such knowledge in the scheduler design of a
practical WiFi network. Moreover, the absence of knowledge on co-channel
interference and the vendor-dependent implementation of WiFi adapters would
also raise challenges in the optimization of scheduling policies.

In this paper, we would like to shed some light on the RL-based scheduling
design for practical WiFi systems suffering from unknown co-channel
interference. Particularly, a framework, namely ReinWiFi, is proposed for the
scheduling of delay-sensitive communication tasks and file delivery tasks in
the application layer of a WiFi network. In ReinWiFi, a controller
periodically collects the past scheduling parameters and average quality-of-
service (QoS) observations of all the application-layer tasks, determines rate
limitation and contention window size for all the transmitters, such that the
total throughput of file delivery tasks is maximized and the latency
requirements of delay-sensitive tasks are ensured. It is shown by the
experiments that the proposed framework can adapt to the variation of task
number, interfering traffic, and link quality, and significantly outperforms
the conventional EDCA mechanism.

##  II System Model

###  II-A Deployment Scenario

The proposed ReinWiFi system is deployed in a WiFi network with multiple
connected access points (APs) and UEs working on the same channel. Denote the
number of the devices, including the APs and UEs, in the WiFi network as $U$,
the set of these devices as $\mathcal{U}=\\{u_{i}|{i}=0,1,\ldots,U-1\\}$, and
the communication link from the $i$-th device to the $j$-th one as the
$({i},{j})$-th link ($\forall u_{i},u_{j}\in\mathcal{U}$). The communication
links can be from UE to AP, from AP to UE, or between UEs (i.e., WiFi Direct).
We define $\mathcal{L}$ as the set of all communication links in the system
and $\mathcal{L}_{i}$ as the set of communication links from the $u_{i}$-th
device. As a remark, one UE could simultaneously maintain the communication
links to the AP and other UEs, where the transmission of the infrastructure
and WiFi Direct modes is separated in the time domain.

The data traffics raised by the applications of UEs in $\mathcal{U}$ are
referred to as communication tasks in this paper. For example, the application
projecting the screen of a mobile phone to a laptop via WiFi Direct will raise
a delay-sensitive task, e.g., Miracast [8], where an application-layer packet
(i.e., video frame) is generated and delivered periodically (the typical
period is $16$ ms). Moreover, file sharing between two devices will raise a
file delivery task. For the elaboration convenience, we define
$\mathcal{T}_{{i},{j}}^{f}$ and $\mathcal{T}_{{i},{j}}^{r}$ as the universal
sets of file delivery tasks and delay-sensitive tasks on the $({i},{j})$-th
link, respectively. A task is in the inactive state if there is no packet
arrival or buffered file at the transmitter.

Because of the transmission latency constraint, the delay-sensitive tasks
should be scheduled with higher priority than the file delivery ones. Hence,
all the transmitters access the channel via the enhanced distributed channel
access (EDCA) mechanism defined in IEEE 802.11e. Particularly, four access
category (AC) queues, namely voice (VI), video (VO), best effort (BE), and
background (BK), are adopted at all the transmitters. The transmission
priorities of the four AC queues are differentiated by values of arbitration
inter-frame spacing (AIFS) and contention window (CW) size. As in the
practical systems, the file delivery tasks are scheduled with the BE priority,
and the delay-sensitive tasks are scheduled with the VI priority. The latter
has smaller AIFS and CW size, leading to a larger successful probability in
channel contention. As a remark, due to the distributive channel contention
mechanism, it is infeasible to accurately control the packet transmission
order among the devices of a WiFi network with commercial network interface
cards (NICs). Instead, the packet transmission in the ReinWiFi system is
scheduled in a stochastic manner by adapting the CW sizes of AC queues in each
device.

There are some other WiFi networks sharing the same channel in the coverage of
the considered network. The traffic in these networks would degrade the QoS of
the considered network, e.g., larger delivery latency and lower throughput.
Denote the set of devices in the interfering networks as $\mathcal{U}_{I}$.
The communications among the devices in $\mathcal{U}_{I}$, namely interfering
traffic, cannot be scheduled by the ReinWiFi system. Instead, the ReinWiFi
system is designed to deduce the interference level and adjust the
transmission accordingly.

###  II-B Task Queuing Model

For each file delivery task, all the information bits to be delivered are
saved in an application-layer buffer, and a user datagram protocol (UDP)
socket is established at the very beginning of transmission. The data dispatch
from the buffer to the UDP socket is controlled by a dispatcher. The UDP
socket encapsulates the received data from the dispatcher into UDP datagrams
and forwards them to the driver of NIC for WiFi transmission accordingly. As a
remark, the new datagrams at the NIC may not be transmitted immediately. In
fact, each NIC maintains four MAC-layer AC queues associated with the four
transmission priorities, respectively. The arrival datagrams are saved in the
corresponding queues and transmitted following the vendor’s protocol. The
queuing status of the NIC is usually not accessible in the application-layer.
Thus, it is infeasible for the proposed system to know when the NIC completely
delivers a datagram; it is, therefore, infeasible for the proposed system to
precisely control the transmission of a UDP datagram or an application-layer
packet. As a result, the scheduling of the proposed system is designed based
on the average observable performance in the application layer.

Specifically, the transmission time is organized into a sequence of scheduling
periods, each with a duration of $T_{s}$ seconds. $T_{s}$ is sufficiently
large to accommodate a number of MAC protocol data unit transmissions. Due to
the invisibility of NIC status, the QoS of a file delivery task is measured by
its application-layer throughput in one scheduling period. Particularly, for
the ${m}$-th file delivery task of the $({i},{j})$-th link, its QoS in the
$t$-th scheduling period $r_{{i},{j}}^{m}(t)$ is defined as the number of
information bits transferred from the task buffer to the associated UDP
socket. The dispatcher is designed to adaptively limit the throughput of the
file delivery task such that delay-sensitive tasks could have a larger chance
to access the channel. Hence, let $b_{{i},{j}}^{m}(t)$ be the throughput
limitation of the ${m}$-th file delivery task of $({i},{j})$-th link in the
$t$-th scheduling slot, the dispatcher would make sure

$\displaystyle r_{{i},{j}}^{m}(t)\leq b_{{i},{j}}^{m}(t).$ (1)

For each delay-sensitive task, a task queue and UDP socket are established at
the very beginning. The application-layer packets arrive at the task queue
periodically with a fixed average data rate. The first packet in the queue is
forwarded to the UDP socket for WiFi transmission as long as the socket is
idle. Due to the lack of MAC-layer status, the measurement of the transmission
latency of a packet could hardly be accurate. Hence, we use the round-trip
time (RTT) as the QoS measurement of delay-sensitive communication tasks.
Particularly, for each delay-sensitive task, an acknowledgment will be sent
back from the receiver to the transmitter when an application-layer packet is
completely received. Hence, the transmitter can calculate the RTTs of all
packet transmissions. For the ${m}$-th delay-sensitive communication task of
the $({i},{j})$-th link
($\forall({i},{j})\in{\mathcal{L}},{m}\in{\mathcal{T}_{{i},{j}}^{r}},$), its
QoS in the $t$-th scheduling period $d_{{i},{j}}^{m}(t)$ is defined as the
average RTT of the packets transmitted in this scheduling period.

###  II-C Scheduling Model

Denote the CW sizes of the VI and BE priorities of the ${i}$-th device at
$t$-th scheduling period as $w_{i}^{\mathtt{VI}}(t)$ and
$w_{i}^{\mathtt{BE}}(t)$ respectively, we shall focus on the joint scheduling
of these channel contention parameters as well as the dispatchers’ throughput
limitation
$\\{b_{{i},{j}}^{m}(t)|\forall({i},{j})\in{\mathcal{L}},{m}\in{\mathcal{T}_{{i},{j}}^{f}}\\}$
in each scheduling period.

Particularly, each transmitter collects the QoS observations of its tasks at
the end of each scheduling period and delivers them to a centralized
controller, which can be implemented in an AP or other device. Not all the
tasks in the universal task sets are in the active state. The average RTTs and
throughputs of the inactive delay-sensitive and file delivery tasks are
represented by a sufficiently large value and $0$, respectively. Hence, the
aggregation of QoS observations received at the controller at the end of the
$t$-th scheduling period can be represented as

$\displaystyle\begin{split}\mathcal{O}_{t}\triangleq&\left\\{r_{{i},{j}}^{m}(t)|\forall({i},{j})\in{\mathcal{L}},{m}\in{\mathcal{T}_{{i},{j}}^{f}}\right\\}\\\
&\cup\left\\{d_{{i},{j}}^{m}(t)|\forall({i},{j})\in{\mathcal{L}},{m}\in{\mathcal{T}_{{i},{j}}^{r}}\right\\}.\end{split}$
(2)

Due to the time-varying traffic of the interfering devices, the scheduling
parameters, including the file throughput limitations and CW sizes, are
adapted at the centralized controller in each schedule according to the
system’s scheduling parameters and QoS observations in the past $N$ scheduling
periods. Specifically, the aggregation of scheduling parameters over a period
is represented as

$\displaystyle\begin{split}\mathcal{A}_{t}\triangleq&\left\\{b_{{i},{j}}^{m}(t)|\forall({i},{j})\in{\mathcal{L}},{m}\in{\mathcal{T}_{{i},{j}}^{f}}\right\\}\\\
&\cup\left\\{w_{i}^{\mathtt{VI}}(t),w_{i}^{\mathtt{BE}}(t)|i=0,1,\ldots,U-1\right\\}.\end{split}$
(3)

Thus, at the very beginning of the $t$-th scheduling period, $\mathcal{A}_{t}$
($\forall t$) is determined based on past scheduling parameters and QoS
observations
$\\{(\mathcal{O}_{t-N},\mathcal{A}_{t-N}),(\mathcal{O}_{t-N+1},\mathcal{A}_{t-N+1}),\ldots,(\mathcal{O}_{t-1},\mathcal{A}_{t-1})\\}$.

##  III Problem Formulation

The proposed ReinWiFi system should successively make scheduling decisions for
each scheduling period. Hence, it is formulated as a Markov decision process
(MDP) in the following.

######  Definition 1 (System State)

In the $t$-th scheduling period ($\forall t$), the system state is defined as
the aggregation of the QoS observations and scheduling parameters of the past
$N$ scheduling periods. Thus,
$\mathcal{S}_{t}\triangleq\left\\{(\mathcal{O}_{t-N},\mathcal{A}_{t-N}),\ldots,(\mathcal{O}_{t-1},\mathcal{A}_{t-1})\right\\}$.

######  Definition 2 (Scheduling Action and Policy)

Denote $\mathcal{A}_{t}$ defined in (3) as the scheduling action in the $t$-th
scheduling period,
$\mathcal{A}_{t}^{i}\triangleq\left\\{b_{i},{j}^{m}(t)|\forall
j\in\mathcal{L}_{i},{m}\in{\mathcal{T}_{{i},{j}}^{f}}\right\\}\cup\left\\{w_{i}^{\mathtt{VI}}(t),w_{i}^{\mathtt{BE}}(t)\right\\}$,
as the local scheduling action of the $i$-th device in the $t$-th scheduling
period. The scheduling policy $\Omega$ is a mapping from state space to action
space as $\Omega(\mathcal{S}_{t})=\mathcal{A}_{t}$.

Moreover, the system cost of the $t$-th scheduling period is defined as

$\displaystyle\begin{split}c_{t}(\mathcal{S}_{t},\mathcal{A}_{t})\triangleq&\sum_{({i},{j})\in{\mathcal{L}}}\sum_{{m}\in{\mathcal{T}_{{i},{j}}^{r}}}\mathds{1}(d_{{i},{j}}^{m}(t)>\mathrm{D}_{{i},{j}}^{m})\\\
&-\omega\sum_{({i},{j})\in{\mathcal{L}}}\sum_{{m}\in{\mathcal{T}_{{i},{j}}^{f}}}r_{{i},{j}}^{m}(t),\end{split}$
(4)

where $\omega$ is a weight, $\mathrm{D}_{{i},{j}}^{m}$ is the maximum
tolerable RTT of the ${m}$-th delay-sensitive task on the $({i},{j})$-th link.
The indicator function $\mathds{1}(\mathcal{E})$ is $1$ if the event
$\mathcal{E}$ is true, and $0$ otherwise. Then, the average overall system
cost is defined as the discounted summation of average system costs for all
the scheduling periods, i.e.,

$\bar{C}(\Omega)=\lim_{T\rightarrow\infty}\mathbb{E}\bigg{[}\sum_{t=1}^{T}\gamma^{t-1}c_{t}(\mathcal{S}_{t},\Omega(\mathcal{S}_{t}))\bigg{]}.$
(5)

For the elaboration convenience, it is assumed that the system has run for at
least $N$ scheduling periods before the first scheduling period, such that
there are sufficient QoS observations in the system state. As a result, the
controller design of the ReinWiFi system can be formulated as

$\text{\bf Problem 1:}\ \min_{\Omega}\ \bar{C}(\Omega).$ (6)

The Bellman’s equations for the above MDP is given by

$\displaystyle
Q(\mathcal{S}_{t},\mathcal{A}_{t})=\mathbb{E}_{\mathcal{S}_{t+1}}\bigg{[}c_{t}(\mathcal{S}_{t},\mathcal{A}_{t})+\gamma\min_{\mathcal{A}^{\prime}}Q(\mathcal{S}_{t+1},\mathcal{A}^{\prime})\bigg{]},$
(7)

where $Q(\mathcal{S}_{t},\mathcal{A}_{t})$ is the Q-function with system state
$\mathcal{S}_{t}$ and action $\mathcal{A}_{t}$. Moreover, the optimal
scheduling is given by

$\Omega^{*}(\mathcal{S})=\operatorname*{arg\,min}\limits_{\mathcal{A}}Q(\mathcal{S},\mathcal{A}).$
(8)

Given the past scheduling actions and QoS observations (i.e., the system
state), it is still difficult to accurately predict the relation between the
scheduling action and task QoS in the current scheduling period. This is
mainly because of the unknown interfering traffic and random channel
contention. As a result, it is impossible to solve the above Bellman’s
equations without any trial on the network performance. In this paper, we
shall rely on the RL method to track the above unknown knowledge with the
assistance of a preliminary observation dataset $\mathscr{S}^{s}$.

Particularly, before the optimization, the dataset $\mathscr{S}^{s}$ is
collected from $M$ scheduling periods experiencing heterogeneous interfering
traffic and link quality (e.g., the distances of links in $\mathcal{L}$ change
due to mobility). Each of the scheduling periods (say the $\tau$-th one) is
divided into two phases. In the first phase, a fixed testing scheduling action
$\mathcal{A}^{p}$ is applied, and corresponding QoS observation
$\mathcal{O}^{p}_{\tau}$ is obtained; in the second phase, a random scheduling
action $\mathcal{A}^{s}_{\tau}$ according to certain distribution is applied,
and another QoS observation $\mathcal{O}^{s}_{\tau}$ is obtained. Hence, the
dataset $\mathscr{S}^{s}$ can be expressed as
$\mathscr{S}^{s}\triangleq\left\\{(\mathcal{O}^{p}_{\tau},\mathcal{A}^{p},\mathcal{O}^{s}_{\tau},\mathcal{A}^{s}_{\tau})|\tau=1,2,\ldots,M\right\\}$.

##  IV Q-Network for Online Scheduling

In this section, a novel Q-network design is proposed to approximate the
Q-function. In order to accelerate the convergence of training and improve the
scheduling performance, all the possible system performance of one scheduling
period is divided into $K$ regions, and the inputs of the Q-network include
not only the system state but also the performance region indices of the past
$N$ scheduling periods.

Hence, the utilization of the proposed Q-network in the transmission
scheduling can be divided into two stages. In the first stage, namely the
offline stage, the performance regions are trained via the preliminary
observation dataset $\mathscr{S}^{s}$, and the Q-network is then trained via
$\mathscr{S}^{s}$ in all the performance regions respectively. In the second
stage, namely the online stage, the Q-network is applied to the transmission
scheduling and fine-trained according to the online QoS observations.

In this section, the performance region quantization is introduced first,
followed by the structure of the Q-network. The hybrid offline and online
training of Q-network is elaborated in Section V.

###  IV-A Performance Region Quantization

The QoS observations with the testing scheduling action $\mathcal{A}^{p}$ are
first extracted from the preliminary observation dataset $\mathscr{S}^{s}$ as
$\mathscr{S}^{p}\triangleq\left\\{(\mathcal{O}^{p}_{\tau},\mathcal{A}^{p})|\tau=1,2,\ldots,M\right\\}$.
The $K$-means classification method [9] is then adopted to classify the QoS
observations in $\mathscr{S}^{p}$ into $K$ clusters. Denote the mean and
variance of the observed throughputs (for the file delivery tasks) in
$\mathscr{S}^{p}$ as $\bar{r}$ and $\sigma_{r}^{2}$ respectively, the mean and
variance of the RTTs (for the delay-sensitive tasks) as $\bar{d}$ and
$\sigma_{d}^{2}$ respectively. The performance region quantization can be
achieved by finding the $K$ cluster centers of the QoS observations in
$\mathscr{S}^{p}$ as follows:

$\\{\mu_{1}^{*},\ldots,\mu_{K}^{*}\\}=\operatorname*{arg\,min}\limits_{\mu_{1},\ldots,\mu_{K}}\
\sum_{k=1}^{K}\sum_{\tau=1}^{M}\|\phi(\mathcal{O}^{p}_{\tau})-\mu_{k}\|^{2},$
(9)

where $\phi(\mathcal{O}^{p}_{\tau})$ denotes the vectorization of the
normalized QoS observations in $\mathcal{O}^{p}_{\tau}$. Particularly,
$\phi(\mathcal{O}^{p}_{\tau})\triangleq\left(\mathbf{r}_{\tau}^{p},\mathbf{d}_{\tau}^{p}\right)$,
where the row vector $\mathbf{r}_{\tau}^{p}$ vectorizes the normalized
throughputs of all file delivery tasks in $\mathcal{O}^{p}_{\tau}$,

$\left\\{\frac{r_{{i},{j}}^{m,p}(\tau)-\bar{r}}{\sigma_{r}}\bigg{|}\forall({i},{j})\in{\mathcal{L}},{m}\in{\mathcal{T}_{{i},{j}}^{f}},r_{{i},{j}}^{m,p}(\tau)\in\mathcal{O}^{p}_{\tau}\right\\},$

and the row vector $\mathbf{d}_{\tau}^{p}$ vectorizes the normalized RTTs of
all the delay-sensitive tasks in $\mathcal{O}^{p}_{\tau}$,

$\left\\{\frac{d_{{i},{j}}^{m,p}(\tau)-\bar{d}}{\sigma_{d}}\bigg{|}\forall({i},{j})\in{\mathcal{L}},{m}\in{\mathcal{T}_{{i},{j}}^{r}},d_{{i},{j}}^{m,p}(\tau)\in\mathcal{O}^{p}_{\tau}\right\\}.$

With $\\{\mu_{1}^{*},\ldots,\mu_{K}^{*}\\}$, the performance region index of a
scheduling period can be determined according to

$\hat{\psi}=\operatorname*{arg\,min}\limits_{k}\
\|\phi(\widehat{\mathcal{O}})-\mu_{k}^{*}\|^{2},$ (10)

where $\widehat{\mathcal{O}}$ is the aggregation of QoS observations with the
testing scheduling action $\mathcal{A}^{p}$ in the scheduling period.

###### Remark 1

Note that the QoS observations of the testing scheduling action
$\mathcal{A}^{p}$ should be collected to determine the performance region
index of one scheduling period. In the online stage, one short slot can be
reversed in each scheduling period to apply the testing scheduling action
$\mathcal{A}^{p}$.

###  IV-B Q-Network Structure

The input of the proposed Q-network is the extended system state of the
current scheduling period, which is defined below:

######  Definition 3 (Extended System State)

In the $t$-th scheduling period ($\forall t$) of either offline or online
training, the extended system state consists of
$\hat{\mathcal{S}}_{t}\triangleq\left\\{(\hat{\psi}_{t-N},\mathcal{O}_{t-N},\mathcal{A}_{t-N}),\ldots,(\hat{\psi}_{t-1},\mathcal{O}_{t-1},\mathcal{A}_{t-1})\right\\}$,
where $\hat{\psi}_{t-i}$ ($i=1,2,...,N$) is the performance region index.

The first part of the Q-network is a multi-head attention layer[10], which is
trained to refine the performance region indices in the extended system state.
The refined extended system state is then used as the input of the following
three fully connected layers with $256$ nodes and ReLU activation function
sequentially.

In order to address the issue of huge action space, we adopt the following
linear approximation structure on the Q-function in the output of the
Q-network:

$Q(\hat{\mathcal{S}},\mathcal{A})\approx\sum_{{i}\in\mathcal{U}}Q^{i}(\hat{\mathcal{S}},\mathcal{A}^{i}),$
(11)

where $Q^{i}(\hat{\mathcal{S}},\mathcal{A}^{i})$ is referred to as the local
Q-function of the ${i}$-th device. Hence, the Q-network output consists of $U$
action clusters for $U$ devices, respectively. Each action cluster provides
the values of the corresponding local Q-function for all possible local
actions. As a result, the optimized local action of the $i$-th device
($\forall i$) in the $t$-th scheduling period of either offline or online
training can be obtained by minimizing the local Q-function, i.e.,

$\mathcal{A}^{i}_{t}=\operatorname*{arg\,min}\limits_{\mathcal{A}^{i}}Q^{i}(\hat{\mathcal{S}}_{t},\mathcal{A}^{i}).$
(12)

##  V Hybrid Q-Learning

$L^{I}(\boldsymbol{\theta}^{w}_{k})=\frac{1}{|\mathcal{O}^{s}_{\tau}|}\left[\alpha\sum_{({i},{j})\in{\mathcal{L}}}\sum_{{m}\in{\mathcal{T}_{{i},{j}}^{f}}}{\left(\hat{r}_{{i},{j}}^{{m}}(\mathcal{A};\boldsymbol{\theta}^{w}_{k})-r_{{i},{j}}^{m}(\tau)\right)}^{2}+\sum_{({i},{j})\in{\mathcal{L}}}\sum_{{n}\in{\mathcal{T}_{{i},{j}}^{r}}}{\left(\hat{d}_{{i},{j}}^{n}(\mathcal{A};\boldsymbol{\theta}^{w}_{k})-\min\left\\{d_{{i},{j}}^{n}(\tau),\beta\mathrm{D}^{n}_{{i},{j}}\right\\}\right)}^{2}\right].$
(13)

$L^{q}(\boldsymbol{\theta}^{q}_{t})=\mathbb{E}\left[{\left(c_{t}\left(\mathcal{S}_{t},\mathcal{A}_{t}\right)+\gamma\sum_{{i}\in\mathcal{U}}\min_{{\mathcal{A}^{{i}}}^{^{\prime}}}Q^{i}\left(\hat{\mathcal{S}}_{t+1},\mathcal{A}^{{i}^{^{\prime}}};\boldsymbol{\theta}^{q,-}_{t}\right)-\sum_{{i}\in\mathcal{U}}Q^{i}\left(\hat{\mathcal{S}}_{t},\mathcal{A}^{i}_{t};\boldsymbol{\theta}^{q}_{t}\right)\right)}^{2}\right].$
(14)

The Q-network is first trained in the offline stage based on the dataset
$\mathscr{S}^{s}$, then tuned in the online stage.

###  V-A Offline Imitation Learning and Q-Network Training

To facilitate the offline training, the performance indices are calculated for
all the scheduling periods in $\mathscr{S}^{s}$ according to (10). Denote the
performance index of the $\tau$-th scheduling period in $\mathscr{S}^{s}$ as
$\hat{\psi}_{\tau}^{s}$, the preliminary dataset $\mathscr{S}^{s}$ can be
rewritten as

$\tilde{\mathscr{S}}^{s}\triangleq\left\\{(\hat{\psi}_{\tau}^{s},\mathcal{O}_{\tau}^{s},\mathcal{A}^{s}_{\tau})|\tau=1,2,\ldots,M\right\\}$
(15)

for notation convenience. Moreover, dataset $\tilde{\mathscr{S}}^{s}$ can be
further divided into $K$ subsets as

$\tilde{\mathscr{S}}^{s}_{k}\triangleq\left\\{(k,\mathcal{O}_{\tau}^{s},\mathcal{A}^{s}_{\tau})|\forall\hat{\psi}_{\tau}^{s}=k\right\\}\subset\tilde{\mathscr{S}}^{s},k=1,\ldots,K.$
(16)

Notice that the subsets $\tilde{\mathscr{S}}^{s}_{k}$ ($k=1,2,\ldots,K$) may
not be sufficiently large for the training of the Q-network in all the
performance regions, the imitation learning method is introduced.
Particularly, we first train $K$ DNN networks (namely imitators), each of
which consists of $10$ fully connected layers and $256$ nodes per layer, to
imitate the relation between the scheduling actions and QoS observations in
the $K$ performance regions, respectively. Denote the imitators as
$f(\mathcal{A};\boldsymbol{\theta}^{w}_{k}),k=1,2,\ldots,K$, where
$\mathcal{A}$ is the input action, and $\boldsymbol{\theta}^{w}_{k}$
represents network parameters. The output of imitator
$f(\mathcal{A};\boldsymbol{\theta}^{w}_{k})$ is trained to approximate the QoS
observations of the system in the $k$-th performance region with input action
$\mathcal{A}$. Then, the Q-network can be trained via the $K$ imitators.

Imitator training: The $k$-th imitator ($k=1,2,...,K$) is trained by
$\tilde{\mathscr{S}}^{s}_{k}$. Let
$\hat{r}_{{i},{j}}^{{m}}(\mathcal{A};\boldsymbol{\theta}^{w}_{k})$ and
$\hat{d}_{{i},{j}}^{n}(\mathcal{A};\boldsymbol{\theta}^{w}_{k})$ be the
throughput and RTT of the ${m}$-th file delivery task and ${n}$-th delay-
sensitive task of the $({i},{j})$-th link in the output of the $k$-th imitator
with input action $\mathcal{A}$. The loss function $L^{I}$ is defined as (13),
where $r^{m}_{{i},{j}}(\tau),d^{n}_{{i},{j}}(\tau)\in\mathcal{O}_{\tau}^{s}$,
$\alpha$ and $\beta$ are both weights, and the minimization is to limit the
range of RTTs.

Offline Q-network training: Based on the imitators, the Q-network can be
trained in each performance region respectively. Particularly, in the $t$-th
scheduling period of offline training with the $k$-th imitator ($\forall
t,k$), providing the scheduling action, the outputs of the imitator are
treated as the QoS observations in the $k$-th performance region, which is
then used to update the extended system state of the $(t+1)$-th scheduling
period in the input of the Q-network. The Q-network is also updated in the
above iterative procedure according to the Q-learning method [11]. The loss
function $L^{q}$ is defined in (14), where
$Q(\cdot,\cdot;\boldsymbol{\theta}^{q}_{t})$ represents the Q-network
parameters in the $t$-th scheduling period, and
$\boldsymbol{\theta}^{q,-}_{t}$ denotes the parameter of target network as in
[11].

In order to efficiently explore the action space, an upper confidence bound
(UCB) based exploration policy is introduced to determine the scheduling
action in the offline training of Q-network. Taking the $t$-th scheduling
period with the $k$-th imitator as the example, we first define the UCB of the
action $\mathcal{A}^{i}$ of ${i}$-th device as

$UCB_{t}(k,\mathcal{A}^{i})=Q_{t}^{i}(\hat{\mathcal{S}}_{t},\mathcal{A}^{i};\boldsymbol{\theta}^{q}_{t})+\sqrt{\frac{4\eta\ln
t}{T_{t}(k,\mathcal{A}^{i})}},$ (17)

where $T_{t}(k,\mathcal{A}^{i})$ counts the number of times the action
$\mathcal{A}^{i}$ is taken up to the $t$-th scheduling period. The hyper-
parameter $\eta$ is used to balance the exploration and exploitation. As a
result, the scheduling action is determined as follows:

$\mathcal{A}_{t}^{i}=\begin{cases}\operatorname*{arg\,min}UCB_{t}(k,\mathcal{A}^{i})&\text{with
probability }1-\epsilon_{t},\\\
\mathcal{A}^{i}\sim\mathrm{Unif}(\mathscr{A}^{i})&\text{with probability
}\epsilon_{t},\end{cases}$ (18)

where $\mathrm{Unif}(\mathscr{A}^{i})$ is the uniform distribution over action
space $\mathscr{A}^{i}$ of ${i}$-th device and exploration rate $\epsilon_{t}$
should satisfy the limit condition $\lim_{t\to\infty}\epsilon_{t}=0$.

###  V-B Online Q-network Training

The online Q-network training with the same loss function as in (14) could be
applied to further improve the performance of the proposed ReinWiFi system.
Particularly, in the $t$-th scheduling period of the online stage, the
scheduling action of the ${i}$-th device, denoted as $\mathcal{A}_{t}^{i}$, is
determined by the $\epsilon$-greedy policy as follows:

$\mathcal{A}_{t}^{i}=\begin{cases}\operatorname*{arg\,min}Q^{i}(\hat{\mathcal{S}}_{t},\mathcal{A}^{i};\boldsymbol{\theta}_{t})&\text{with
probability }1-\epsilon_{t},\\\
\mathcal{A}^{i}\sim\mathrm{Unif}(\mathscr{A}^{i})&\text{with probability
}\epsilon_{t},\end{cases}$ (19)

where $\epsilon_{t}$ and $\mathrm{Unif}(\mathscr{A}^{i})$ are defined in (18).

##  VI Experiments

The proposed ReinWiFi system is implemented in a WiFi network with one HONOR
XD30 AP and $3$ UEs each equipped with a TP-Link TL-WDN6200 USB WiFi adapter
in the experiment111The source code of implementation is available online in
https://github.com/QianrenLi/ReinWiFi.. Denote the AP as $u_{0}$ and the three
UEs as $u_{1},u_{2},u_{3}$, respectively. The network is working on the $5$G
WiFi band following the IEEE 802.11ac specification. The real-time controller
is implemented in a laptop with Intel Core i7-8750H CPU and Ubuntu 20.04
operating system. An Ethernet connection with a maximum data rate of $1$ Gbps
is employed to facilitate communication between the controller and the AP.
Moreover, we implement a Linux module to adapt the CW sizes of TL-WDN6200
adapters in real-time from user space. Hence, the controller can collect the
QoS observations from UEs and notify the scheduling actions via WiFi, such
that the UEs’ transmission scheduling can be adjusted accordingly.

Both file delivery tasks and delay-sensitive tasks are tested in the
experiment. The former tasks with a sufficient backlog are transmitted with
the BE priority. The latter tasks, consisting of two types, are delivered with
the VI priority. The data rates of type I and II delay-sensitive tasks are
$\lambda_{1}=50$Mbps and $\lambda_{2}=25$Mbps, respectively. The packet
arrival intervals of the two types are both $16$ ms. Moreover, the maximum
tolerable RTTs are $16$ms and $28$ms, respectively. The universal set of
communication tasks tested in the experiment includes a delay-sensitive task
with arrival data rate $\lambda_{1}$ (Task 1) and a file delivery task (Task
2) on the $(u_{1},u_{0})$-th link; a delay-sensitive task with arrival data
rate $\lambda_{2}$ (Task 3) on the $(u_{2},u_{0})$-th link; a delay-sensitive
task with arrival data rate $\lambda_{2}$ (Task 4) on the $(u_{3},u_{0})$-th
link. The quality of the $(u_{1},u_{0})$-th, $(u_{2},u_{0})$-th, and
$(u_{3},u_{0})$-th links depend on their distances and the propagation
environment, which could be changed in the experiment.

In the experiment, the duration of the scheduling period is 1 second, the CW
size takes values from $\\{2^{i}-1\mid i=1,2,\ldots,10\\}$, and throughput
limitation takes values from $\\{\frac{i}{20}r_{{i},{j}}^{{m},\max}\mid
i=0,1,\ldots,20\\}$, where $r_{{i},{j}}^{{m},\max}$ = 600 Mbps. Moreover, in
addition to the background interference, the interfering traffic between two
interfering UEs, denoted as $u_{4}$ and $u_{5}$, is generated with a random
data rate and BE priority in the same channel.

The preliminary observation dataset $\mathscr{S}^{s}$ is collected from the
following three different traffic patterns (TPs): (1) Tasks 1 and 2 are
activated; (2) Tasks 1, 2, and 3 are activated; and (3) Tasks 1, 2, 3 and 4
are activated. In all the TPs, the communication distances of the links are
altered to exploit the diversity of link rates. In the collection of
$\mathscr{S}^{s}$, the testing scheduling action $\mathcal{A}^{p}$ is first
applied in the first half of the scheduling period, where the CW size and
throughput limitations are $7$ and $300$ Mbps respectively. Then, a randomized
action is applied in the second half. QoS observations of both actions are
collected in each scheduling period.

Based on dataset $\mathscr{S}^{s}$, the performance of the three TPs are
quantized into 3, 6, and 6 regions, respectively. Then, $15$ QoS imitators are
trained according to Section V with $\alpha=1$, $\beta=3$. Given the trained
QoS imitators, the Q-network is further trained as elaborated in Section V
with $\omega=1/{r}^{m,\max}_{{i},{j}}$.

To demonstrate the performance gain, the proposed framework is compared with
two baselines. The first baseline, namely Standard EDCA, relies on the
conventional 802.11 EDCA protocol. The second baseline, namely Rate Control
Only, adapts the throughput limitation of file delivery tasks via the proposed
framework with the CW sizes following the 802.11 EDCA protocol. The
performance evaluation and comparison are conducted in $11$ distinct test
scenarios listed in Table I, where only the first $5$ scenarios have been
measured in the preliminary observation dataset $\mathscr{S}^{s}$.

Scenario | TP |  | Link Rate
---
(Mbps)
Scenario | TP |  | Link Rate
---
(Mbps)
1 | 1 | 563, 499, 572 | 7 | 3 | 563, 424, 572
2 | 2 | 563, 499, 572 | 8 | 2 | 563, 400, 346
3 | 3 | 563, 499, 572 | 9 | 3 | 563, 400, 346
4 | 2 | 563, 370, 572 | 10 | 2 | 459, 499, 572
5 | 3 | 563, 370, 572 | 11 | 3 | 459, 499, 572
6 | 3 | 563, 499, 476 |  |  |
TABLE I: Table of test scenarios, where the link rate refers to the maximum
data rates of the $(u_{1},u_{0})$-th, $(u_{2},u_{0})$-th, and
$(u_{3},u_{0})$-th links. Figure 1: Performance comparison in scenarios $1\sim
5$.

The performance comparison of the proposed framework and the two baselines in
the first $5$ test scenarios is illustrated in Fig. 1, where the online
training of the Q-network is not applied in the proposed framework and the
Baseline 2. It can be observed that the proposed Q-network offline trained via
imitators significantly outperforms the conventional EDCA mechanism. Moreover,
the performance gain of the Baseline 2 over Baseline 1 demonstrates the
necessity of the throughput limitation, which has never been investigated in
the existing literature.

The performance comparison in the test scenarios $6$ to $11$ is illustrated in
Fig. 2. Since these test scenarios are not measured in the preliminary
observation dataset $\mathscr{S}^{s}$, the performance gain of the proposed
scheme over the Baseline 1 demonstrates the good generalization capability of
the proposed Q-network. It can also be observed that the online training could
further improve the scheduling performance of the Q-network, which has already
been trained in the offline stage.

Figure 2: Performance comparison in scenarios $6\sim 11$.

##  VII Conclusion

In this paper, a reinforcement-learning-based framework, namely ReinWiFi, is
proposed for the application-layer QoS optimization of WiFi networks. Due to
the absence of PHY-layer and MAC-layer status, the historical scheduling
parameters and QoS observations are considered as the system state in the
determination of the current scheduling parameters. Because of the unknown
interference and vendor-dependent implementations, a novel Q-network is
proposed to track the relation between the system state, scheduling parameter,
and the overall QoS. Moreover, an imitation learning method is introduced to
improve the training efficiency. It is demonstrated via the testbed that the
proposed framework, with the dynamic adaptation of CW size and throughput
limitation, significantly outperforms the convention EDCA mechanism.

## References

  * [1] A. Kumar, G. Verma, C. Rao, A. Swami, and S. Segarra, “Adaptive contention window design using deep $Q$-learning,” in _IEEE Int. Conf. Acoust., Speech Signal Process.(ICASSP)_. IEEE, Jun. 2021, pp. 4950–4954.
  * [2] L. Zhang, H. Yin, Z. Zhou, S. Roy, and Y. Sun, “Enhancing WiFi multiple access performance with federated deep reinforcement learning,” in _IEEE Veh. Technol. Conf. (VTC Fall)_. IEEE, Nov. 2020, pp. 1–6.
  * [3] T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson, “Monotonic value function factorisation for deep multi-agent reinforcement learning,” _J. Mach. Learn. Res._ , vol. 21, no. 1, pp. 7234–7284, Jan. 2020.
  * [4] Z. Guo, Z. Chen, P. Liu, J. Luo, X. Yang, and X. Sun, “Multi-agent reinforcement learning-based distributed channel access for next generation wireless networks,” _IEEE J. Sel. Areas Commun._ , vol. 40, no. 5, pp. 1587–1599, May 2022.
  * [5] R. Ali, N. Shahin, Y. B. Zikria, B.-S. Kim, and S. W. Kim, “Deep reinforcement learning paradigm for performance optimization of channel observation–based MAC protocols in dense WLANs,” _IEEE Access_ , vol. 7, pp. 3500–3511, 2018.
  * [6] H. v. Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double $Q$-learning,” in _Proc. AAAI Conf. Artif. Intell._ , Feb. 2016, p. 2094–2100.
  * [7] S.-C. Chen, C.-Y. Li, and C.-H. Chiu, “An experience driven design for IEEE 802.11ac rate adaptation based on reinforcement learning,” in _Proc. IEEE Int. Conf. Comput. Commun. (INFOCOM)_ , May 2021, pp. 1–10.
  * [8] Wi-Fi Alliance, Wi-Fi Display Technical Task Group, “Wi-Fi display technical specification v1.2n,” 2011.
  * [9] J. MacQueen, “Some methods for classification and analysis of multivariate observations,” in _Proc. Berkeley Symp. Math. Statist. Probab._ , vol. 1, Oakland, CA, USA, 1967, pp. 281–297.
  * [10] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Proc. Int. Conf. Neural Inf. Process. Syst. (NIPS)_ , Dec. 2017, p. 6000–6010.
  * [11] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep reinforcement learning,” _Nature_ , vol. 518, no. 7540, pp. 529–533, Feb. 2015.