# Deep reinforcement learning for automatic run-time adaptation of UWB PHY
radio settings

Dieter Coppens, Adnan Shahid, , Eli De Poorter D. Coppens, A. Shahid, and E.
De Poorter are with IDLab, Department of Information Technology, Ghent
University—imec, 9052 Ghent, Belgium (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Ultra-wideband technology has become increasingly popular for indoor
localization and location-based services. This has led recent advances to be
focused on reducing the ranging errors, whilst research focusing on enabling
more reliable and energy efficient communication has been largely unexplored.
The IEEE 802.15.4 UWB physical layer allows for several settings to be
selected that influence the energy consumption, range and reliability.
Combined with the available link state diagnostics reported by UWB devices,
there is an opportunity to dynamically select PHY settings based on the
environment. To address this, we propose a deep Q-learning approach for
enabling reliable UWB communication, maximizing packet reception rate (PRR)
and minimizing energy consumption. Deep Q-learning is a good fit for this
problem, as it is an inherently adaptive algorithm that responds to the
environment. Validation in a realistic office environment showed that the
algorithm outperforms traditional Q-learning, linear search and using a fixed
PHY layer. We found that deep Q-learning achieves a higher average PRR and
reduces the ranging error while using only 14% of the energy compared to a
fixed PHY setting in a dynamic office environment.

###### Index Terms:

UWB, localization, deep reinforcement learning

This work has been submitted to the IEEE for possible publication. Copyright
may be transferred without notice, after which this version may no longer be
accessible

##  I Introduction

Ultra-wideband (UWB) technology has recently attracted the attention of not
only the research community, but also the public. This is due to the addition
of the technology to several smartphones and other consumer products [1]. UWB
uses a much wider bandwidth i.e., more than 500 MHz, compared to traditional
narrowband techniques such as Narrowband Internet of things (NB-IoT), Sigfox,
LoRa, etc. This has benefits such as higher robustness to multipath effects,
high temporal resolution and high channel capacity [2]. The high temporal
resolution allows UWB to be used for precision ranging and enables
applications like hands-free access-control and indoor navigation. While UWB
is more robust to multipath effects (compared to narrowband techniques), the
performance is still highly dependent on the environment. Having Line-of-sight
(LOS) or Non-line-of-sight (NLOS) conditions and destructive interference
influence the calculated ranges and reliability dramatically [3]. This has led
to a large quantity of work to develop complex algorithms in order to maximize
the ranging accuracy of UWB in these situations. Several approaches for this
have been explored, such as auto-encoders [4], probabilistic learning [5] and
path detection algorithms [6].

The IEEE 802.15.4 UWB physical layer (PHY) standard [7] contains several
settings that can be selected: for example, channel, data rate (DR) and
preamble symbol repetitions (PSR). These settings have shown to be highly
influential on the radio sensitivity and energy consumption [8] and could
enable more reliable and energy efficient UWB communication. However, it is
challenging to find a reliable setting when a poor link is detected. For
example, multipath reflections can cause destructive interference and to avoid
it, a change of channel (frequency) is necessary even if it would be in theory
a less reliable channel. Currently, hard-coded UWB PHY settings are used in
most scientific papers and practical UWB systems, and these are not
dynamically adapted with respect to a change in the environment [8]. This
makes them ineffective at sustaining a high reception rate [9]. A second
factor is that Qorvo DW1000-based devices report numerous link quality
measures for each received packet, such as the received (RX) power, first path
(FP) power, channel impulse response (CIR) and so on [10]. This information
could be used to characterize the environment and aid in selecting the optimal
PHY settings. The combination of the influence of the PHY layer settings and
the available link state information gives the opportunity to dynamically
select PHY settings based on the environment. By doing this separately on
different UWB links, we can enable a decentralized solution for adaptive UWB
communication in complex, dynamic environments without the need for a central
gateway to handle all communication. This dynamic link adaption is still
mostly unexplored for UWB communication in scientific literature, certainly
compared to accuracy improvement algorithms.

Since the aim is to solve the problem in a decentralized way, i.e., directly
on anchors, reinforcement learning (RL) is well suited as it can reach an
optimal solution for a control problem by interaction with the environment. A
RL agent interacts with the environment in discrete time steps. At each time,
it takes an action and then receives feedback for that action in the form of a
reward. The agent uses a policy to maximize the expected cumulative rewards by
rewarding desired behavior and punishing undesired ones. In RL, the
environment can be formulated as a Markov Decision Process (MDP), the value of
the current state should suffice to determine transition probabilities and
rewards following an action selection. This enables it to perceive and
interpret its environment without requiring a model of the environment.
Q-learning [11] is a commonly used RL algorithm that has already been used for
several cognitive communication applications [12, 13, 14, 15, 16]. However,
Q-learning is only suitable for problems with a low-dimensional state space
because the learned policy is stored in a table. This is unfeasible for large
action and state spaces, as it a) lacks generalization [17] and b) increases
the size of the Q-table, which would result in slower convergence. To mitigate
this, the table can be replaced by a neural network that takes the state as
input and approximates the Q-table values. This method is called deep
Q-learning, and it has also shown its potential in cognitive communication
applications [18, 19, 17].

The main contributions of the paper are the following:

  * •

Design of a decentralized approach for solving adaptive UWB settings using
Deep Q- and Q-learning that is executed at the anchor level without the need
of information from the tag.

  * •

Identification of the most relevant UWB link features for adaptive UWB PHY
problems.

  * •

Comparison of deep Q-learning, Q-learning and linear search in terms of PRR
and energy consumption in static and dynamic environments. Algorithmic
complexity and time complexity of the proposed approaches are also presented.

  * •

Collection of UWB dataset in a realistic environment using 72 different UWB
PHY settings, which is made publicly available111The dataset will be made
available together with the final version of the paper, and is currently
available on request. https://github.com/dietercoppens/UWB-DRL-PHY-Runtime-
Adaptation-dataset.

The remainder of this paper is organized as follows. Section II discusses the
related work for both UWB PHY adaptation and deep Q-learning. In section I,
the UWB communication problem and the system model are described. Section IV
describes the proposed deep Q-learning and Q-learning algorithms. Next,
sections V-A and V-B describe the environment in which the dataset is gathered
and how the measurements are performed. Section V-C discusses the performance
of the developed algorithms in a static and dynamic environments. This is
followed by the conclusion in Section VI

##  II Related Work

The related work is divided into a) adaptive UWB communication and b)
Q-learning and Deep Q-learning for wireless communication.

###  II-A Adaptive UWB communication

Although numerous papers aim to optimize the ranging accuracy by using
advanced algorithms [5, 6, 4], e.g. by performing error correction or
detecting LOS or NLOS, these solutions typically do not consider link
reliability or energy consumption aspects. This section provides an overview
of recent papers that investigate the influence UWB PHY settings on these
metrics.

The authors of [8] present a study of the influence of different PHY settings
on the reliability and energy efficiency of UWB communication. For example,
increasing the pulse repetition frequency (PRF) provides a slight improvement
in reliability, but it reduces the energy efficiency and increasing the PSR
provides a greater increase in reliability, but it also has a worse effect on
the energy efficiency. This study shows that by adjusting these settings, the
radio sensitivity can be increased, but this generally comes at the cost of
energy efficiency. Next, they estimate the link quality based on the CIR and
try to extract characteristics of the surrounding environment from this.
Finally, a scheme is designed to adapt the PHY setting based on the estimated
link quality. The only used link state metric is CIR, and the whole logic is
based on experimental data in only LOS conditions, which is not realistic for
practical applications where NLOS conditions cannot be ignored. For
characterization of LOS/NLOS situations to adapt the PHY setting, the authors
refer to other papers using machine learning techniques. This paper aims to
solve these limitations by using several metrics to characterize the
environment, using realistic experimental data in both LOS and NLOS situations
and removing the need for separate algorithms to detect LOS/NLOS. In [20], the
energy consumption of UWB communication is analyzed based on the PHY layer
setting. The authors show that more energy is consumed for lower data rates
due to the higher number of pulses that need to be transmitted, which causes
the system to be turned on longer. [21] focuses on the influence of the UWB
PHY settings on the ranging accuracy when using asynchronous two-way ranging.
The results were obtained using an extensive measurement campaign where more
than 200 UWB PHY settings were tested. The used radio channel, DR and PRF were
found to strongly influence the ranging accuracy, but no algorithm was
provided to dynamically modify the setting based on the environment. The
authors of [22] present a method to give a reliability indication to an UWB
distance measurement. For this, diagnostic information available on Qorvo’s
DW1000 UWB radio chip is used. The diagnostic information, such as CIR and
leading edge detection (LDE)—threshold, are processed together to determine
the estimated quality of the ranging measurement. In [23], a framework to make
UWB more robust is proposed. The framework uses the distance between the first
path and the ambient noise to determine low quality ranging. If low quality
ranging is detected, a linear search algorithm that tries out all UWB settings
within certain data rate, energy consumption, error rates, and robustness
requirements. The best performing setting is then selected. This approach has
some drawbacks, the UWB settings are only changed when a certain quality
threshold is passed, which means that the performance is not continuously
evaluated and optimized and the linear search algorithm means that every
possible UWB PHY settings needs to be tried out first before the best
performing one is found and configured.

The authors of these papers discuss the influence of the PHY settings on the
reliability and energy consumption and how to give a quality indication to a
range. However, they do not elaborate on how to use the link state information
and link diagnostics to dynamically improve the UWB reliability and energy
consumption of UWB communication. We will address these limitations by
developing a RL algorithm that dynamically selects the best performing setting
without a) the need to try out every possible setting and b) requiring
additional knowledge of the link state parameters. The best performing setting
is continuously chosen by the algorithm.

###  II-B Q-learning and Deep Q-learning for wireless communication

Q-learning and Deep Q-learning have been used for several resource management
wireless communication problems. The authors of [18] proposed Deep Q-learning
for a resource allocation mechanism for V2V communications. In [17] it is used
for downlink power allocation in multicell systems and the authors of [19]
proposed Deep Q-learning for circumstance-independent resource allocation with
efficient scheduling and power allocation. These papers show that RL, and in
particular Q- and deep Q-learning, can be successful in wireless communication
resource management. However, it has not yet been used for UWB communication,
as it requires a different system and algorithm design and the proposed
algorithms in wireless communication resource management problems cannot be
directly applied for solving the adaptation of UWB PHY settings problem. There
is a clear research gap in (1) developing a decentralized algorithm that
adapts the UWB PHY setting in runtime using deep Q-learning while taking into
account both reliability (PRR) and energy consumption and (2) evaluating the
algorithm on a realistic dataset containing both LOS and NLOS situations.

##  III Problem and system description

TABLE I: List of DW1000 provided link state information parameters Features (Abbreviation) | Description
---|---
$CIR$ | The array of accumulated complex Channel Impulse Response (CIR)
$F_{1}$ | The amplitude of the first harmonic of the first path (FP) signal
$F_{2}$ | The amplitude of the second harmonic of the FP signal
$F_{3}$ | The amplitude of the third harmonic of the FP signal
$CIR\\_{power}$ | The CIR power
$N_{c}$ | The standard deviation of the noise reported in the CIR accumulator
$RX_{pacc}$ | The preamble accumulation count at the receiver
$fpindex$ | The index of the detected FP
$FP_{p}$ | The estimated FP power level in $dBm$ and can be calculated as
| FP power
level$=10\log_{10}\left(\frac{F_{1}^{2}+F_{2}^{2}+F_{3}^{2}}{N^{2}}\right)-A$
| where $A$ is $113.77$ for a PRF of $16MHz$ and 121.74 for a PRF of $64MHz$,
|  $N$ represents preamble accumulation count
$RX_{p}$ | The estimated RX power level in $dBm$ and can be calculated as
| RX power level = $10\log_{10}\left(\frac{CIR_{power}\times
2^{17}}{N^{2}}\right)-A$
$NLOS$ | The power difference between the RX power and FP power and can be calculated as
| $\textrm{NLOS}=\textrm{RX power level}-\textrm{FP power level}$
| Can be used as an indicator for NLOS. If greater than 10 dB, likely to be in
NLOS
$PR$ | The power ratio between the RX power and FP power and can be calculated as
| $\textrm{PR}=\textrm{RX power level}/\textrm{FP power level}$
$\hat{d}$ | The estimated range and can be calculated as $\hat{d}=c\times\tau$
| where $c$ represents the speed of light in $m/s$ and $\tau$ the signal
propagation time from tag to anchor.
$LDE$ | Leading edge detection (LDE) threshold used to find the first path, based on an estimate of the noise.
$PP$ | Amplitude of the peak path (PP) in the CIR.
$PP\\_index$ | Index of the PP in the CIR.
$Q_{1}$ | The ratio between $F_{2}$ and $N_{c}$
| $Q_{1}=F_{2}/N_{c}$
| Comparing the noise with $F_{2}$ can give additional indication of the
quality of the FP measurement.
$Q_{2}$ | The ratio between $F_{2}$ and $LDE\\_threshold$
| $Q_{1}=F_{2}/LDE\\_threshold$
| Comparing the $LDE\\_threshold$ with $F_{2}$ gives additional indication of
the quality of the $LDE\\_threshold$

Figure 1 illustrates environmental challenges that cause problems for UWB
communication in indoor environments. Obstacles in the environment can cause
attenuated NLOS paths between the devices, and reflections can cause multipath
and interference. These effects have a major impact on both the reliability
and ranging accuracy.

Figure 1: An illustrative picture of a complex indoor environment causing
multipath and attenuated communication

###  III-A UWB link state information

The Qorvo DW1000 UWB radio chip reports several diagnostic link state
parameters that can be used to characterize the environment and determine the
state of the link. Table I gives an overview of the link state parameters that
are available. All diagnostics can be found in the user manual [24] as well.
Using these parameters, estimates can be made of link quality [8, 22] and the
presence of LOS/NLOS [25, 10].

Figure 2: Structure of a UWB PHY frame in the IEEE 802.15.4 standard [7].

###  III-B UWB PHY settings

In UWB communication, several PHY settings can be configured that have an
influence on the sensitivity and energy consumption. Important to mention is
that in this section, we focus on the settings that are supported by the Qorvo
DW1000 chip that is used to gather the dataset.

  * •

Channel: The IEEE 802.15.4 UWB PHY defines 16 different channels. A channel is
a combination of center frequency and bandwidth. Changing the center frequency
and bandwidth influences the perceived noise and can be an important parameter
to increase the robustness.

  * •

PSR: The UWB PHY packet (shown in Figure 2) starts with a synchronization
header containing a preamble (used for frame synchronization and detection)
and a start-of-frame delimiter (SFD) that indicates the end of the preamble
and the time of arrival used for ranging. The length of the preamble can be
changed by changing the amount of repetitions of the preamble symbol that are
sent. A higher PSR increases the robustness because there is a higher chance
of reception, but it increases the radio’s energy expenditure due to the
longer length.

  * •

PRF: In UWB communication, a bit is transmitted using a train of pulses. The
speed at which these pulses are sent is a configurable setting. By increasing
the PRF, the amount of pulses sent per unit of time increases. This means that
there are more threshold decision events during, and thus a higher accuracy
and robustness. This comes at the cost of higher energy consumption.

  * •

Data rate (DR): Lowering the DR increases the reliability of the
communication, but causes a longer transmission time and thus higher energy
consumption.

  * •

Transmit power gain ($P_{tx}$): The transmit power gain can be adjusted, a
higher power gain means reliable communication can be achieved for longer
distance between the devices, but this also comes at the cost of higher energy
consumption.

Summarizing the information from these settings, we can see that there is a
trade-off between the reliability and energy consumption [8]. However, it is
challenging to find a reliable PHY setting when a poor link is detected. For
example, multipath reflections can cause destructive interference and to avoid
this, a change of channel (carrier frequency) is necessary even if it would be
in theory a less reliable channel (higher carrier frequency).

###  III-C System description

The list of notations used in the paper is given in Table II. We consider a
link between two UWB devices, a tag and an anchor (as shown in Figure 1) using
the IEEE 802.15.4 UWB PHY standard. The goal is to evaluate the link state and
adjust the PHY settings (in real time) on the anchor to ensure that the UWB
communication is always as good as possible, in terms of reliability (PRR) and
energy consumption, even in dynamic and changing environments. This PHY
setting is then communicated to the tag UWB device and finally configured on
both. The PHY setting can be combined into a single settings variable $A$ as
shown in (1).

$A=\left\\{C,PSR,PRF,DR,P_{tx}\right\\},$ (1)

where $C\in\\{3,5,7\\}$, $PSR\in\\{128,1024,4096\\}$, $PRF\in\\{16,64\\}$,
$DR\in\\{110,6800\\}$ and $P_{tx}\in\\{0,10.5\\}$. These values are determined
by our collected dataset.

TABLE II: List of notations and abbreviations used in the paper. Notation/Abbreviation | Description
---|---
$e$ | Environment
C | Channel
DR | Data rate
$A$ | UWB PHY setting variable
TX | Transmit
RX | Receive
$P$ | Power consumption
$I$ | current consumption
$S_{p}$ | The number of symbols in the complete
| preamble
$S_{s}$ | The number of symbols in the SFD
$S_{d}$ | The number of symbols in the data
| part of the frame
$b_{p}$ | The number of bits in the PHR
$B_{p}$ | The number of bytes in the payload
$R_{FEC}$ | The forward error correction rate
$T_{sym}$ | The duration of a symbol
$T_{p}$ | The total duration of the preamble
$T_{d}$ | The total duration of the data part
$E_{p}$ | Energy consumption of the preamble
$E_{d}$ | Energy consumption of the data part
$E_{rx}$ | Energy consumption during transmission
$E_{tx}$ | Energy consumption during reception
$P_{p}$ | Power consumption during the preamble
$P_{d}$ | Power consumption during the data part
$E$ | The total energy consumption for a range
$G$ | The complete system with respect to $A$-th
| setting and $e$-th environment
$s_{t}$ | State at time t
$f$ | List of all available features
$Xcor$ | The cross correlation between two vectors.
$\alpha$ | Learning rate in Bellman equation
$\gamma$ | Discount factor in Bellman equation
$F$ |  $F$-value of a feature
$R_{t}$ | The reward at time $t$
$\epsilon$ | The epsilon-greedy parameter
$\theta$ | Weights of main Deep Q-network
$\hat{\theta}$ | Weights of target Deep Q-network
$\zeta$ | Importance sampling prioritization
| parameter
$\beta$ | Importance sampling bias compensation
| parameter
$p_{i}$ | Priority or TD-error of experience $i$
$Prob(i)$ | Sampling probability of experience $i$
$w_{i}$ | Importance sampling weight of experience $i$
TABLE III: Overview of I($A$) for the different modes in the system [10] Channel |  | PRF
---
(MHz)
| Data rate
---
(Mbps)
| Preamble
---
TX
current (mA)
| Preamble
---
RX
current (mA)
| Data
---
TX
current (mA)
| Data
---
RX
current (mA)
3 | 16 | 0.11 | 68 | 113 | 35 | 59
3 | 16 | 6.8 | 68 | 113 | 50 | 118
3 | 64 | 0.11 | 83 | 113 | 40 | 72
3 | 64 | 6.8 | 83 | 113 | 52 | 118
5 | 16 | 0.11 | 74 | 118 | 42 | 62
5 | 16 | 6.8 | 74 | 118 | 57 | 123
5 | 64 | 0.11 | 89 | 118 | 46 | 75
5 | 64 | 6.8 | 89 | 118 | 59 | 123
7 | 16 | 0.11 | 74 | 118 | 42 | 62
7 | 16 | 6.8 | 74 | 118 | 57 | 123
7 | 64 | 0.11 | 95 | 124 | 52 | 81
7 | 64 | 6.8 | 95 | 124 | 65 | 129

The environment is described by parameter $e$, this indicates the influence of
the environment on the system. The parameter is adjusted when changes happen
in the environment in which the UWB ranging is taking place. The ranging
method used in the system is called Asymmetric Double Sided TWR (ADS-TWR)
[26], which means that for one range estimation, there are three packets
transmitted (TX) and received (RX). Combining this with the configured setting
$A$, the energy consumption can be determined. Table III shows the current
consumption ($I$) for the different settings during TX and RX for both the
preamble and data parts in mA. Using this information and the constant supply
voltage of 3.3 V, the power $P$ during the preamble and data parts of the
frame for both TX and RX can be calculated using (2). Note that we use $P(A)$
which means the power with respect to the PHY setting $A$. This notation is
used for other quantities as well.

$P(A)=3.3\>\cdot I(A).$ (2)

The number of symbols in the preamble $S_{p}(A)$ and data parts $S_{d}(A)$ can
be calculated by (3) and (4), using the 12 data bytes ($B_{p}$), the number of
bits in the PHR ($b_{p}$) the forward error correction rate ($R_{FEC}=0.87$)
and the SFD symbols ($S_{s}$) (64 for 110 kbps and 8 for other data rates).

TABLE IV: DW1000 symbol durations in the separate parts of the frame [10] | PRF
---
(MHz)
| Data Rate
---
(Mbps)
| $\boldsymbol{T_{sym}}$
---
SHR (ns)
| $\boldsymbol{T_{sym}}$
---
Data & PHR
(ns)
16 | 0.11 | 993.59 | 8205.13
16 | 6.8 | 993.59 | 1025.64
64 | 0.11 | 1017.63 | 8205.13
64 | 6.8 | 1017.63 | 128.12

$S_{p}(A)=PSR+S_{s}(A).$ (3) $S_{d}(A)=(b_{p}+\frac{B_{p}\cdot 8}{R_{FEC}}).$
(4)

The duration of the preamble $T_{p}(A)$ and data $T_{d}(A)$ parts (in seconds)
can then be calculated using (5) and (6) where the symbol duration ($T_{sym}$)
is computed from Table IV.

$T_{p}(A)=S_{p}(A)\cdot T_{sym}(A).$ (5) $T_{d}(A)=S_{d}(A)\cdot T_{sym}(A).$
(6)

Using the power and duration of the separate parts, we can calculate the
energy consumption of the preamble $E_{p}(A)$ and data $E_{d}(A)$ parts as
shown in (7) and (8).

$E_{p}(A)=P_{p}(A)\cdot T_{p}(A).$ (7) $E_{d}(A)=P_{d}(A)\cdot T_{d}(A).$ (8)

The energy consumption is the combination of the data and preamble parts as
shown in (9).

$E(A)=E_{p}(A)+E_{d}(A).\>$ (9)

Finally, the total energy consumption (in Joules) for a range using a certain
setting is found using (10). For the energy consumption during TX, an extra
factor $10^{\frac{P_{tx}}{10}}$ is added to account for the $P_{tx}$ gain in
the setting.

$E(A)=3\cdot(E_{rx}(A)+E_{tx}(A)\cdot 10^{\frac{P_{tx}}{10}}).$ (10)

The second factor to optimize in the system is the PRR, this factor depends on
both the environment $e$ and used setting A. The complete system $G(A,e)$ with
respect to the $A$-th setting and $e$-th environment can be described by (12).
It is a combination of the PRR and the scaled and normalized energy
consumption ($E$).

$PRR=\frac{received\>packets}{total\>transmitted\>packets}.$ (11)

$G(A,e)=PRR(A,e)+\left(1-\frac{E(A)-min(E(A)}{max(E(A))-min(E(A))}\right).$
(12)

###  III-D Problem description

The goal is to enable reliable UWB communication while consuming as little
energy as possible with the available settings. This is depicted below.

Maximize: $\displaystyle
G(A,e)=PRR(A,e)+\left(1-\frac{E(A)-min(E(A)}{max(E(A))-min(E(A)}\right)$
Constrained by: $\displaystyle A$

##  IV Algorithm design

In this section, the proposed deep Q-learning algorithm for runtime adaptation
of UWB PHY settings while maximizing $G(A,e)$ as introduced in Section III.
First, the key parts of RL are introduced. Then, the feature selection step is
shown. Next, we will discuss why the discrete nature of Q-learning limits to
what extent link estimation can be optimized. Finally, a more advanced deep
Q-learning algorithm is proposed.

###  IV-A Reinforcement learning

Figure 3: High-level overview of RL system for UWB PHY run-time adaptation

As shown in Figure 3, the RL framework is composed of an agent and an
environment interacting with each other. An agent considers the link between
an anchor and a tag UWB device and is executed at the anchor. Anything in the
area around the two UWB devices that could possibly affect communications is
regarded as the environment. As in Figure 3, at each time $t$, the UWB link
observes a state $s_{t}$, and takes an action $a_{t}$. The action selection is
determined by a state-action function, $Q(s_{t},a_{t})$. The environment
transitions to the next state $s_{t+1}$ and receives a reward $R_{t}$ based on
the action taken. Specifically for this problem, the objective function $G$
can be seen as a first approximation of the reward. The UWB PHY setting,
expressed as:

$A=\left\\{C,PSR,PRF,DR,P_{tx}\right\\},$

is the action space from which action $a_{t}$ is selected. The state is a
combination of the link state parameters from Table I, expressed as:

$s_{t}=\left\\{CIR,F_{1},F_{2}...,Q_{2}\right\\}.$

###  IV-B State feature selection

The DW1000 UWB chip reports a significant amount of diagnostics for assessing
the link quality. Using all of them for the state $s_{t}$ would considerably
increase the complexity of the proposed Q-learning and deep Q-learning
algorithms. Therefore, a feature selection step is necessary. Reinforcement
learning consists of two phases, namely an initial exploration phase, where
many actions are tried out to find out in general which actions perform well
and the optimization phase, where the system tries to find the optimal action
to select. To select features that are representative for both phases, there
are two factors based upon which features can be prioritized: (1) for
exploration, their impact on selecting which action to take and (2) to find
the optimal action, their influence on the objective function $G$. To reflect
these two factors, feature selection is performed for both classifying which
setting to select and predicting the value of $G$. To select the best features
to predict $G$, we use the F-value. This calculation is performed in two
steps:

Figure 4: Overview of the most important features, based on F-value, for both
classification of the setting (action) and prediction of the G used in the
run-time adaptation of the UWB PHY layer

  1. 1.

The cross-correlation $Xcor$ between each feature $f(i)$ in list of features f
and target $G$ is calculated using (13).

  2. 2.

The cross correlation is converted to the F-value using equation 14.

$Xcor(i)=\frac{E\left[(f[i]-mean(f[i])\cdot(G-mean(G))\right]}{std(f[i])\cdot
std(G)}$ (13) $F(i)=\frac{Xcor(i)^{2}}{1-Xcor(i)^{2}}\cdot\nu,$ (14)

where $\nu$ is the degrees of freedom. ($\nu=n-1$ with $n$ is the number of
features $f$ in the dataset). The features with the highest F-value have the
highest correlation to the value of $G$ and will thus be the most important
features. To select the best features to classify which setting is the best,
the Analysis of variance (ANOVA) F-value can be used. This F-value is similar
to the previously explained F-value, only now the result is a categorical
value (the setting) instead of a numerical value. The F-values for the six
most important features for both F-value calculations are shown in Figure 4.
From this, we can select the most important features to be used in our system:
$RX_{p}$ (The estimated RX power level), $FP_{p}$ (The estimated FP power
level), $NLOS$ (The difference between $RX_{p}$ and $FP_{p}$), $N_{c}$ (The
standard deviation of the noise), $Q_{1}$ (The ratio between the amplitude of
the second harmonic ($F_{2}$) and $N_{c}$), $RX_{pacc}$ (The preamble
accumulation count), $LDE$ (leading edge detection threshold, $Q_{2}$ (The
ratio between the amplitude of the second harmonic ($F_{2}$) and $LDE$, PRR
(The ratio of received packets to total transmitted packets).

These will be used as the state of the RL system, expressed as:

$s_{t}=\left\\{RX_{p},FP_{p},NLOS,N_{c},Q_{1},RX_{pacc}.LDE,Q_{2},PRR\right\\},$

###  IV-C Q-learning

Q-learning is a model-free RL algorithm that learns the value of an action in
a certain state. The most important part of the algorithm is the Q-table. Each
row in the table represents a state of the system, and each column represents
an action. Each value in the table represents the ’quality’ of a particular
state-action pair. A more detailed schematic of the Q-learning algorithm is
given in Figure 5.

Figure 5: Schematic showing the Q-table and Bellman update equation for the
Q-learning run-time adaptation of UWB PHY settings

####  IV-C1 Updating the Q-table

The foundation of the Q-learning algorithm is the Bellman equation that is a
value update function using the newly received information and weighted old
value:

$\displaystyle Q_{new}(s_{t},a_{t})$ $\displaystyle\leftarrow
Q_{old}(s_{t},a_{t})$ (15)
$\displaystyle+\>\alpha\left[R_{t}+\gamma\max_{a}Q(s_{t+1},a_{t})-Q_{old}(s_{t},a_{t})\right].$

The following parameters are used in the equation:

  * •

$\alpha$: learning rate. This factor determines the weight that is given to
the newly acquired information and how much old information can be overridden.

  * •

$\gamma$: discount factor. This factor determines the weight that is given to
newly acquired information.

Using this function, the values in the Q-table are filled in. After enough
iterations, the values in the table will reflect the quality of the state-
action pairs and the expected value of the total received rewards will be
maximized. The reward clearly drives the behavior of the algorithm and the
reward function, a mapping of a state-action pair to a numerical value that
indicates the desirability, is thus crucial. Applied to this problem, the
reward function needs to describe the value of selecting a certain setting
based on the link state information. The objective function $G$ discussed
before is a potential reward function. However, this function has a problem.
It could lead to the selection of a certain setting based purely on a low
energy consumption. Having a low energy consumption is meaningless when no
communication is possible using that low energy setting. To mitigate this
problem, the following reward function is proposed:

$R_{t}=PRR+PRR\cdot\left(1-\frac{E-min(E)}{max(E)-min(E)}\right).$ (16)

In this function, the energy consumption factor is multiplied by the PRR. This
causes the energy consumption to influence the reward proportionally to the
reliability of the communication.

At each point in time, the agent selects an action to be taken using the
Q-table. This action selection is a fundamental trade-off in RL. Initially,
the agent does not yet know the outcome of the possible actions. Hence, high
enough exploration (not selecting action with highest Q-value) is required.
Once the agent has learned more information, the exploration can be reduced by
exploiting (selecting the action with highest Q-value) the learned
information. However, only exploiting is a dangerous approach, as the agent
can get stuck in a suboptimal state.

During the training stage, the epsilon-greedy policy is used. In this policy,
the best action (highest Q-value) is selected with a probability of
1-$\epsilon$. With a probability of $\epsilon$ a random action is chosen
uniformly.

For evaluation, the epsilon-greedy policy is modified. Now, in exploration,
the random selection is changed to a random selection among the 10 actions
with the highest Q-value.

####  IV-C2 State determination

The biggest drawback in using Q-learning is that we need to determine a
discrete state number from continuous state variables. The more granularity we
introduce in the states, the more states are needed and the size of the
Q-table increases as well. The state consists of the PRR and the selected
features, each split up in 3 categories: low, middle and high. This gives a
total of 19,683 possible link states or rows in the Q-table. Multiplying this
with the number of actions, there is a total of 1,417,176 cells. The Q-table
is huge, while we have only introduced very limited granularity on the
features and performed feature selection. Without feature selection and the
same limited granularity, there would be $83.7e^{9}$ cells in the Q-table.
This demonstrates the need for the feature selection step. The second drawback
is that we also need to determine the low, middle and high separation for each
feature manually. This means that some ’expert knowledge’ of UWB diagnostics
is necessary that has a large influence on the performance of the algorithm.

The pseudocode for the complete Q-learning algorithm is given in Algorithm 1.
The Q-learning algorithm was trained for 500,000 steps with $\alpha=0.8$ and
$\gamma=0.5$. $\epsilon$ (from the epsilon-greedy policy) changed during the
training, following an exponential decay (17).With $\lambda$ decay constant
and ’step’ the amount of training steps that have already been performed.

$\epsilon=\epsilon_{min}+(\epsilon_{max}-\epsilon_{min})\cdot e^{-\lambda\cdot
step}$ (17)

where $\epsilon_{min}=0.01$, $\epsilon_{max}=1$ and $\lambda=3.91e^{-5}$.

Algorithm 1 Q-learning for UWB PHY

1:

2:Initialize Q-table

3:Initialize parameters $\alpha$, $\gamma$ and $\epsilon$

4:

5:Select random UWB link and action (setting)

6:Determine start state $s_{t}$

7:while step $<$ training steps do

8: Every 100 steps: select new random UWB link

9: With probability $\epsilon$ select random action $a_{t}$

10: Otherwise $a_{t}$ = $arg\max_{a}(Q(s,a))$

11: Perform action $a_{t}$ (configure setting)

12: Measure link quality indicators

13: Determine reward $R_{t}$ and new state $s_{t+1}$

14: $Q_{new}(s_{t},a_{t})\leftarrow Q_{old}(s_{t},a_{t})$

15:$+\alpha\>\left[R_{t}+\gamma\cdot\max_{a}Q(s_{t+1},a_{t})-Q(s_{t},a_{t})\right]$

16: Update $\epsilon$

17: $s_{t}$=$s_{t+1}$

18:end while

###  IV-D Deep Q-learning

To mitigate the previously mentioned drawbacks, the Q-learning algorithm can
be modified to a Deep Q-learning algorithm. In Deep Q-learning, the Q-table is
replaced by a neural network that approximates the Q-value function. As
illustrated in Figure 6, the state is given as input and the Q-value of all
possible actions is the output.

Figure 6: Illustration of input and output of a Deep Q-network

Using a neural network instead of a table has several benefits:

  * •

The continuous values of the link state measurements can now be used directly
as input (only scaling between 0 and 1).

  * •

No ’expert knowledge’ is necessary to determine categories for the inputs.

  * •

No loss of information due to discretization of measurements.

  * •

No need for a table containing millions of cells.

####  IV-D1 Target network

In supervised learning, the label of an input does not change over time. This
stable condition for input and output allows it to perform well. In RL, both
the input and the target to obtain change constantly during the learning
process. This makes training unstable because the target or ’label’ of the
output $Q$ depends on $Q$ itself. (18) shows the target value based on the
Bellman equation with $\theta$, the weights of the $Q$ neural network.

$target=R_{t}+\gamma\cdot\max_{a}Q(s_{t+1},:,\theta)).$ (18)

This means that the target will move as the $Q$ approximation improves. The
moving target could cause the ’chasing your own tail’ problem [27]. To
mitigate this issue, a target network can be introduced. The target network is
used to calculate the $Q$-values and fix the target for as long as we fix the
target network. The main network updates its weights $\theta$ using the
training data to minimize the following loss function (19) on a dataset $D$.

$Loss(\theta)=\sum_{s_{t},a_{t}\in D}(target-Q(s_{t},a_{t},\theta))^{2}.$ (19)

After several updates of the main network, the weights of the main network are
copied to the weights of the target network $\hat{\theta}$. In practice, we do
not have two separate networks, but two instances of the neural network
weights. This prevents the training process from spiraling and increases the
chances of convergence [28]. The neural network architecture is given in Table
V. The input to the neural network consists of 14 values: a) 8 state features,
b) the 5 current configured PHY layer settings and c) the current PRR. The
output is an approximation of the Q-values for all actions in this state.

TABLE V: Neural network architecture DNN |
---|---
Layer | Output dimension
Input | 14
Dense(128), ReLu | 128
Dense(256), ReLu | 256
Dense(512), ReLu | 512
Dense(256), ReLu | 256
Dense(128), ReLu | 128
Dense(72), linear | 72
Output | 72

####  IV-D2 Weighted importance sampling

Updating and training a neural network is quite different from updating a cell
in a Q-table. Updating the neural network at every time step with one sample
would be very inefficient. Therefore, the network is updated on batches of
data that are sampled from a replay memory containing experiences
$(s_{t},a_{t},R_{t},s_{t+1})$ of the $Q$-agent. The sampling of the
experiences from the memory can be done in several ways. Popular methods are
random sampling, prioritized sampling (experiences with the highest reward)
and weighted importance sampling [29].

For this problem, we opted for weighted importance sampling. The central
component of this is the criterion on which the experiences are selected. The
optimal criterion would be the amount the $Q$-agent can learn from using an
experience. While this measure is not available, a reasonable approximation is
the temporal-difference (TD)-error, which indicates how surprising a certain
transition is. However, greedy TD-error prioritization has some issues. First,
to avoid expensive sweeps over the entire replay memory, TD errors are only
updated for the transitions that are replayed, which means that transitions
that have a low TD error on the first visit may not be replayed for a very
long time (or ever). Further, greedy prioritization focuses on a small subset
of the experience. This lack of diversity makes the system prone to
overfitting. To overcome these issues, a stochastic sampling method that
interpolates between pure greedy prioritization and uniform random sampling
can be used [29]. The priority or TD-error $p_{i}$ of an experience is
calculated in (20) and the sampling probability of each experience is given in
(21).

$p_{i}=R_{t}+\gamma\cdot\max_{a}\hat{Q}(s_{t+1},:,\hat{\theta}))-Q(s_{t},a_{t}).$
(20) $Prob(i)=\frac{p_{i}^{\zeta}}{\sum_{k}p_{k}^{\zeta}},$ (21)

where $\zeta$ is a hyperparameter that determines how much prioritization is
used. This priority is saved with the experience. Prioritized learning in this
way introduces bias, as it changes the solution that the estimates will
converge to [29]. This bias can be corrected by using importance sampling
weights (w) shown in (22).

$w_{i}=\left(\frac{1}{N}\cdot\frac{1}{Prob(i)}\right)^{\beta},$ (22)

where $N$ is the amount of experiences in the memory, and $\beta$ a
hyperparameter that determines how much the non-uniform probabilities $P(i)$
are compensated.

####  IV-D3 Update function

Combining this, the update function to generate the training data is given by
equation (23)

$Q(s,a_{t},\theta)=Q(s,a_{t},\theta)+\alpha\cdot p_{i}\cdot w_{i}$ (23)

The pseudocode for the Deep Q-learning algorithm is given in algorithm 2. This
algorithm was trained for 200,000 steps, with $\alpha=0.8$, $\gamma=0.5$.
$\epsilon$ changed during training following equation 17 with
$\epsilon_{min}=0.01$, $\epsilon_{max}=1$ and $\lambda=1.96e^{-5}$.

Algorithm 2 Deep Q-learning for UWB PHY

1:

2:Initialize replay memory $D$ to capacity $N$

3:Initialize action-value function $Q$ with random weights $\theta$

4:Initialize target action-value function $\hat{Q}$ with $\hat{\theta}$ =
$\theta$

5:

6:Select random UWB link and start action (setting)

7:Determine start state $s_{t}$

8:while step $<$ training steps do

9: Every 100 steps: select new random UWB link

10: With probability $\epsilon$ select random action $a_{t}$

11: Otherwise $a_{t}$ = $arg\max_{a}(Q(s,a,\theta))$

12: Perform action $a_{t}$ (configure setting)

13: Measure link quality indicators

14: Determine reward $R_{t}$, new state $s_{t+1}$ and priority $p_{t}$

15: Store $e_{t}$ = ($s_{t}$, $a_{t}$, $R_{t}$, $s_{t+1}$) with priority
$p_{t}$ in $D$

16: Every M steps:

17: Sample minibatch b of $e_{j}$ $\sim
Prob(j)=\frac{p_{j}^{\zeta}}{\sum_{k}p_{k}^{\zeta}}$

18: For all $e_{j}$ in b: current_Qs: $Q(s,:,\theta)$

19: For all $e_{j}$ in b: future_$Q$: $\hat{Q}(s_{t+1},:,\hat{\theta}))$

20: X = $s_{t}$ for all $e_{j}$ in $b$

21: for all $e_{j}$ in b:

22: $max\\_future\\_q=R_{t}+\gamma\cdot\max(future\\_Qs)$

23: $p_{i}=|$max_future_q - current_Qs[$a_{t}$]$|$

24: $w_{i}=\left(\frac{1}{Prob(i)}\cdot\frac{1}{N}\right)^{\beta}$

25: current_Qs[$a_{t}$] = current_Qs[$a_{t}$] + $\alpha\cdot p_{i}\cdot w_{i}$

26: $Y$ = current_$Q$s for all $e_{j}$ in b

27: Update weights $\theta$: fit $Q$ using $X$ and $Y$.

28: Every T steps: reset $\hat{Q}$ = $Q$, i.e., $\hat{\theta}$ = $\theta$

29:end while

##  V Experimental evaluation

In this section, we evaluate the proposed Q-learning and deep Q-learning
algorithms to assess their performance compared to a linear search algorithms
and fixed UWB PHY settings. First, the environment in which the data for the
experiments is gathered is described. Then, we describe how the measurements
are performed, and finally we discuss and analyze the results.

###  V-A Office lab

The OfficeLab from of imec - IDLab - Ghent University [30] offers a test
environment which includes 3 floors that are equipped with 40 Intel NUC nodes,
supporting several Wi-Fi and sensor technologies including UWB. In our
evaluation, we use a single floor that has a total area of around 41 x 26 m2
and 15 UWB nodes placed in corridors, meeting rooms, and offices. All nodes
are placed at the same height of around 2.6 m above the floor. The walls
separating the rooms consist of heterogeneous materials, ranging from plywood
to reinforced concrete, resulting in a very heterogeneous environment. An
overview of the placement of the nodes is shown in Figure 7.

Figure 7: Data gathering setup with 15 UWB nodes (A0-A14) and the anchor-tag
links selected for dynamic evaluation distributed over the 9th floor of the
OfficeLab at imec - IDLab - Ghent University

###  V-B Dataset

The dataset was gathered using Wi-PoS: a Low-Cost UWB Hardware Platform with
Long Range Sub-GHz Backbone [31]. All 15 nodes are programmed once as tag and
try to range with the other 14 nodes (anchors) using the following settings
shown in Table VI.

TABLE VI: Parameters used in static test scenario for runtime adaptation of UWB PHY settings Parameter | Values
---|---
Channel | 3,5,7
PSR | 128, 1024, 4096
PRF | 16, 64 MHz
Data rate | 110, 6800 kbps
Transmit power gain | 0,10.5 dB

The combination of these settings give a total of 72 different UWB PHY
settings. ADS-TWR [26] is used to enable accurate ranging. In the
bidirectional communication, both nodes log their raw responses, which are
then combined and converted to a more readable file format, e.g., csv. Each
row of the dataset corresponds to one range between two nodes and contains the
distance estimation and the available link state information that is needed in
the algorithms. All devices have ranged with each other for a total of 500
range attempts per combination. This resulted in a total of more than 605
thousand ranges overall. This is considerably less than the total attempts, as
on average each node could only range with 7 other nodes due to obstacles like
walls.

###  V-C Results

In this section, we evaluate the proposed Q-learning and deep Q-learning
algorithms to assess their impact on (1) the time it takes to find the best
setting in a static situation (the environment and link is fixed) compared to
a linear search algorithm and (2) the reliability (PRR) and energy consumption
in a dynamic or changing environment compared to using fixed UWB PHY settings.

####  V-C1 Static environment

This experiment tries to evaluate the ability of the proposed model to
quickly, in terms of RL iterations (or amount of settings configured),
determine the best PHY layer setting when UWB communication is started between
two devices. For every possible pair of UWB devices in the dataset, the models
try to determine the best PHY setting in a certain amount of settings that can
be configured. We compare our models with a linear search algorithm that
sequentially goes through all 72 possible PHY settings and then selects the
one that performed the best. The parameters of the models used during this
evaluation are shown in Table VII.

TABLE VII: Parameters used in static test scenario for runtime adaptation of UWB PHY settings Parameter | Value
---|---
Update rate main network | 5
Update rate target network | 25
Learning rate | 0.7
Discount factor | 0.7
Batch size | 10

In Figure 8, the percentage of UWB links that found settings within 5% of the
highest possible reward (for that link) in function of the number of
iterations is shown for Q-learning and Deep Q-learning. The y-axis value is
explained mathematically in (24), with $D$ the complete dataset, $l$ a link in
between two UWB devices, $R$_selected($l$) the reward of the UWB PHY setting
selected by the algorithm for link $l$ and $R$_best($l$) the reward of the
setting with the highest reward for link $l$. A link is considered ’optimal’
if the reward of the selected UWB PHY setting is within 5% of the setting with
the highest possible reward, $Optimal(l)$ returns 1 in this case.

$\textrm{Optimal(l)}=\begin{cases}1&\text{R\\_selected(l)
}\geq\textrm{R\\_best(l)}\times 0.95\\\ 0&\text{otherwise}\end{cases}$ (24)

$\text{y}=\frac{\sum\limits_{\forall l\in
D}\text{Optimal(l)}}{\sum\limits_{\forall l\in D}1}$ (25)

The performance of a linear search algorithm is indicated on the figure as
well. This algorithm tries out every possible setting and selects the best
performing one in the end. In this scenario, there are 72 possible settings,
which means that the algorithm needs 72 iterations but is 100% sure that the
best setting is selected. Q- and deep Q-learning try to predict the best
setting based on the link state parameters. Figure 8 clearly shows that Deep
Q-learning performs better than Q-learning. After the first iteration, which
means starting at a random action (iteration 0) measuring the link state and
then configuring a setting, the Q-learning algorithm selects an optimal
setting for 39% of the links between UWB devices and Deep Q-learning for 84%.
After 10 iterations, the percentage has increased to 92% for Deep Q-learning,
while the Q-learning is still only around 40%. Comparing the algorithms when
they have used the same amount of iterations as linear search shows that Deep
Q-learning finds the best setting in 95% of the cases and Q-learning 70%.
These results show the importance of using Deep Q-learning instead of
Q-learning. The drawbacks of using Q-learning as discussed in IV-C are clearly
visible in these results.

Deep Q-learning shows a clear advantage over the linear search algorithm, as
it can select an optimal setting with a high accuracy after only a few
iterations. While the linear search algorithm requires 72 iterations.

Figure 8: The percentage of optimal configurations selected in terms of number
of iterations

####  V-C2 Dynamic environment

To test the performance of the algorithms in a dynamic environment, a test
scenario is defined as shown in Figure 7. To simulate a walk around the
office, every 5 seconds, one of the anchor-tag UWB nodes is switched to
abruptly change the situation and test how well the algorithm can adapt.
Harder links, containing more obstacles, require more energy. The color used
for the tag-anchor link indicates the minimal amount of energy necessary to
enable communication, as shown by the legend. The parameters used during this
experiment are shown in Table VIII.

TABLE VIII: Parameters used in dynamic test scenario for runtime adaptation of UWB PHY settings Parameter | Value
---|---
Ranging update rate | 50 Hz
Update rate main neural network | 20
Update rate target neural network | 100
Mini batch size | 256
Learning rate | 0.55
Discount factor | 0.55

In Figure 9, the PRR and the energy consumption during this test scenario are
shown for different cases, namely deep Q-learning, Q-learning and three fixed
PHY settings. These were chosen because they represent the difference in
performance of high and low energy settings and the influence of the
channel.The fixed PHY settings are (1) a high energy consuming setting on
channel 7 {7, 4096, 64, 6800, 10.5}, (2) a high energy consuming setting on
channel 3 {7, 4096, 64, 6800, 10.5} and (3) a low energy consuming setting on
channel 7 {7, 128, 64, 6800, 0}.

Figure 9: Comparison of the proposed Q-learning and deep Q-learning algorithms
with constant high energy and low energy PHY settings in terms of PRR and
energy consumption.

Figure 9 shows that using a fixed PHY setting does not guarantee good
reliability. Communication drops off completely at several links for the fixed
PHY settings on channel 7, even for the high energy consuming one. The high
energy consuming setting on channel 3 has better reliability, except for the
single drop to 40%.

Q-learning has no links where the communication drops off completely, but
fails to provide true reliability as the PRR is never 100%. This means that
frames are continuously lost. However, it still uses considerably less energy
than the two high energy consuming ones. However, it does not select very low
energy consuming settings.

Figure 9 shows that the Deep Q-learning algorithm can ensure reliable
communication, by keeping the PRR constantly very close to 100%, while
dynamically selecting settings with higher energy consumption when necessary.
This is well demonstrated by the sharp increase in energy consumption at time
intervals 5-10 and 35-40 seconds of the evaluation. In these time intervals,
the UWB communication is happening over a red link (number 2 and 8) in 7.
Between 40 and 65 seconds, we can clearly see that the algorithm can also make
the distinction between links with a smaller difference in required energy
consumption (green vs. yellow). However, deep Q-learning never selects a PHY
setting with an energy consumption as low as the low energy setting, even
though this setting can enable reliable communication for certain links.

Table IX compares the average performance of the five different cases in the
figure in terms of PRR, energy consumption and ranging error.

TABLE IX: Average performance comparison of the different methods in the test scenario  Fixed radio configuration / Proposed algorithm | Average energy | Average | Average
---|---|---|---
| consumption (W$\mu$ s) | PRR (%) | ranging error (mm)
Channel 7, small preamble and low $P_{tx}$ | 373.92 | 30.59 | 295.018
Channel 7, large preamble and high $P_{tx}$ | 49572.15 | 61.23 | 318.02
Channel 3, large preamble and high $P_{tx}$ | 43497.42 | 94.25 | 235.92
Q-learning | 17782.15 | 93.26 | 243.45
Deep Q-learning | 5898.92 | 99.31 | 240.65

Deep Q-learning clearly performs best, it has the highest average PRR, the
lowest energy consumption and good ranging error performance. The high energy
consuming setting on channel 3 comes closest based on PRR and has slightly
better ranging error performance, but it consumes more than seven times the
amount of energy. While Q-learning performs considerably worse than deep
Q-learning, it has a similar PRR and ranging performance as the high energy
consuming setting on channel 3 and consumes less than half the amount of
energy. The fixed settings on channel 7 perform the worst as the PRR is low,
even when consuming a lot of energy.

###  V-D Complexity analysis

####  V-D1 Algorithmic complexity

The complexity of the Q-learning and Deep Q-learning is mainly determined by
the complexity of determining the Q-value approximation belonging to a certain
state [32] and updating the model or table. For Q-learning, this is simply
selecting a row from a table $O(1)$ and changing one value in the table. For
the Deep Q-learning, the complexity depends on the structure and
hyperparameters of the neural network. For a neural network composed by fully-
connected layers, the complexity is given by $O(mn$ log $n)$, with $m$ the
number of layers and $n$ the number of units per layer [33].

####  V-D2 Time complexity

For Q-learning, the time to calculate the optimal setting and update the
Q-table based on the received information is dependent on the time to select
the maximal value in a row and update one value in the Q-table. When executing
the algorithm on a Windows laptop containing an Intel(R) Xeon(R) E3-1200/1500
v5/6th Gen, this takes on average 6.89 ms. For the Deep Q-learning algorithm,
this time is dependent on the inference time of the neural network and the
time to update the neural network with a new batch of data. This time depends
on the size of the model, as well as the batch size and update rate of the
neural network. The main factor contributing to this is the update rate of the
model. As retraining, the model is the most extensive task. A higher update
rate makes the model more dynamic, but has the drawback of longer time
complexity. For the settings used during the evaluation shown in Table VIII
the average time is 23.18 ms, while the training time with one batch takes on
average 320 ms.

##  VI Conclusion

For the increasingly prominent indoor localization technology UWB, accurate
ranging has been extensively researched. However, enabling reliable, low
energy consuming UWB communication in dynamic environments is largely
unexplored. This work proposes a deep Q-learning algorithm for improved
reliability in dynamic environments while minimizing energy consumption by
changing the PHY layer settings based on link state measurements. This method
outperforms using a fixed PHY layer setting and an exhaustive search, which
are currently the most common ways to set the PHY setting. We found that deep
Q-learning can achieve higher PRR while using considerably less energy and
whilst reducing the ranging error. We also concluded that using traditional
Q-learning does not suffice to solve this problem. This work can be extended
by improving upon the current reward function to enable deep Q-learning to
distinguish better between settings that have smaller differences in energy
consumption. In addition, future work could employ RL for UWB localization
systems using Time Difference of Arrival (TDoA) instead of two-way ranging or
multi-agent localization problems.

##  VII Acknowledgment

This work was supported by the imec ICON WISH project HBC.2021.0664.

## References

  * [1] D. Coppens, E. De Poorter, A. Shahid, S. Lemey, B. Van Herbruggen, and C. Marshall, “An overview of uwb standards and organizations (ieee 802.15. 4, fira, apple): Interoperability aspects and future research directions,” _IEEE Access_ , 2022.
  * [2] A. Alarifi, A. Al-Salman, M. Alsaleh, A. Alnafessah, S. Al-Hadhrami, M. A. Al-Ammar, and H. S. Al-Khalifa, “Ultra wideband indoor positioning technologies: Analysis and recent advances,” 2016.
  * [3] B. Denis, J. Keignart, and N. Daniele, “Impact of nlos propagation upon ranging precision in uwb systems,” in _IEEE Conference on Ultra Wideband Systems and Technologies, 2003_ , 2003, pp. 379–383.
  * [4] J. Fontaine, M. Ridolfi, B. Van Herbruggen, A. Shahid, and E. De Poorter, “Edge inference for uwb ranging error correction using autoencoders,” _IEEE access_ , vol. 8, pp. 139 143–139 155, 2020.
  * [5] C. Mao, K. Lin, T. Yu, and Y. Shen, “A probabilistic learning approach to uwb ranging error mitigation,” in _2018 IEEE Global Communications Conference (GLOBECOM)_ , 2018, pp. 1–6.
  * [6] S. Wu, Y. Ma, Q. Zhang, and N. Zhang, “Nlos error mitigation for uwb ranging in dense multipath environments,” in _2007 IEEE Wireless Communications and Networking Conference_ , 2007, pp. 1565–1570.
  * [7] “Ieee standard for low-rate wireless networks,” _IEEE Std 802.15.4-2020 (Revision of IEEE Std 802.15.4-2015)_ , pp. 1–800, 2020.
  * [8] B. Großwindhager, C. Alberto Boano, M. Rath, and K. Römer, “Enabling runtime adaptation of physical layer settings for dependable uwb communications,” in _2018 IEEE 19th International Symposium on ”A World of Wireless, Mobile and Multimedia Networks” (WoWMoM)_ , 2018, pp. 01–11.
  * [9] B. Kempke, P. Pannuto, and P. Dutta, “Polypoint: Guiding indoor quadrotors with ultra-wideband localization,” in _Proceedings of the 2nd International Workshop on Hot Topics in Wireless_ , 2015, pp. 16–20.
  * [10] D. Ltd, _DW1000 User Manual Version 2.18_ , 2017.
  * [11] C. J. Watkins and P. Dayan, “Q-learning,” _Machine learning_ , vol. 8, no. 3, pp. 279–292, 1992.
  * [12] X.-L. Huang, Y.-X. Li, Y. Gao, and X.-W. Tang, “Q-learning-based spectrum access for multimedia transmission over cognitive radio networks,” _IEEE Transactions on Cognitive Communications and Networking_ , vol. 7, no. 1, pp. 110–119, 2021.
  * [13] O. J. Pandey, T. Yuvaraj, J. K. Paul, H. H. Nguyen, K. Gundepudi, and M. K. Shukla, “Improving energy efficiency and qos of lpwans for iot using q-learning based data routing,” _IEEE Transactions on Cognitive Communications and Networking_ , vol. 8, no. 1, pp. 365–379, 2022.
  * [14] A. Shahid, S. Aslam, H. S. Kim, and K.-G. Lee, “A docitive q-learning approach towards joint resource allocation and power control in self-organised femtocell networks,” _Transactions on Emerging Telecommunications Technologies_ , vol. 26, no. 2, pp. 216–230, 2015.
  * [15] M. Girmay, V. Maglogiannis, D. Naudts, A. Shahid, and I. Moerman, “Coexistence scheme for uncoordinated lte and wifi networks using experience replay based q-learning,” _Sensors_ , vol. 21, no. 21, p. 6977, 2021.
  * [16] V. Maglogiannis, D. Naudts, A. Shahid, and I. Moerman, “A q-learning scheme for fair coexistence between lte and wi-fi in unlicensed spectrum,” _IEEE Access_ , vol. 6, pp. 27 278–27 293, 2018.
  * [17] K. I. Ahmed and E. Hossain, “A deep q-learning method for downlink power allocation in multi-cell networks,” 2019. [Online]. Available: https://arxiv.org/abs/1904.13032
  * [18] H. Ye, G. Y. Li, and B.-H. F. Juang, “Deep reinforcement learning based resource allocation for v2v communications,” _IEEE Transactions on Vehicular Technology_ , vol. 68, no. 4, pp. 3163–3173, 2019.
  * [19] H.-S. Lee, J.-Y. Kim, and J.-W. Lee, “Resource allocation in wireless networks with deep reinforcement learning: A circumstance-independent approach,” _IEEE Systems Journal_ , vol. 14, no. 2, pp. 2589–2592, 2020.
  * [20] P. Losco, S. Bourdel, J. Gaubert, N. Dehaese, S. Meillère, O. Ramos, R. Vauche, and H. Barthelemy, “Analysis of the ieee 802.15.4a uwb phy layer for optimizing the power consumption of the transmitter,” in _2013 IEEE International Conference on Ultra-Wideband (ICUWB)_ , 2013, pp. 189–194.
  * [21] K. Mikhaylov, J. Petäjäjärvi, M. Hämäläinen, A. Tikanmäki, and R. Kohno, “Impact of ieee 802.15.4 communication settings on performance in asynchronous two way uwb ranging,” _International Journal of Wireless Information Networks_ , 2017.
  * [22] F. Hartmann, C. Enders, and W. Stork, “Ranging errors in uwb networks and their detectability,” in _2016 39th International Conference on Telecommunications and Signal Processing (TSP)_ , 2016, pp. 194–198.
  * [23] H. Mohammadmoradi, M. Heydariaan, and O. Gnawali, “Uwb physical layer adaptation for best ranging performance within application constraints,” _Proceedings of the 2nd International Conference on Smart Digital Environment_ , 2018. [Online]. Available: https://doi.org/10.1145/3289100.3289120
  * [24] D. Ltd., “Dw1000 user manual version 2.18.”
  * [25] J. Schroeder, S. Galler, K. Kyamakya, and K. Jobmann, “Nlos detection algorithms for ultra-wideband localization,” in _2007 4th Workshop on Positioning, Navigation and Communication_ , 2007, pp. 159–166.
  * [26] Y. Jiang and V. C. Leung, “An asymmetric double sided two-way ranging for crystal offset,” in _2007 International Symposium on Signals, Systems and Electronics_ , 2007, pp. 525–528.
  * [27] O. M. Omisore, T. Akinyemi, W. Duan, W. Du, and L. Wang, “A novel sample-efficient deep reinforcement learning with episodic policy transfer for pid-based control in cardiac catheterization robots,” _arXiv preprint arXiv:2110.14941_ , 2021.
  * [28] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski _et al._ , “Human-level control through deep reinforcement learning,” _nature_ , vol. 518, no. 7540, pp. 529–533, 2015.
  * [29] T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized experience replay,” _arXiv preprint arXiv:1511.05952_ , 2015.
  * [30] “Ghent university - idlab research infrastructure: Officelab.” [Online]. Available: https://www.ugent.be/ea/idlab/en/research/research-infrastructure/officelab.htm
  * [31] B. Van Herbruggen, B. Jooris, J. Rossey, M. Ridolfi, N. Macoir, Q. Van den Brande, S. Lemey, and E. De Poorter, “Wi-pos: A low-cost, open source ultra-wideband (uwb) hardware platform with long range sub-ghz backbone,” _Sensors_ , vol. 19, no. 7, 2019. [Online]. Available: https://www.mdpi.com/1424-8220/19/7/1548
  * [32] H.-S. Lee, J.-Y. Kim, and J.-W. Lee, “Resource allocation in wireless networks with deep reinforcement learning: A circumstance-independent approach,” _IEEE Systems Journal_ , vol. 14, no. 2, pp. 2589–2592, 2020.
  * [33] Y. Wang, C. Ding, Z. Li, G. Yuan, S. Liao, X. Ma, B. Yuan, X. Qian, J. Tang, Q. Qiu, and X. Lin, “Towards ultra-high performance and energy efficiency of deep learning systems: An algorithm-hardware co-optimization framework,” 2018\. [Online]. Available: https://arxiv.org/abs/1802.06402