# DRL meets DSA Networks: Convergence Analysis and Its Application to System
Design

Ramin Safavinejad, Hao-Hsuan Chang, and Lingjia Liu R. Safavinejad and L. Liu
are with Wireless@Virginia Tech, Bradley Department of Electrical and Computer
Engineering, Virginia Tech. H. Chang is with the Standards and Mobility
Innovation Laboratory of Samsung Research America.

###### Abstract

In dynamic spectrum access (DSA) networks, secondary users (SUs) need to
opportunistically access primary users’ (PUs) radio spectrum without causing
significant interference. Since the interaction between the SU and the PU
systems are limited, deep reinforcement learning (DRL) has been introduced to
help SUs to conduct spectrum access. Specifically, deep recurrent Q network
(DRQN) has been utilized in DSA networks for SUs to aggregate the information
from the recent experiences to make spectrum access decisions. DRQN is
notorious for its sample efficiency in the sense that it needs a rather large
number of training data samples to tune its parameters which is a
computationally demanding task. In our recent work, deep echo state network
(DEQN) has been introduced to DSA networks to address the sample efficiency
issue of DRQN. In this paper, we analytically show that DEQN comparatively
requires less amount of training samples than DRQN to converge to the best
policy. Furthermore, we introduce a method to determine the right
hyperparameters for the DEQN providing system design guidance for DEQN-based
DSA networks. Extensive performance evaluation confirms that DEQN-based DSA
strategy is the superior choice with regard to computational power while
outperforming DRQN-based DSA strategies.

###### Index Terms:

Deep reinforcement learning (DRL), Covering Numbers, Dynamic Spectrum Access
(DSA), Echo State Network (ESN), Recurrent Neural Network, 5G Beyond and 6G

##  I Introduction

Dynamic spectrum access (DSA) was originally introduced in cognitive radio
networks (CRN) to allow secondary users (SUs) to share the spectrum licensed
by primary users (PUs) without causing harmful interference to the PUs [1].
Through PU-SU coexistence, DSA is regarded as a promising technology to
improve the overall spectrum utilization of a CRN significantly [2]. Similar
ideas have been extended to allow spectrum sharing and coexistence among
different wireless systems. For instance, through licensed assisted access
(LAA) [3] the 4G LTE-Advanced system is able to coexist with Wi-Fi systems in
the unlicensed spectrum, while 4G and 5G systems are able to dynamically share
the licensed spectrum [4]. Meanwhile, coexistence between different wireless
systems are introduced to allow different wireless users potentially from
different services (e.g., IoT network, 5G, Wi-Fi etc) to share the radio
spectrum to improve the spectrum utilization. Therefore, shared spectrum
access, especially the dynamic spectrum access, that promotes the most
effective and efficient spectrum uses has been regarded as one of the key
technology enablers for 6G networks [5]. Still, the potential of DSA is yet to
be fully realized in practice because of the technological barriers in coping
with the complex and highly dynamic radio environments that SU devices have to
interact with to gain real-time channel access and interference control. This
makes the coexistence of primary and secondary systems extremely challenging.
With the recent surge of artificial intelligence (AI), it becomes possible to
employ secondary devices to sense and access the channels smartly in highly
dynamic networks. Among available AI/machine learning (ML) tools, deep
reinforcement learning (DRL) becomes a natural framework to solve for DSA
networks [6, 7]. To be specific, with the help of DRL, SU devices will be able
to adapt their spectrum access strategies to unknown dynamic radio
environments without modeling the underlying environments.

On the other hand, developing DRL-based DSA strategies for mobile networks is
usually difficult due to the following challenges.

  * •

Challenge 1: Partial observability: SU devices can be resource constrained IoT
devices that are not able to conduct accurate spectrum sensing across all
available bandwidth simultaneously. This will lead to a partially observable
radio environment to SU devices with noisy inputs.

  * •

Challenge 2: Limited over-the-air (OTA) training: OTA training data is
generally costly in mobile networks since it imposes control overhead [6].
Meanwhile, the highly dynamic nature of the radio environment also makes the
collected OTA training data outdated in a relatively short time.

Combining recurrent neural network (RNN) and deep Q-network (DQN), the deep
recurrent Q-network (DRQN), has been introduced as a DRL technique to
aggregate information from past observations in a partially observable Markov
decision process (POMDP) environment [8]. Action-specific Deep Recurrent
Q-Network (ADRQN) [9] further enhances DRQN by using both observation and
action as inputs of RNNs. Therefore, both techniques have been introduced to
enable DRL-based DSA to address Challenge 1 [10]. However, the training of the
underlying RNN of both methods suffers from poor convergence requiring a lot
of training samples [11], which is infeasible in a highly dynamic environment
with limited OTA training data —– Challenge 2. Meanwhile, generative models of
the environment have been introduced at the DRL agents to allow them to learn
and to infer the belief states using the learned models [9, 12]. However,
these methods are not suitable for DSA networks since the learned generative
model will become outdated quickly in such a highly dynamic environment.
Therefore, it is of critical importance to design a fast-converging DRL
strategy that can work efficiently in a partially observable and highly
dynamic environment to jointly address both challenges of the DRL-based DSA
strategies for mobile networks.

In light of the challenges, a new DRL strategy called deep echo state Q
network (DEQN) has been introduced in [13]. Echo state network (ESN) [14] has
been adopted in DEQN as the kernel of the Q-network to reduce the training
time and the required training data. Note that ESN is a special type of RNN
that only trains the output weights while leaving input weights and recurrent
weights untrained. ESNs have been shown to achieve comparable performance with
RNNs, especially in some applications requiring fast learning [15].
Furthermore, the hidden states of ESNs are unchanged during the training
process due to the fact that both input weights and recurrent weights are
fixed. In this way, DEQNs can pre-store hidden states, while DRQNs have to
recalculate hidden states in every training iteration. Accordingly, DEQNs can
substantially decrease the training time and the amount of training data.

In this paper, we will focus on DEQN to show that it can be used to address
both of the challenges for DRL-based DSA strategies and introduce a method to
determine the right hyperparameters for the DEQN providing system design
guidance for DEQN-based DSA schemes. This is done through the notion of
covering numbers, which capture the complexity of a given function class such
as that of RNNs or ESNs. Covering numbers portray this complexity by
determining the least number of hyperballs of radius $\epsilon$ we would need
to cover the output space of a function class given a fixed input set. If a
function class needs more hyperballs for it to be covered, then it is a richer
and, consequently, more complex function class. It is shown in the literature
that the covering number of a Q-network can upper bound a notion called the
one step approximation error [16], which shows how fast DRL would converge
with that specific Q-network as its kernel. For the theoretical convergence
analysis of this paper, we first upper bound the covering number of RNNs and
ESNs, which is outlined in Section III. We then use this upper bound on the
covering number to characterize the convergence rate. The work in [17]
provides an upper bound on the covering number of a few variations of RNNs.
However, that work does not consider the effect of the number of training
sequences that are used for training. Therefore, we cannot make use of the
upper bound that is obtained in [17] for our work.

In summary, the main contributions of our work are as follows:

  1. 1.

An upper bound has been obtained on the covering number of single layer RNNs
and ESNs.

  2. 2.

Theoretical convergence analysis of both DEQN and DRQN have been conducted by
means of covering numbers to demonstrate the faster convergence rate of DEQN
compared to DRQN.

  3. 3.

A bias-variance trade-off has been characterized for DEQN where the variance
term is characterized by its covering number and the bias term is determined
empirically through a quadratic optimization problem.

  4. 4.

A method has been introduced for determining the spectral norm of the
recurrent weights of the ESN kernel of DEQN to minimize the bias plus variance
error. It will provide theoretical foundation for choosing hyperparameter for
the underlying DEQN-based DSA strategies.

  5. 5.

Extensive performance evaluation has been conducted for both DEQN and DRQN on
a highly dynamic DSA environment to demonstrate the benefits of DEQN-based DSA
strategy over its DRQN counterpart.

In what follows, we will go over the DRL problem formulation, introduction of
DEQN and its training method in Section II, provide theoretical analysis of
DEQNs and DRQNs in Section III, formulate the 5G DSA problem in Section IV,
showcase numerical results and performance evaluation in Section V and
finally, conclude the paper in Section VI.

##  II Deep Echo State Q Network

In this section we first formally state the reinforcement learning (RL)
problem for our theoretical analysis and then introduce the DEQN’s update
equations and training algorithm.

###  II-A RL Problem Setup

For the theoretical analysis, we assume the underlying environment has the
Markov property and the problem can be formulated as a Markov Decision Process
(MDP). we define the problem as an MDP rather than a POMDP, due to how general
a POMDP can be. In the literature, theoretical analysis of POMDPs is mostly
done with knowledge of the model and without learning based approaches (see
e.g. [18]). However, we will also apply the introduced learning methods to
partially observable environments in our performance evaluation. An MDP model
is characterized by a tuple ($\mathcal{S}$, $\mathcal{A}$, $\mathcal{T}$, $R$,
$\gamma$), where $\mathcal{S}$, $\mathcal{A}$, $\mathcal{T}$, $R$ and
$\gamma\in(0,1)$ are the state space, action space, state transition
probability, reward function and discount factor for future rewards
respectively. To put it in more detail, the agent observes state
$s_{t}\in\mathcal{S}$ at time $t$ and takes the action $a_{t}\in\mathcal{A}$.
After taking the action $a_{t}$, the agent receives a reward
$r_{t}=R(s_{t},a_{t})$, and then the system shifts to the next state $s_{t+1}$
based on the state transition probability $\mathcal{T}(s_{t+1}|s_{t},a_{t})$.
The the agent’s goal is to find a policy $\pi$ for making a sequence of action
decisions that maximize the expected discounted reward,
$\mathbb{E}_{\pi}\left[\sum_{t=1}^{\infty}\gamma^{t-1}r_{t}\right]$. Also,
denote $\Omega_{\leq T}$ as the space spanned by
$\\{s_{1},s_{2},\dots,s_{T}\\}$.

For the practical implementation setup, we know that the environment is, in
general, a POMDP. Therefore, the agent only has a partial observation of the
environment, rather than the full state of it. However, we will still denote
the observation of the agent as $s_{t}$. We avoid introducing another notation
for the observation as to not confuse the reader when comparing the
theoretical analysis with the practical implementation. The rest of the setup
is the same as the theoretical RL setup.

###  II-B Update Equations of DRQN/DEQN

The action-value function in DRQN/DEQN is approximated by the Q-network
$Q_{\theta}$ with parameter
$\theta=\left(W_{\text{in}},W_{\text{rec}},W_{\text{out}}\right)$, where
$W_{\text{in}}\in\mathbb{R}^{d_{h}\times d_{x}}$ is the input weights,
$W_{\text{rec}}\in\mathbb{R}^{d_{h}\times d_{h}}$ is the recurrent weights,
and $W_{\text{out}}\in\mathbb{R}^{d_{y}\times d_{h}}$ is the output weights. A
sequence of observations $s_{1},\ldots,s_{t}\in\mathbb{R}^{d_{x}}$ is input to
the Q-network to generate a sequence of outputs
$y_{1},\ldots,y_{t}\in\mathbb{R}^{d_{y}}$ as follows:

$\displaystyle h_{t}$
$\displaystyle=\sigma_{h}\left(W_{\text{in}}s_{t}+W_{\text{rec}}h_{t-1}\right)\in\mathbb{R}^{d_{h}},$
(1) $\displaystyle y_{t}$
$\displaystyle=\sigma_{y}\left(W_{\text{out}}h_{t}\right)\in\mathbb{R}^{d_{y}},$

where $\sigma_{h}$ and $\sigma_{y}$ are the recurrent and output units
activation functions, and $h_{t}$ is the hidden state. $\sigma_{h}$ is often
chosen as ReLU or tanh activation functions, and $\sigma_{y}$ is chosen as the
identity function in the case of ESNs. The hidden state $h_{t}$ is used to
represent a summary of the past sequence of observations up to $t$ and we set
$h_{0}=\boldsymbol{0}$. Note that the $a^{\text{th}}$ element of $y_{t}$ is
equal to the estimated action-value function of selecting action $a$, i.e.,
$y_{t}^{a}=Q_{\theta}\left(s_{\leq t},a\right)$, where $s_{\leq
t}=\left(s_{1},\ldots,s_{t}\right)$. We define the action-value function as
the sum of discounted rewards from the current time step until the future,
given that the sequence of states until now is $s_{\leq t}$ and the action
taken is $a_{k}$, in other words

$\displaystyle Q_{\theta}\left(s_{\leq
t},a\right)=\mathbb{E}\Bigg{[}\sum_{\tau=0}^{\infty}{\gamma^{\tau}r_{t+\tau}}\bigg{|}S_{0}=s_{0},\dots,S_{t}=s_{t},A_{t}=a_{t}\Bigg{]}$
(2)

where $r_{t}$ is the reward received at time step $t$. We also define the
operator $\mathcal{B}$ as the Bellman operator given as

$\displaystyle\mathcal{B}Q_{\theta}(s_{\leq
t},a_{t})=r_{t}+\gamma\max_{a}Q_{\theta}(s_{\leq t+1},a)$ (3)

If for a policy $\pi_{L}^{*}$ we have
$\mathcal{B}Q_{\theta}^{\pi_{L}^{*}}(s_{\leq
t},a_{t})=Q_{\theta}^{\pi_{L}^{*}}(s_{\leq t},a_{t})$, we call $\pi_{L}^{*}$
an optimal policy and define $Q^{*}=Q^{\pi_{L}^{*}}$.

We therefore define the loss function for training $\theta$ as

$\left(r_{t}+\gamma\max_{a}Q_{\theta^{-}}(s_{\leq t+1},a)-Q_{\theta}(s_{\leq
t},a_{t})\right)^{2},$ (4)

where $\theta^{-}$ is the parameter of a target Q-network initially identical
to the main Q-network. To stabilize the training, $\theta^{-}$ is periodically
synchronized with $\theta$ instead of being updated in each training
iteration.

###  II-C Training Algorithm for DEQN

BPTT involves unfolding the network in time into a computational graph that
has a repetitive structure, which suffers from the slow convergence rate due
to vanishing and exploding gradients [11]. Furthermore, DRQN requires a large
amount of training data to ensure the learning agent converges to an
appropriate policy. Unfortunately, in the DSA problem for 5G, the DRL agent
can only collect limited training data and the 5G wireless environment is
highly time varying. To quickly adapt to such an environment using limited
training data, we adopt the online training algorithm for DEQN originally
introduced in [13].

Algorithm 1 The online training algorithm for DEQN.

Initialize a Q-network $\theta$ and a target Q-network $\theta^{-}$.

for $l=1,...,L$ do

Set $\theta^{-}=\theta$ and empty the replay buffer $D$.

Observe $o_{1}$ from the environment.

for $t=1,...,T$ do

Input $s_{t}$ and $h_{t-1}$ to $Q_{\theta}$ to calculate $h_{t}$ and $y_{t}$.

Execute $a_{t}$ based on $y_{t}$ and then receive $r_{t}$.

Observe $s_{t+1}$ from the environment.

Input $s_{t+1}$ and $h_{t}^{-}$ to $Q_{\theta^{-}}$ to calculate
$h_{t+1}^{-}$.

Store ($h_{t}$, $a_{t}$, $r_{t}$, $h_{t+1}^{-}$) in replay buffer $D$.

end for

Sample ($h_{j}$, $a_{j}$, $r_{j}$, $h_{j+1}^{-}$) from replay buffer $D$.

Update $\theta$ by performing gradient descents on
$\left(r_{j}+\gamma\max\limits_{a}Q_{\theta^{-}}(h_{j+1}^{-},a)-Q_{\theta}(h_{j},a_{j})\right)^{2}$.

end for

The designed DEQN training algorithm is stated in Algorithm 1. First, the
input weights and the recurrent weights of ESNs are initialized randomly, and
then they remain untrained. Only the output weights of ESNs are trained so the
training is extremely fast by avoiding BPTT. Second, given
$(s_{1},\ldots,s_{t})$, we can observe that the hidden states
$(h_{1},\ldots,h_{t})$ are unchanged during the training process from (1)
because the input weights and recurrent weights are fixed. In contrast to
DRQNs that waste some training samples and computational resources to
recalculate the hidden states in every training iteration, DEQNs can pre-store
the hidden states in the replay buffer and use them for training. Therefore,
the DEQN training is much more sample and computationally efficient than the
DRQN training. Third, the DEQN training can randomly sample from the replay
buffer to create a training batch because the hidden states are unchanged and
pre-stored. On the other hand, the DRQN training has to sample continuous
sequences to create a training batch. Breaking the temporal correlations of
sampled data during training is crucial for reducing generalization error, as
the stochastic optimization algorithms usually assume i.i.d. data. Lastly, to
deal with the highly dynamic and time varying environment, the outdated
training data in the replay buffer will be removed and the policy will be
updated continually by the effective data collected from the latest
environment.

##  III Theoretical Analysis for DRQN and DEQN

In this section we will theoretically study the convergence properties of DEQN
and DRQN, compare them with each other and provide a method for designing an
ESN structure that optimizes the performance of a DEQN.

For the theoretical analysis, we consider the problem of batch reinforcement
learning, which is normally considered in the literature for theoretical
analysis of RL problems (e.g. [16, 19]). In batch RL, the agent has had a
multitude of transitions in the environment and then the training of the
Q-network is done after all the transitions have finished. The agent would
sample a batch of size $n$ from those transitions and will do one iteration of
training on the neural network using those $n$ transitions. It is also assumed
that sampling from this accumulation of transitions is the same as sampling a
transition from the environment with a fixed distribution $\sigma$. Note that,
in our case, the agent collects streams of experiences over the course of
multiple episodes and stores them in the experience replay buffer. We assume
that each episode consists of a sequence of size $T$ of actions taken in the
environment. We denote $\\{s_{i,1},s_{i,2},\dots,s_{i,T}\\}_{i=1}^{n}$ as a
batch of $n$ i.i.d sample sequences that will be used for one training
iteration. The training continues for another i.i.d batch of sequences for a
total of $L$ iterations. Algorithm 2 outlines the full training procedure that
will be used in this section and is regarded as the Neural Fitted Q-Iteration
Algorithm. We refer the reader to [16] for a detailed explanation on why this
algorithm effectively characterizes Deep Q-Learning.

Algorithm 2 Neural Fitted Q-Iteration Algorithm.

Initialize $T$ Q-networks with parameters
$\\{\theta_{0}^{(t)}\in\Theta\\}_{t=1}^{T}$.

for $t=1,\dots,T$ do

for $l=1,...,L$ do

Sample $n$ sequences of size $T$ of transitions in the environment.

Store $\left\\{(s_{i,t},a_{i,t},r_{i,t},s_{i,t+1}\right)\\}_{i=1}^{n}$ in
replay buffer $D$.

Update the action-value function:
$\theta_{l}^{(t)}\leftarrow\operatorname*{\arg\\!\min}\limits_{\theta_{l}^{(t)}\in\Theta}\frac{1}{n}\sum_{i=1}^{n}\left(z_{i,t}-Q_{\theta_{l}^{(t)}}\left(s_{i,\leq
t},a_{i,t}\right)\right)^{2}$, where
$z_{i,t}=r_{i,t}+\gamma\max_{a}Q_{\theta_{l-1}^{(t)}}\left(s_{i,\leq
t+1},a\right)$.

end for

Define $\pi_{L}^{(t)}$ as the greedy policy with respect to
$Q_{\theta_{L}^{(t)}}$.

end for

Before proceeding with the analysis, we first introduce the notion of covering
number, which is widely known in the statistical learning theory community.

######  Definition 1.

Let $\mathcal{F}$ be a function class defined from $\mathbb{R}^{d_{x}}$ to a
metric space $B$ with metric $\text{dis}(\cdot)$. For a given set of sample
observations $\\{\textbf{x}_{1},\dots,\textbf{x}_{n}\\}$, the
$\epsilon$-covering number of this function class is denoted by
$\mathcal{N}\left(\mathcal{F},\epsilon,\text{dis}(\cdot)\right)$ and is
defined as the least cardinality $m$ of a set of vectors
${V}_{m}=\\{\textbf{v}_{i}:\textbf{v}_{i}\in B^{n},i\in[m]\\}$, such that for
any $f\in\mathcal{F}$ there exists $\textbf{v}\in{V}_{m}$ such that

$\displaystyle\left(\sum_{i=1}^{n}\text{dis}\left(f(\textbf{x}_{i}),\textbf{v}_{(i)}\right)^{2}\right)^{1/2}\leq\epsilon$

where $\textbf{v}_{(i)}$ is the $i$-th element of v

The above definition is not the most general form of Covering Number
definition, but is sufficient for our analysis. The reader is referred to [20]
and references therein for more general forms of definitions.

In order to analyze the convergence properties of a DQN, we make use of the
following two theorems from [16]:

######  Theorem 1.

(Theorem 6.1 in [16]) Let $\sigma$ be the sampling distribution over
$\Omega_{\leq T}\times\mathcal{A}$ in Algorithm 2, $\mu$ be a fixed
probability distribution over $\Omega_{\leq T}\times\mathcal{A}$, and
$R_{\text{max}}$ be the maximum reward value. Then we have

$\mathbb{E}_{\mu}\left[\big{|}Q^{*}-Q^{\pi_{L}}\big{|}\right]\leq\frac{2\phi_{\mu,\sigma}\gamma}{(1-\gamma)^{2}}\cdot\eta_{\text{max},T}+\frac{4\gamma^{L+1}}{(1-\gamma)^{2}}\cdot
R_{\text{max}},$ (5)

where $\phi_{\mu,\sigma}$ is the concentration coefficient of $\mu$ and
$\sigma$, $\pi_{L}$ is the output policy of Algorithm 2, and
$\eta_{\text{max},T}$ is the maximum one-step approximation error given by
$\eta_{\text{max},T}=\max_{t\in T}\|\mathcal{B}Q_{t-1}-Q\|$, where
$\mathcal{B}(\cdot)$ denotes the Bellman optimality operator.

######  Theorem 2.

(Theorem 6.2 in [16]) Let $\epsilon>0$ and
$V_{\text{max}}=R_{\text{max}}/(1-\gamma)$. From Theorem 6.2 in [16], the
upper-bound of $\eta_{\text{max},t}$ can be written as

$\displaystyle\eta_{\text{max},T}^{2}\leq$ $\displaystyle
4\sup_{\theta^{\prime}\in\Theta}\inf_{\theta\in\Theta}\mathbb{E}_{\sigma}\left[\left(\mathcal{B}Q_{\theta^{\prime}}-Q_{\theta}\right)^{2}\right]+C_{1}\cdot
V_{\text{max}}^{2}/T\cdot\text{log}N_{\epsilon,T}+C_{2}\cdot
V_{\text{max}}\cdot\epsilon,$ (6)

where $\mathcal{B}$ is the Bellman optimality operator, $N_{\epsilon,t}$ is
the $\epsilon$-covering number of RNN/ESN with respect to the Euclidean norm,
and $C_{1}$ and $C_{2}$ are constants given by

$\displaystyle
C_{1}=\left(8\sqrt{2T}+256/V_{max}\right)~{},~{}C_{2}=\left(4\sqrt{2n}+52\right)$

The maximum one step approximation error characterizes the distance between
the Q-value at a time instant and the Bellman optimality operator operating on
the Q-value of the previous time instant. The smaller this distance, the
closer the Q-network is to approximating the actual Q-values of the
environment. We denote the first term in the RHS of 6 as the bias term, and
the second and third terms as the variance term. In what follows next, we
derive an upper bound on the covering number which characterizes the variance
term, and provide a method for evaluating the bias term.

###  III-A Upper bounding the covering number of RNN/ESN

We define the class of single layer RNN functions with sequence length $t$ as
all the functions that do the operation stated in (1), their output is $y_{t}$
and have the matrices $W_{in}$, $W_{rec}$ and $W_{out}$ as trainable
parameters but with the constraints

$\displaystyle\|W_{in}\|_{F}\leq B_{in},\|W_{rec}\|_{F}\leq
B_{rec},\|W_{out}\|_{F}\leq B_{out}$ (7)

We denote this class as $\mathcal{H}_{RNN,t}$. Similarly, we denote the class
of single reservoir ESN functions with sequence length $t$ as
$\mathcal{H}_{ESN,t}$. It is defined exactly the same way as
$\mathcal{H}_{RNN,t}$ except for the fact that $W_{in}$ and $W_{rec}$ are
fixed and each function in this class is distinguished by $W_{out}$ only. Note
that the constraints in (7) hold in this class as well. We also pose a
constraint on the input states and let $\|s_{i}\|\leq B_{X};i\in[T]$.

######  Proposition 1.

The $\epsilon$-covering number of a single layer RNN is upper bounded by

$\displaystyle\ln\left(\mathcal{N}(\mathcal{H}_{RNN,t},\epsilon,\|\cdot\|)\right)$
$\displaystyle\leq\left(\dfrac{9\rho_{y}^{2}\rho_{h}^{2}B_{out}^{2}B_{in}^{2}B_{X}^{2}\left(\rho_{h}^{t}B_{rec}^{t}-1\right)^{2}}{\left(\rho_{h}B_{rec}-1\right)^{2}\epsilon^{2}}\right)\ln(2d_{y}d_{h})$
$\displaystyle+\dfrac{9t^{2}\rho_{y}^{2}\rho_{h}^{2}B_{out}^{2}B_{in}^{2}B_{X}^{2}B_{rec}^{2}\rho_{h}^{2}}{\epsilon^{2}}\left(\dfrac{\rho_{h}^{2t}B_{rec}^{2t}-1}{\rho_{h}^{2}B_{rec}^{2}-1}\right)\ln(2d_{h}d_{x})$
$\displaystyle+\dfrac{9(t-1)^{2}\rho_{y}^{2}\rho_{h}^{2}B_{W}^{2}B_{X}^{2}B_{out}^{2}}{\epsilon^{2}\left(\rho_{h}B_{rec}-1\right)^{2}}\bigg{(}(t-1)(\rho_{h}B_{rec})^{2t}-2\rho_{h}^{t+1}$
$\displaystyle\cdot
B_{rec}^{t+1}\dfrac{(\rho_{h}B_{rec})^{t-1}-1}{\rho_{h}B_{rec}-1}+\rho_{h}^{2}B_{rec}^{2}\dfrac{(\rho_{h}B_{rec})^{2t-2}-1}{\rho_{h}B_{rec}-1}\bigg{)}\ln(2d_{h}^{2})$
(8)

Proof: See Appendix A for a detailed proof.

######  Proposition 2.

The $\epsilon$-covering number of a single reservoir ESN is upper bounded by

$\displaystyle\ln\left(\mathcal{N}(\mathcal{H}_{ESN,t},\epsilon,\|\cdot\|)\right)$
$\displaystyle\leq\dfrac{\rho_{y}^{2}\rho_{h}^{2}B_{in}^{2}B_{X}^{2}B_{out}^{2}d_{y}}{\epsilon^{2}}\left(\dfrac{\rho_{h}^{t}B_{rec}^{t}-1}{\rho_{h}B_{rec}-1}\right)^{2}\ln(2d_{h}d_{y})$
(9)

Proof: Refer to Appendix B for a detailed proof.

The first term in (1), which gets multiplied by $\ln(2d_{y}d_{h})$,
corresponds to the output weight $W_{out}$. The other two terms that get
multiplied by $\ln(2d_{h}d_{x})$ and $\ln(2d_{h}^{2})$ correspond to $W_{in}$
and $W_{rec}$ respectively. As can be observed, the upper bound on the ESN’s
covering number in (2) is the same as the first term in the RNN’s covering
number bound in (1) up to a constant factor. The other two terms do not appear
in the ESN covering number upper bound. This is because the output weights
$W_{out}$ are the only parameters that are trained in the ESN and $W_{in}$ and
$W_{rec}$ remain untrained. This has reduced the richness of the ESN function
class compared to RNN, and thus caused a reduction in its covering number.
This reiterates that ESNs are a less complex function class compared to RNNs.

The upper bounds on the covering number allow us to upper bound the variance
term in (6) (i.e. the last two terms). This means that if two Q-network
kernels have comparable performance in terms of their best performance, then
the one that has a higher covering number will need more training data for the
RL algorithm to converge. In other words, the convergence rate of the one with
higher covering number will be lower. This is a critical property that a
Q-network kernel would need in order to adapt to fast-changing environments.

###  III-B Empirical derivation of the bias term

The bias term in (6) represents how good the Q-network can approximate any
function in the function space created by the Bellman operator operating on
all the Q-functions formable by the Q-network. The bias term is highly
dependent on the RL environment and it is usually very difficult to find an
upper bound on. For example, the work in [21] provides an upper bound on the
average error we would get with a particular way of initializing an ESN.
However, that work requires prior knowledge on the function we are trying to
approximate, i.e. the Q-function in our case. Specifically, since the Fourier
transform of the Q-function of an the RL environment is unknown to us, we
cannot make use of the upper bounds that are mentioned there.

Therefore, we propose to find the bias term empirically during the first
episode of the RL training. We will show in this section that the infimum of
the bias error of an ESN can be formulated as a quadratic optimization
problem. It is worth mentioning that such a straightforward approach is not
available for RNNs. The reason for this is that in an RNN there are multiple
non-linearities that prevent the optimization problem from becoming even
convex. For the case of ESNs, however, only the output weights are trainable.
This means that the output $y_{t}$ in (1) is just a linear combination of the
non-linearly derived $h_{t}$, which for a given set of sample states, is
constant.

According to Algorithm 2, the training update step for the DEQN can be
expressed as

$\displaystyle\theta_{\text{new}}\leftarrow\operatorname*{\arg\\!\min}\limits_{\theta\in\Theta}\sum_{i=1}^{n}\left(z_{i}-Q_{\theta}\left(\textbf{h}_{i},a_{i}\right)\right)^{2}$
(10)

where $\theta$ is the parameters of our ESN output weights, $n$ is the number
of experience replay samples used for one training step, $\textbf{h}_{i}$ is
the output of the reservoir for the $i^{\text{th}}$ sample, $a_{i}$ is the
action taken for the $i^{\text{th}}$ experience replay sample and $z_{i}$ is
defined as

$\displaystyle
z_{i}=r_{i}+\gamma\max_{a}Q_{\theta_{\text{old}}}(\textbf{h}_{i}^{{}^{\prime}},a)$
(11)

where $r_{i}$ is the reward received in the $i^{\text{th}}$ sample,
$\theta_{old}$ is the ESN parameters of the previous training iteration, which
in this case is just the initial parameters, and
$\textbf{h}_{i}^{{}^{\prime}}$ is the reservoir output of the next state.

Let’s consider the optimization problem in (10). Substituting the Q function
with the output of the ESN shown in (1), we get

$\displaystyle\operatorname*{\arg\\!\min}\limits_{W_{\text{out}}}\sum_{i=1}^{n}\left(W_{\text{out},a_{i}}\textbf{h}_{i}-z_{i}\right)^{2}$
(12)

where $W_{\text{out},a_{i}}$ is the row in $W_{\text{out}}$ that leads to the
Q value of action $a_{i}$ when applied to the reservoir output
$\textbf{h}_{i}$. If we assume $a_{i}\in\\{1,2,\dots,|\mathcal{A}|\\}$, the
optimization in (12) can be rewritten as

$\displaystyle\operatorname*{\arg\\!\min}\limits_{W_{\text{out}}}\sum_{k=1}^{|\mathcal{A}|}\sum_{j=1}^{n_{k}}\left(W_{\text{out},k}\textbf{h}_{k,j}-z_{k,j}\right)^{2}$
(13)

where $n_{k}$ is the number of samples in the experience replay that had led
to action $a_{k}$, $\textbf{h}_{k,j}$ is the $j^{\text{th}}$ reservoir output
among the outputs that had led to action $a_{k}$ and $z_{k,j}$ is defined
similarly. Since the optimization parameters $W_{\text{out},k}$ are mutually
exclusive in $k$, we have

$\displaystyle\min_{W_{\text{out}}}\sum_{k=1}^{|\mathcal{A}|}\sum_{j=1}^{n_{k}}\left(W_{\text{out},k}\textbf{h}_{k,j}-z_{k,j}\right)^{2}$
(14)
$\displaystyle=\sum_{k=1}^{|\mathcal{A}|}\min_{W_{\text{out,k}}}\sum_{j=1}^{n_{k}}\left(W_{\text{out},k}\textbf{h}_{k,j}-z_{k,j}\right)^{2}$

Therefore, it is equivalent to minimize each of the inner objective functions
separately. In other words

$\displaystyle{W_{\text{out,k}}^{*}}=\operatorname*{\arg\\!\min}_{W_{\text{out,k}}}\sum_{j=1}^{n_{k}}\left(W_{\text{out},k}\textbf{h}_{k,j}-z_{k,j}\right)^{2}$
(15)

Let
$L_{k}=\sum_{j=1}^{n_{k}}\left(W_{\text{out},k}\textbf{h}_{k,j}-z_{k,j}\right)^{2}$
and denote $\textbf{w}_{k}=W_{\text{out},k}^{T}$. We have

$\displaystyle\min_{\textbf{w}_{k}}L_{k}=\min_{\textbf{w}_{k}}\sum_{j=1}^{n_{k}}\left(\textbf{w}_{k}^{T}\textbf{h}_{k,j}-z_{k,j}\right)^{2}$
$\displaystyle\equiv\min_{\textbf{w}_{k}}\sum_{j=1}^{n_{k}}\bigg{\\{}\left(\textbf{w}_{k}^{T}\textbf{h}_{k,j}\right)^{2}-2\textbf{w}_{k}^{T}\textbf{h}_{k,j}z_{k,j}\bigg{\\}}$
$\displaystyle=\min_{\textbf{w}_{k}}\textbf{w}_{k}^{T}\textbf{H}_{k}\textbf{H}_{k}^{T}\textbf{w}_{k}-2\textbf{w}_{k}^{T}\textbf{H}_{k}\textbf{z}_{k}$
(16)

where
$\textbf{H}_{k}\coloneqq[\textbf{h}_{k,1}~{}\textbf{h}_{k,2}~{}\dots~{}\textbf{h}_{k,n_{k}}]$
and $\textbf{z}_{k}\coloneqq[z_{k,1}~{}\dots~{}z_{k,n_{k}}]^{T}$. The
optimization problem in (III-B) is a quadratic one in the form of
$\frac{1}{2}\textbf{w}_{k}^{T}\textbf{A}_{k}\textbf{w}_{k}+\textbf{w}_{k}^{T}\textbf{b}_{k}$
with $\textbf{A}_{k}=2\textbf{H}_{k}\textbf{H}_{k}^{T}$ and
$\textbf{b}_{k}=-2\textbf{H}_{k}\textbf{z}_{k}$. The solution to this
optimization problem when $\textbf{A}_{k}$ is positive semi definite is
$\textbf{w}_{k}^{\text{opt}}=-\textbf{A}_{k}^{-1}\textbf{b}_{k}$ (see e.g.
[22]). However, $\textbf{A}_{k}$ might become seriously ill conditioned
especially when the number of reservoir outputs that had led to action $a_{k}$
are not plenty enough. In such a case, we would get extremely high values for
the elements of the $\textbf{w}_{k}$ vector. Consequently, we will add a
regularizer term to the optimization problem in (III-B) to mitigate this
issue. We therefore try to solve the following problem instead

$\displaystyle\min_{\textbf{w}_{k}}\textbf{w}_{k}^{T}\textbf{H}_{k}\textbf{H}_{k}^{T}\textbf{w}_{k}-2\textbf{w}_{k}^{T}\textbf{H}_{k}\textbf{z}_{k}+\frac{1}{2}\alpha\textbf{w}_{k}^{T}\textbf{w}_{k}$
$\displaystyle=$
$\displaystyle\min_{\textbf{w}_{k}}\frac{1}{2}\textbf{w}_{k}^{T}\left(2\textbf{H}_{k}\textbf{H}_{k}^{T}+\alpha\textbf{I}\right)\textbf{w}_{k}+\textbf{w}_{k}^{T}\left(-2\textbf{H}_{k}\textbf{z}_{k}\right)$
(17)

where $\alpha$ is the regularization parameter. By setting
$\tilde{\textbf{A}}_{k}=2\textbf{H}_{k}\textbf{H}_{k}^{T}+\alpha\textbf{I}$ we
would have a much more stable solution of
$\textbf{w}_{k}=-\tilde{\textbf{A}}_{k}^{-1}\textbf{b}_{k}$.

With the necessary building blocks set in place, we can run the RL framework
for one episode of $T$ time steps and gather enough samples to form the loss
minimization in (14). We then find the minimum value of the bias term through
finding the optimum weights of each of the rows of the output weight matrix
$W_{\text{out}}$ as was outlined in (III-B). We repeat this for multiple
choices of the recurrent and input weight matrices and compare their bias plus
variance error and check to see which one has the least error. We then choose
that ESN as our main Q network and go forward with the RL training in the next
episodes using Algorithm 1.

##  IV RL Problem Formulation for 5G DSA

In this section, we first outline two challenges that we face in solving real
world problems using RL. We then introduce the DSA problem and discuss the
setting that we choose for our RL formulation.

###  IV-A Motivations

In many real-world scenarios, we encounter environments with a highly dynamic
behaviour, where the agent has to deal with rewards that change behavior over
time and variable transition probabilities between system states through time.
These kinds of environments degrade the performance of DRL-based strategies
drastically because the learned policy is not suitable for the latest
environment. Furthermore, the high variance poses a big challenge to learning
stability because the collected training data become obsolete as the
environment changes [23], which means the effective training data is limited.
Therefore, we adopt an online training algorithm that only relies on the very
recent experiences and removes outdated training data from the replay buffer
to update the policy, so that it can adapt to the environment quickly. Most
existing DRL algorithms require a large number of samples for the training to
converge. This may not be an issue for applications such as computer games
where samples can be easily obtained. However, in 5G networks, data collection
is usually expensive making it challenging to directly apply DRL algorithms.
Furthermore, the 5G environment is constantly changing where the collected
past data become obsolete and thus there is very limited valid training data
available. As a result, developing efficient DRL algorithms requiring limited
training data with fast convergence is critical to make DRL applicable to 5G
networks. With these motivations in mind, we will adopt the DEQN to tackle the
DSA problem. We will also use the method outlined in section III to determine
the best hyperparameter $B_{rec}$ in our DEQN framework.

###  IV-B DSA Problem formulation

DSA is a promising technology to improve the utilization of radio spectrum.
DSA allows an SU to access the radio spectrum if PUs only receive tolerable
interference. The primary system consists of a primary base station (PBS) and
$M$ PUs, and the secondary system consists of a secondary base station (SBS)
and one or more SUs. The primary system has a license to operate on $C$
wireless channels. It is assumed that each user can only transmit on one
channel at a particular time. Users cause interference to one another if they
transmit on the same channel at a particular time. Our goal is to develop a
spectrum access strategy of the SU to increase the overall spectrum
utilization without generating intolerable interference to PUs.

A spectrum opportunity occurs on a channel for two cases: 1) There is no PU
that conducts data transmission on that channel. 2) The SU can share a channel
with the PU if the interference to the PU is tolerable. Unfortunately,
obtaining this control information is costly in 5G mobile wireless networks.
Furthermore, the control information is outdated quickly in the non-stationary
5G networks. Therefore, it is impractical to design a DSA strategy by assuming
that all the control information is known.

The SU causes interference to PUs if they conduct data transmission on the
same channel at the same time. To protect the primary system, the SU performs
spectrum sensing to detect the activity of PUs before accessing a channel. We
assume that the SU has low communication capabilities and can only sense one
channel at a particular time. There has been studies on definition of reduced
capability devices in [24]. The DRL agent will use the sensing outcomes as the
input to learn a spectrum access strategy in order to maximize the spectrum
utilization while enabling the protection to PUs. To enhance the protection to
the primary system, we assume that the primary system will broadcast a warning
signal if it experiences strong interference. The warning signal contains
information related to which PU experiences interference so that the SU is
aware of the issue.

###  IV-C DRL-based solution for DSA

We now formulate the DSA problem using the DRL framework. To be specific, we
assume that the SU has a DRL agent that takes the sensing outcomes as
observations and learns how to perform spectrum sensing and access actions in
order to maximize its cumulative reward. Since the spectrum sensing outcomes
are incomplete and noisy, the environment is partially observable. Suppose
that there are a total of $C$ channels available for transmission. The
observation of the SU at time $t$ is denoted by
$s_{t}=\left(c_{t},q_{t}\right)$, where $c_{t}\in\\{0,1\\}$ is the sensing
outcome at time $t$ and $q_{t}$ is a one-hot $C$-dimensional vector
representing what channel was sensed at time $t$. We assume that the SU
performs spectrum sensing and then it makes the spectrum access decision. The
action of the SU at time $t$ is denoted by $a_{t}=\left(u_{t},i_{t}\right)$,
where $u_{t}\in\\{0,1\\}$ represents whether the SU will access the current
sensed channel ($u_{t}=1$) or stay idle ($u_{t}=0$) at time $t$,
$i_{t}\in\\{1,...,C\\}$ represents the channel index that the SU will sense at
time $t+1$. In other words, the SU makes two decisions: $u_{t}$ decides
whether to conduct data transmission in the current sensed channel at time $t$
and $i_{t}$ decides which channel to sense at time $t+1$.

The reward for the SU is designed to maximize the spectrum utilization while
preventing harmful interference to PUs, which is based on the underlying
achieved modulation and coding (MCS) strategy adopted in the 3GPP LTE/LTE-
Advanced standard [25]. To be specific, a user measures the quality of the
wireless connection and determines the MCS for data transmission. The selected
MCS corresponds to a specific spectral-efficiency, which is defined as the
transmitted information bits per symbol in a range from $0.1523$ to $5.5547$.
The reward function depends on the average spectral-efficiency of PUs, and we
choose $3$ as the threshold for determining if the interference to the primary
system is tolerable. Let $\bar{e}_{c,t}^{\text{PU}}$ be the average spectral-
efficiency of PUs on channel $c$. The reward of the SU at time $t$ is written
as

${r_{t}}=\begin{cases}0,&\text{SU is idle},\\\ -1,&\text{SU accesses channel
$c$ and }\bar{e}_{c,t}^{\text{PU}}<3,\\\ 1,&\text{SU accesses channel $c$ and
}\bar{e}_{c,t}^{\text{PU}}\geq 3.\end{cases}$ (18)

With the state space, action space and the reward function defined, we are
able to set an SU as an RL agent and let it learn from the environment the
best actions to take. In the next section we will provide simulation results
of applying the mentioned RL framework.

##  V Performance Evaluation

In this section, we evaluate the performance of the introduced DEQN
methodology in the DSA scenario of a relevant 5G network. We develop a network
simulator to provide practical radio environments in 5G networks by
incorporating field measurements obtained from a $10$-cell ray-tracing area in
a city and real-world spectrum occupancy database [26]. Furthermore, all users
are moving based on the random waypoint model [27], which is commonly used for
modeling the user mobility in 5G networks.

(a)

(b)

Figure 1: A snapshot of the DSS network geometry at the beginning of the
simulation. (a) First simulation scenario with 4 PUs and 1 SU. (b) Second
simulation scenario with 12 PUs and 3 SUs.

Fig. 1 shows the DSA scenarios in our experiment, where there are $1$ SU and
$4$ PUs in scenario (a), and $3$ SUs and $12$ PUs in scenario (b) for a
400m$\times$400m area. The primary system has a license to operate on $3$
wireless channels ($d_{x}=4$ and $d_{y}=6$), while each SU has to access the
channels without generating intolerable interference to the primary system.
Each SU adopts an online training algorithm that learns its spectrum access
strategy continually to adapt to the non-stationary 5G environment. To be
specific, each SU collects $T=500$ training samples in one episode and stores
them in its memory buffer. Then the SU trains its DRL agent for $100$
iterations to update its policy and applies the updated policy in the next
episode. Since the environment gradually changes over time, the SU removes the
outdated training samples from the memory buffer and collects another $T$
training samples in the new episode.

Clearly, the second scenario is a more dynamic environment and due to the
presence of multiple agents, it becomes even more dynamic as the agents also
change their behavior over time. We analyze the effectiveness of our
theoretical framework in scenario (a), because there is only one agent in the
environment and the underlying assumption of the system having the Markov
property is not voided. Specifically, we apply the method outlined in section
III-B to find the best $B_{rec}$ to use for our ESN kernel of the Q-network.
We then check if the suggested $B_{rec}$ provides good performance for us.

In the second scenario, however, we aim to evaluate the effect of using ESN as
the recurrent Q-network, in comparison with two baselines that use different
recurrent Q-network structures, which are DRQN [8] and ADRQN [28]. ADRQN
extends DRQN by using both observation and action as inputs of the Q-network.
Although we assume vanilla RNN in theoretical analysis for DRQN to simplify
the analysis, Long Short Term Memory (LSTM) is usually preferred to be used in
DRQN. Therefore, LSTM is used as the Q-network for DRQN and ADRQN. To have a
fair comparison, the LSTM of DRQN/ADRQN and the ESN of DEQN have the same
number of neurons $d_{h}=64$ and the learning rate that we saw has the best
performance on each kernel. This was $0.01$ for the DEQN and $0.001$ for the
DRQN and ADRQN. Furthermore, we let $B_{X}=1$, $B_{\text{in}}=0.5$,
$B_{\text{out}}=10$, $\gamma=0.9$, $R_{\text{max}}=1$, and $\epsilon=1.5$. The
choice of $\epsilon$ should usually be around $1/10$ of
$V_{max}=R_{max}/(1-\gamma)$.

From the method outlined in section III-B, we calculate the sum of the squared
bias and the variance in scenario (a) under different $B_{\text{rec}}$ in Fig.
2. We observe that $B_{\text{rec}}=0.7$ achieves the lowest upper bound on one
step approximation error. Fig. 3 shows the rolling window average reward
achieved under different choices of $B_{\text{rec}}$ with a window length of
160 decision steps. We can see that the choice of $B_{\text{rec}}=0.7$
ultimately achieves better performances than other choices while maintaining a
good convergence rate. We similarly set the $B_{rec}$ to be $0.7$ in scenario
(b) too.

Figure 2: The bound on the one step approximation error with respect to ESN’s
recurrent weights spectral radius, for the case of 4 PUs and 1 SU. Figure 3:
Rolling average reward under different recurrent weights’ spectral radius per
each step of decision making in the case of 4 PUs and 1 SU. Figure 4: Average
reward per episode with 4 PUs and 1 SU. Figure 5: Simulation results with 4
PUs and 1 SU. (a) Average reward per episode. (b) Success and failure
probability per episode for accessing a channel.

We now do the comparison between DEQN and DRQN in terms of their performance
in a DSA environment. Specifically, we will compare the average received
reward per episode and the probability of success and failure in accessing a
channel. All the results are compared while randomly initializing the
networks. Each episode is considered to be 500 steps of decision making.

The curves of the average reward per episode and success/failure probability
is shown in Figs. (4) and (5) respectively for each Q-network kernel. From
both of these figures we observe that DEQN and ADRQN have roughly the same
convergence rate while DRQN needs many more training samples to reach its best
performance. The comparable performance of DEQN with ADRQN is while DEQN has a
simple vanilla RNN type structure and the hidden state variables are not even
trained, compared to the other two methods that use LSTM with trainable
parameters. Assuming that the device has low computational capabilities, it is
important to choose a learning method that has less complexity. Therefore, we
will also compare the total simulation time of different learning methods.
Table I compares the average total CPU run time for $200$ episodes using the
same machine (2.71 GHz Intel i5 CPU) in scenario (a). DEQN has the shortest
training time because it only trains the output weights and avoids
recalculating hidden states. This is also an important aspect of DEQNs, as we
need computationally inexpensive algorithms that can be used in regular low
cost user equipment.

TABLE I: Average CPU run time with 4PUs and 1 SU. DEQN | DRQN | ADRQN
---|---|---
$330.4$sec |  $1255.9$sec |  $1417.5$sec

We emphasize that the the key benefit of using ESNs as kernel is its higher
convergence rate. This feature is mostly prominent in highly varying and
dynamic environments. Therefore, we will showcase the results for the case of
scenario (b) too. Similarly, Figs. (6) and (7) show the average reward and
success/failure probability per episode respectively. We can see that DEQN
reaches it’s best performance after around 25 episodes, while the other two
methods will need more than 150 episodes of training to reach their best
performance. Note that the most received reward for every DQN structure has
decreased compared to scenario (a). This is because the environment has more
PUs that are constantly transmitting data and the agents seem to have decided
to stay idle more than the case in scenario (a). On the other hand, the
success rate of DRQN and ADRQN reaches its best relatively quickly, while it
takes a large number of samples for them to reduce their failure rate. This is
while DEQN has a much faster convergence rate and has already learned to lower
its failure rate while maintaining a high success rate. Therefore, DEQN can
find the most spectrum opportunities. Similarly to scenario (a), we compare
the average total CPU run time in scenario (b) (summed over all agents) in
Table II. It is apparent that DEQN takes much less time to train its network
weights than DRQN.

Figure 6: Average reward per episode with 12 PUs and 3 SUs. Every plot is an
average over secondary users. Figure 7: Success and failure probability per
episode for accessing a channel with 12 PUs and 3 SUs. Every plot is an
average over secondary users.

Since all methods have the same number of training samples and training
iterations, the evaluation results confirm the theoretical findings and
clearly demonstrate that DEQN has the benefit of faster learning under limited
training data than DRQN in practice.

TABLE II: Average CPU run time with 12 PUs and 3 SUs DEQN | DRQN | ADRQN
---|---|---
$630.5$sec |  $2806.4$sec |  $2911.4$sec

##  VI Conclusion

In this work, we studied the performance of DRQNs and DEQNs and compared their
performance using the mathematical notion of covering numbers and how it
affects the one step approximation error of a DQN. We also proposed a method
for determining the hyper parameters of a DEQN by minimizing the empirically
derived bias error plus the theoretically upper bounded variance error. We
showed that DEQNs can provide faster convergence rates due to a lower covering
number bound. Our simulation of a dynamic spectrum sharing network with
multiple SUs and PUs also verified this as there were multiple agents in the
environment, adding to its non-stationarity. It is absolutely critical to use
a framework that converges faster in these environments as the training data
becomes outdated quickly. DEQNs provide us such a framework, allowing us to
not only train our network with less training data, but also incur low
computational compelxity for the user equipment.

##  Appendix A Proof of RNN covering number bound

In order to move forward with the proof, we first need to introduce a Lemma
from [29] which provides an upper bound for linear predictors handled with
matrix multiplications.

######  Lemma 1.

(Lemma 3.2 in [29]) Let $X\in\mathbb{R}^{d\times n}$ be a given data matrix
with $\|X\|_{F}\leq b$, where $d$ is the number of features and $n$ is the
number of samples. Then

$\displaystyle\ln\mathcal{N}\left(\Big{\\{}WX:W\in\mathbb{R}^{m\times
d},\|W\|_{F}\leq a\Big{\\}},\epsilon,\|\cdot\|_{2}\right)$
$\displaystyle\leq\left(\dfrac{a^{2}b^{2}m}{\epsilon^{2}}\right)\ln(2dm)$ (19)

In what follows, we first find a cover set for the whole network by
introducing a cover for each matrix multiplication in the RNN. Then, we will
tune each of the introduced covers’ granularity such that the whole cover
would have the desired granularity.

###  A-A Covering the whole network

We will start by forming a cover set for each of the inputs, i.e. $X_{i},1\leq
i\leq t$. For each input $X_{i}$, we form an $\varepsilon_{i}$-cover set
$\mathscr{X}_{i}$ of $WX_{i}$. The cardinality of $\mathscr{X}_{i}$ is

$\displaystyle|\mathscr{X}_{i}|$
$\displaystyle=\mathcal{N}\Big{\\{}WX_{i}:W\in\mathbb{R}^{d_{h}\times
d_{x}},\|W\|_{F}\leq B_{in},\varepsilon_{i},\|\cdot\|_{2}\Big{\\}}$
$\displaystyle\leq N_{i}$ (20)

where $N_{i}$ is obtained from the RHS of Lemma 1.

We then inductively form cover sets for each of the time slots of RNN. For
each element $F\in\mathscr{X}_{i}$, we form a $\mu_{1}$ cover
$\mathscr{U}_{1}(F)$ of $U(\sigma_{h}(F))$. The cardinality of
$\mathscr{U}_{1}(F)$ is

$\displaystyle|\mathscr{U}_{1}(F)|=\mathcal{N}\Big{\\{}$ $\displaystyle
U(\sigma_{h}(F)):U\in\mathbb{R}^{d_{h}\times d_{h}},\|U\|_{F}\leq
B_{rec},\mu_{1},$ $\displaystyle\|\cdot\|_{2}\Big{\\}}\leq
M_{1};F\in\mathscr{X}_{1}$ (21)

where $M_{1}$ is obtained from the RHS of Lemma 1. We then form the cover
$\mathscr{H}_{1}\coloneqq\bigcup_{F\in\mathscr{X}_{i}}\mathscr{U}_{1}(F)$
whose cardinality is

$\displaystyle\big{|}\mathscr{H}_{1}\big{|}=\Bigg{|}\bigcup_{F\in\mathscr{X}_{i}}\mathscr{U}_{1}(F)\Bigg{|}\leq\sum_{F\in\mathscr{X}_{i}}\big{|}\mathscr{U}_{1}(F)\big{|}\leq
M_{1}N_{1}$ (22)

Next, for every cover element $F\in\mathscr{X}_{i}$ and every cover element
$\tilde{F}\in\mathscr{H}_{i-1}$, we form a $\mu_{i}$ cover
$\mathscr{U}_{i}(F,\tilde{F})$ of $U(\sigma_{h}(F+\tilde{F}))$. The
cardinality of $\mathscr{U}_{i}(F,\tilde{F})$ is

$\displaystyle|\mathscr{U}_{i}(F,\tilde{F})|=\mathcal{N}\Big{\\{}U(\sigma_{h}(F+\tilde{F})):U\in\mathbb{R}^{d_{h}\times
d_{h}},\|U\|_{F}\leq B_{rec},$
$\displaystyle\mu_{i},\|\cdot\|_{2}\Big{\\}}\leq
M_{i};i\in\\{2,\dots,t\\},F\in\mathscr{X}_{i},\tilde{F}\in\mathscr{H}_{i-1}$
(23)

where $M_{i}$ is obtained from the RHS of Lemma 1. We then form the cover
$\mathscr{H}_{i}\coloneqq\bigcup_{F\in\mathscr{X}_{i},\tilde{F}\in\mathscr{H}_{i-1}}\mathscr{U}_{i}(F,\tilde{F})$
whose cardinality is

$\displaystyle\big{|}\mathscr{H}_{i}\big{|}=\Bigg{|}\bigcup_{\begin{subarray}{c}F\in\mathscr{X}_{i}\\\
\tilde{F}\in\mathscr{H}_{i-1}\end{subarray}},\mathscr{U}_{i}(F,\tilde{F})\Bigg{|}$
$\displaystyle\leq\sum_{F\in\mathscr{X}_{i}}\sum_{\tilde{F}\in\mathscr{H}_{i-1}}\big{|}\mathscr{U}_{1}(F,\tilde{F})\big{|}$
$\displaystyle\leq M_{i}N_{i}\big{|}\mathscr{H}_{i-1}\big{|}$ (24)

Therefore, by an induction on (24) given the first expression as (22), we
would have

$\displaystyle\big{|}\mathscr{H}_{i}\big{|}\leq\prod_{k=1}^{i}M_{k}N_{k}$ (25)

Finally, for every cover element $F\in\mathscr{X}_{t}$ and every cover element
$\tilde{F}\in\mathscr{H}_{t-1}$ we form the $\delta$-cover set
$\mathscr{Y}_{t}(F,\tilde{F}$) of $V(\sigma_{h}(F+\tilde{F}))$. The
cardinality of $\mathscr{Y}_{t}(F,\tilde{F})$ is

$\displaystyle|\mathscr{Y}_{t}(F,\tilde{F})|=\mathcal{N}\Big{\\{}V(\sigma_{h}(F+\tilde{F})):V\in\mathbb{R}^{d_{y}\times
d_{h}},\|V\|_{F}\leq B_{out},$ $\displaystyle\delta,\|\cdot\|_{2}\Big{\\}}\leq
Q;F\in\mathscr{X}_{t},\tilde{F}\in\mathscr{H}_{t-1}$ (26)

where $Q$ is obtained from the RHS of Lemma 1. We then form the whole cover of
the RNN function class by taking a union over $F$ and $\tilde{F}$ in (26),
i.e.
$\mathscr{F}_{t}\coloneqq\bigcup_{F\in\mathscr{X}_{t},\tilde{F}\in\mathscr{H}_{t-1}}\mathscr{Y}_{t}(F,\tilde{F})$.
The cardinality of $\mathscr{F}_{t}$ is lower bounded by

$\displaystyle\big{|}\mathscr{F}_{t}\big{|}=\Bigg{|}\bigcup_{\begin{subarray}{c}F\in\mathscr{X}_{t}\\\
\tilde{F}\in\mathscr{H}_{t-1}\end{subarray}}\mathscr{Y}_{t}(F,\tilde{F})\Bigg{|}$
$\displaystyle\leq\sum_{F\in\mathscr{X}_{t}}\sum_{\tilde{F}\in\mathscr{H}_{t-1}}\big{|}\mathscr{Y}_{t}(F,\tilde{F})\big{|}$
$\displaystyle\leq N_{t}\Bigg{(}\prod_{k=1}^{t-1}M_{k}N_{k}\Bigg{)}Q$ (27)

###  A-B Tuning the spacing between the cover elements

In this section we will intentionally set $\varepsilon_{i}$’s, $\mu_{i}$’s and
$\delta$ in such a way that $\mathscr{F}_{t}$ would be an $\varepsilon$-cover
for $Y_{t}$. In other words, we need those variables to be set in such a way
that for any given set of network parameters, we would be able to present a
$\hat{Y}_{t}\in\mathscr{F}_{t}$ such that
$|Y_{t}-\hat{Y}_{t}|\leq\varepsilon$.

To proceed, we will fix arbitrary values for $U$, $W$ and $V$. This way, we
would fix a function in our function class and we would like to find a
$\hat{Y}_{t}$ which is close to the output of this function given a specific
set of inputs $\\{X_{1},X_{2},\dots,X_{t}\\}$. We recursively define the
outputs of each time-series layer to be

$\displaystyle F_{i}\coloneqq WX_{i}$ $\displaystyle G_{0}\coloneqq 0$
$\displaystyle G_{i}\coloneqq U(\sigma_{h}(F_{i}+G_{i-1}))$ $\displaystyle
J_{t}\coloneqq V(\sigma_{h}(F_{i}+G_{i-1}))$ (28)

Next, we choose a cover element close to the output of each input layer. Then,
we recursively choose cover elements close the output of each time-series
layer, given the cover elements of the previous layer as inputs. The choices
are as follows

  * •

Choose $\hat{F}_{i}\in\mathscr{X}_{i}$ such that
$\|WX_{i}-\hat{F}_{i}\|<\varepsilon_{i}$

  * •

Choose $\hat{G}_{i}\in\mathscr{H}_{i}$ such that
$\|U(\sigma_{h}(\hat{F}_{i}+\hat{G}_{i-1}))-\hat{G}_{i}\|<\mu_{i}$

  * •

Choose $\hat{J}_{t}\in\mathscr{Y}_{t}$ such that
$\|V(\sigma_{h}(\hat{F}_{t}+\hat{G}_{t-1}))-\hat{J}_{t}\|<\delta$

  * •

Finally, let $\hat{Y}_{t}=\sigma_{y}(\hat{J}_{t})$

Now that we have properly chosen our cover elements, we need to make sure the
cover element chosen for the output is close enough to the actual output.
Therefore, we will write

$\displaystyle|Y_{t}-\hat{Y}_{t}|\leq\rho_{y}|J_{t}-\hat{J}_{t}|$
$\displaystyle\leq\rho_{y}\Big{|}V\big{(}\sigma_{h}(UH_{t-1}+WX_{t})\big{)}-V\big{(}\sigma_{h}(\hat{G}_{t-1}+\hat{F}_{t})\big{)}$
$\displaystyle+V\big{(}\sigma_{h}(\hat{G}_{t-1}+\hat{F}_{t})\big{)}-\hat{J}_{t}\Big{|}$
$\displaystyle\leq\rho_{y}\Big{(}|V|_{op}\rho_{h}\big{|}UH_{t-1}+WX_{t}-\hat{G}_{t-1}-\hat{F}_{t}\big{|}+\delta\Big{)}$
$\displaystyle\leq\rho_{y}\rho_{h}|V|_{op}\Big{(}\big{|}UH_{t-1}-\hat{G}_{t-1}\big{|}+\varepsilon_{t}\Big{)}+\rho_{y}\delta$
$\displaystyle=\rho_{y}\delta+\rho_{y}\rho_{h}|V|_{op}\varepsilon_{t}+\rho_{y}\rho_{h}|V|_{op}\big{|}UH_{t-1}-\hat{G}_{t-1}\big{|}$
(29)

The first two terms in (29) can be controlled using $\delta$ and
$\varepsilon_{t}$. In order to control the last term, however, we will conduct
the following analysis

$\displaystyle\big{|}UH_{i}-\hat{G}_{i}\big{|}=\Big{|}UH_{i}-U\big{(}\sigma_{h}(\hat{G}_{i-1}+\hat{F}_{i})\big{)}$
$\displaystyle+U\big{(}\sigma_{h}(\hat{G}_{i-1}+\hat{F}_{i})\big{)}-\hat{G}_{i}\Big{|}$
$\displaystyle\leq|U|_{op}\Big{|}H_{i}-\sigma_{h}\big{(}\hat{G}_{i-1}+\hat{F}_{i}\big{)}\Big{|}+\mu_{i}$
$\displaystyle=|U|_{op}\Big{|}\sigma_{h}\big{(}UH_{i-1}+WX_{i}\big{)}-\sigma_{h}\big{(}\hat{G}_{i-1}+\hat{F}_{i}\big{)}\Big{|}+\mu_{i}$
$\displaystyle\leq\rho_{h}|U|_{op}\big{|}UH_{i-1}-\hat{G}_{i-1}+WX_{i}-\hat{F}_{i}\big{|}+\mu_{i}$
$\displaystyle\leq\rho_{h}|U|_{op}\big{|}UH_{i-1}-\hat{G}_{i-1}\big{|}+\rho_{h}|U|_{op}\varepsilon_{i}+\mu_{i}$
(30)

Therefore, by an induction on (30) while considering the fact that
$\big{|}UH_{0}-\hat{G}_{0}\big{|}=0$, we would have

$\displaystyle\big{|}UH_{t-1}-\hat{G}_{t-1}\big{|}\leq\sum_{k=1}^{t-1}{\rho_{h}^{k}|U|_{op}^{k}\varepsilon_{t-k}}+\sum_{k=0}^{t-2}\rho_{h}^{k}|U|_{op}^{k}\mu_{t-1-k}$
(31)

Inserting (31) into (29) and considering the fact that for every matrix $A$,
$\|A\|_{op}\leq\|A\|_{F}$, we get

$\displaystyle|Y_{t}-\hat{Y}_{t}|\leq\rho_{y}\delta+\rho_{y}\rho_{h}|V|_{F}\varepsilon_{t}+\rho_{y}\rho_{h}|V|_{F}\sum_{k=1}^{t-1}{\rho_{h}^{k}|U|_{F}^{k}\varepsilon_{t-k}}$
$\displaystyle+\rho_{y}\rho_{h}|V|_{F}\sum_{k=0}^{t-2}\rho_{h}^{k}|U|_{F}^{k}\mu_{t-1-k}$
$\displaystyle\leq\rho_{y}\delta+\rho_{y}\rho_{h}B_{out}\left(\sum_{k=0}^{t-1}{\rho_{h}^{k}B_{rec}^{k}\varepsilon_{t-k}}+\sum_{k=0}^{t-2}\rho_{h}^{k}B_{rec}^{k}\mu_{t-1-k}\right)$
(32)

Now we have reached a point where we can fully control how close the cover
element in $\mathscr{F}_{t}$ can be to $Y_{t}$. We simply set

$\displaystyle\delta=\frac{\varepsilon/3}{\rho_{y}}$ (33)
$\displaystyle\varepsilon_{i}=\frac{\varepsilon/3}{t\rho_{y}\rho_{h}B_{out}\rho_{h}^{t-i}B_{rec}^{t-i}},i\in\\{1,\dots,t\\}$
(34)
$\displaystyle\mu_{i}=\frac{\varepsilon/3}{(t-1)\rho_{y}\rho_{h}B_{out}\rho_{h}^{t-1-i}B_{rec}^{t-1-i}},i\in\\{1,\dots,t-1\\}$
(35)

so that we would have

$\displaystyle|Y_{t}-\hat{Y}_{t}|\leq\varepsilon$ (36)

Now that we have covered the output of an RNN, it is required that we
characterize $N_{i}$, $M_{i}$ and $Q$ so that we could evaluate (27). For this
end, we need to get an upper bound on the norm of the output of each layer.

Starting with (20), we have $\|X\|_{F}\leq B_{X}$. For the coverings in (21),
(23) and (26), the input of the covers are
$\sigma_{h}(H_{i})=\sigma_{h}(WX_{i}+UH_{i-1})$, which is spectrally bounded
by

$\displaystyle\|\sigma_{h}(WX_{i}+UH_{i-1})\|\leq\rho_{h}(\|WX_{i}\|+\|UH_{i-1}\|)$
$\displaystyle\leq\rho_{h}\|W\|\|X_{i}\|+\rho_{h}\|U\|\|H_{i-1}\|$
$\displaystyle=\rho_{h}\|W\|\|X_{i}\|+\rho_{h}\|U\|\|\sigma_{h}(WX_{i-1}+UH_{i-2})\|,$
$\displaystyle i\in\\{1,\dots,t\\},X_{0}=H_{0}=H_{-1}=0$ (37)

We can then use induction to conclude that

$\displaystyle\|\sigma_{h}(H_{i})\|=\|\sigma_{h}(WX_{i}+UH_{i-1})\|$
$\displaystyle\leq\sum_{k=1}^{i}{\rho_{h}^{i-k+1}}\|W\|\|X_{k}\|\|U\|^{i-k}\leq
B_{W}B_{X}\sum_{k=1}^{i}{\rho_{h}^{i-k+1}}B_{rec}^{i-k}$
$\displaystyle=\rho_{h}B_{W}B_{X}\dfrac{\rho_{h}^{i}B_{rec}^{i}-1}{\rho_{h}B_{rec}-1}\eqqcolon
B_{H,i},i\in\\{1,\dots,t\\}$ (38)

Therefore, we would have

$\displaystyle\ln(N_{i})\coloneqq\left(\dfrac{B_{in}^{2}B_{X}^{2}d_{h}}{\epsilon_{i}^{2}}\right)\ln(2d_{h}d_{x})$
(39)
$\displaystyle\ln(M_{i})\coloneqq\left(\dfrac{B_{rec}^{2}B_{H,i}^{2}d_{h}}{\mu_{i}^{2}}\right)\ln(2d_{h}^{2})$
(40)
$\displaystyle\ln(Q)\coloneqq\left(\dfrac{B_{out}^{2}B_{H,t}^{2}d_{y}}{\delta^{2}}\right)\ln(2d_{y}d_{h})$
(41)

Combining (33 \- 35) and (38) with (39 \- 41) and substituting in (27) we get

$\displaystyle\ln\left(|\mathscr{F_{t}}|\right)=\ln(Q)+\sum_{k=1}^{t}\ln(N_{i})+\sum_{k=1}^{t-1}\ln(M_{k})$
(42)

in which we have

$\displaystyle\ln(Q)=\left(\dfrac{9\rho_{y}^{2}\rho_{h}^{2}B_{out}^{2}B_{W}^{2}B_{X}^{2}\left(\rho_{h}^{t}B_{rec}^{t}-1\right)^{2}}{\left(\rho_{h}B_{rec}-1\right)^{2}\epsilon^{2}}\right)\ln(2d_{y}d_{h})$
(43) $\displaystyle\sum_{k=1}^{t}\ln(N_{i})=\sum_{k=0}^{t-1}\ln(N_{t-k})$
$\displaystyle=\sum_{k=0}^{t-1}\dfrac{9t^{2}\rho_{y}^{2}\rho_{h}^{2}B_{out}^{2}B_{in}^{2}B_{X}^{2}B_{rec}^{2}\rho_{h}^{2k}}{\epsilon^{2}}\ln(2d_{h}d_{x})$
$\displaystyle=\dfrac{9t^{2}\rho_{y}^{2}\rho_{h}^{2}B_{out}^{2}B_{in}^{2}B_{X}^{2}B_{rec}^{2}\rho_{h}^{2}}{\epsilon^{2}}\left(\dfrac{\rho_{h}^{2t}B_{rec}^{2t}-1}{\rho_{h}^{2}B_{rec}^{2}-1}\right)\ln(2d_{h}d_{x})$
(44) $\displaystyle\sum_{k=1}^{t-1}\ln(M_{i})=\sum_{k=1}^{t-1}\ln(M_{t-k})$
$\displaystyle=\sum_{k=1}^{t-1}\dfrac{9(t-1)^{2}\rho_{y}^{2}\rho_{h}^{2k}B_{out}^{2}B_{rec}^{2k}B_{H,t-k}^{2}d_{h}}{\epsilon^{2}}\ln(2d_{h}^{2})$
$\displaystyle=\dfrac{9(t-1)^{2}\rho_{y}^{2}\rho_{h}^{2}B_{W}^{2}B_{X}^{2}B_{out}^{2}}{\epsilon^{2}\left(\rho_{h}B_{rec}-1\right)^{2}}\sum_{k=1}^{t-1}\rho_{h}^{2k}B_{rec}^{2k}\left(\rho_{h}^{t-k}B_{rec}^{t-k}-1\right)^{2}\cdot\ln(2d_{h}^{2})$
$\displaystyle=\dfrac{9(t-1)^{2}\rho_{y}^{2}\rho_{h}^{2}B_{W}^{2}B_{X}^{2}B_{out}^{2}}{\epsilon^{2}\left(\rho_{h}B_{rec}-1\right)^{2}}\bigg{(}(t-1)(\rho_{h}B_{rec})^{2t}-2\rho_{h}^{t+1}$
$\displaystyle\cdot
B_{rec}^{t+1}\dfrac{(\rho_{h}B_{rec})^{t-1}-1}{\rho_{h}B_{rec}-1}+\rho_{h}^{2}B_{rec}^{2}\dfrac{(\rho_{h}B_{rec})^{2t-2}-1}{\rho_{h}B_{rec}-1}\bigg{)}\cdot\ln(2d_{h}^{2})$
(45)

And so the proof of Proposition 1 is complete. $\blacksquare$

##  Appendix B Proof of ESN covering number bound

Covering an ESN is pretty similar to covering an RNN. Since $W_{in}$ is random
but fixed, we do not need to cover the input layer’s output as there is no
parameter tuning feature for it. The same can be said for the recurrent
layers. Therefore, we do not need to define cover sets for those layers as we
have done in (20), (21) and (23). The only tunable parameters are those in the
output layer, thus we form a $\lambda_{t}$-cover set $\mathscr{Z}_{t}$ of
$W_{out}H_{t}$, which its cardinality will be bounded as

$\displaystyle|\mathscr{Z}_{t}|=\mathcal{N}\Big{\\{}W_{out}H_{t}:W_{out}\in\mathcal{W}_{out},\lambda_{t},\|\cdot\|_{2}\Big{\\}}\leq
R$ (46)

Where $R$ is equal to the RHS of [Lemma in Bartlett paper]. Now we will fix
$W_{out}$ so that we can find a cover element that is close to the output of
the ESN. For every input sequence $\\{X_{1},X_{2},\dots,X_{t}\\}$ we would
choose $\hat{Z}_{t}\in\mathscr{Z}_{t}$ such that
$|Y_{t}-\sigma_{y}(\hat{Z}_{t})|\leq\epsilon_{t}$. Knowing that

$\displaystyle|Y_{t}-\sigma_{y}(\hat{Z}_{t})|=|\sigma_{y}(H_{t})-\sigma_{y}(\hat{Z}_{t})|\leq\rho_{y}|H_{t}-\hat{Z}_{t}|\leq\rho_{y}\lambda_{t}$
(47)

we only need to set $\lambda_{t}=\epsilon/\rho_{y}$ so that we would have a
cover element $\hat{Y}_{t}=\sigma_{y}(\hat{Z}_{t})$ that is in
$\epsilon$-proximity of $Y_{t}$. We remark that $H_{t}$ is fixed given a fixed
input sequence. The only thing that determines $R$ in (46) is $H_{t}$’s
spectral norm which has already been shown to be upper bounded by the
expression in (38). Inserting this into Lemma 1 we would have the metric
entropy of an ESN upper bounded by

$\displaystyle\ln\left(\mathcal{N}(\mathcal{H}_{ESN,t},\epsilon,\|\cdot\|)\right)\leq\dfrac{\rho_{y}^{2}\|H_{t}\|^{2}B_{out}^{2}d_{y}}{\epsilon^{2}}\ln(2d_{h}d_{y})$
$\displaystyle\leq\dfrac{\rho_{y}^{2}\rho_{h}^{2}B_{in}^{2}B_{X}^{2}B_{out}^{2}d_{y}}{\epsilon^{2}}\left(\dfrac{\rho_{h}^{t}B_{rec}^{t}-1}{\rho_{h}B_{rec}-1}\right)^{2}\ln(2d_{h}d_{y})$
(48)

And this completes the proof of Proposition 2. $\blacksquare$

## References

  * [1] S. Haykin and P. Setoodeh, “Cognitive radio networks: The spectrum supply chain paradigm,” _IEEE Transactions on Cognitive Communications and Networking_ , vol. 1, no. 1, pp. 3–28, 2015.
  * [2] T. Chakraborty, I. S. Misra, and T. Manna, “Design and implementation of VoIP based two-tier cognitive radio network for improved spectrum utilization,” _IEEE Systems Journal_ , vol. 10, no. 1, pp. 370–381, 2016\.
  * [3] N. Rupasinghe and İ. Güvenç, “Reinforcement learning for licensed-assisted access of lte in the unlicensed spectrum,” in _2015 IEEE Wireless Communications and Networking Conference (WCNC)_. IEEE, 2015, pp. 1279–1284.
  * [4] F. M. Abinader et al., “Enabling the coexistence of lte and wi-fi in unlicensed bands,” _IEEE Communications Magazine_ , vol. 52, no. 11, pp. 54–61, 2014.
  * [5] “National 6G Roadmap,” ATIS NextG Alliance, 02 2022.
  * [6] R. Shafin, L. Liu, V. Chandrasekhar, H. Chen, J. Reed, and J. C. Zhang, “Artificial intelligence-enabled cellular networks: A critical path to beyond-5g and 6g,” _IEEE Wireless Communications_ , vol. 27, no. 2, pp. 212–217, 2020.
  * [7] H.-H. Chang, H. Song, Y. Yi, J. Zhang, H. He, and L. Liu, “Distributive dynamic spectrum access through deep reinforcement learning: A reservoir computing-based approach,” _IEEE Internet of Things Journal_ , vol. 6, no. 2, pp. 1938–1948, 2018.
  * [8] M. Hausknecht and P. Stone, “Deep recurrent Q-learning for partially observable MDPs,” in _2015 aaai fall symposium series_ , 2015.
  * [9] M. Igl, L. Zintgraf, T. A. Le, F. Wood, and S. Whiteson, “Deep variational reinforcement learning for pomdps,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 2117–2126.
  * [10] Y. Xu, J. Yu, and R. M. Buehrer, “The application of deep reinforcement learning to distributed spectrum access in dynamic heterogeneous environments with partial observations,” _IEEE Transactions on Wireless Communications_ , vol. 19, no. 7, pp. 4494–4506, 2020.
  * [11] R. Pascanu, T. Mikolov, and Y. Bengio, “On the difficulty of training recurrent neural networks,” in _International conference on machine learning_. PMLR, 2013, pp. 1310–1318.
  * [12] S. Huang, H. Su, J. Zhu, and T. Chen, “Svqn: Sequential variational soft q-learning networks,” in _International Conference on Learning Representations_ , 2019.
  * [13] H.-H. Chang, L. Liu, and Y. Yi, “Deep echo state Q-network (DEQN) and its application in dynamic spectrum sharing for 5G and beyond,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 33, no. 3, pp. 929–939, 2022.
  * [14] H. Jaeger, “The “echo state” approach to analysing and training recurrent neural networks-with an erratum note,” _Bonn, Germany: German National Research Center for Information Technology GMD Technical Report_ , vol. 148, no. 34, p. 13, 2001.
  * [15] G. Tanaka, T. Yamane, J. B. Héroux, R. Nakane, N. Kanazawa, S. Takeda, H. Numata, D. Nakano, and A. Hirose, “Recent advances in physical reservoir computing: A review,” _Neural Networks_ , vol. 115, pp. 100–123, 2019.
  * [16] J. Fan, Z. Wang, Y. Xie, and Z. Yang, “A theoretical analysis of deep q-learning,” in _Learning for Dynamics and Control_. PMLR, 2020, pp. 486–489.
  * [17] M. Chen, X. Li, and T. Zhao, “On generalization bounds of a family of recurrent neural networks,” in _International Conference on Artificial Intelligence and Statistics_. PMLR, 2020, pp. 1233–1243.
  * [18] M. Lauri, D. Hsu, and J. Pajarinen, “Partially observable markov decision processes in robotics: A survey,” _IEEE Transactions on Robotics_ , 2022\.
  * [19] Y. Duan, C. Jin, and Z. Li, “Risk bounds and rademacher complexity in batch reinforcement learning,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 2892–2902.
  * [20] T. Zhang, “Covering number bounds of certain regularized linear function classes,” _Journal of Machine Learning Research_ , vol. 2, no. Mar, pp. 527–550, 2002.
  * [21] L. Gonon, L. Grigoryeva, and J.-P. Ortega, “Approximation bounds for random neural networks and reservoir systems,” _arXiv preprint arXiv:2002.05933_ , 2020.
  * [22] S. Boyd, S. P. Boyd, and L. Vandenberghe, _Convex optimization_. Cambridge university press, 2004.
  * [23] J. Foerster, N. Nardelli, G. Farquhar, T. Afouras, P. H. Torr, P. Kohli, and S. Whiteson, “Stabilising experience replay for deep multi-agent reinforcement learning,” in _International conference on machine learning_. PMLR, 2017, pp. 1146–1155.
  * [24] 3GPP, “Technical Specification Group Radio Access Network; study on support of reduced capability NR devices (Release 17). TR 38.875,” 3rd Generation Partnership Project (3GPP), Mar. 2021, version 17.0.0.
  * [25] ——, “Evolved universal terrestrial radio access (E-UTRA); Physical Layer Procedures. Technical Specification (TS) 36.213,” 3rd Generation Partnership Project (3GPP), Jun. 2019, version 15.6.0.
  * [26] M. Wellens and P. Mähönen, “Lessons learned from an extensive spectrum occupancy measurement campaign and a stochastic duty cycle model,” _Mobile networks and applications_ , vol. 15, no. 3, pp. 461–474, 2010.
  * [27] E. Hyytiä and J. Virtamo, “Random waypoint mobility model in cellular networks,” _Wireless Networks_ , vol. 13, no. 2, pp. 177–188, 2007.
  * [28] P. Zhu, X. Li, P. Poupart, and G. Miao, “On improving deep reinforcement learning for pomdps,” _arXiv preprint arXiv:1804.06309_ , 2018.
  * [29] P. L. Bartlett, D. J. Foster, and M. J. Telgarsky, “Spectrally-normalized margin bounds for neural networks,” _Advances in neural information processing systems_ , vol. 30, 2017.