# Risk-Sensitive Reinforcement Learning for URLLC Traffic in Wireless Networks

Nesrine Ben-Khalifa⋄, Mohamad Assaad⋄, Mérouane Debbah∗
⋄TCL chair on 5G, Laboratoire des Signaux et Systèmes (L2S), Centrale Supélec,
France
∗Huawei Technologies, Boulogne-Billancourt, France

###### Abstract

In this paper, we study the problem of dynamic channel allocation for URLLC
traffic in a multi-user multi-channel wireless network where urgent packets
have to be successfully transmitted in a timely manner. We formulate the
problem as a finite-horizon Markov Decision Process with a stochastic
constraint related to the QoS requirement, defined as the packet loss rate for
each user. We propose a novel weighted formulation that takes into account
both the total expected reward (number of successfully transmitted packets)
and the risk which we define as the QoS requirement violation. First, we use
the value iteration algorithm to find the optimal policy, which assumes a
perfect knowledge of all the model parameters, namely the channel statistics.
We then propose a Q-learning algorithm where the controller learns the optimal
policy without having knowledge of neither the CSI nor the channel statistics.
We illustrate the performance of our algorithms with numerical studies.

###### Index Terms:

URLLC, risk-sensitivity, resource allocation, constrained MDP, reinforcement
learning

##  I Introduction

In the fifth generation (5G) wireless networks, there are new service
categories with heterogeneous and challenging requirements, among them the
Ultra Reliable Low Latency (URLLC) traffic [6], designed for delay and
reliability sensitive applications like real-time remote control, autonomous
driving, and mission-critical traffic. In URLLC traffic, the End-to-End (E2E)
latency defined by 3GPP is lower than $1$ ms along with a reliability
requirement of $1-10^{-5}$ to $1-10^{-9}$ [6, 15].

A plausible solution to address the latency requirement issue is to make
transmissions without Channel State Information (CSI) knowledge at the
transmitter side. To increase reliability, exploiting frequency diversity is
beneficial, and occurs by making parallel transmissions of the same packet
over different subcarriers in an Orthogonal Frequency Division Multiplexing
(OFDM) system where each subcarrier experiences different channel
characteristics.

However, this solution is costly in terms of system capacity. Therefore, the
number of parallel transmissions should not be fixed in advance but should
rather be variable and depending on many parameters such as the position of a
user in the cell, or the statistics about his packet losses over the previous
time slots. For example, if a user experienced a high number of packet losses
in the previous time slots, it should be allocated a high number of
subchannels to increase his success probability, whereas a user with a low
number of dropped packets may be assigned a low number of subcarriers. Hence,
it is crucial to design efficient dynamic schemes able to adapt the number of
parallel transmissions for each user to his experienced QoS.

In this work, we study the problem of dynamic channel allocation for URLLC
traffic in a multi-user multi-channel wireless network under QoS constraints.
A channel here refers to a frequency band or a subcarrier in an OFDM system,
and the QoS is related to the packet loss rate for each user, defined as the
average number of dropped packets. Besides, we introduce the notion of risk
related to the violation of the QoS requirements; more precisely, a risk
occurs or equivalently, a risk state is reached when the QoS requirement is
violated for a user. Furthermore, we consider that the transmitter does not
have neither the CSI nor the channel statistics at the transmission moment. In
fact, due to the urgency of URLLC packets mentioned previously, there is not
enough time for the BS to make channel estimation and probing techniques like
in conventional wireless communications.

###  I-A Related Work

The issue of deadline-constrained traffic scheduling has been investigated by
several works including [10, 17, 9, 8]. For example, in [8], the authors study
the problem of dynamic channel allocation in a single user multi-channel
system with service costs and deadline-constrained traffic. They propose
online algorithms to enable the controller to learn the optimal policy based
on Thompson sampling for multi-armed bandit problems. The MDP framework and
reinforcement learning approaches for downlink packet scheduling are
considered in [10, 9, 1, 14, 3, 2]. In [10], the authors propose an MDP for
deadline-constrained packet scheduling problem and use dynamic programming to
find the optimal scheduling policies. The authors do not consider QoS
constraints in the scheduling problem.

Most risk-sensitive approaches consist in analyzing higher order statistics
than the average metric such as the variance of the reward [6, 13, 16, 5]. For
instance, a risk-sensitive reinforcement learning is studied in [20] in
millimeter-wave communications to optimize both the bandwidth and transmit
power. The authors consider a utility (data rate) that incorporates both the
average and the variance to capture the tail distribution of the rate, useful
for the reliability requirement of URLLC traffic. The authors do not exploit
frequency diversity.

In this work, we consider an alternative approach to the risk which consists
in minimizing the risk state visitation probability. In fact, due to the
stochastic nature of the problem (time-varying channel and random arrival
traffic in our context), giving a low reward to an undesirable or a risk-state
may be insufficient to minimize the probability of visiting such state [12].
Therefore, in addition to the maximization of the total expected reward, we
propose to consider a second criterion which consists in minimizing the
probability of visiting risk states where a risk state here is related to the
violation of QoS requirements.

###  I-B Addressed Issues and Contribution

In this work, we address the following issues:

  * $\bullet$

We formulate the dynamic channel allocation problem for URLLC traffic as a
finite-horizon MDP wherein the state represents the QoS of the users, that is,
the average number of dropped packets or packet loss rate of the users. The
decision variable is the number of channels to assign to each user. We define
a risk state as any state where the QoS requirement is violated for at least
one user. Besides, we define a stochastic constraint related to the risk state
visitation probability.

  * $\bullet$

Assuming the channel statistics are known to the controller, we use the
finite-horizon value iteration algorithm to find the optimal policy to the
weighted formulation of the problem, which takes into account both the total
expected reward over the planning horizon and the risk criterion (QoS
requirement violation probability).

  * $\bullet$

When the channel statistics are unknown to the controller, we propose a
reinforcement learning algorithm (Q-learning) for the weighted formulation of
the problem, which enables the controller to learn the optimal policy. We
illustrate the performance of our algorithms with numerical studies.

###  I-C Paper Structure

In Section II, we present the system model for the multi-user multi-channel
wireless network with URLLC packets and time-varying channels along with the
QoS definition. In Section III, we introduce the constrained MDP formulation
with all its components. In Section IV, we present both the finite-horizon
value iteration algorithm and the reinforcement learning algorithm. Section V
is devoted to numerical results. Finally, we conclude the paper in Section VI.

##  II System Model

We consider a multi-user multi-channel wireless network where URLLC packets
have to be transmitted over time-varying and fading channels. Due to the
strict latency requirement of URLLC packets in 5G networks mentioned
previously, there is not enough time for the BS to estimate the channel, and
the packets are then immediately transmitted in the absence of CSI at the
transmitter side. When a packet is successfully decoded, the receiver sends an
acknowledgment feedback, which is assumed to be instantaneous and error-free.
We consider a centralized controller which dynamically distributes the
channels to the users based on their QoS (see Fig. 1).

Controller user $1$ user $K$
$\rho_{1}(t+1)$$\rho_{K}(t+1)$$\vdots$$a_{1}(t)$$a_{K}(t)$$\ell_{1}(t)$$\ell_{K}(t)$
Figure 1: Dynamic allocation of channels $(\ell_{1},..,\ell_{K})$ to the users
based on their QoS $(\rho_{1},..,\rho_{K})$.

Furthermore, we make the following assumptions:

Packet arrival process: the packet arrival process is considered as an
independent and identically distributed (i.i.d.) random process over a finite
set $\mathcal{I}=\\{0,1,..,A_{max}\\}$, where $A_{max}$ is a positive
constant, and is identical for all the users. Let $\alpha_{a}$ denote the
probability that $a\in\mathcal{I}$ packets arrive for a given user at the
beginning of a time slot.

Deadline-constrained traffic: regarding the strict URLLC latency requirement
specified by 3GPP (lower than 1 ms), each packet has a lifetime of one time
slot and can either be served or dropped; if there are available channels, the
packet will be transmitted, otherwise, it will be dropped because after one
time slot it becomes outdated and useless. Furthermore, one packet is
transmitted per channel.

Channel model: we consider i.i.d. Bernoulli channels with a mean
$\mu\in[0,1]$. In millimeter-wave communications, the links are characterized
by their intermittence and high sensitivity, and this channel model reflects
the existence of a light-of-sight (LOS) channel state[8, 4]. To increase
reliability, a user can be assigned more channels than the number of waiting
packets (depending on his experienced QoS). Some packets are then
simultaneously sent over multiple parallel channels.

Channel split: for each user, all the packets are equally important: when the
number of available channels is larger than that of waiting packets, we assume
that some packets are picked uniformly at random to be replicated. A packet is
obviously more likely to be successfully transmitted when sent over many
channels simultaneously. However, assigning more channels to a user will
affect the QoS experienced by the other users. Note that channel split across
the packets (which occurs in the same manner for all the users) should not be
confused with the channel split across the users (which takes into account the
QoS perceived by the users).

For user $k$, the distribution of available channels $\ell_{k}$ over the
waiting packets $a_{k}$ occurs as follows: each packet is transmitted over
($\ell_{k}\wedge a_{k}$) channels and may be furthermore replicated once with
a probability ($\frac{\ell_{k}\vee a_{k}}{a_{k}}$), where the symbol
$\ell_{k}\wedge a_{k}$ denotes the larger integer $m$ such that
$ma_{k}\leqslant\ell_{k}$, and $\ell_{k}\vee a_{k}$ denotes the remaining
integer of the division of $\ell_{k}$ by $a_{k}$.

The probability that a packet is successfully transmitted given that there are
$a_{k}$ waiting packets at the transmitter and $\ell_{k}$ assigned channels
can then be expressed by

$\displaystyle\nu_{k}(a_{k},\ell_{k})$ $\displaystyle=$
$\displaystyle\left(1-\frac{\ell_{k}\vee
a_{k}}{a_{k}}\right)\left(1-(1-\mu)^{(\ell_{k}\wedge a_{k})}\right)$ (1)
$\displaystyle+\left(\frac{\ell_{k}\vee
a_{k}}{a_{k}}\right)\left(1-(1-\mu)^{1+(\ell_{k}\wedge a_{k})}\right).$

The expected number of successfully transmitted packets for user $k$ is then
given by

$\displaystyle\mathbb{E}\left[N_{k}(\ell_{k})\right]=\sum_{a_{k}\in\mathcal{I}}{a_{k}\alpha_{a_{k}}\nu_{k}(a_{k},\ell_{k})}.$
(2)

QoS criterion: for each user $k$, we define the packet loss rate at time slot
$t$, $\rho_{k}(t)$, as follows

$\displaystyle\rho_{k}(t)=\frac{1}{t}\sum_{i=0}^{t-1}{\frac{n_{k}(i)}{a_{k}(i)}},\quad
t\geqslant 1,$ (3)

where $n_{k}(t)$ denotes the number of lost packets for user $k$ at time slot
$t$. Note that $\rho_{k}\in[0,1]$ ($n_{k}(t)\leqslant a_{k}(t)$). A packet is
lost when either of the two following events occurs:

  * (i)

it is not transmitted because of insufficient available channels,

  * (ii)

is transmitted but ACK feedback is not received.

The parameter $\rho_{k}$ reflects the QoS perceived by user $k$: higher values
of $\rho_{k}$ mean a higher number of lost packets and poor QoS whereas lower
values of $\rho_{k}$ mean good QoS. To ensure good QoS for the users, the
resource allocation scheme should take account of their experienced QoS and
keep this parameter values for all users within an acceptable range.

Finally, the decision variable is the number of channels associated to each
user $k$ at each time slot, denoted by $\ell_{k}$, which satisfies

$\displaystyle\sum_{k=1}^{K}{\ell_{k}(t)}=L,$ (4)

where $L$ denotes the number of available channels.

##  III Constrained MDP Framework

The stochastic nature of the wireless channel incites us to consider an MDP
framework to solve the decision problem. In this section, we first introduce
the constrained MDP formulation along with its components. We then derive the
optimality equations.

###  III-A Model Formulation

We define the following finite-horizon MDP

  * $\bullet$

State Space: is the finite set $\mathcal{T}\times\mathcal{S}$ where
$\mathcal{T}=\\{0,..,T\\}$,
$\mathcal{S}=\left\\{\rho_{1}\times..\times\rho_{K}\right\\}$, $\rho_{k}$ for
$k=1,..,K$ is defined in (3), and the symbol $\times$ stands for the Cartesian
product.

  * $\bullet$

Action Space: is the finite set $\mathcal{L}=\\{(\ell_{1},..,\ell_{K})\mbox{
satisfying }\eqref{contrainte}\\}$, where $\ell_{k}$ denotes the number of
channels assigned to user $k$.

  * $\bullet$

Reward: we define the reward $r$ at time slot $t$, when the controller chooses
action $\ell\in\mathcal{L}$ in state $s_{t}$, as the expected total number of
successfully transmitted packets over all the users, that is,

$\displaystyle
r(s_{t},\ell)=\mathbb{E}\left[{\sum_{k=1}^{K}{N_{k}}(\ell_{k})}\right].$ (5)

Note that the reward depends only on the number of channels allocated for each
user (the action), and not on the current state $s_{t}$. Besides, the reward
is a non-linear function of the action.

  * $\bullet$

Transition Probabilities:

First, we define the probability that $n$ packets are lost for user $k$ as a
function of the number of waiting packets $a_{k}$ and the number of assigned
channels $\ell_{k}$ at a given time slot as follows

$\displaystyle\sigma_{k}(n,a_{k},\ell_{k})=\binom{a_{k}}{n}\Big{(}1-\nu_{k}(a_{k},\ell_{k})\Big{)}^{n}\nu_{k}(a_{k},\ell_{k})^{a_{k}-n},$

where $n\leqslant a_{k}$ and $\binom{a_{k}}{n}$ denotes the binomial
coefficient. The state transition probability for user $k$ is given by

$\displaystyle
p(\rho^{\prime}_{k}\,|\,\rho_{k}(t),\ell_{k})=\alpha_{a_{k}}\sigma_{k}(n,a_{k},\ell_{k}),$
(6)

where

$\displaystyle\rho^{\prime}_{k}=\frac{t}{t+1}\rho_{k}+\frac{1}{t+1}\frac{n}{a_{k}}.$
(7)

Finally, let $s_{t+1}=\rho^{\prime}_{1}\times..\times\rho^{\prime}_{K}$ and
$s_{t}=\rho_{1}\times..\times\rho_{K}$, the transition probability from state
$s_{t}$ to state $s_{t+1}$ given when action $l$ is taken, is then given by

$\displaystyle
p(s_{t+1}\,|\,s_{t},\ell)=\prod_{k=1}^{K}{p(\rho^{\prime}_{k}\,|\,\rho_{k}(t),\ell_{k})}.$
(8)

Regarding the strict requirements of URLLC packets described earlier, we
introduce in the following the notion of a risk-state.

######  Definition 1.

We define a risk state any state where $\rho_{k}>\rho_{max}$ for any
$k\in\\{1,..,K\\}$ with $\rho_{max}>0$ is constant fixed by the controller.
The set of risk states $\Phi$ is then,

$\displaystyle\Phi=\\{\rho_{1}\times..\times\rho_{K}\mbox{ where there
$\exists$ }k\mbox{ such that }\rho_{k}>\rho_{max}\\}.$

Besides, a risk-state is an absorbing state, that is, the process ends when it
reaches a risk state [12].

A deterministic policy $\pi$ assigns at each time step and for each state an
action. Our goal is to find an optimal deterministic policy $\pi^{*}$ which
maximizes the total expected reward $\mathcal{V}_{T}^{\pi}(s)$ given by

$\displaystyle\mathcal{V}_{T}^{\pi}(s)=\mathbb{E}^{\pi}\left[\sum_{t=0}^{T}{r(s_{t},\pi(s_{t}))|\,s_{0}=s}\right],$
(9)

with the reward $r$ is defined in (5), while satisfying the QoS constraint
given by

$\displaystyle\eta^{\pi}(s)<w,$ (10)

where $\eta^{\pi}(s)$ denotes the probability of visiting a risk state over
the planning horizon, given that the initial state (at time slot $0$) is $s$
and policy $\pi$ is followed, and $w$ is a positive constant. Formally,

$\displaystyle\eta^{\pi}(s)=\mathbb{P}^{\pi}\;(\exists\,t\mbox{ such that
}s_{t}\in\Phi|s_{0}=s).$ (11)

In order to explicitly characterize $\eta^{\pi}(s)$, we introduce in the
following the risk signal $\overline{r}$.

######  Definition 2.

We define a risk signal $\overline{r}$ as follows

$\overline{r}(s_{t},\ell_{t},s_{t+1})=\left\\{\begin{array}[]{cc}1&\quad\mbox{
if }s_{t+1}\in\Phi\\\ 0&\quad\mbox{ otherwise,}\end{array}\right.$ (12)

where $s_{t}$ and $\ell_{t}$ denote the state and action at time slot $t$,
respectively, and $s_{t+1}$ denotes the subsequent state.

######  Proposition 1.

The probability of visiting a risk-state, $\eta^{\pi}(s)$, is given by

$\displaystyle\eta^{\pi}(s)=\overline{\mathcal{V}}_{T}^{\pi}(s),$ (13)

where we set

$\displaystyle\overline{\mathcal{V}}_{T}^{\pi}(s)=\mathbb{E}^{\pi}\left[\displaystyle\sum_{t=0}^{T}{\overline{r}\left(s_{t},\pi(s_{t}),s_{t+1}\right)\,|\,s_{0}=s}\right].$
(14)

###### Proof.

The random sequence $\overline{r}(t=0)$, $\overline{r}(t=1)$,..,
$\overline{r}(t=T)$ may contain $1$ if a risk state is visited, otherwise all
its components are equal to zero (recall that a risk state is an absorbing
state). Therefore, $\sum_{t=0}^{T}{\overline{r}(t)}$ is a Bernoulli random
variable with a mean equal to the probability of reaching a risk state, that
is, relation (13) holds. ∎

###  III-B Optimality Equations

By virtue of Proposition 1, we associate a state value function
$\overline{\mathcal{V}}_{T}^{\pi}$ to the probability of visiting a risk
state. Now, we define a new weighted value function
$\mathcal{V}_{\xi,T}^{\pi}$, which incorporates both the reward and the risk,
as follows

$\displaystyle\mathcal{V}_{\xi,T}^{\pi}(s)=\xi\mathcal{V}_{T}^{\pi}(s)-\overline{\mathcal{V}}_{T}^{\pi}(s),$
(15)

where $\xi>0$ is the weighting parameter, determined by the risk level the
controller is willing to tolerate. The function $\mathcal{V}^{\pi}_{\xi,T}$
can be seen as a standard value function associated to the reward $\xi
r-\overline{r}$. The case $\xi=0$ corresponds to a minimum-risk policy whereas
the case $\xi\rightarrow\infty$ corresponds to a maximum-value policy.

Let $\Pi$ denote the set of deterministic policies, and define

$\displaystyle\mathcal{V}_{T}^{*}(s)=\underset{\pi\in\Pi}{\mbox{ max
}}\mathcal{V}_{T}^{\pi}(s),\quad\mathcal{\overline{V}}_{T}^{*}(s)=\underset{\pi\in\Pi}{\mbox{
min }}\mathcal{\overline{V}}_{T}^{\pi}(s),$
$\displaystyle\mathcal{V}^{*}_{\xi,T}(s)=\underset{\pi\in\Pi}{\mbox{ max
}}\mathcal{V}^{\pi}_{\xi,T}(s).$

Besides, we define $u^{\pi}_{t}$, $\overline{u}_{t}^{\pi}$, and
$u^{\pi}_{\xi,t}$ for $0\leqslant t\leqslant T$ respectively by

$\displaystyle u^{\pi}_{t}(s)$ $\displaystyle=$
$\displaystyle\mathbb{E}^{\pi}\left[\sum_{i=t}^{T}{r(s_{i},\pi(s_{i}))}|\,s_{t}=s\right],$
(16) $\displaystyle\overline{u}^{\pi}_{t}(s)$ $\displaystyle=$
$\displaystyle\mathbb{E}^{\pi}\left[\sum_{i=t}^{T}{\overline{r}(s_{i},\pi(s_{i}),s_{i+1})}|\,s_{t}=s\right],$
(17) $\displaystyle u^{\pi}_{\xi,t}(s)$ $\displaystyle=$ $\displaystyle\xi
u^{\pi}_{t}(s)-\overline{u}^{\pi}_{t}(s).$ (18)

Note that $\mathcal{V}^{\pi}_{T}$ incorporates the total expected reward over
the entire planning horizon whereas $u_{t}$ incorporates the rewards from
decision epoch $t$ to the end of the planning horizon only. Besides,
$\overline{u}_{t}(s)$ is the probability of visiting a risk state given that
at time $t$ the system is in state $s\in\\{\mathcal{S}/\Phi\\}$, and is thus a
measure of the risk.

The optimality equations are given by (the proof is similar to that in [18],
chap. 4 and skipped here for brevity)

$\displaystyle u_{t}^{*}(s)$ $\displaystyle=$
$\displaystyle\underset{\ell\in\mathcal{L}}{\mbox{max
}}\Big{\\{}r(s_{t},\ell)+\sum_{j\in\mathcal{S}}{p(j|s_{t},\ell)u_{t+1}^{*}(j)}\Big{\\}}\;\;\;\;\;$
(19) $\displaystyle{\overline{u}}_{t}^{*}(s)$ $\displaystyle=$
$\displaystyle\underset{\ell\in\mathcal{L}}{\mbox{min
}}\Big{\\{}\sum_{j\in\mathcal{S}}{p(j|s_{t},\ell)\Big{(}\overline{r}(s_{t},\ell,j)+\overline{u}_{t+1}^{*}(j)\Big{)}}\Big{\\}}\;\;\;\;\;\;\;\;$
(20) $\displaystyle u_{\xi,t}^{*}(s)$ $\displaystyle=$
$\displaystyle\underset{\ell\in\mathcal{L}}{\mbox{max
}}\Big{\\{}\sum_{j\in\mathcal{S}}{p(j|s_{t},\ell)\Big{(}\xi
r(s_{t},\ell)-\overline{r}(s_{t},\ell,j)}$ (21)
$\displaystyle+u_{\xi,t+1}^{*}(j)\Big{)}\Big{\\}},$

for $t=0,..,T-1$. For the boundary conditions, that is at time slot $T$,
$u^{*}_{T}(s)$, $\overline{u}^{*}_{T}(s)$, and $u_{\xi,T}^{*}(s)$ are set to
zero for each $s\in\mathcal{S}$.

In a non-risk state, the reward $r$ is given in (5) and the risk signal is
equal to zero whereas in a risk state the reward $r$ is set to zero and the
risk signal $\overline{r}$ is set to one.

##  IV Algorithm Design

In this section, we present two algorithms: (i) finite-horizon value iteration
algorithm which assumes that all the model parameters are known to the
controller, namely the channel statistics (model-based algorithm), and (ii)
reinforcement learning algorithm which does not require the controller
knowledge of channel statistics (model-free algorithm).

###  IV-A Value Iteration Algorithm

In order to find a policy that maximizes the weighted value function defined
in (15), we use the value iteration algorithm [18]. In this algorithm, we
proceed backwards: we start by determining the optimal action at time slot $T$
for each state, and successively consider the previous stages, until reaching
time slot $0$ (see Algorithm 1).

Algorithm 1 Finite-Horizon Value Iteration Algorithm

1:Initialization: for each s

2:$u_{T}^{*}(s)\leftarrow 0$, $\overline{u}^{*}_{T}(s)\leftarrow 0$,
$u_{\xi,T}^{*}(s)\leftarrow 0$

3:Endfor

4:$t\leftarrow T-1$

5:while $t\geqslant 0$

6:For each s

7:update $u_{t}^{*}(s)$, $\overline{u}_{t}^{*}(s)$, and ${u}_{\xi,t}^{*}(s)$
according to (19), (20), and (21), respectively

8:EndFor

9:$t\leftarrow t-1$

10:EndWhile

###  IV-B Risk-Sensitive Reinforcement Learning Algorithm

Environment (Wireless Channel) Learning Controllernewstaterisksignal
$\overline{r}$actionInteractionObservation$r$ Figure 2: Reinforcement Learning
Model.

During the learning phase, the controller gets estimates of the value of each
state-action pair. It updates its estimates through the interaction with the
environment where at each iteration it performs an action and then observes
the reward, risk signal $\overline{r}$, and the next state (see Fig. 2).

The learning controller chooses an action at each learning step following the
$\varepsilon$-greedy policy, that is, it selects an action that maximizes its
current estimate with probability $1-\varepsilon$, or a random action with
probability $\varepsilon$. The parameter $\varepsilon$ captures the
exploration-and-exploitation trade-off: when $\varepsilon\rightarrow 0$, the
controller tends to choose an action that maximizes its current state’s
estimated value; whereas when $\varepsilon\rightarrow 1$, the controller tends
to choose randomly an action and to favor the exploration for optimality.

The state-action value function is given by [21, 19]

$\displaystyle
Q^{\pi}(s_{t},\ell)=r(s_{t},\ell)+\sum_{j\in\mathcal{S}}{p(j|s_{t},\ell)\,{u}^{\pi}_{t+1}(j)},$

where the first term denotes the immediate reward, that is the number of
successfully transmitted packets over all the users, when the action $l$ is
performed in state $s_{t}$; and the second term denotes the expected reward
when the policy $\pi$ is followed in the subsequent decision stages. Similarly
to the state-action value function associated to the reward, we define the
state-action value function associated to the risk $\overline{Q}^{\pi}$ as

$\displaystyle\overline{Q}^{\pi}(s_{t},\ell)=\sum_{j\in\mathcal{S}}{p(j|s_{t},\ell)\left(\overline{r}(s_{t},\ell,j)+\overline{u}^{\pi}_{t+1}(j)\right)}.$

Note that the introduction of the signal risk $\overline{r}$ enabled us to
define a state-action value function, $\overline{Q}$ to the risk.

Besides, the state-action value function associated to the weighted
formulation, $Q_{\xi}^{\pi}$, is given by

$\displaystyle Q_{\xi}^{\pi}(s_{t},\ell)=\xi
Q^{\pi}(s_{t},\ell)-\overline{Q}^{\pi}(s_{t},\ell).$

Finally, the Q-function updates at the learning step $n$ (which should not be
confused with the decision epoch $t$) are given by [21]

$\displaystyle Q^{(n+1)}(s_{t},\ell)$ $\displaystyle\leftarrow$
$\displaystyle\big{[}1-\alpha_{n}(s_{t},\ell)\big{]}Q^{(n)}(s_{t},\ell)+$ (22)
$\displaystyle\alpha_{n}(s_{t},\ell)\big{[}r+\underset{\ell\in\mathcal{L}}{\mbox{
max }}\\{Q^{(n)}(s_{t+1},\ell)\\}\big{]},\;\;\;\;\;\;$
$\displaystyle\overline{Q}^{(n+1)}(s_{t},\ell)$ $\displaystyle\leftarrow$
$\displaystyle\big{[}1-\alpha_{n}(s_{t},\ell)\big{]}\overline{Q}^{(n)}(s_{t},\ell)+$
(23)
$\displaystyle\alpha_{n}(s_{t},\ell)\big{[}\overline{r}+\underset{\ell\in\mathcal{L}}{\mbox{
min }}\\{\overline{Q}^{(n)}(s_{t+1},\ell)\\}\big{]},\;\;\;\;\;\;$

and,

$\displaystyle Q_{\xi}^{(n+1)}(s_{t},\ell)$ $\displaystyle\leftarrow$
$\displaystyle\big{[}1-\alpha_{n}(s_{t},\ell)\big{]}Q_{\xi}^{(n)}(s_{t},\ell)+$
$\displaystyle\alpha_{n}(s_{t},\ell)\big{[}\xi
r-\overline{r}+\underset{\ell\in\mathcal{L}}{\mbox{ max
}}\\{Q_{\xi}^{(n)}(s_{t+1},\ell)\\}\big{]},$

where $\alpha_{n}(s_{t},\ell)$ denotes the learning rate parameter at step $n$
when the state $s_{t}$ and action $\ell$ are visited.

The learning algorithm converges to the optimal state-action value function
when each state-action pair is performed infinitely often and when the
learning rate parameter satisfies for each $(s_{t},\ell)$ pair (the proof is
given in [21, 7] and skipped here for brevity),

$\displaystyle\sum_{n=1}^{\infty}{\alpha_{n}}(s_{t},\ell)=\infty,\quad\mbox{
and }\quad\sum_{n=1}^{\infty}{\alpha_{n}^{2}(s_{t},\ell)}<\infty.$

In this case, the Q-functions are related to the value functions as follows

$\displaystyle\underset{\ell\in\mathcal{L}}{\mbox{max}}\left\\{Q(s_{t},\ell)\right\\}=u_{t}^{*}(s_{t}),\quad\underset{\ell\in\mathcal{L}}{\mbox{min}}\left\\{\overline{Q}(s_{t},\ell)\right\\}=\overline{u}_{t}^{*}(s_{t}),$
$\displaystyle\underset{\ell\in\mathcal{L}}{\mbox{max}}\left\\{Q_{\xi}(s_{t},\ell)\right\\}=u_{\xi,t}^{*}(s_{t}).$

When a risk state is reached during the learning phase, the system is
restarted according to the uniform distribution to a non-risk state. In
addition, when $t\geqslant T$, we consider that an artificial absorbing state
is reached and we reinitialize $t$ (see Algorithm 2).

Algorithm 2 Q-learning Algorithm

1:Initialization $t\leftarrow 0$, $s_{0}\leftarrow s$, $n\leftarrow 1$,

2: for each $\ell\in\mathcal{L}$

3:$Q(s_{0},\ell)\leftarrow 0$, $\overline{Q}(s_{0},\ell)\leftarrow 0$,
$Q_{\xi}(s_{0},\ell)\leftarrow 0$

4:End for

5:Repeat

6:observe current state $s_{t}$

7:select and perform action $\ell$ in state $s_{t}$

8:observe the new state $s_{t+1}$, reward $r$ and the risk $\overline{r}$

9:update the Q-functions  $Q(s_{t},l)$, $\overline{Q}(s_{t},l)$,
$Q_{\xi}(s_{t},\ell)$ according to (22), (23), (LABEL:q3) respectively

10:$t\leftarrow t+1$

11:$n\leftarrow n+1$

12:update $\alpha_{n}$

13:if $t=T$, then $t\leftarrow 0$  _artificial absorbing state reached_

14:if $s_{t}\in\Phi$, then $s_{t}\sim\mbox{Unif}\\{\mathcal{S}/\Phi\\}$
_absorbing state reached_

15:until convergence

##  V Performance Evaluation

In this section, we present the numerical results obtained with the value
iteration and the learning algorithms in a variety of scenarios. We consider
the setting of two users along with a number of channels $L=5$. For the
arrival traffic, we consider the following truncated Poisson distribution

$\mbox{Prob}(a=m)=\left\\{\begin{array}[]{cc}\frac{\lambda^{m}/m!}{\sum_{i=0}^{A_{max}}{\lambda^{i}/i!}}&\mbox{
if }m\leqslant A_{max}\\\ \mbox{ zero }&\mbox{otherwise,}\end{array}\right.$
(25)

where $\lambda=3$ and $A_{max}=6$. The mean of the Bernoulli channel $\mu$ and
the value of the parameter $\rho_{max}$ throughout this section are fixed to
$0.6$ and $0.55$ respectively.

###  V-A Minimum-risk vs maximum-value policy

First, we compare the performance of the minimum-risk policy (obtained when
$\xi=0$), maximum-value policy (obtained when $\xi\rightarrow\infty$),
weighted policy (when $\xi>0$), and the fixed policy which consists is
assigning the same number of channels for each user at each time slot
($\ell_{1}=2$ and $\ell_{2}=3$).

Figure 3: Performance of the minimum-risk policy $\overline{\pi}^{*}$, the
maximum-value policy $\pi^{*}$, the weighted-policy $\pi_{\xi}^{*}$ with
$\xi=0.1$, and the fixed policy $\pi_{f}$. On the top, $u_{t}(s)$, on the
bottom, $\overline{u}_{t}(s)$ where $s=0.3\times 0$ and $T=9$.

We depict in Fig. 3-top the reward $u_{t}(s)$ given in (19) as a function of
time when $s=0.3\times 0$ and different policies are followed. We observe that
the maximum-value policy clearly outperforms the fixed and the minimum-risk
policy. In Fig. 3-bottom showing $\overline{u}_{t}(s)$ given in (20), we
observe that the probability of visiting a risk-state when the fixed policy is
followed is much higher than that obtained when the minimum-risk policy
$\overline{\pi}^{*}$ is performed. For example, at the time step $t=5$,
$\overline{u}_{t}(s)$ is equal to $0.42$ when the policy $\pi_{f}$ is
performed whereas this value reduces to $0.02$ when the policy
$\overline{\pi}^{*}$ is followed. In fact, the fixed policy does not take
account of the experienced QoS of the users, and therefore, it is the policy
which results in the highest risk-state visitation probability. Besides, this
probability decreases over time for all the policies. In fact, as time goes
on, the probability of entering a risk-state over the remaining time steps
decreases.

The reward $u_{t}(s)$ increases for the lower values of $t$ until reaching a
maximum value and then it decreases, for all the policies. In fact, for the
lower values of $t$, the probability of visiting a risk-state is high, and
this affects the expected value of the reward (recall that in the risk state,
the reward is equal to zero). As time goes on, this probability decreases, and
thus the expected reward increases. However, at the further time steps, the
number of remaining decision stages is low and hence the expected reward
(total number of successfully transmitted packets over the remaining time
slots) decreases.

###  V-B Learning

In the learning algorithm, we simulate the wireless channel with a Bernoulli
random variable with a number of trials equal to the number of channels
associated to each packet for each user. For the learning rate parameter
$\alpha_{n}$, we considered the following expression [11]:

$\displaystyle\alpha_{n}=\frac{1}{(1+n(s_{t},\ell))^{\gamma}},$ (26)

where $n(s_{t},\ell)$ denotes the number of times the state-action pair
$(s_{t},\ell)$ was visited until iteration $n$, and $\gamma$ is a positive
parameter $\in[0.5,1]$[11].

Figure 4: Optimal policy $\ell_{1}^{*}$ as a function of time steps and
$\rho_{1}$ with $\rho_{2}=0$ and $T=5$.

We depict in Fig. 4 the optimal (minimum-risk) policy (number of channels to
assign to user $1$ , $\ell_{1}\in[0,..,5]$) computed by the learning
algorithm, as a function of time steps (decision epochs) and $\rho_{1}$, when
$\rho_{2}$ is fixed to $0$. The figure shows a monotony property: the number
of channels to assign to user $1$ increases with time and with $\rho_{1}$. In
fact, as the QoS of user $1$ degrades ($\rho_{1}$ increases), more channels
are assigned to it to compensate for this degradation; and as time goes on,
this policy is more sensitive to this degradation as more channels are
assigned for the same values of $\rho_{1}$, but at further time steps.

##  VI Conclusion

In this work, we studied the problem of dynamic channel allocation for URLLC
traffic in a multi-user multi-channel wireless network within a novel
framework. Due to the stochastic nature of the problem related to time-
varying, fading channels and random arrival traffic, we considered a finite-
horizon MDP framework. We determined explicitly the probability of visiting a
risk state and we wrote it as a cumulative return (risk signal). We then
introduced a weighted global value function which incorporates two criteria:
reward and risk. By virtue of the value iteration algorithm, we determined the
optimal policy. Furthermore, we used a Q-learning algorithm to enable the
controller to learn the optimal policy in the absence of channel statistics.
We illustrated the performance of our algorithms with numerical studies, and
we showed that by adapting the number of parallel transmissions in a smart
way, the performance of the system can be substantially enhanced.

In the future work, we would like to take account of spatial diversity in the
dynamic allocation scheme where both the BS and the user terminals can be
equipped with multiple antennas to enhance the system performance.

## References

  * [1] R. Aggarwal, M. Assaad, C. E. Koksal, and P. Schniter. OFDMA downlink resource allocation via ARQ feedback. In Conference Record of the Forty-Third Asilomar Conference on Signals, Systems and Computers, pages 1493–1497, Pacific Grove, CA, Nov 2009\.
  * [2] A. Ahmad and M. Assaad.  Joint resource optimization and relay selection in cooperative cellular networks with imperfect channel knowledge.  In Proc. of IEEE 11th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC).
  * [3] A. Ahmad and M. Assaad.  Optimal resource allocation framework for downlink OFDMA system with channel estimation error.  In Proc. of IEEE Wireless Communication and Networking Conference, pages 1–5, Sydney, April 2010.
  * [4] M. R. Akdeniz, Y. Liu, M. K. Samimi, S. Sun, S. Rangan, T. S. Rappaport, and E. Erkip.  Millimeter wave channel modeling and cellular capacity evaluation.  IEEE Journal on Selected Areas in Communications, 32(6):1164–1179, June 2014.
  * [5] M. Assaad, A. Ahmad, and H. Tembine.  Risk sensitive resource control approach for delay limited traffic in wireless networks.  In Proc. of IEEE Global Telecommunications Conference (GLOBECOM), pages 1–5, Kathmandu, Dec 2011.
  * [6] Mehdi Bennis, Mérouane Debbah, and H. Vincent Poor.  Ultra-reliable and low-latency wireless communication: tail, risk and scale.  Technical Report, ArXiv, may 2018.
  * [7] D. P. Bertsekas and J. Tsitsiklis.  Neuro-dynamics Programming.  MIT Press, Cambridge, Massachusetts, 1996.
  * [8] S. Cayci and A. Eryilmaz.  Learning for serving deadline-constrained traffic in multi-channel wireless networks.  In Proc. of the 15th International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt), pages 1–8, May 2017.
  * [9] A. Destounis, G. S. Paschos, J. Arnau, and M. Kountouris.  Scheduling URLLC users with reliable latency guarantees.  In Proc. of the 16th International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt), pages 1–8, May 2018.
  * [10] A. Dua and N. Bambos.  Downlink wireless packet scheduling with deadlines.  IEEE Transactions on Mobile Computing, 6(12):1410–1425, Dec 2007\.
  * [11] Eyal Even-Dar and Yishay Mansour.  Learning rates for Q-learning.  J. Mach. Learn. Res., 5:1–25, 2004.
  * [12] Peter Geibel and Fritz Wysotzki.  Risk-sensitive reinforcement learning applied to control under constraints.  J. Artif. Int. Res., 24(1):81–108, jul 2005.
  * [13] A. Gosavi.  Finite horizon Markov control with one-step variance penalties.  In Proc. of the 48th Annual Allerton Conference on Communication, Control, and Computing, pages 1355–1359, Sept 2010.
  * [14] N. U. Hassan and M. Assaad.  Dynamic resource allocation in multi-service OFDMA systems with dynamic queue control.  IEEE Transactions on Communications, 59(6):1664–1674, June 2011\.
  * [15] H. Ji, S. Park, J. Yeo, Y. Kim, J. Lee, and B. Shim.  Ultra-reliable and low-latency communications in 5G downlink: Physical layer aspects.  IEEE Wireless Communications, 25(3):124–130, June 2018.
  * [16] A. Kumar, V. Kavitha, and N. Hemachandra.  Finite horizon risk sensitive MDP and linear programming.  In Proc. of the 54th IEEE Conference on Decision and Control (CDC), pages 7826–7831, Dec 2015.
  * [17] N. Nomikos, N. Pappas, T. Charalambous, and Y. Pignolet.  Deadline-constrained bursty traffic in random access wireless networks.  In SPAWC, May 2018.
  * [18] Martin L. Puterman.  Markov Decision Processes: Discrete Stochastic Dynamic Programming.  John Wiley & Sons, 2005.
  * [19] Richard S. Sutton and Andrew G. Barto.  Reinforcement Learning: An Introduction.  MIT Press, Cambridge, Massachusetts, 1998.
  * [20] T. K. Vu, M. Bennis, M. Debbah, M. Latva-aho, and C. S. Hong.  Ultra-reliable communication in 5G mmWave networks: A risk-sensitive approach.  IEEE Communications Letters, 22(4):708–711, April 2018.
  * [21] Christopher J. C. H. Watkins and Peter Dayan.  Q-learning.  Machine Learning, 8(3):279–292, May 1992.