# Age of Loop for Wireless Networked Control Systems Optimization

Pedro M. de Sant Ana1, Nikolaj Marchenko1, Petar Popovski2 and Beatriz Soret2
1Corporate Research and Advanced Engineering, Robert Bosch GmbH, Renningen,
Germany
Email: {Pedro.MaiadeSantAna<EMAIL_ADDRESS>2Department of
Electronic Systems, Aalborg University, Aalborg, Denmark
Email: {petarp<EMAIL_ADDRESS>
###### Abstract

Joint design of control and communication in Wireless Networked Control
Systems (WNCS) is a promising approach for future wireless industrial
applications. In this context, Age of Information (AoI) has been increasingly
utilized as a metric that is more representative than latency in the context
of systems with a sense-compute-actuate cycle. Nevertheless, AoI is commonly
defined for a single communication direction, Downlink or Uplink, which does
not capture the closed-loop dynamics. In this paper, we extend the concept of
AoI by defining a new metric, _Age of Loop (AoL)_ , relevant for WNCS closed-
loop systems. The AoL is defined as the time elapsed since the piece of
information causing the latest action or state (depending on the selected time
origin) was generated. We then use the proposed metric to learn the WNCS
latency and freshness bounds and we apply such learning methodology to
minimize the long term WNCS cost with the least amount of bandwidth. We show
that, using the AoL, we can learn the control system requirement and use this
information to optimize network resources.

##  I Introduction

Networked control systems (NCS) are an essential part of many industrial
domains such as factory automation, logistics, or transportation. Wireless
NCSs (WNCS) enable mobile control applications were wiring is not possible or
high flexibility is required. However, due to the nature of the wireless
medium, reliability of WNCS remains an open challenge, in particular for low-
latency applications.

Decisions taken at a system level can have a direct impact on the
communication medium and vice-versa, formulated by Witsenhausen as
counterexample for distributed control problem [1]. In recent work on WNCS,
authors have been increasingly exploring the inter-relation between the
control and communication schemes. In [2] and [3], for example, the authors
demonstrate how the latency and reliability trade-off directly impacts the
system level stability, proposing a co-design of both control and
communication entities. Specifically in [2], authors have demonstrated a
counter-intuitive conclusion that the plant can still be stabilized with an
arbitrarily large delay under certain channel conditions. Another interesting
finding was presented in [4], where authors elucidate an example how one can
optimize long-term system performance by assuming more risks with less
reliable transmissions in exchange for lower latency. The authors of this
publication analyzed the remote AGV control problem in [5].

These findings outline that the traditional notion of latency of the radio
link, in which we attempt to allocate a small, almost deterministic, part of
the latency budget to the radio link may not be the most efficient way to go
from a system viewpoint. That is why other timing measures have been defined,
also in 3GPP [6] and 5G-ACIA [7], such as the survival time and recovery time.

In other words, the conventional system design for low-latency and high
reliability leads to over-provisioning communication network resources by
decoupling the control and communication entities. Alternatively, the Age of
Information (AoI) [8] has been proven to be a more representative metric for
systems with a sense-compute-actuate cycle like the ones considered in this
paper, where the receiver is interested in fresh knowledge of the remotely
controlled system, rather than the individual packet delay. The AoI defines
the time that has elapsed since the newest update available at the destination
was generated at the source, and it captures not only the communication delays
but also the impact of the packet generation at the controlled process. The
other observation regarding the use of resources is that, from the perspective
of a Base Station (BS) situated at the controller, WNCS have been
traditionally optimized separately for the two transmission directions, uplink
(UL) and downlink (DL). However, WNCS applications and many others are
inherently two-way, such that there is a closed-loop where the UL
communication can affect the DL and vice-versa, impacting either on system
performance or in the use of network resources. In this context, the goal of
our paper is to learn the optimal configuration of the communication network
that ensures stability of the control system. For this purpose, we consider
the two-way nature of the problem and the control-communications interplay.

This paper contains two main contributions:

  1. 1.

We propose a new metric, the Age of Loop (AoL), which extends the current AoI
definition to take into consideration both UL and DL of the control loop in
WNCS, and thus can provide a more precise system state estimation.

  2. 2.

We demonstrate how to apply the AoL metric for joint WNCS optimizations with
the application example of a remotely controlled inverted pendulum system [9].
With a Reinforcement Learning (RL) approach, we find the bandwidth allocation
policy based on the AoL state, which significantly outperforms policies based
on fixed latency requirements.

The rest of this paper is organized as following: in the next two sections, we
introduce the system and WNCS model, respectively. In section III, we define
the AoL and show how we can evaluate the control system performance using the
proposed metric. Finally, in section V, we formulate the bandwidth allocation
problem and propose a solution, where the results are analyzed in section VI.

##  II System Model

We consider the classical inverted pendulum system model, a widely used
benchmark problem in both control and RL domain. As illustrated in Figure 1, a
pole is attached by a joint to a cart, which can be moving along a
frictionless track. The pendulum starts upright at a random initial angle
$\theta_{0}\in(\theta_{0,min},\theta_{0,max})$, and the goal is to prevent it
from falling over by applying a force to the cart. While conceptually simple,
the system dynamics are highly unstable and continuously requires fast control
cycles to keep stability. When, in turn, being controlled over a wireless
channel, the problem becomes an illustrative model of strict timing
requirement.

Figure 1: Inverted pendulum system model.

###  II-A Control System Model

The system dynamics can be represented by the differential equations [10]:

$\displaystyle\ddot{\theta}=\frac{g\cdot\text{sin}(\theta)+\text{cos}(\theta)\left(\frac{-F-m_{p}l\dot{\theta}^{2}\text{sin}(\theta)}{m_{c}+m_{p}}\right)}{l(\frac{4}{3}-\frac{m_{p}\text{cos}^{2}(\theta)}{m_{c}+m_{p}})}$
(1)
$\displaystyle\ddot{x}=\frac{F+m_{p}l(\dot{\theta}^{2}\text{sin}(\theta)-\ddot{\theta}\text{cos}(\theta))}{m_{c}+m_{p}}$

where $x$ and $\theta$ are, respectively, the cart position coordinates and
the pole angle from vertical reference. The mass of the cart is $m_{c}$, and
the mass of the pendulum is $m_{p}$, while $l$ is the length of the pendulum,
and $F$ is the force applied to the cart under gravity $g$. We use the
Newton’s notation ($\dot{\Box},\ddot{\Box}$) to represent derivatives w.r.t
time.

By defining a state space vector $X=[x,\dot{x},\theta,\dot{\theta}]$, we can
design a standard optimal controller in two steps. First, computing the
Jacobian of (1) around the operating point $X=[0,0,0,0]$ to linearize the
plant, so that the system dynamic takes the linear time invariant form:

$\begin{cases}\dot{X}=AX+Bu+w\\\ u=-KX\end{cases}$ (2)

where $u$ is the linear state feedback control policy of gain $K$, $w$ is a
process disturbance modeled as a zero-mean and one-standard deviation Gaussian
white noise, $A$ and $B$ are the system transition matrix, respectively given
by:

$A=\begin{bmatrix}0&1&0&0\\\ 0&0&\frac{-12m_{p}g}{13m_{c}+m_{p}}&0\\\
0&0&0&1\\\ 0&0&\frac{12(m_{p}g+m_{c}g)}{l(13m_{c}+m_{p})}&0\\\
\end{bmatrix},\,B=\begin{bmatrix}0\\\ \frac{13}{13m_{c}+m_{p}}\\\ 0\\\
\frac{-12}{l(13m_{c}+m_{p})}\\\ \end{bmatrix}$ (3)

The second step consists of finding the optimal control policy, $u^{*}$,
subject to (2) that minimizes the cost function,

$J(u)=\int_{0}^{\infty}\left(X^{T}QX+u^{T}Ru\right)dt,$ (4)

where $R$ and $Q$ are arbitrary positive defined matrices in which we can
assign weights to state space variables and control signal. In control theory
this kind of problem formulation is known as Linear-Quadratic-Regulator (LQR)
[11].

The optimal control policy then can be defined by solving the Algebraic
Riccati Equation [11] as:

$\displaystyle A^{T}P+PA-PBR^{-1}B^{T}P+Q=0$ (5) $\displaystyle
K^{*}=R^{-1}B^{T}P$ $\displaystyle u^{*}=K^{*}X$

For $(A,B)$ controllable, the infinite horizon LQR with ${Q,R>0}$ gives a
convergent closed-loop system [11], where the stability can be easily
guaranteed.

###  II-B Networked Control Model

As defined in [5], we adopt a similar NCS model to define the system behavior
over the wireless medium operating in Frequency Division Duplexing (FDD) mode
with separated frequency bands for the uplink (UL) and downlink (DL)
directions, which makes the medium access for UL and DL independent from each
other in time domain. Figure 2 illustrates the proposed model, showing the
details of the interaction between the communication and application control
loop.

Figure 2: WNCS Model.

First, the sensor readings of the application describe the system states,
$X_{i}$, which are stored in memory and communicated to the controller over
the uplink channel. The readings and transmissions of sensor values are done
strictly periodically with the cycle time $\Delta T_{\text{in}}$, as it is
commonly done across various control systems [12].

At the controller, the received sensor values are also stored into the memory.
The control application gets the recent values, and produces a control signal
$u_{i}$ according to (5). Immediately after producing a control command, the
controller sends it over a downlink channel to the controlled system. After
finishing the current transmission, the controller keeps waiting for the next
state update from the controlled device, and starts the procedure once again.

At the controlled system side, the received command $u_{i}$ is stored in the
memory. The output application for actuators control (e.g., motor drives) is
called periodically with the time interval $\Delta T_{\text{out}}$, calls the
most recently stored command values from the memory and applies them to the
application drives, producing the system dynamics of (1).

###  II-C Wireless channel model

Both the DL and UL transmissions can suffer latency while delivering the
information, which, in this model, depends on two main factors: the current
channel quality and the total bandwidth allocated for the transmission. To
evaluate this behavior, we consider the 3GPP 4-bit CQI Table 7.2.3-1 [13],
where we can estimate the amount of time to deliver the data considering both
the channel quality indicator (CQI) and the total bandwidth allocated at the
transmission. The following two assumptions have been adopted: a) the UL
finishes its transmission within $\Delta T_{\text{in}}$, and b) the DL only
starts a new transmission after finishing the current one. The details of the
bandwidth allocation problem are discussed later in Section V.

###  II-D System Model Discussion

It is important to emphasize that (4) is guaranteed to be bounded according to
the Riccati-equation [11]. However, the combination of two main factors might
affect the system LQR performance. The first is the uplink effect, which
represents the level of knowledge the controller has about the plant, meaning
that, if $\Delta T_{\text{in}}$ is too high or the uplink takes overly long to
deliver sensor data, the controller will compute the control signal using old
state feedback, causing the control command to be ineffective or even harmful
for the plant. The second is the downlink effect, or simply the overall delay
to deliver the control signal. This is important because if the plant applies
outdated control commands for too long, the stability of the controlled system
might also be compromised.

Each of these factors might affect the plant in different ways and cannot be
independently decoupled, which means that a delay in the UL will impact the DL
transmission, provoking cumulative effects at the plant and at the network
resources.

##  III Age of Information and Age of Loop

Age of information (AoI) provides a measure for quantifying the freshness of
the knowledge we have about the status of a remote system. It represents the
time duration between the generation time of the freshest received data and
the current time. We can refer to its formal definition as in [8, 14], where,
at time $t$, if the newest data (i.e., with the largest generation time)
received at the destination was generated at time $U(t)$, the AoI $\Delta(t)$
is defined as $\Delta(t)=t-U(t)$.

The formal AoI definition, however, is inherited to a single communication
link. Papers which so far explored WNCS related problems using AoI are limited
to specific analysis over only the UL [15, 16, 3] or DL [4, 2] transmissions.
However, wireless networked control systems, as the one considered in this
paper, rely intrinsically on both DL and UL with a closed-loop, where the UL
communication can affect the DL and vice-versa, impacting system performance
and the use of network resources. A simple intuitive example that can
illustrate this idea is that a high UL AoI implicates less knowledge that the
controller has about the plant, which demands more urgency to deliver the
control signal and, as a consequence, more network resources usage by the DL
link. To address this implications, we propose a new metric to evaluate the
overall age of a WNCS closed-loop, which we refer to as Age-of-Loop (AoL).

Specifically, we can first verify that the state values $X_{i}$ are
periodically generated and transmitted at time intervals of
${t_{i}={\\{i\cdot\Delta T_{\text{in}}\\},\forall i\in\mathbb{N}^{+}}}$, ,
where we can define ${\\{X_{i},t_{i}\\}}$ the sequence of generated state
values and its respective time step. The control signal, in turn, is
asyncrhonous and must finish the current DL transmission to start a new one
upon reception of a new status update. We can define a sequence
${\\{u_{j},\hat{t}_{j}\\}\,\forall j\in\mathbb{N}^{+}}$ with aperiodically
generated control commands $u_{j}$ at time step $\hat{t}_{j}$. If
${\\{X_{i},t_{i}\\}}$ is the freshest state feedback that spawned a new
control signal, we can extend the DL transmission definition to include state
time information, i.e., ${\text{DL}:\\{u_{j},\hat{t}_{j},t_{i}\\}}$.
Reciprocally, every state feedback also occurs under the input of the freshest
control command, so that we can also extend the UL transmission definition to
include control time information, i.e.,
${\text{UL}:\\{X_{i},t_{i},\hat{t}_{j}\\}}$.

We consider two plausible definitions of the AoL depending on the selected
time origin: the DL-AoI is DL-initiated, meaning that the time origin is a new
control command; the UL-AoI is UL-initiated, i.e., the time origin is a new
status update in the UL. The DL AoL metric captures the time elapsed since the
control command that led to the latest received update in the controller was
generated. Analogously, the UL AoL metric refers to the time elapsed since the
status update that caused the latest applied control command was generated at
the sensor. Mathematically, if the origin is the DL, the current AoL is the
difference between the current time $t$ and the time when the freshest
received control command was generated:

$\text{DL AoL}(t)=t-\hat{t}_{j}$ (6)

Likewise, if the time origin is the UL, the AoL is calculated as the
difference between the current time and the time when the freshest received
state was spawned:

$\text{UL AoL}(t)=t-t_{i}$ (7)

Essentially, the main idea of AoL is to establish a metric that encompasses
the behavior of two separated and locally measured entities (DL and UL) into a
single instance, in which we can observe from different perspectives. It is
important to note that, in the case of two independent AoI links, we need an
instantaneous feedback to the source to know the instantaneous age at the
destination, thus making complex and potentially imprecise the union of two
directions; AoL fixes this. In practice, it also offers the possibility to
design solutions that enclose the whole closed-loop behavior by checking the
loop age from either an UL or DL perspective. For example, we can potentially
design a power allocation policy for the UL by observing the current UL AoL
status. Likewise, we are able to define a modulation coding scheme algorithm
for the DL transmissions by observing the DL AoL. It will be proven that they
are both valid to optimize the stability of the WNCS. To illustrate the
proposed concept, Figure 3 shows a representative time diagram of the AoL
behavior.

Figure 3: Time Diagram of AoL Behavior.

##  IV AoL Evaluation

We aim to estimate the performance of the control system measured according to
(4) using the current AoL status calculated at the controller (DL AoL). More
formally, we can use the value function definition [17] to estimate the
expected LQR cost, i.e.,

$V(\Delta_{\text{AoL}}(t))=\int_{t}^{\infty}(X^{T}QX+u^{T}Ru)dt$ (8)

Since the control policy is unchangeable over time and the plant operation is
sampled at cycles of $\Delta T_{\text{out}}$, (8) becomes the recursion
problem:

$\displaystyle\begin{split}V(\Delta_{\text{AoL}}(t))=\int\displaylimits_{t}^{t+\Delta
T_{\text{out}}}(X^{T}QX&+u^{T}Ru)dt\\\ &+V(\Delta_{\text{AoL}}(t+\Delta
T_{\text{out}})),\end{split}$ (9)

where we can solve iteratively using a temporal difference (TD) learning
algorithm [17] with actual state transitions, such that:

$\displaystyle V(\Delta_{\text{AoL}}(t))\leftarrow
V(\Delta_{\text{AoL}}(t))+\alpha\Bigg{[}\int\displaylimits_{t}^{t+\Delta
T_{\text{out}}}(X^{T}QX+u^{T}Ru)dt$ $\displaystyle+\,\gamma
V(\Delta_{\text{AoL}}(t+\Delta
T_{\text{out}}))-V(\Delta_{\text{AoL}}(t))\Bigg{]}$ (10)

where $\alpha$ and $\gamma$ are, respectively, the learning rate and the
discount factor of future values. We can emphasize that (10) converges
asymptotically to the correct predictions with probability 1 if the step-size
$\alpha$ decreases according to the usual stochastic approximation conditions
[17].

###  IV-A Numerical Evaluation

Considering the following inverted pendulum configuration: ${m_{c}=1.0}$ kg,
${m_{p}=0.1}$ kg, ${l=0.5}$ m, ${g=9.8}\,m/s^{2}$ and ${\Delta
T_{\text{out}}=1}$ ms, we evaluated the expected LQR behavior for different
AoL states using (10).

Figure 4 illustrates the obtained result, where we can emphasize three
extensive conclusions. First, low AoL values, as expected, provide the best
system performance, such that the theoretical LQR upper bound is achieved if
the AoL is close to zero. Second, prior to an AoL around $40$ ms, the LQR
slightly decrease. After that point, however, the system starts to
progressively become less tolerable to additional AoL delays. The third and
most relevant conclusion is the fact that between $10$ and $40$ ms, there is
no considerable variation at the system performance, meaning that we can avoid
over-provisioning network resources by learning the system robustness.

Figure 4: Expected LQR Cost vs Age of Loop.

###  IV-B AoI vs AoL

We performed the same numerical evaluation using a state space comprised of a
single DL AoI or a single UL AoI. The goal is to verify the estimation error
of the value function when we change the state space for a single AoI metric
instead of AoL. We analyze the value function estimation by verifying the
Temporal Difference (TD) error, given by:

$\displaystyle\begin{split}\int\displaylimits_{t}^{t+\Delta
T_{\text{out}}}(X^{T}QX&+u^{T}Ru)dt\\\ &+\gamma V(\Delta_{\text{AoL}}(t+\Delta
T_{\text{out}}))-V(\Delta_{\text{AoL}}(t)),\end{split}$ (11)

which indicates, for each state, how far the predicted value function deviates
from the actual value. For example, the learning rule in (10) adjusts state
value in a direction that tends to reduce the TD error.

Lower TD errors indicates better accuracy about the value function estimation
over each state, which is ultimately important for learning better policies,
especially in RL context. Figure 5 illustrates the obtained result along
training episodes. We can verify that, as expected, the estimated values are
more precise when the whole loop age is considered. The DL and UL AoL values
can be merely different, especially because of $\Delta T_{\text{in}}$. After
generation, the DL data might spend time between state transmissions before
finishing the loop, which does not happen in UL case. Thus explaining the
slight different behavior of both values in Figure 5.

Figure 5: Mean value function estimation error over training episodes.

##  V Bandwidth Allocation Problem

As discussed in section IV, the AoL status of a WNCS can provide an estimation
of the system LQR performance, so that we can use the learned value function
to build a policy. In this work, we explore the bandwidth allocation problem
of a remote controller, where two main objectives must be satisfied: minimize
the LQR cost while using the minimum amount of bandwidth.

In more details, we can define
${B=\\{b_{1},b_{2},\dots,b_{i},\dots,b_{N}\\},b_{i+1}>b_{i}}$ a set of
bandwidths in which the controller, for every DL transmission, must decide for
a certain bandwidth allocation $b\in B$ given the current AoL state
information and the current channel quality. So, for
$T={\\{t_{1},t_{2},\dots,t_{i},\dots t_{N}\\}\,t_{i+1}>t_{i}}$ the time
instances where control packets starts transmission and
${C=\\{c_{1},c_{2},\dots,c_{i},\dots,c_{N}\\}}$ the corresponding CQI of each
transmission, the goal is to find an allocation policy
${\pi:\\{\Delta_{\text{AoL}}(t_{i}),c_{i}\rightarrow b_{i}}\\},\,\forall
t_{i}\in T,\,\forall c_{i}\in C,\,\forall b_{i}\in B$ that minimizes the
infinite-horizon LQR cost plus the amount of bandwidth usage over the system
trajectory, i.e.,

$\displaystyle\pi^{*}=\operatorname*{arg\,min}_{\pi}\left(\int\displaylimits_{0}^{\infty}(X^{T}QX+u^{T}Ru)dt+\sum_{i=1}^{N}\frac{b_{i}}{b_{N}}\right)$
$\displaystyle\text{s.t.}\;(\ref{eq:system_dynamics_cartpole}),(\ref{eq:are})$
(12)

###  V-A Solution Proposal

We can decompose the problem in (12) into sub-problems, where between two
consecutive control transmissions ${[t_{i},t_{i+1}),\forall t_{i}\in T}$, we
select at $t_{i}$ a bandwidth $b_{i}\in B$ based on the AoL and CQI state
${\\{\Delta_{\text{AoL}}(t_{i}),c_{i}\\}}$. Receiving, as consequence, a one-
stage decision cost of:

$\int_{t_{i}}^{t_{i+1}}(X^{T}QX+u^{T}Ru)dt+\frac{b_{i}}{b_{N}},$ (13)

which depends only on the present state and the decision taken on that state.
Such decision-making model is a typical Markov Decision Proces (MDP) [17],
where we can optimally solve each sub-problem with actual state transitions
and overlap those solutions to build the overall optimal solution. In this
context, we can define the following MDP configuration:

####  V-A1 State Space

Comprised of 20 AoL values, each representing regions of 5 ms from 0 to 100
ms. In addition, 15 possible CQI values for each AoL, resulting in a total of
300 states.

####  V-A2 Action Space

Represented by the bandwidth set with ten possible values:
$B={\\{100,200,300,\dots,1000\\}}$ kHz.

####  V-A3 Reward

The immediate cost as defined in (13).

####  V-A4 Scenario

We evaluate the proposed MDP considering the NCS model described in section
II-B, assuming the following inverted pendulum configuration: ${m_{c}=1.0}$
kg, ${m_{p}=0.1}$ kg, ${l=0.5}$ m, ${g=9.8}$ m/s2, control packet size of 1024
bits and ${\Delta T_{\text{out}}=1}$ ms. For each run, the CQI is randomly
chosen ${\\{1,2,3,\dots,15\\}}$. The evaluation is also performed under
different sensor feedback ${\Delta T_{\text{in}}=1,5,10,15\text{ and }20}$ ms.

To solve the proposed MDP, we advocate a RL methodology for two main reasons.
First, the MDP transitions probabilities are not easily tractable since the
AoL variation will simultaneously depend on the channel and resource
allocation of both UL and DL links. So, the UL behavior might be analytically
unpredictable from the DL perspective and vice-versa. Second, learning a value
function from the AoL states means that we have a prediction of system
performance given the current AoL condition. In other words, this methodology
offers the possibility for the network to essentially learn the control system
behavior, where the bandwidth allocation policy is just one of multiple
network functions in which it can serve. We could easily extend the learned
values to find optimal polices, for example, in terms of packet length, power
allocation, antenna configuration and so on.

Hence, we solved the proposed MDP by applying a TD RL algorithm, based on a
$\epsilon$-greedy decision making during training, with exponential learning
and exploration rate decay. [17].

##  VI Results

We compare the proposed solution with a bandwidth allocation scheme based on
predefined delay requirements, which is the general solution currently used in
industry. In more details, given an arbitrary requirement of $T_{r}$ ms for
the control packet to be delivered, we can directly calculate the minimum
amount of bandwidth to achieve the necessary requirement using the 3GPP 4-bit
CQI Table 7.2.3-1 [13] and the total packet size. These baseline approaches,
as well as the RL scheme, were evaluated on the scenario described in Section
V.

We analyze the results for three common network requirements,
${T_{r}=1\,\text{ms}}$, ${T_{r}=5\,\text{ms}}$ and ${T_{r}=10}$ ms. In each
case, we analyzed the total bandwidth usage and the total LQR cost, which are
respectively illustrated in Figure 6 and Figure 7.

Figure 6: Total amount of bandwidth usage for each method. Figure 7: Total
amount of LQR cost for each method.

The immediate conclusion we can verify is that the RL scheme was capable to
learn the system delay requirement, such that we can relate the LQR cost in
Figure 7 with the result in Figure 4 to show that it is operating around the
LQR edge performance (around -8) in order to save bandwidth. The second
conclusion is that, as expected, strict latency requirement (${T_{r}=1}$ ms)
demands more bandwidth usage. Compared to ${T_{r}=10}$ ms, however, the RL
scheme could still save $36$% more bandwidth, which is an indication that $10$
ms is still a sub-optimal requirement, but we can learn it from the RL
algorithm.

##  VII Conclusions

In this work, we proposed a new metric to evaluate the age of an WNCS closed-
loop, and we applied this metric, the Age of Loop, to track the LQR
performance of a inverted pendulum control system. Furthermore, we also
propose a bandwidth allocation policy based on the age of loop and channel
quality information, showing that we can learn the system robustness in order
to avoid over-provisioning of network resources on a networked control system.

As future works, we intend to explore a joint DL and UL RL methodology where
both cooperate to optimize system performance and network resources.

## References

  * [1] H. S. Witsenhausen, “A counterexample in stochastic optimum control,” _SIAM Journal on Control_ , 1968.
  * [2] W. Liu, G. Nair, Y. Li, D. Nesic, B. Vucetic, and H. V. Poor, “On the latency, rate and reliability tradeoff in wireless networked control systems for IIoT,” _IEEE Internet of Things Journal_ , 2020.
  * [3] K. Gatsis, H. Hassani, and G. J. Pappas, “Latency-reliability tradeoffs for state estimation,” _IEEE Transactions on Automatic Control_ , 2020.
  * [4] Huang, Kang and Liu, Wanchun and Li, Yonghui and Savkin, Andrey and Vucetic, Branka, “Wireless feedback control with variable packet length for industrial IoT,” _IEEE Wireless Communications Letters_ , 2020.
  * [5] P. M. de Sant Ana, N. Marchenko, P. Popovski, and B. Soret, “Wireless control of autonomous guided vehicle using reinforcement learning,” in _IEEE Global Communications Conference_ , 2020.
  * [6] 3GPP, “Technical Specification Group Services and System Aspects;Release 15 Description,” 3rd Generation Partnership Project (3GPP), Technical Specification (TS) 21.915, 10 2019, version 15.0.0.
  * [7] “White Paper: 5G for Connected Industries and Automation,” 5G Alliance for Connected Industries and Automation, Tech. Rep., 02 2019.
  * [8] A. M. Bedewy, Y. Sun, and N. B. Shroff, “Minimizing the age of information through queues,” _IEEE Transactions on Information Theory_ , 2019.
  * [9] A. G. Barto, R. S. Sutton, and C. W. Anderson, “Neuronlike adaptive elements that can solve difficult learning control problems,” _IEEE Transactions on Systems, Man, and Cybernetics_ , 1983.
  * [10] R. V. Florian, “Correct equations for the dynamics of the cart-pole system,” _Center for Cognitive and Neural Studies (Coneural), Romania_ , 2007.
  * [11] F. L. Lewis, D. Vrabie, and V. L. Syrmos, _Optimal control_. John Wiley & Sons, 2012.
  * [12] P. Park, S. C. Ergen, C. Fischione, C. Lu, and K. H. Johansson, “Wireless network design for control systems: A survey,” _IEEE Communications Surveys & Tutorials_, 2017.
  * [13] 3GPP, “Evolved Universal Terrestrial Radio Access (E-UTRA); Physical layer procedures,” 3rd Generation Partnership Project (3GPP), Technical Specification (TS) 36.213, 10 2014, version 12.3.0.
  * [14] R. D. Yates, Y. Sun, D. R. Brown, S. K. Kaul, E. Modiano, and S. Ulukus, “Age of information: An introduction and survey,” _IEEE Journal on Selected Areas in Communications_ , 2021.
  * [15] J. P. Champati, M. H. Mamduhi, K. H. Johansson, and J. Gross, “Performance characterization using AoI in a single-loop networked control system,” in _IEEE Conference on Computer Communications Workshops_ , 2019.
  * [16] M. Klügel, M. H. Mamduhi, S. Hirche, and W. Kellerer, “AoI-penalty minimization for networked control systems with packet loss,” in _IEEE Conference on Computer Communications Workshops_ , 2019.
  * [17] R. S. Sutton and A. G. Barto, _Reinforcement learning: An introduction_. MIT press, 2018.