# Semantic-aware Transmission Scheduling: a Monotonicity-driven Deep
Reinforcement Learning Approach

Jiazheng Chen,, Wanchun Liu*, Daniel E. Quevedo, Yonghui Li, Branka Vucetic,
J. Chen, W. Liu, Y. Li, and B. Vucetic are with the School of Electrical and
Information Engineering, The University of Sydney, Sydney, NSW 2006, Australia
(e-mails: {jiazheng.chen, wanchun.liu, yonghui.li,
branka.vucetic}@sydney.edu.au). D. E. Quevedo is with the School of Electrical
Engineering and Robotics, Queensland University of Technology (QUT), Brisbane,
Australia. (e-mail: dquevedo@ieee.org). (W. Liu is the corresponding author.)

###### Abstract

For cyber-physical systems in the 6G era, semantic
communications connecting distributed devices for dynamic control and remote
state estimation are required to guarantee application-level performance, not
merely focus on communication-centric performance. Semantics here is a measure
of the usefulness of information transmissions. Semantic-aware transmission
scheduling of a large system often involves a large decision-making space, and
the optimal policy cannot be obtained by existing algorithms effectively. In
this paper, we first establish the monotonicity of the Q function of the
optimal semantic-aware scheduling policy and then develop advanced deep
reinforcement learning (DRL) algorithms by leveraging the theoretical
guidelines. Our numerical results show a 30% performance improvement compared
to benchmark algorithms.

###### Index Terms:

Deep reinforcement learning, sensor scheduling, age of information, semantic
communications.

##  I Introduction

Different from the first generation to the 5G communications, which focused on
improving data-oriented performance (e.g., data rate, latency and
reliability), the 6G is envisioned to support many emerging applications and
adopt semantic-related metrics for application-level performance optimization
[1]. The potential applications supported by 6G semantic communications can be
divided into two categories: human-centered systems (e.g., Metaverse and XR
[2]) and cyber-physical systems (CPSs) (e.g., autonomous driving and networked
robotics [3]). For the former, semantics is a measure of the meaning of
information that a human can understand; for the latter, semantics
characterizes the usefulness of messages with respect to the goal of data
exchange.

In CPSs, semantic-aware transmission aims to minimize the distortion and
timing mismatch of the data between the edge devices and the remote
destination [3]. Thus, the freshness of messages is critical, and monotonic
functions of age of information (AoI) are often introduced as semantic quality
metrics [4, 5]. Semantic-aware (and AoI-aware) dynamic scheduling of cyber-
physical systems over limited communication resources has attracted much
attention in the communications society [1]. In [4], an optimal scheduling
problem of a multi-sensor remote estimation system was considered to minimize
the overall long-term estimation mean-square error (MSE). Importantly, the
instantaneous MSE of each sensor is a function of the corresponding AoI. The
scheduling problem was formulated as a Markov decision process (MDP) that can
be solved by conventional value and policy iteration methods. In [6], an
optimal AoI minimization scheduling problem was considered under an average
transmission power constraint and was formulated into a constrained MDP. Note
that the conventional MDP algorithms can only solve very small-scale problems
due to the high computational complexity introduced by the curse of
dimensionality. Heuristic scheduling policies without optimality guarantees
were proposed in were proposed in [7, 8].

The recent development of deep reinforcement learning (DRL) allows us to solve
large MDPs effectively with deep neural networks (NNs) for function
approximations. In [9], deep Q-network (DQN) was adopted to solve optimal
scheduling problems of multi-sensor-multi-channel remote estimation systems.
In particular, those solutions can achieve better performance (in terms of
MSE) than heuristic methods. Advanced DRL algorithms with an actor-critic
framework (consisting of an actor NN and a critic NN), such as soft actor-
critic (SAC) and deep deterministic policy gradient (DDPG), were introduced to
deal with large-scale semantic-aware scheduling problems that cannot be
handled by DQN [10, 11]. _A major drawback of applying existing DRL methods in
large-scale scheduling problems is that training is usually very time-
consuming and can often get stuck in local minima. This motivates us to have a
more profound understanding of the structural properties of optimal scheduling
policies (such as the monotonicity of the critic NN), which can then be
leveraged to guide DRL algorithms towards faster and more effective
convergence._

In this paper, we focus on developing advanced actor-critic DRL algorithms to
optimize semantic-aware scheduling policies. The contributions include: First,
we mathematically prove that a well-trained critic NN of an optimal scheduling
policy should be a monotonic function in terms of the AoI states and channel
states of the system. Such a property has never been established in the open
literature. Second, by leveraging this theoretical property, we propose two
monotonic critic NN training methods: network architecture-enabled
monotonicity and regularization-based monotonicity. The proposed training
methods can be directly applied to the baseline DDPG and its variants. Last,
our numerical results show that the proposed monotonicity-enforced DDPG
algorithms can save training time and improve training performance
significantly compared to the baseline DDPG.

##  II System Model

We consider semantic communication in a CPS with $N$ edge devices (e.g.,
sensors and robots) and a remote destination (e.g., a remote estimator or
controller) connected by a wireless network with $M<N$ channels (e.g., sub-
carriers).

###  II-A Wireless Communication Model

At each time step $t$, the transmission scheduling action for device $n$ is
denoted by

$a_{n,t}=\left\\{\begin{array}[]{ll}0&\text{if device $n$ is not
scheduled,}\\\ m&\text{if device $n$ is scheduled to channel
$m$.}\end{array}\right.\vspace{-0.1cm}$ (1)

We assume that each device can be scheduled to at most one channel and that
each channel is assigned to one device, i.e.,

$\sum_{m=1}^{M}\bm{\mathbbm{1}}\left(a_{n,t}=m\right)\leq
1,\quad\sum_{n=1}^{N}\bm{\mathbbm{1}}\left(a_{n,t}=m\right)=1,\vspace{-0.1cm}$
(2)

where $\bm{\mathbbm{1}}(\cdot)$ is the indicator function.

We consider independent and identically distributed (i.i.d.) block fading
channels, where the channel quality is fixed during each packet transmission
and varies packet by packet independently. The channel state of the system at
time $t$ is denoted by a $N\times M$ matrix $\mathbf{H}_{t}$, where the $n$th
row and $m$th column element
$h_{n,m,t}\in\mathcal{H}\triangleq\left\\{1,2,\dots,\hat{h}\right\\}$
represents the channel state between device $n$ and the destination at channel
$m$ quantized into $\hat{h}$ levels. The distribution of the channel state
$h_{n,m,t}$ is given as

$\operatorname{Pr}(h_{n,m,t}=j)=q^{(n,m)}_{j},\forall t,\vspace{-0.1cm}$ (3)

where $\sum_{j=1}^{\hat{h}}q^{(n,m)}_{j}=1,\forall n,m$. A higher channel
state leads to a large packet drop probability. Let $p_{n,m,t}$ denote the
packet drop probability at the channel state $h_{n,m,t}$. The instantaneous
full channel state $\mathbf{H}_{t}$ is available at the remote estimator based
on standard channel estimation methods.

###  II-B Semantic Communication Performance

First, we define $\tau_{n,t}\in\\{1,2,\dots\\}$ as the AoI of device $n$ at
time $t$, i.e., the time elapsed since its latest device packet was
successfully received [5, 6, 7]:

$\\!\\!\\!\\!\\!\tau_{n,t+1}\\!=\\!\\!\begin{cases}\\!1,&{\\!\\!\\!\\!\\!\text{at
time $t$, device $n$'s packet is received,}}\\\
\\!\tau_{n,t}\\!+\\!1,&{\\!\\!\\!\\!\text{otherwise.}}\end{cases}\\!\\!\\!\\!\\!\vspace{-0.1cm}$
(4)

Next, we define a _positive non-decreasing cost_ (or penalty) function
$g_{n}(\tau_{n,t}),\forall n\in\\{1,\dots,N\\}$ as a semantic measurement of
device $n$’s information. A higher AoI state corresponds to an increased cost
function, indicating poorer performance in semantic communications. Semantic
communication systems with varying functionalities possess distinct
$g_{n}(\cdot)$ functions. The scheduling algorithms proposed in the rest of
this paper are general and applicable to any positive and non-decreasing cost
function.

An example below shows how to determine $g_{n}(\cdot)$ in a specific semantic
communication system.

######  Example 1 (A remote state estimation system).

Such a system consists of $N$ sensors, each measuring a physical process, and
a remote estimator for signal reconstruction. Each physical process dynamics
and its measurement are modeled as a discrete-time linear time-invariant (LTI)
system:

$\displaystyle\mathbf{e}_{n,t+1}=\mathbf{A}_{n}\mathbf{e}_{n,t}+\mathbf{w}_{n,t},\
\mathbf{z}_{n,t}=\mathbf{C}_{n}\mathbf{z}_{n,t}+\mathbf{v}_{n,t},\vspace{-0.1cm}$
(5)

where $\mathbf{e}_{n,t}\in\mathbb{R}^{l_{n}}$ is process $n$’s state at time
$t$, and $\mathbf{z}_{n,t}\in\mathbb{R}^{c_{n}}$ is the state measurement of
sensor $n$; $\mathbf{A}_{n}\in\mathbb{R}^{l_{n}\times l_{n}}$ and
$\mathbf{C}_{n}\in\mathbb{R}^{c_{n}\times l_{n}}$ are the system matrix and
the measurement matrix, respectively; $\mathbf{w}_{n,t}\in\mathbb{R}^{l_{n}}$
and $\mathbf{v}_{n,t}\in\mathbb{R}^{c_{n}}$ are the process disturbance and
the measurement noise modeled as i.i.d zero-mean Gaussian random vectors
$\mathcal{N}(\mathbf{0},\mathbf{W}_{n})$ and
$\mathcal{N}(\mathbf{0},\mathbf{V}_{n})$, respectively. Due to the noisy
measurement in (5), sensor $n$ applies a local Kalman filter (KF) to pre-
process the raw measurement before sending it to the remote estimator. Due to
the dynamic scheduling and the random packet dropouts, the remote estimator
cannot always receive sensor $n$’s measurement and thus uses a minimum mean
square error estimator for state estimation, say $\hat{\mathbf{e}}_{n,t}$.
Then, the estimation error covariance is given as

$\displaystyle\mathbf{P}_{n,t}$
$\displaystyle\triangleq\mathbb{E}\left[\left(\hat{\mathbf{e}}_{n,t}-\mathbf{e}_{n,t}\right)\left(\hat{\mathbf{e}}_{n,t}-\mathbf{e}_{n,t}\right)^{\top}\right]=f_{n}^{\tau_{n,t}}(\bar{\mathbf{P}}_{n}),\vspace{-0.1cm}$
(6)

where
$f_{n}(\mathbf{X})=\mathbf{A}_{n}\mathbf{X}\mathbf{A}_{n}^{\top}+\mathbf{W}_{n}$
and $f^{\tau+1}_{n}(\cdot)=f_{n}(f^{\tau}_{n}(\cdot))$, where
$\bar{\mathbf{P}}_{n}$ is a constant matrix determined by $\mathbf{A}_{n}$,
$\mathbf{C}_{n}$, $\mathbf{W}_{n}$ and $\mathbf{V}_{n}$.

To characterize the remote estimation performance, we define the cost function
as the estimation MSE of process $n$

$g_{n}(\tau_{n,t})\triangleq\operatorname{Tr}(\mathbf{P}_{n,t})=f_{n}^{\tau_{n,t}}(\bar{\mathbf{P}}_{n}),\vspace{-0.1cm}$
(7)

which has been proved to be a positive non-decreasing function of
$\tau_{n,t}$, see [5] for further details.

##  III Semantic-aware Transmission Scheduling

We aim to find a dynamic scheduling policy $\pi(\cdot)$ that minimizes the
expected total discounted cost function of all $N$ devices over the infinite
time horizon.

######  Problem 1.

$\min_{\pi}\lim_{T\to\infty}\mathbb{E}^{\pi}\left[\sum_{t=1}^{T}\sum_{n=1}^{N}\gamma^{t}g_{n}(\tau_{n,t})\right],\vspace{-0.1cm}$
(8)

where $\gamma\in(0,1)$ is a discount factor.

###  III-A Markov Decision Process Formulation

Problem 1 is a sequential decision-making problem with the Markovian property
and hence can be cast as an MDP as below.

1) The state of the MDP consists of both the AoI and channel states as
$\mathbf{s}_{t}\triangleq\left(\bm{\tau}_{t},\mathbf{H}_{t}\right)\in\mathbb{N}^{N}\times\mathcal{H}^{M\times
N}$, where
$\bm{\tau}_{t}=(\tau_{1,t},\tau_{2,t},\dots,\tau_{N,t})\in\mathbb{N}^{N}$ is
the AoI state vector.

2) The overall schedule action of the $N$ devices is defined as
$\mathbf{a}_{t}=(a_{1,t},a_{2,t},\dots,a_{N,t})\in\left\\{0,1,2,\dots,M\right\\}^{N}$
under the constraint (2). The policy $\pi(\cdot)$ maps a state to an action as
$\mathbf{a}_{t}=\pi(\mathbf{s}_{t})$.

3) The immediate reward (i.e., the negative sum cost of all edge devices) at
time $t$ is defined as $-\sum_{n=1}^{N}g_{n}(\tau_{n,t})$.

4) The transition probability
$\operatorname{Pr}(\mathbf{s}_{t+1}|\mathbf{s}_{t},\mathbf{a}_{t})$ is the
probability of the next state $\mathbf{s}_{t+1}$ given the current state
$\mathbf{s}_{t}$ and the action $\mathbf{a}_{t}$. Since the optimal policy of
an infinite-horizon MDP is stationary [12, Chapter 6], the state transition is
independent of the time index. Thus, we drop the subscript $t$ and use
$\mathbf{s}\triangleq(\bm{\tau},\mathbf{H})$ and
$\mathbf{s}^{+}\triangleq(\bm{\tau}^{+},\mathbf{H}^{+})$ to represent the
current and the next states, respectively. Due to the i.i.d. fading channels
and the AoI state transition, we have

$\operatorname{Pr}(\mathbf{s}^{+}|\mathbf{s},\mathbf{a})=\operatorname{Pr}(\bm{\tau}^{+}|\bm{\tau},\mathbf{H},\mathbf{a})\operatorname{Pr}(\mathbf{H}^{+}),$
(9)

where
$\operatorname{Pr}(\bm{\tau}^{+}|\bm{\tau},\mathbf{H},\mathbf{a})=\prod_{n=1}^{N}\operatorname{Pr}(\tau^{+}_{n}|\tau_{n},\mathbf{H},a_{n})$
and

$\displaystyle\\!\\!\\!\\!\\!\operatorname{Pr}(\tau^{+}_{n}|\tau_{n},\mathbf{H},a_{n})\\!=\\!\begin{cases}1-p_{n,m},&\\!\\!\\!\text{if
$\tau^{+}_{n}=1,a_{n}=m$,}\\\ p_{n,m},&\\!\\!\\!\text{if
$\tau^{+}_{n}=\tau_{n}\\!+\\!1,a_{n}\\!=\\!m$,}\\\ 1,&\\!\\!\\!\text{if
$\tau^{+}_{n}=\tau_{n}\\!+\\!1,a_{n}\\!=\\!0$,}\\\
0,&\\!\\!\\!{\text{otherwise.}}\end{cases}\vspace{-0.3cm}\\!\\!\\!\\!$ (10)

###  III-B Value Functions of the Optimal Policy

We define the V function (also called “state-value function”),
$V(\mathbf{s}_{t}):\mathcal{S}\to\mathbb{R}$ and the Q function (the “action-
value function”),
$Q(\mathbf{s}_{t},\mathbf{a}_{t}):\mathcal{S}\times\mathcal{A}\to\mathbb{R}$
of the optimal policy $\pi^{*}(\cdot)$. Given the current state
$\mathbf{s}_{t}$, the V function is the maximum expected discounted sum of the
future reward achieved by the optimal policy, i.e.,
$V(\mathbf{s}_{t})=\mathbb{E}\left[\sum_{t^{\prime}=t}^{\infty}\gamma^{t^{\prime}-t}r\left(\mathbf{s}_{t^{\prime}}\right)\Big{|}\mathbf{a}_{t^{\prime}}=\pi^{*}(\mathbf{s}_{t^{\prime}}),\forall
t^{\prime}\geq t\right].$ Given the current state-action pair,
$(\mathbf{s}_{t},\mathbf{a}_{t})$, the Q function is the maximum expected
discounted sum of the future reward under the optimal policy:

$\\!Q(\mathbf{s}_{t},\mathbf{a}_{t})\\!=\\!\mathbb{E}\left[\sum_{t^{\prime}=t}^{\infty}\gamma^{t^{\prime}-t}r(\mathbf{s}_{t^{\prime}})\Big{|}\mathbf{a}_{t},\mathbf{a}_{t^{\prime}}\\!=\\!\pi^{*}(\mathbf{s}_{t^{\prime}}),\forall
t^{\prime}\\!>\\!t\right].$ (11)

Thus, the V function measures the long-term performance given the current
state, while the Q function can be used for comparing the long-term
performance affected by different current actions.

The relation between the two value functions is given by [13]

$Q(\mathbf{s}_{t},\mathbf{a}_{t})=r(\mathbf{s}_{t})+\gamma\sum_{\mathbf{s}_{t+1}}\operatorname{Pr}(\mathbf{s}_{t+1}|\mathbf{s}_{t},\mathbf{a}_{t})V(\mathbf{s}_{t+1}).\vspace{-0.1cm}$
(12)

In particular, the Q function satisfies the Bellman equation:

$Q(\mathbf{s}_{t},\mathbf{a}_{t})=r(\mathbf{s}_{t})+\mathbb{E}_{\mathbf{s}_{t+1}}\left[\gamma\max_{\mathbf{a}_{t+1}}Q\left(\mathbf{s}_{t+1},\mathbf{a}_{t+1}\right)\right].\vspace{-0.1cm}$
(13)

###  III-C DDPG for scheduling policy optimization

A DDPG agent has two neural networks (NNs): an actor NN with the parameter set
$\bm{\alpha}$ and a critic network with the parameter set $\bm{\beta}$. In
particular, the actor NN aims to approximate the optimal policy
$\pi^{*}(\mathbf{s})$ by $\pi(\mathbf{s};\bm{\alpha})$ after well-training,
while the critic NN approximates $Q(\mathbf{s}_{t},\mathbf{a}_{t})$ by
$Q(\mathbf{s}_{t},\mathbf{a}_{t};\bm{\beta})$. The overall training process
for the two NNs works iteratively based on sampled data named _transitions_.
Each transition
$\mathcal{T}_{i}\triangleq(\mathbf{s}_{i},\mathbf{a}_{i},r_{i},\mathbf{s}_{i+1})$,
consists of the state, action, reward, and next state. To train the actor NN,
the critic NN’s Q values are utilized for characterizing the performance of
the actor NN’s policy over the sampled data; to train the critic NN, the
transitions generated by the actor NN’s policy are utilized to evaluate the
accuracy of Q function approximation. In particular, the approximation error
given $\mathcal{T}_{i}$ is defined as the temporal difference (TD) error

$\mathsf{TD}_{i}^{2}\triangleq\left(y_{i}-Q(\mathbf{s}_{i},\mathbf{a}_{i};\bm{\beta})\right)^{2},\vspace{-0.1cm}$
(14)

where
$y_{i}=r_{i}+\gamma\max_{{\mathbf{a}}_{i+1}}Q(\mathbf{s}_{i+1},{\mathbf{a}}_{i+1};\bm{\beta}^{-})$
is the estimation of the Q function at next step and $\bm{\beta}^{-}$ is the
critic NN parameters from the previous iteration. In other words, the training
of the critic NN is to minimize the TD errors over the transitions, making the
approximated Q function satisfy the Bellman equation (13) [13].

######  Remark 1.

Our work focuses on the effective training of the critic NN by utilizing the
partial monotonicity of the Q function discussed in the section below, and the
training method for the actor NN is identical to the baseline DDPG.
Furthermore, the proposed training methods can be applied to solve other DRL
problems that involve monotonic critic NNs and can seamlessly integrate with
actor-critic DRL algorithms like Twin Delayed DDPG (TD3) and Proximal Policy
Optimization (PPO).

##  IV Partial Monotonicity of Q Function

Before proceeding further, we need the following technical lemma related to
the V function monotonicity.

######  Lemma 1 (Monotonicity of V function w.r.t. AoI states [10]).

Consider states $\mathbf{s}=(\bm{\tau},\mathbf{H})$ and
$\mathbf{s}^{\prime}_{\text{AoI}}=(\bm{\tau}^{\prime}_{(n)},\mathbf{H})$,
where
$\bm{\tau}^{\prime}_{(n)}=(\tau_{1},\dots,\tau^{\prime}_{n},\dots,\tau_{N})$
and $\tau^{\prime}_{n}\geq\tau_{n}$. For any $n\in\\{1,\dots,N\\}$, the
following holds

$V(\mathbf{s}^{\prime}_{\text{AoI}})\leq V(\mathbf{s}).\vspace{-0.1cm}$ (15)

Now we establish the partial monotonicity of the Q function in terms of the
AoI and channel states as below.

######  Theorem 1 (Monotonicity of Q function w.r.t. AoI states).

Consider action $\mathbf{a}=(a_{1},\dots,a_{N})$, and states
$\mathbf{s}=(\bm{\tau},\mathbf{H})$ and
$\mathbf{s}^{\prime}_{\text{AoI}}=(\bm{\tau}^{\prime}_{(n)},\mathbf{H})$,
where
$\bm{\tau}^{\prime}_{(n)}=(\tau_{1},\,\dots,\,\tau^{\prime}_{n},\,\dots,\tau_{N})$,
$\tau^{\prime}_{n}\geq\tau_{n}$. The following holds

$\displaystyle Q(\mathbf{s}^{\prime}_{\text{AoI}},\mathbf{a})$
$\displaystyle\leq Q(\mathbf{s},\mathbf{a}).\vspace{-0.2cm}$ (16)

###### Proof.

For notation simplicity, we assume that $a_{n}=m,\forall m\in\\{1,\dots,M\\}$.
From (3) and (4), we have

$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\operatorname{Pr}(\bm{\tau}^{+}|\bm{\tau}\\!,\\!\mathbf{H},\\!\mathbf{a})$
$\displaystyle\\!=\\!\prod_{n=1}^{N}\operatorname{Pr}(\tau_{n}^{+}|\tau_{n},\mathbf{h}_{n},a_{n})$
(17)
$\displaystyle\\!\\!=\\!\operatorname{Pr}(\tau_{n}^{+}|\tau_{n},\\!\mathbf{h}_{n},\\!a_{n})\\!\operatorname{Pr}(\bm{\tau}_{\backslash
n}^{+}|\bm{\tau}_{\backslash n},\\!\mathbf{H}_{\backslash
n},\\!\mathbf{a}_{\backslash n}),$ (18)

where $\bm{\tau}_{\backslash
n}=(\tau_{1},\dots,\tau_{n-1},\tau_{n+1},\dots,\tau_{N})$ and
$\mathbf{a}_{\backslash n}=(a_{1},\dots,a_{n-1},a_{n+1},\dots,a_{N})$
represent the AoI states and the actions of all devices except device $n$,
respectively; $\mathbf{h}_{n}=(h_{n,1},h_{n,2},\dots,h_{n,M})$ is the vector
channel state between device $n$ and the remote estimator and
$\mathbf{H}_{\backslash
n}=(\mathbf{h}_{1},\dots,\mathbf{h}_{n-1},\mathbf{h}_{n+1},\dots,\mathbf{h}_{N})$.
By using (9), (12), and (18), it can be derived that

$\displaystyle\\!\\!Q(\mathbf{s},\mathbf{a})\\!=\\!r(\mathbf{s})+$
$\displaystyle\gamma\\!\sum_{\mathbf{H}^{+}}\sum_{\bm{\tau}_{\backslash
n}^{+}}\sum_{\tau_{n}^{+}}\operatorname{Pr}(\mathbf{H}^{+})\operatorname{Pr}(\bm{\tau}_{\backslash
n}^{+}|\bm{\tau}_{\backslash n},\mathbf{H}_{\backslash
n},\mathbf{a}_{\backslash n})\\!\\!$ (19)
$\displaystyle\times\operatorname{Pr}(\tau_{n}^{+}|\tau_{n},\mathbf{h}_{n},a_{n})V(\mathbf{s}^{+}).$
(20)

Since the state $\mathbf{s}^{\prime}_{\text{AoI}}$ is identical to the state
$\mathbf{s}$ except the device $n$’s AoI state $\tau_{n}$, to prove (16) from
(20) and $r(\mathbf{s}^{\prime}_{\text{AoI}})\leq r(\mathbf{s})$, we only need
to prove the following inequality

$\displaystyle\sum_{{\tau_{n}^{\prime}}^{+}}\operatorname{Pr}({\tau_{n}^{\prime}}^{+}|\tau_{n},\mathbf{h}_{n},a_{n})V({\mathbf{s}^{\prime}_{\text{AoI}}}^{+})$
(21)
$\displaystyle\leq\sum_{{\tau_{n}}^{+}}\operatorname{Pr}({\tau_{n}}^{+}|\tau_{n},\mathbf{h}_{n},a_{n})V({\mathbf{s}}^{+}).$
(22)

From (4), each of $\mathbf{s}^{+}$ and
${\mathbf{s}^{\prime}_{\text{AoI}}}^{+}$ has two different states in (22)
depending on the different transition of device $n$’s AoI state. Then, by
taking (10) into (22), the latter is equivalent to

$\displaystyle\\!\\!\\!\\!(1\\!-p_{n,m})V\\!(1,\\!\bm{\tau}_{\backslash
n}^{+},\\!{\mathbf{H}}^{+})+p_{n,m}V\\!({\tau_{n}^{\prime}}\\!+\\!1,\\!\bm{\tau}_{\backslash
n}^{+},\\!{\mathbf{H}}^{+})$ (23)
$\displaystyle\\!\\!\\!\leq(1\\!-p_{n,m})V\\!(1,\\!\bm{\tau}_{\backslash
n}^{+},\\!\mathbf{H}^{+})\\!+\\!p_{n,m}V\\!({\tau_{n}}\\!+\\!1,\\!\bm{\tau}_{\backslash
n}^{+},\\!\mathbf{H}^{+}).$ (24)

By using Lemma 1, we have
$V\\!({\tau_{n}^{\prime}}\\!+\\!1,\\!\bm{\tau}_{\backslash
n}^{+},\\!{\mathbf{H}}^{+})\leq
V\\!({\tau_{n}}\\!+\\!1,\\!\bm{\tau}_{\backslash n}^{+},\\!\mathbf{H}^{+})$.
The inequality (24) holds directly. ∎

######  Theorem 2 (Monotonicity of Q function w.r.t. channel states).

Consider action $\mathbf{a}=(a_{1},\dots,a_{N})$, and states
$\mathbf{s}=(\bm{\tau},\mathbf{H})$ and
$\mathbf{s}^{\prime}_{\text{Ch}}=(\bm{\tau},\mathbf{H}^{\prime}_{n,m})$ where
$\mathbf{H}^{\prime}_{n,m}$ is identical to $\mathbf{H}$ except the
device-$n$-channel-$m$ state with $h^{\prime}_{n,m}>h_{n,m}$.

  * (i)

If $a_{n}=m$, then the following holds

$\displaystyle Q(\mathbf{s}^{\prime}_{\text{Ch}},\mathbf{a})\leq
Q(\mathbf{s},\mathbf{a}).\vspace{-0.1cm}$ (25)

  * (ii)

If $a_{n}\neq m$, then the equality in (25) holds, i.e., we have

$\displaystyle
Q(\mathbf{s}^{\prime}_{\text{Ch}},\mathbf{a})=Q(\mathbf{s},\mathbf{a}).\vspace{-0.1cm}$
(26)

###### Proof.

(i) Based on (20), we only need to prove that

$\displaystyle\sum_{{\mathbf{H}^{\prime}}^{+}}\sum_{{\tau_{n}}^{+}}\operatorname{Pr}({\mathbf{H}^{\prime}}^{+})\operatorname{Pr}({\tau_{n}}^{+}|\tau_{n},\mathbf{h}^{\prime}_{n},a_{n})V({\mathbf{s}^{\prime}_{\text{Ch}}}^{+})$
(27)
$\displaystyle\leq\sum_{{\mathbf{H}}^{+}}\sum_{{\tau_{n}}^{+}}\operatorname{Pr}(\mathbf{H}^{+})\operatorname{Pr}({\tau_{n}}^{+}|\tau_{n},\mathbf{h}_{n},a_{n})V({\mathbf{s}}^{+}),$
(28)

which is equivalent to

$\displaystyle(1-p^{\prime}_{n,m})\sum_{{\mathbf{H}^{\prime}}^{+}}\operatorname{Pr}({\mathbf{H}^{\prime}}^{+})V(1,\bm{\tau}_{\backslash
n}^{+},{\mathbf{H}^{\prime}}^{+})$ (29)
$\displaystyle+p^{\prime}_{n,m}\sum_{{\mathbf{H}^{\prime}}^{+}}\operatorname{Pr}({\mathbf{H}^{\prime}}^{+})V({\tau_{n}}+1,\bm{\tau}_{\backslash
n}^{+},{\mathbf{H}^{\prime}}^{+})$ (30)
$\displaystyle\leq(1-p_{n,m})\sum_{{\mathbf{H}}^{+}}\operatorname{Pr}({\mathbf{H}}^{+})V(1,\bm{\tau}_{\backslash
n}^{+},{\mathbf{H}}^{+})$ (31)
$\displaystyle+p_{n,m}\sum_{{\mathbf{H}}^{+}}\operatorname{Pr}({\mathbf{H}}^{+})V({\tau_{n}}+1,\bm{\tau}_{\backslash
n}^{+},{\mathbf{H}}^{+}).$ (32)

From (3), we have

$\sum_{{\mathbf{H}^{\prime}}^{+}}\operatorname{Pr}({\mathbf{H}^{\prime}}^{+})V(\bm{\tau}^{+},{\mathbf{H}^{\prime}}^{+})=\sum_{{\mathbf{H}}^{+}}\operatorname{Pr}({\mathbf{H}}^{+})V(\bm{\tau}^{+},{\mathbf{H}}^{+}),$
(33)

where $\bm{\tau}^{+}=(1,\bm{\tau}_{\backslash n}^{+})$ or
$(\tau_{n}+1,\bm{\tau}_{\backslash n}^{+})$. By using (33), the inequality
(32) is simplified as

$\displaystyle(p^{\prime}_{n,m}\\!-\\!p_{n,m})\\!\sum_{{\mathbf{H}}^{+}}\operatorname{Pr}({\mathbf{H}}^{+})\Big{[}$
$\displaystyle V({\tau_{n}}\\!+\\!1,\\!\bm{\tau}_{\backslash
n}^{+},\\!{\mathbf{H}}^{+})$ (34) $\displaystyle-V(1,\\!\bm{\tau}_{\backslash
n}^{+},\\!{\mathbf{H}}^{+})\Big{]}\\!\leq\\!0.$ (35)

Due to the facts that $V\\!({\tau_{n}}\\!+\\!1,\\!\bm{\tau}_{\backslash
n}^{+},\\!{\mathbf{H}}^{+})\\!\leq\\!V\\!(1,\\!\bm{\tau}_{\backslash
n}^{+},\\!{\mathbf{H}}^{+})$ and $p^{\prime}_{n,m}\geq p_{n,m}$, (35) holds
directly.

(ii) Let $a_{n}=\bar{m}\neq m$. Similar to the proof of (i), after
simplifications, we only need to prove the following equality

$\displaystyle(p_{n,\bar{m}}\\!-\\!p_{n,\bar{m}})\\!\sum_{{\mathbf{H}}^{+}}\operatorname{Pr}({\mathbf{H}}^{+})\Big{[}$
$\displaystyle V({\tau_{n}}\\!+\\!1,\\!\bm{\tau}_{\backslash
n}^{+},\\!{\mathbf{H}}^{+})$ (36) $\displaystyle-V(1,\\!\bm{\tau}_{\backslash
n}^{+},\\!{\mathbf{H}}^{+})\Big{]}=0,$ (37)

which is directly valid. ∎

##  V Partially Monotonic Critic Neural Network

Building on the partial monotonicity of the Q function, we aim to develop
novel training methods guaranteeing the monotonicity of the critic NN that can
help solve Problem 1 more effectively compared to conventional learning
methods. We consider two partially monotonic critic NN training methods: (i)
network architecture-enabled monotonicity and (ii) regularization-based
monotonicity in the sequel.

###  V-A Network Architecture-enabled Monotonicity

We propose a three-layer critic NN architecture (a shallow NN) that directly
guarantees the partial monotonicity of the Q function, as shown in Fig. 2 (a).
The critic NN integrates the outputs of two NN processing the state and the
action inputs separately. In particular, the state-processing part (on the top
of Fig. 2(a)) has to be a monotonic decreasing function. We adopt the
monotonic NN in [14], which guarantees that the output is a monotonic
decreasing function of the input with the following conditions: 1) the
activation functions are non-decreasing (e.g., sigmoid or ReLU), and 2) the
weights satisfy

$\beta^{(1,2)}_{\bar{i},\bar{j}}\beta^{(2,3)}_{\bar{j}}\leq
0,\forall\bar{i}\in\\{1,N(M+1)\\},\bar{j}\in\\{1,\dots,U\\}.$ (38)

In (38), $\beta^{(1,2)}_{\bar{i},\bar{j}}$ denotes the weight for the
connection between $\bar{i}$-th input node and the $\bar{j}$-th hidden node,
$\beta^{(2,3)}_{\bar{j}}$ denotes the weight for the connection between the
$\bar{j}$-th hidden node and the output node in the critic NN, and $U$ is the
size of the hidden layer. There is no specific requirement on the parameters
of the action-processing part (at the bottom of Fig. 2(a)).

First, we adopt the sigmoid activation function with positive output. Then, to
satisfy the constraint (38) and to ensure negative Q value outputs (see the
definition in (11)) during training, we restrict the weights
$\beta^{(1,2)}_{\bar{i},\bar{j}}$ and $\beta^{(2,3)}_{\bar{j}}$ to be non-
negative and non-positive, respectively, i.e.,
$\beta^{(1,2)}_{\bar{i},\bar{j}}\leftarrow\max(0,\beta^{(1,2)}_{\bar{i},\bar{j}})$
and
$\beta^{(2,3)}_{\bar{i},\bar{j}}\leftarrow\min(0,\beta^{(2,3)}_{\bar{i},\bar{j}})$.

The advantage of the presented approach is that the monotonicity is strictly
and automatically ensured by the NN architecture during the training at each
time step. A potential drawback is that the “shallow” monotonic critic NN has
a single hidden layer, limiting its capability for approximating complex
functions, which are required when solving scheduling problems with large
state and action spaces. Note that although there are some deep monotonic NN,
such as deep lattice networks [15], these NNs have very complex architectures
and are computationally expensive for large inputs. Therefore, we propose the
following approach for training a “deep” monotonic critic NN, which is a
multi-layer fully connected NN without a specific NN architecture requirement.

Figure 1: Partially monotonic critic NNs. (a) A shallow critic NN with network
architecture-enabled monotonicity, where the NN has only one hidden layer. (b)
A deep critic NN with regularization-based monotonicity.

###  V-B Regularization-based Monotonicity

We introduce penalty terms in the loss function of the critic NN training to
enforce a monotonically _decreasing_ relationship between the state input and
the output. In particular, we propose two types of penalty for monotonicity
regularization.

####  V-B1 Positive derivative penalty (type I)

Let $s_{i,j}$ denote the $j$-th state of the length-$N(M+1)$ state vector
$\mathbf{s}_{i}$, which can be an AoI state (i.e., $j\leq N$) or a channel
state (i.e., $j>N$). The partial derivative of the Q function
$Q(\mathbf{s}_{i},\mathbf{a}_{i};\bm{\beta})$ at the state input $s_{i,j}$ is
given as

$\mathsf{QD}_{i,j}=\nabla_{s_{i,j}}Q(\mathbf{s}_{i},\mathbf{a}_{i};\bm{\beta})\quad\forall
j\in\\{1,\dots,N(M+1)\\}.$ (39)

To enforce the critic NN approximating a monotonically decreasing function at
$s_{i,j}$, the corresponding positive derivative penalty is
$\max(0,\mathsf{QD}_{i,j})$.

The computation of the derivative (39) requires backward propagation, which
has a higher computational complexity than forward propagation, i.e.,
evaluating a Q value by feeding the critic NN with an input. Thus, we propose
a forward-propagation-only penalty scheme below.

####  V-B2 Positive increment penalty (type II)

We introduce the Q value increment from states $\mathbf{s}_{i}$ to
$\mathbf{s}^{\prime}_{i,j}$ as

$\mathsf{QI}_{i,j}=Q(\mathbf{s}^{\prime}_{i,j},\mathbf{a}_{i};\bm{\beta})-Q(\mathbf{s}_{i},\mathbf{a}_{i};\bm{\beta}),$
(40)

where $\mathbf{s}^{\prime}_{i,j}$ is identical to $\mathbf{s}_{i}$ except for
the $j$-th state with one-step increment, $s^{\prime}_{i,j}=s_{i,j}+1$. To
enforce the critic NN approximating a monotonically decreasing function from
$\mathbf{s}_{i}$ to $\mathbf{s}^{\prime}_{i,j}$, we define the positive
increment penalty as $\max(0,\mathsf{QI}_{i,j})$.

Potentially, given the vector state $\mathbf{s}_{i}$ and based on the penalty
defined above related to $s_{i,j}$, one can calculate penalty terms for all
$j\in\\{1,\dots,N(M+1)\\}$, leading to high computational cost when $N$ and
$M$ are large. Therefore, we propose a novel scheme to compute the penalty
effectively.

_Stochastic penalty sampling scheme._ First, given the state action pair
$(\mathbf{s}_{i},\mathbf{a}_{i})$, we define an effective set
$\mathcal{J}_{i}\subset\\{1,\dots,N(M+1)\\}$ of the individual states for
penalty calculation. From (39) and (40), a penalty term is effective if
$\mathsf{QD}_{i,j}$ or $\mathsf{QI}_{i,j}$ is positive and the Q value changes
dramatically. From (26) of Theorem 2, channel $h_{n,m}$’s quality improvement
does not change the Q value at all, if device $n$ is not scheduled at channel
$m$. Thus, we can eliminate those cases when calculating the penalty. We
define an indicator on whether a channel state $s_{i,j}$ with $j>N$ should be
considered for penalty evaluation or not, i.e.,

$\\!\\!\\!\\!I(s_{i,j})=\begin{cases}j&\text{\\!$\mathbf{a}_{i}$ utilizes
$s_{i,j}$'s corresponding channel,}\\\
\O&\text{\\!otherwise.}\end{cases}\\!\\!$ (41)

Then, the effective set for penalty calculation is

$\mathcal{J}_{i}\\!=\\!\\{1,\dots,N,I(s_{i,N+1}),\dots,I(s_{i,N(M+1)})\\}.$
(42)

Second, we randomly generate a uniformly down-sampled set
$\bar{\mathcal{J}}_{i}$ with size $K\ll N(M+1)$ based on $\mathcal{J}_{i}$.
From (39) and (40), the loss function of the transition $\mathcal{T}_{i}$
regularized by the penalty terms determined by $\bar{\mathcal{J}}_{i}$ is

$L_{i}(\bm{\beta})\\!=\\!\\!\begin{cases}\\!\mathsf{TD}_{i}^{2}\\!+\\!\sum_{j\in\bar{\mathcal{J}}_{i}}\max\left(0,\mathsf{QD}_{i,j}\right)&{\text{type
I regularized}},\\\
\\!\mathsf{TD}_{i}^{2}\\!+\\!\sum_{j\in\bar{\mathcal{J}}_{i}}\max\left(0,\mathsf{QI}_{i,j}\right)&{\text{type
II regularized}}.\end{cases}$ (43)

A well-trained critic NN $\bm{\beta}$ should minimize both the TD error and
the penalty term in the loss function above.

TABLE I: Summary of Hyperparameters of different DDPG algorithms Hyperparameters | Value
---|---
Mini-batch size, $B$ | 128
Time horizon for each episode | 500
Experience replay memory size, $K$ | 20000
Learning rate of actor network and critic network | 0.0001 and 0.001
Decay rate of learning rate | 0.001
Soft parameter for target update, $\delta$ | 0.005
Input dimension of actor network | $N+N\times M$
Output dimension of actor network | $N$
Input dimension of critic network | $2N+N\times M$
Output dimension of critic network | $1$

##  VI Numerical Experiments

In this section, we evaluate and compare the performance of the three
monotonicity-enforced training methods for DDPG proposed in the previous
section, namely monotonic architecture-based DDPG (MA-DDPG), type I
monotonicity-regularized DDPG (MRI-DDPG), and type II monotonicity-regularized
DDPG (MRII-DDPG).

The computing platform for running the numerical experiments has an Intel Core
i5 9400F CPU @ 2.9 GHz, 16GB RAM, and an NVIDIA RTX 2070 GPU. The experiments
are built on the remote estimation system described in Example 1. In terms of
the system parameters, each process has a two-dimensional state and scalar
measurement, i.e., $l_{n}=2$ and $c_{n}=1$. The spectral radius of
$\mathbf{A}_{n}$ and the entries of $\mathbf{C}_{n}$ are randomly sampled from
the range (1, 1.3) and (0,1), respectively. The channel states are quantized
into $\bar{h}=5$ levels from the Rayleigh fading channel with the scale
parameter drawn uniformly from the range (0.5, 2) [10]. The packet drop
probabilities of different levels are set as 0.2, 0.15, 0.1, 0.05, and 0.01.
The critic NN for a 6-sensor-3-channel system and other cases has 1 hidden
layer and 3 hidden layers, respectively, each with 1024 nodes. The actor NN
for all cases has 3 hidden layers with 1024 nodes. All the NNs are fully
connected NNs. The number of penalty terms evaluated is $K=2$. The setting of
the other hyperparameters for the different DDPG algorithms are shown in Table
I.

TABLE II: Performance Comparison of the Different DDPG Algorithms System Scale $(N,M)$ |  Algorithms |  Training time/episode (second) | Experiment 1 | Experiment 2 | Experiment 3 | Experiment 4 | Experiment 5
---|---|---|---|---|---|---|---
NEC | MSE |  NEC | MSE |  NEC | MSE |  NEC | MSE |  NEC | MSE
$(6,3)$ | DDPG | 3.10 | $-$ | $-$ | $-$ | $-$ | 107 | 133.7559 | $-$ | $-$ | $-$ | $-$
| MA-DDPG | 3.17 | 58 | 50.4761 | 80 | 80.3567 | 30 | 56.3479 | 56 | 66.5171 | 83 | 74.1661
| MRI-DDPG | 4.69 | 74 | 57.3178 | $-$ | $-$ | 82 | 61.4528 | $-$ | $-$ | $-$ | $-$
| MRII-DDPG | 4.01 | 66 | 57.4541 | $-$ | $-$ | 53 | 61.4805 | $-$ | $-$ | $-$ | $-$
$(14,7)$ | DDPG | 6.16 | 189 | 160.6451 | 162 | 136.5146 | $-$ | $-$ | $-$ | $-$ | 132 | 127.8284
| MRI-DDPG | 8.15 | 148 | 122.1874 | 105 | 114.3175 | 204 | 140.6513 | 176 | 207.1918 | 97 | 110.1273
| MRII-DDPG | 7.20 | 87 | 112.3492 | 60 | 108.8440 | 160 | 131.0342 | 113 | 187.2761 | 68 | 103.4623
$(20,10)$ | DDPG | 8.32 | 193 | 210.6116 | 188 | 199.5522 | 213 | 175.2343 | $-$ | $-$ | $-$ | $-$
| MRI-DDPG | 9.83 | 158 | 172.7708 | 163 | 181.6881 | 182 | 144.6849 | 127 | 180.3464 | 273 | 216.3028
| MRII-DDPG | 9.37 | 116 | 155.7731 | 109 | 170.6473 | 131 | 134.4982 | 103 | 167.6880 | 201 | 181.9388

Figs. 2 and 3 show the average sum MSE of all processes during the training
achieved by different DDPG algorithms with $N=6$, $M=3$ and $N=14$, $M=7$,
respectively. Fig. 2 illustrates that the MA-DDPG reduces the average sum MSE
by $15\%$ when compared to the MRI-DDPG and MRII-DDPG, while the baseline DDPG
cannot converge with the shallow critic NN. Thus, for a small-scale system,
MA-DDPG is the most effective algorithm. Fig. 3 shows that MRI-DDPG and MRII-
DDPG save about $15\%$ and $50\%$ training episodes for convergence,
respectively, and also reduce the average sum MSE by $30\%$ when compared to
DDPG. MRII-DDPG performs better than MRI-DDPG, since the state space is
discrete, and MRII-DDPG directly regularizes the Q function monotonicity at
the discrete state space. Note that MA-DDPG cannot converge due to its shallow
NN architecture.

Figure 2: Average sum MSE during training with $N=6,M=3$. The critic NN has
only one hidden layer – a shallow NN setup. Figure 3: Average sum MSE during
training with $N=14,M=7$. The critic NN has three hidden layers – a deep NN
setup.

In Table II, we evaluate the training time, the number of episodes for
convergence (NEC), and the performance of different DDPG algorithms (based on
20000-step simulations) over different system scales and parameters. We see
that for the 20-sensor-10-channel systems, MRII-DDPG can reduce the average
MSE by $27\%$ when compared to DDPG (at the cost of the increased training
time per episode by $13\%$), whilst saving more than $30\%$ time for
convergence (i.e., training time/episode $\times$ NEC), despite the extended
training duration of each episode. In particular, MRII-DDPG solves the
scheduling problem in Experiments 4 and 5 effectively, while the baseline DDPG
fails to converge.

##  VII Conclusion

In this work, building on the monotonicity of the Q function of the optimal
semantic-aware scheduling policy, we have developed monotonicity-driven DRL
algorithms to solve the scheduling problems effectively and reduce the MSE by
about $30\%$ compared to the conventional DRL algorithm. For future work, we
will investigate semantic-aware resource allocation problems over practical
time-correlated fading channels.

## References

  * [1] D. Gündüz, Z. Qin, I. E. Aguerri, H. S. Dhillon, Z. Yang, A. Yener, K. K. Wong, and C.-B. Chae, “Beyond transmitting bits: Context, semantics, and task-oriented communications,” _IEEE J. Sel. Areas Commun._ , vol. 41, no. 1, pp. 5–41, Jan. 2023.
  * [2] J. Chen, J. Wang, C. Jiang, and J. Wang, “Age of incorrect information in semantic communications for NOMA aided XR applications,” _IEEE J. Sel. Top. Signal Process._ , early access, Jun. 2023.
  * [3] M. Kountouris and N. Pappas, “Semantics-empowered communication for networked intelligent systems,” _IEEE Commun. Mag._ , vol. 59, no. 6, pp. 96–102, Jun. 2021.
  * [4] S. Wu, X. Ren, S. Dey, and L. Shi, “Optimal scheduling of multiple sensors over shared channels with packet transmission constraint,” _Automatica_ , vol. 96, pp. 22–31, Oct. 2018.
  * [5] W. Liu, D. E. Quevedo, Y. Li, K. H. Johansson, and B. Vucetic, “Remote state estimation with smart sensors over Markov fading channels,” _IEEE Trans. Autom. Control_ , vol. 67, no. 6, pp. 2743–2757, Jun. 2022.
  * [6] H. Tang, J. Wang, L. Song, and J. Song, “Minimizing age of information with power constraints: Multi-user opportunistic scheduling in multi-state time-varying channels,” _IEEE J. Sel. Areas Commun._ , vol. 38, no. 5, pp. 854–868, May. 2020.
  * [7] Z. Qian, _et al._ , “Minimizing age of information in multi-channel time-sensitive information update systems,” in _IEEE Conf. Comput. Commun. (INFOCOM)_. IEEE, Jul. 2020.
  * [8] J. Tong, L. Fu, and Z. Han, “Age-of-Information oriented scheduling for multichannel IoT systems with correlated sources,” _IEEE Trans. Wirel. Commun._ , vol. 21, no. 11, pp. 9775–9790, Nov. 2022.
  * [9] L. Yang, H. Rao, M. Lin, Y. Xu, and P. Shi, “Optimal sensor scheduling for remote state estimation with limited bandwidth: A deep reinforcement learning approach,” _Inf. Sci._ , vol. 588, pp. 279–292, Apr. 2022.
  * [10] J. Chen, W. Liu, D. E. Quevedo, S. R. Khosravirad, Y. Li, and B. Vucetic, “Structure-enhanced DRL for optimal transmission scheduling,” _accepted by IEEE Trans. Wirel. Commun._ , 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2211.10827
  * [11] M. Akbari, M. R. Abedi, R. Joda, M. Pourghasemian, N. Mokari, and M. Erol-Kantarci, “Age of information aware VNF scheduling in industrial IoT using deep reinforcement learning,” _IEEE J. Sel. Areas Commun._ , vol. 39, no. 8, pp. 2487–2500, Aug. 2021.
  * [12] M. L. Puterman, _Markov decision processes: discrete stochastic dynamic programming_. John Wiley & Sons, 2014\.
  * [13] R. S. Sutton and A. G. Barto, _Reinforcement learning: An introduction_. MIT press, 2018.
  * [14] H. Daniels and M. Velikova, “Monotone and partially monotone neural networks,” _IEEE Trans. Neural Netw._ , vol. 21, no. 6, pp. 906–917, Jun. 2010.
  * [15] S. You, D. Ding, K. Canini, J. Pfeifer, and M. Gupta, “Deep lattice networks and partial monotonic functions,” in _NeurIPS_ , 2017. [Online]. Available: https://doi.org/10.48550/arXiv.1709.06680