# Task-oriented Communication System Design in Cyber-Physical Systems:
A Survey on Theory and Applications
††thanks: The authors are with the Centre for Security Reliability and Trust,
University of Luxembourg, Luxembourg. Emails: {arsham.mostaani, thang.vu,
<EMAIL_ADDRESS>
Arsham Mostaani, , Thang X. Vu, , and Symeon Chatzinotas,

###### Abstract

Communication system design has been traditionally guided by task-agnostic
principles, which aim at efficiently transmitting as many correct bits as
possible through a given channel. However, in the era of cyber-physical
systems, the effectiveness of communications is not dictated simply by the
bitrate, but most importantly by the efficient completion of the task in hand,
e.g., controlling remotely a robot, automating a production line or
collaboratively sensing through a drone swarm. In parallel, it is projected
that by 2023 half of the worldwide network connections will be among machines
rather than humans. In this context, it is crucial to establish a new paradigm
for designing communications strategies for multi-agent cyber-physical
systems. This is a daunting task, since it requires a combination of
principles from information, communication, control theories and computer
science theory in order to formalize a general framework for task-oriented
communication design. In this direction, this paper reviews and structures the
relevant theoretical work across a wide range of scientific communities.
Subsequently, it proposes a general conceptual framework for task-oriented
communication design, along with its specializations according to the targeted
use case. Furthermore , it provides a survey of relevant contributions in
dominant applications, such as of Industrial Internet of Things, multi-UAV
systems, Tactile Internet and Federated Learning. Finally, it highlights the
most important open research topics from both framework and application points
of view.

###### Index Terms:

Task-oriented communications, goal-oriented communications, cyber-physical
systems, machine learning for communication, multiagent systems, reinforcement
learning, Industrial Internet of Things, multi-UAV systems, Tactile Internet ,
Federated Learning.

##  I Introduction

Traditionally, communication system design has been guided by task-agnostic
principles, which aim at efficiently transmitting as many correct bits as
possible through a given channel. The design approaches have been largely
based on information and coding theories, where the former sets the upper
bounds on the system capacity, whereas the latter focuses on achievable
techniques to approach the bounds with infinitesimal error probabilities.
Despite the abstraction level of these theories, they have been successfully
extended to an impressive number of communication network topologies. In this
direction, digital communications have made extraordinary leaps in terms of
performance, allowing robust information transfer even in the face of adverse
channel conditions.

However, in the era of cyber-physical systems, the effectiveness of
communications is not dictated simply by the throughput performance indicators
(e.g. bitrate, latency, jitter, fairness etc.), but most importantly by the
efficient completion of the task in hand, e.g., remotely controlling a robot,
automating a production line, collaboratively sensing/communicating through a
drone swarm etc. It should be noted that according to projections, by 2023
half of the worldwide network connections will be among machines rather than
humans [1]. Machines and its components (e.g. sensors, processors, actuators)
- in contrast to humans - operate based on objective quantifiable processes,
which in theory can be modelled and used as side information during the
communication design process. Moreover, coordination through communication
messages will be imperative in terms of achieving the completion of complex
tasks with the help of multiple agents, be it integrated modules such as
robots, drones, vehicles or individual components thereof. In this future
cyber-physical world, it becomes critical to investigate a new paradigm for
designing communications strategies for multi-agent cyber-physical systems,
which can be adapted or tailored on case by case basis by analyzing jointly
the nature of the targeted collaborative task objective and the constraints of
the underlying communication infrastructure.

Looking back to conventional communication systems, the design of the vast
majority is currently based on the source-channel separation principle, which
suggests that the source information can be compressed independently of the
communication channel and subsequently suitable redundancy should be added to
combat the adversities of the channel itself. However, it should be noted that
this principle only applies under strict conditions [2, 3]. More importantly,
the source compression is based on the statistical properties of the input
distributions, which do not reveal the importance/value of each sample with
respect to the task completion. At the same time, current communication
infrastructure largely depends on the concept of layering, which is meant to
create abstractions which lead to simplified system design. However, the same
abstractions create rigid interfaces that prevent the higher layers (i.e.
application/tasks) from directly affecting/adapting the lower layers of
communication system design.

Incorporating this view of the problem in the design process could be a
daunting task, since it requires a combination of principles from information,
communication, control and theories and computer science theory. In fact, all
of the aforementioned scientific communities have already realised the value
of this new paradigm and have made initial steps to address it from their own
point of view. Nevertheless, a common framework for task-oriented
communication design is still lacking, mainly due to the following challenges:

  * •

Divergence of model assumptions e.g. communication model ( channel, network
topology) , statistical system models (Partially Observable Markov Decision
process, Independent Identically Distributed processes), local vs global
rewards/utility functions

  * •

Divergence of performance metrics e.g. mutual information, discounted
empirical error/risk, error/cost functions within time horizons

  * •

Divergence of mathematical tools e.g. rate-distortion theory, strong-weak
coordination theories, dynamic programming, successive approximation,
stochastic optimization and reinforcement learning.

###  I-A Contributions

In this survey, we make a first step towards highlighting the importance of
the new paradigm and surveying various approaches from the wider scientific
community, which can help towards a common understanding. In this direction,
we formulate a common conceptual framework to clarify and justify the selected
terminology, assumptions and definitions, which will then form the basis for
the general problem description. To validate the framework, we focus on
specific use cases, where we have collected the major literature which can be
studied under the prism of this new paradigm. The end objective is to list a
number of critical open research topics within the proposed framework for
task-oriented communication design along with the necessary tools to derive
near-optimal achievable strategies and policies. Properly addressed, the
implications of these topics can be far reaching across a range of real-world
applications, such as Industrial Internet of Things, multi-UAV systems,
Tactile Internet, and Federated Learning.

###  I-B Organization

The present survey is structured as follows. Firstly, in section II we review
the relevant literature from a theoretical perspective, classifying the
contributions across information/communication, control theories and computer
science theory. Secondly, in section III we introduce a common conceptual
framework along with the corresponding assumptions to clearly delimit the
problems that are addressed within this survey. Subsequently, in section IV,
we focus on concrete application areas by specifying the general framework to
match the underlying system model and we structure the relevant literature.
Last but not least, section V offers a list of open research topics and
challenges pertinent to task-oriented communication system design.

###  I-C Notations and acronyms

Table I: Table of notations Symbol | Meaning
---|---
${\bf x}(t)$ | A generic random variable generated at time $t$
${\mathsf{x}}(t)$ | Realization of ${\bf x}(t)$
$\mathcal{X}$ | Alphabet of x(t)
$|\mathcal{X}|$ | Cardinality of $\mathcal{X}$
$p_{{\bf x}}\big{(}{\mathsf{x}}(t)\big{)}$ | Shorthand for $\mathrm{Pr}\big{(}{\bf x}(t)={\mathsf{x}}(t)\big{)}$
$H\big{(}{\bf x}(t)\big{)}$ | Information entropy of ${\bf x}(t)$ (bits)
$I\big{(}{\bf x}(t);{\bf y}(t)\big{)}$ | Mutual information of ${\bf x}(t)$ and ${\bf y}(t)$
$\mathbb{E}_{p({\mathsf{x}})}\\{{\bf x}\\}$ |  Expectation of the random variable $X$ over the probability distribution $p({\mathsf{x}})$
${\bf x}(t_{1}^{n})$ | Sequence of ${\bf x}(t_{1}),{\bf x}(t_{2}),...,{\bf x}(t_{n})$

The notations used throughout the paper are listed in Table I. In general,
bold font is used for matrices or scalars which are random and their
realizations follows simple font. For ease of presentation, we represent the
sequence $\big{\\{}{\bf x}(t)\big{\\}}_{t=1}^{T}$ as ${\bf x}(t^{T}_{1})$.

##  II Theoretical Concepts for Task-oriented Communication Design

This section focuses on theoretical problems and insights which can have
implications or applications on task-oriented communication design. Although
the lines are often blurry, we classify contributions across three main axes:
information/communication theory, computer science and control theor. Fig. 1
presents a summary of the classification, main theories and selected
references.

Figure 1: Main theories and selected references

###  II-A Information/Communication Theory

Relevant results in Information Theory can be traced all the way back to 1971
[4], where the state of the system as well as the observations of agents are
modelled as information sources with memory and approached through the lens of
rate distortion theory. The paper introduces a quantitative measure to
captures the amount of information that is stored in average in the memory of
the information source. It is then shown that when source coding is designed
subject to any constraint on the distortion, the best achievable rate
(corresponding to that distortion constraint) is lower than the achievable
rate for a similar source which is memoryless. The difference between the
achievable rate for the a source with memory and a similar source without
memory is quantified to be less than or equal to the measure of the memory of
the information source. The results of the paper are general enough to apply
to the sources with finite memory of any size, e.g. Markov sources with memory
size of $L=1$. Due to the wealth of literature, this section focuses mainly on
recent results and surveys, which can be used by the reader to trace back
other useful references.

The concept of common randomness [5] is recurrent through the information-
theoretic approaches relevant to our framework. Even though its applications
are much wider, we focus here on the problem of generating shared uniformly
distributed bits across a network of agents, using initial correlated
observations as side information, complemented with interactive communication.
Common or shared randomness can facilitate the execution of numerous
distributed tasks, such as secret key generation, distributed computation,
channel coding over arbitrary channels, synchronization, consensus, leader
election etc.

Coordination Capacity: In the information theory community, the probing of the
system through different agents is often modelled as a joint distribution of
actions that has to achieved across the agents given constraints on the
communication rates. In this context, the work in [6] introduces the concepts
of empirical and strong coordination. In this work, the concept of common
randomness plays as important role and it is defined as a source of random
samples which is available at multiple nodes even if there is no communication
among them. Empirical coordination is tightly connected with distortion
theory, since the objective it to achieve a desired joint probability
distribution in a set of nodes driven by the communicated random samples
generated in another set of nodes. The authors begin with toy examples of two
or three nodes by examining various topologies such as the cascade and the
broadcast channel. Strong coordination extends the paradigm to temporal
sequences of samples. The authors in [7] introduce another variation, termed
imperfect empirical coordination, aiming to bound the total variation between
the joint type of actions and the desired distribution.

Anytime Capacity: Another notable contribution in [8] focuses on the
intersection of information and control theory, by studying the concept of
anytime capacity. In this case, the aim of communication is specifically
targeted on stabilizing an unstable linear process (e.g. a plant control
loop). While previous works have focused on erasure channels [9], the work in
hand addresses noisy channels. The focus is on a small-scale scalar system
model with a single observer who communicated over a noisy channel with a
single controller. The controller can send both control signals to the system
and feedback to the observer with a one step delay. The main result is the
“equivalence” between stabilization over noisy feedback channels and reliable
communication over noisy channels with feedback. A key point in the model is
that the decoder of the controller has to provide increasingly reliable
estimates for all received past messages, as there is no side information
about the message timing as required by the system. The reliability of the
messages should increase sufficiently rapidly as time evolves to that the
system remains stable.

Information Bottleneck: The information bottleneck [10] is an interesting
construct with hidden implications towards task-oriented communication design.
Let us consider the following formulation which considers three random
variables.

$\displaystyle\underset{{{\bf t}}\in{\mathcal{T}}}{\text{max}}$ $\displaystyle
I({\bf x};{\bf t})$ (1) $\displaystyle\mathtt{s.t.}$ $\displaystyle I({\bf
t};{\bf y})<R.$

The aim of the information bottleneck is to compress the information in ${\bf
y}$ into ${\bf t}$ following the rate constraint $R$, such that ${\bf t}$ can
provide the most useful information about ${\bf x}$ in the mutual information
sense. In this context, it has been recognized that information bottleneck
problem, provides a method to extract the information of ${\bf y}$ which is
most relevant for estimating ${\bf x}$ [11] or to extract the information in
${\bf y}$ which is most relevant in approximating ${\bf x}$. Now let us
consider a toy example of sequential decision making for two agents, where one
receives an observation ${\bf y}$ that has to be, at least partly,
communicated to the other one in order to maximize the expected cumulative
reward. The random variable ${\bf x}$ can be seen as the expected cumulative
reward of the system given action $a$ and the current state of the system,
where the state is jointly defined by $\langle{\bf y},{\bf
y}^{{}^{\prime}}\rangle$. The parameter ${\bf t}$ is the communication message
that the first agent is about to transmit based on its observation ${\bf y}$
to facilitate the control decision $a$ by the agent $j$. Accordingly, by
solving a (conditional) information bottleneck problem, at the side of
encoder, we can optimize the communication of observations ${\bf y}$ of agent
$i$, by compressing them to ${\bf t}$ while ensuring that the compressed
communication message ${\bf t}$ has yet the maximum possible information about
the conditional expected return ${\bf x}$. Fig. 2 illustrates the setting of
this problem. When switch (1) and (2) are both off the problem reduces to a
rate-distortion problem with memory at encoder of the agent $i$. The work done
in [12] was first to show the connection of the information bottleneck problem
and rate-distortion with logarithmic loss. To stay consistent with the
framework that will be proposed in section III, we assume the switch (1) to be
always on. Accordingly, conditioned on the switch (2) being on, the coding in
the encoder of agent $j$ should be done such that it maximizes the conditional
mutual information $I\big{(}{\bf x};{\bf t}|{\bf y}^{{}^{\prime}}\big{)}$. A
solution to the conditional information bottleneck problem is provided by
[13].

Figure 2: Distributed task based source coding

Multi-agent coordination under imperfect observations: The contribution in
[14], [15] lies in the intersection of information and communication theory.
The focus is on multi-agent systems that have to coordinate to maximize their
long-term utility functions based on system observations impaired by an i.i.d.
process. The particular example focuses on distributed power control and the
system state contains the channel gains, which are partially and imperfectly
known to each agent. The authors formalize the optimal problem and provide an
achievable solution based sequential best response dynamics.

###  II-B Control Theory

After the advent of the networked control systems [16, 17], the pioneering
work [18] done by Tatikonda and Mitter, was successful in developing a
cohesive data rate theorem.The paper studies the stability of a control system
under a rate-limited but reliable communication channel. It is assumed that
the rate of communication is time invariable, the state process is a discrete
random variable, and the disturbances that the system is subject to are
bounded. They show that the necessary conditions on the rate of the
communication channel is independent of the algorithms used for encoding and
decoding the communication messages. It has been also shown that these
necessary conditions on the channel rate is also independent from the
information patterns of the networked control system.

There has been a plethora of subsequent works afterwards to extend this
setting, to more realistic scenarios. Authors in paper [19] extended this
framework to cases were the support of system disturbances is unbounded.
Martins accommodated time-varying rates for the communication channel in the
framework, however, their results were limited to first order linear systems
with bounded disturbances [20]. Paper [21] widened the applicability of the
data rate theorem to finite dimensional linear systems with unbounded
disturbance and time varying rate for the communication channel. In contrast
to the previous works [19, 20, 21], Liu in [22] considers the joint effect of
all the two parameters of latency and data rate by finding a region of
stability that indicates the necessary and sufficient values of data-rate as
well as latency. Huang also considered the joint effect of all the three
parameters of latency, reliability and data rate on the control system [23].
It was shown by Kostina that for a fully observable linear system, a lower
bound for the rate-cost function can be computed even when the system
disturbances are not generated by a Gaussian process [24]. The results
provided by their research can even be generalized to partially observed
linear systems, when the observation noise is Gaussian. The rate-cost function
is the minimum required bit-rate that can guarantee the system state to be
upper-bounded by a certain value.

Interested readers can find more details about the solutions proposed to the
control of a linear system over a communication network in the following
surveys and books [25]. While most of the works discussed above consider the
state process to be generated by a linear system with added Gaussian
system/measurement noise, fewer works have targeted the state processes which
are generated by Markov jump linear processes (MJLP) [26, 27, 28, 29] or
Markov Decision Process [30, 31, 32]

State aggregation for dynamic programming has been studied for long among the
communities of control theory and operations research [33, 34, 35, 31].
Successive approximation is one of the main tools leveraged to form a trade-
off between the accuracy of the solution of the dynamic programming problem
and its computational complexity [36, 33]. Later, adaptive algorithms for
state aggregation were proposed which could recompose the aggregation of
states during the iterations [31]. The major driving force for researchers, in
the past, to work on this problem has been to enhance the computational
efficiency of the algorithms. However, the algorithms that they have made
available, can now be used to efficiently represent the state of environment
while minimizing the degradation of the objective function. This area is also
well explored by the community of computer science and is addressed in the
next subsection.

Shafieepoorfard and Raginsky, have studied the problem of controlling a Markov
Decision Process while the agent is subject to observation constraint [37]. In
particular, the observations of the agent here are considered to have a
limited mutual information with the state of the system. Most of the
literature in control theory society either treats a given medium for
observations as a given constraint or considers transmissions of the sensory
system as an additional cost [38]. The work in [37] together with [39] can be
considered among the very first few papers which solve the problem of finding
a stochastic control function in conjunction with the control problem. The
problem of joint design of the observation and control policy is studied in
this paper in its very general form as the policies are not considered as
deterministic but as stochastic functions, where the transitions of the system
are also considered to be stochastic. The problem of one-shot control-
communication policy optimization under mutual information constraint, is
first shown by authors to be a form of rate-distortion problem. Armed with
this analytical result, authors consequently use the Bellman equations to
reformulated the core problem as a one-shot control policy optimization under
mutual information constraint. This particular way that the infinite horizon
control problem under observation constraint is formulated by [37] was first
introduced by Sims in his seminal work [40] which won him a Nobel prize.

In [40] Sims could explain the limited correlation between the behavior of
economic agents and the information they have access to, observing the
information through a rate limited channel. One application for this novel way
of viewing/modeling economic agents is to solve the permanent income problem
where there is limited information about the labour as well as the wealth,
which is a dynamic programming problem with mutual information constraint on
the state information of the system. In [41, 42] the concept of information
constrained dynamic programming is brought into the context of multi-agent
coordination. Agents are limited to local observations but are allowed to
communicate through rate-limited channels. Therein, the authors have developed
a state aggregation algorithm which enables each agent to compress its
generated communication message while maintaining their performance in the
collaborative task. Similarly in [43], authors introduce task-based joint
source channel coding to solve the problem of multiagent coordination over
noisy communication channels. To understand how these works are relevant to
our framework, it should be noticed that, in fact, the information constraint
on the observation of the agents in the aforementioned works can be translated
as the limitation of on the communication channel between the observer of the
environment and the controller.

###  II-C Computer Science

Function approximation has played an essential role in the reinforcement
learning (RL) literature to overcome the limitations of the Q-learning method.
The authors of [44, 45] have built the foundation of function approximation
RL. Therein, the convergence of function approximations which are linearly
combined from basis functions over a state space are established. This result
opens up a wide range applications of RL as it only requires a compact
representation of the cost-to-go function, in which there are fewer number of
parameters than states. In [46], a neural network based on reinforcement
learning, namely neural fitted Q (NFQ), is proposed. NFQ comprises of a multi-
layer perceptron which is able to store and reuse the transition experience.
It is shown therein that NFQ can effectively train a model-free Q-value and
achieve the control policy after few communications rounds. The authors of
[47] consider similar fitted Q-iteration applied to continuous state and
continuous actions batch reinforcement learning. The goal is to achieve a good
policy generalized from a sufficient number of generated trajectories. The
authors have developed first finite-time bound for value-function based
algorithms applied to continuous state and action problems. The authors of
[48] extend the temporal-difference learning proposed in [44] to stochastic
control settings via convergence analysis of several variations of Q-learning
under function approximation. Therein, the condition for the approximate
methods to converge with probability one is identified. The advantages of
approximate RL have been successfully demonstrated in real-world environments
in Atari game in [49, 50]. Therein, the deep Q-network was able learn the
policies directly from the pixels and the game score and outperforms all
existing learning models.

A lossless compression scheme has been proposed by [51, 52, 53] for the
collaborative tasks where the observations of the distributed decision makers
are generated by a (Decentralized) partially observable Markov decision
process (Dec-)POMDP. Authors suggest an optimal clustering scheme to partition
the history of observations of an agent such that no loss in the team
objective occurs. While the main intention of the paper is to reduce the
complexity of distributed computations done at each individual decision maker,
the proposed algorithm can have substantial applications in data compression
when a group of collaborative decision makers intend to communicate through
rate-limited channels. Similar to these works, [41] also proposes k-median
clustering to find a proper abstraction of agents’ observations before they
are communicated to other agents through a communication channel. It is shown
that the problem of agents’ observation abstraction in a multiagent setting
is, in fact, a generalized version of rate-distortion problem.

A general multi-agent framework has been proposed in [54] for mixed
cooperative and competitive multi-agent environments. Moving from the common
assumption in multi-agent studies, which assumes observations and policies of
all agents are available at every agent, this work proposed to learn the
approximation of other agents’ policies via the maximization of the log
probability of the agent. In order to maintain the robustness of the of
proposed policy, the agents employ policy ensembles that are trained for a
wide range of policies. Nevertheless, the developed framework allows to
evaluate the impact of the communication among the agents on the learning
policy by limiting the observed information from other agents.

Efficient communication strategies have been studied in [55, 56] to save the
communications in distributed learning systems. Therein, the authors have
proposed a so-called LASG (Lazily aggregated stochastic gradients), which
adaptively determines when to communicate. The main idea of LASG is based on
the new communication rule that determines the informative content of the
gradients after each round based on the different of the fresh and staled
gradients. By properly implementing the rules, both downlink and uplink load
can be reduces, since only nodes with certain informative update communicate.
It was shown that the proposed LASG therein achieves similar convergence as
the conventional SGD and significantly reduces the communications load.

The authors of [57] proposed a framework for states abstraction via
aggregating original states into abstract states to reduces states and action
spaces. Based on the unified state abstraction framework in [58], the authors
therein proposed and proved the existence of four abstractions approximate
types that guarantee the gap to the reward using the true state value
function. The developed framework therein was tested in five different
problems which shows significant reduction in abstract state space and
suboptimal value function. In [59], a representation learning scheme was
proposed for hierarchical reinforcement learning in which a high-level
controller learns to communicate the goals to the lower-level policy that is
trained to achieve the goals. It was shown that a good choice of presentation
learning policy leads to a bounded suboptimality. Although developed for the
single-agent system, these frameworks are useful to reduce the communication
message in multi-agent networks.

##  III Task-oriented Communication Framework and Scope

As was surveyed in the preceding section, the task-oriented communication
design has been investigated through many different viewpoints. There can be
found many direct/indirect task-oriented design schemes of the communication
system in the literature of computer science, information/communication theory
and control theory. While the theoretical tools that were introduced in
section II, were mostly focused on indirect schemes, the succeeding section
will explore direct schemes. In contrast to indirect schemes, the direct
schemes aim at guaranteeing or improving the performance of the cyber-physical
system at a particular task by designing a tailored communication strategy.
Since direct schemes are specifically designed for a particular task, they can
rarely be generalized to other application scenarios. On the other hand,
direct schemes can take the advantage of the available side knowledge about
the particular task they are meant to facilitate. As an example, it is seen
that one widely used technique to reduce the communication load in distributed
machine learning systems, is for distributed devices to not communicate the
gradients of magnitude below a certain threshold [60, 61, 62]. The distributed
devices can then locally aggregate the gradients which were not communicated
until the magnitude of the aggregated gradients surpass the threshold. This
communication strategy is usually used in federated learning, since the
information that are not communicated (to the server) do not (significantly)
lose their value through time in this application. Moreover, the distributed
devices know what exact process will be done by the parameter server and can
replicate it (at least partly) based on their available local information.
These conditions may not hold in other applications that require low latency
communications or entail sophisticated processes in the central controller.

Therefore, we believe that having a unified problem formulation would allow us
to find commonalities among various task-oriented communication problems, use
the available methods in the literature to solve a wider range of task-
oriented communication problems and analytically distinguish the differences
between them. In this section, we formulate a framework, which is sufficiently
generic to capture various examples of task-oriented communication design, as
will be shown later in Section IV. Furthermore, the proposed framework will
establish a common terminology, clarify the underlying assumptions and delimit
the targeted problem set.

Figure 3: Task-oriented communication design framework.

We start first by explaining the framework of task-oriented communication
presented in Fig. 3. There are $K\geq 2$ agents which interact with the
system. An agent $i$ at any time step $t$ can locally observe the state ${\bf
s}(t)\in\mathcal{S}$ of the system through the observation signal ${\bf
o}_{i}(t)\in\Omega_{i}$. The local observation of the agent is revealed via
the mutual information between the system state and local observation signal,
which is given as

$H\big{(}{\bf s}(t)\big{)}>H\big{(}{\bf o}_{i}(t)\big{)}\geq I\big{(}{\bf
o}_{i}(t);{\bf s}(t)\big{)}>0,$ (2)

where $H\big{(}{\bf x}(t)\big{)}$ is the information entropy of the random
variable ${\bf x}(t)$ (in bits) and $I\big{(}{\bf x}_{i}(t);{\bf y}(t)\big{)}$
the mutual information between ${\bf x}(t)$ and ${\bf y}(t)$ (in bits). In
parallel, an agent $i$ at any time step $t$ can probe the system by sending
the probing signal ${\bf p}_{i}(t)\in\mathcal{P}_{i}$ to the system. As the
result, the agents can affect the overall state of the system and/or the
response signal ${\bf r}(t)$ of it. Each agent $i$ can communicate their
observations to each other at any time step $t$ by sending a communication
message ${\bf c}_{i}(t)\in\mathcal{C}$, where the received message from agent
$j$ is denoted by $\tilde{{\bf c}}_{ji}(t)\in\mathbb{C}$. While $\mathcal{C}$
stands for the alphabet from which an agent can select its communication
message, the set $\mathbb{C}$ is the set of complex numbers as the distorted
version of ${\bf c}_{i}(t)$ can be a complex number.

We do not assume a specific model for the state evolution of the system,
although any task-oriented design of the inter-agent communication requires a
consistent and unique objective function to be in place. The objective
function here is the basis upon which we can measure the performance of the
agent’s collaborative probing of the system and it is assumed to be shown as a
side information based on the targeted task. In some example scenarios, the
objective function can be the accomplishment of an industrial task with enough
precision in an industrial IoT framework, not missing or mistaking the target
within a time horizon in a UAV object tracking framework, or having a minimal
sum of errors throughout a limited time horizon in the central server of a
distributed training system in the framework of federated learning, see
Section IV for further details on these examples.

To delimit the problems that fall under our introduced framework, we set a
list of fundamental assumptions in the following:

  * •

The whole framework consists of at least $k\geq 2$ agents.

  * •

There is a single, common, and consistent single-variate objective function.
The objective function should not vary through the time horizon for which the
problem is solved. The study for competing or non-aligned objectives for
different agents is not considered by our framework.

  * •

We assume that there are $k^{\prime}\geq 1$ number of agents for whom the eq.
(2) holds strictly otherwise there will be no need for communication among
agents e.g., in the case of centralized design, the action selections for all
agents could be carried out locally.

  * •

We assume that the local/global response signal of the system is available to
all agents at no cost. In case that agents have local response signals, we
assume that the global response signal can be represented as a function of the
local responses.

_Remark_ : We do not assume that the objective function is known in its
analytic form by the task-oriented communication designer. Potential side
information about the equations governing the evolution of the system state
and the objective function can be exploited to design task-oriented
communication policies using optimization/dynamic programming techniques.
Nevertheless, even if we only have access to sampled data points of the
objective function, one could resort to machine learning techniques for data-
aided communication policy design. Machine learning techniques can also be
promising to give rather generic solutions to the problem of task-oriented
communication design, since they can generalize over tasks and systems that
are not analytically tractable.

###  III-A Centralized Task-Based Communication Design

The aim of this subsection is to cast the problem of task-oriented
communication design in a centralized manner and to provide some insights
about how it can generalize many interesting communication design problems. To
this end, we start by formulating the probing problem at any time $t_{0}$ as
an exact/empirical risk minimization of the functional

$\displaystyle\mathbb{E}\Big{\\{}\mathcal{J}\Big{(}{\bf
s}(t_{t_{0}-n}^{t_{0}}),{\bf p}_{1}(t_{0}),...,{\bf
p}_{K}(t_{0})\Big{)}\Big{\\}},$ (3)

over all possible functions
$z_{i}:\mathcal{S}^{n+1}\times\mathcal{P}_{i}\rightarrow[0,1]$, for all
$i\in\\{1,2,...,K\\}$, where $n+1$ indicates the order of the memory of the
system, equivalently how many of the past states of the system can be
sufficient to statistically represent it. Each function $z_{i}(\cdot)$
provides a stochastic policy which selects ${\bf
p}_{i}(t_{0})={\mathsf{p}}_{i}\in\mathcal{P}_{i}$ with a certain probability
i.e., $z_{i}\big{(}{\bf s}(t_{t_{0}-n}^{t_{0}}),{\mathsf{p}}_{i}\big{)}$. The
risk function itself can be defined in terms of the system response signal
$r(\cdot):\times_{i=1}^{K}\mathcal{P}_{i}\times\mathcal{S}^{n+1}\rightarrow\mathbb{R}$,
defined as

$\displaystyle\mathcal{J}\Big{(}{\bf s}(t_{{t_{0}}-n}^{t_{0}}),{\bf
p}_{1}({t_{0}}),...,{\bf p}_{K}({t_{0}})\Big{)}=$ (4)
$\displaystyle\sum_{t_{0}=1}^{t_{0}=h}r\Big{(}{\bf
s}(t_{{t_{0}}-n}^{t_{0}}),{\bf p}_{1}({t_{0}}),...,{\bf
p}_{K}({t_{0}})\Big{)},$

where each agent can have access to the global system response signal
$r(\cdot)$ and $h\in(1,\infty]$ is the time horizon over which the risk is
computed. In the case of the access of the agents to the local reward signals
$r_{i}(\cdot):\times_{i=1}^{K}\mathcal{P}_{i}\times\mathcal{S}^{n+1}\rightarrow\mathbb{R}$,
see e.g., references [63, 64] and [65, Sec. II. B. 3], the risk function can
be defined as $\sum_{i=1}^{K}\sum_{t_{0}=1}^{t_{0}=h}r_{i}\Big{(}{\bf
s}(t_{{t_{0}}-n}^{t_{0}}),{\bf p}_{1}({t_{0}}),...,{\bf
p}_{K}({t_{0}})\Big{)}$.

Now let’s assume that each function $z^{*}_{i}\big{(}{\bf
s}(t_{{t_{0}}-n}^{t_{0}})\big{)}:\mathcal{S}^{n+1}\times\mathcal{P}_{i}\rightarrow[0,1]$
for all $i\in\\{1,2,...,K\\}$ is the optimal solution to the problem (3).
Accordingly, the function $\mathcal{J}\Big{(}{\bf
s}(t_{{t_{0}}-n}^{t_{0}}),z^{*}_{1}\big{(}{\bf
s}(t_{{t_{0}}-n}^{t_{0}})\big{)},...,z^{*}_{k}\big{(}{\bf
s}(t_{{t_{0}}-n}^{t_{0}})\big{)}\Big{)}=\mathcal{J}^{*}\big{(}{\bf
s}(t_{{t_{0}}-n}^{{t_{0}}})\big{)}$ yields the optimal expected value of the
risk functional $\mathcal{J}(\cdot)$.

To solve the probing problem, we assume to have access to the global state of
the system. As this assumption might be violated due to the limited rate of
the communication channel and local observability of the agents, we have to
solve this problem with approximated value of the state of the system instead
of the true value. Consequently, the goal is to minimize the cost of the
difference between the optimal risk of the probing problem and the best risk
value attainable for the approximated probing problem. In particular, we can
formulate the joint source channel coding and probing problem at time $t$ as
the following exact/empirical risk minimization:

$\displaystyle{\underset{\zeta,\psi}{\text{min}}}$
$\displaystyle\mathbb{E}\Big{\\{}\\!D\Big{(}\\!\mathcal{J}^{*}\big{(}{\bf
s}(t_{t-n}^{t})\big{)},\ddot{\mathcal{J}}_{\zeta,\psi}\big{(}\hat{{\bf
s}}(t_{t-n}^{t})\big{)}\Big{)}\\!\Big{\\}},$ (5)

where the $\ddot{\mathcal{J}}_{\zeta,\psi}\\!\big{(}\hat{{\bf
s}}(t_{t-n}^{t})\\!\big{)}$ itself is the optimal functional obtained by
solving the variational approximated probing problem, which is stated as
follows:

$\displaystyle\underset{z_{1}(\cdot),...,z_{k}(\cdot)}{\text{min}}$
$\displaystyle\mathbb{E}\Big{\\{}\mathcal{J}\Big{(}\hat{{\bf
s}}(t_{t-n}^{t}),{\bf p}_{1}(t),...,{\bf p}_{K}(t)\Big{)}\Big{\\}},$ (6)

where the tuple $\zeta=\big{(}\zeta_{1}(\cdot),...,\zeta_{k}(\cdot)\big{)}$ is
composed of $k$ functions and for all $i\in\\{1,2,...,k\\}$ the function
${\zeta_{i}(\cdot)\\!:}\ \Omega_{i}\times\mathcal{C}\rightarrow[0,1]$ does the
task of the joint source-channel coding in the transmitter of agent $i$.
Moreover, the function
$\psi(\cdot):\mathbb{C}^{k}\times\mathcal{S}\rightarrow[0,1]$ does the task of
soft decoding in the receiver of the centralized controller and $D(\cdot)$ is
an arbitrary distortion function. Function $D(\cdot)$ measures the cost of
difference between the optimal objective function $\mathcal{J}^{*}\big{(}{\bf
s}(t_{t-n}^{t})\big{)}$ and the functional
$\ddot{\mathcal{J}}_{\zeta,\psi}\big{(}\hat{{\bf s}}(t_{t-n}^{t})\big{)}$
obtained by solving (6). The subscripts $\zeta$ and $\psi$ of
$\ddot{\mathcal{J}}(\cdot)$ present the dependency of the solution to the
problem (6) on the joint source-channel coding and decoding policies since

$\displaystyle\ddot{\mathcal{J}}_{\zeta,\psi}\\!\big{(}\hat{{\bf
s}}(t_{t-n}^{t})\\!\big{)}=\ddot{\mathcal{J}}_{\zeta,\psi}\\!\Big{(}\psi\Big{(}\zeta_{1}\big{(}{{\bf
o}}_{1}(t_{t-n}^{t})\big{)},...,\zeta_{k}\big{(}{{\bf
o}}_{k}(t_{t-n}^{t})\big{)}\Big{)}\\!\Big{)}.$

In fact, finding the optimal answer to the problem (6), in its general
functional form $\ddot{\mathcal{J}}_{\zeta,\psi}\\!\big{(}\hat{{\bf
s}}(t_{t-n}^{t})\\!\big{)}$, for any particular application is not a trivial
task to accomplish. However, if functional form of
$\ddot{\mathcal{J}}_{\zeta,\psi}\\!\big{(}\hat{{\bf
s}}(t_{t-n}^{t})\\!\big{)}$ could be found, we could decouple the control
problem in (6) and the joint source-channel coding problem in (5). As a
result, problem (5) can be solved using the existing techniques for joint
source channel coding [66, 67, 68]. One special case of the problem (5) is
where $\zeta_{i}(\cdot)$ does only the task of source coding. As such, problem
(5) can be easily translated as a generalized form of rate-distortion problem
which can be solved using Blahut-Arimoto Algorithm [69, 70], if the source of
information ${\bf s}(t_{t-n}^{t})$ is memoryless, or using [71, 72, 73] if the
source of information has memory.

The differences between different task-oriented communication problems come
from the variety of factors. The distortion function $D(\cdot)$ and the memory
of the information source to be communicated are among those factors. As
before, the communication channel over which a task-oriented design of the
communication can be carried out is also of the essence. Moreover, the system
model or the process by which state of the system is generated becomes another
reason to have different task-oriented communication problems. Different parts
of the communication stack that can be jointly designed with the probing
problem provide us with another spectrum of task-oriented communication
problems. Hereafter, we will attempt to characterize many of different task-
oriented communication problems, studied in the literature, by specifying the
unique aspects of the problem (1) that they consider. We believe this can be a
step forward to understand what is the underlying structure of all task-
oriented communication problems despite the different look they may have.

Figure 4: Joint source channel coding and decoding and separate computational
unit.

###  III-B Agent Architectures

For a point-to-point communication problem, when there is no constraint on the
delay of the communications, it is proven that the design of source and
channel codes can be performed separately [74, 2] under not very restrictive
conditions [75]. In addition to their provable performance, separately
designed source and channel codes are practically appealing. Accordingly,
separate design of all the blocks of a communication stack, source coding,
channel coding and modulation which is performed independent also from the
design of control policy is what conventional communication systems offer.
Fig. 4 depicts an example architecture of an agent in which the design of the
communication sub-systems as well as the control policy is implemented
separately.

Figure 5: Task-based communication agent with separate computational unit.
Figure 6: Joint computing and communication agent.

It is, however, well-known that the separate design of source and channel
coding does not result in desirable performance over a network of users [76,
77]. Also joint design of channel coder/decoder and modulator/demodulator is
known to obtain much better coding gains while not sacrificing the spectral
efficiency [78, 79, 80]. This encourages the co-design of some communication
sub-systems [78, 79, 80, 81, 82]. Fig. 5 depicts the architecture of an agent
in which the design of all or some of the transceiver sub-systems is done
jointly, yet the design of controller is done separately.

Similarly, the design of the control policy and receiver/transmitter is often
carried out separately [83, Ch. 8.3]. It is shown under some conditions, the
design of transceiver and controller can be decoupled [84], or the separation
of source (de)coding design (for a source generated by a Markov decision
process) from the controller is conditionally possible [41]. However, the
conditions governing the separability of the control policy and the
communication under general conditions is largely unknown. The massive
emerging literature on ”networked control”, as surveyed in section II-B, is
studying the co-design of communication and control policies for a network of
controllers and attempts to measure the effect of communication design on the
stability of the system to be controlled and the performance of the controller
in achieving its task, minimizing the cost function.

(a) Collocated plants and sensors (b) Collocated controller and sensors

Figure 7: An industrial internet of things problem illustrated using the Task-
Based Communication framework.

All the works which have achieved a performance gain by jointly designing the
control policy and a part of the subsystem of communication of a
plant/agent/user/robot/decision maker motivate the joint design of computing
and communication agents. Fig. 6 is the very extreme end of joint design of
communication and control policies. Since the communication stack is composed
of many inter-dependant sub-systems, on both receiving and transmitting ends,
there are numerous ways to jointly/separately design the whole sub-systems of
the stack. Any combination of the communication stack sub-systems and the
control policy can potentially be jointly designed, which constructs a wide
range of possible architectures for an agent. The figures demonstrated in this
subsection are just some of the realizations of these many possible
architectures.

##  IV Applications

In this section, we demonstrate how the proposed framework captures the most
popular applications of task-oriented communication networks, which includes
_Industrial IoTs_ , _UAV Communications Networks_ , _Tactile Internet_ , and
_Federated Learning_.

###  IV-A Industrial IoTs

The block diagram of the task-oriented communications for the industrial IoTs
is depicted in Fig. 7. In this figure, the system response $r(t)$ is the
negative of the stage cost function received after the system is probed by the
plant $i$ through the signal $p_{i}(t)$.

In [64, 85], an industrial IoT system is studied, where a number of plants
e.g., chemical plants or robots, are controlled by a single controlling unit
through slow/fast fading channels. The aim is to reduce the communication
latency to meet the ultra low-latency requirements of industry. The authors
propose a coding-free communication scheme of the central controller with the
plants to optimize a shared cost function among all the plants under a power
constraint at the central controller. A power control algorithm is proposed to
solve the problem. In [64], the global stage cost function is not locally
observable by each plant. Similarly, authors in [63] also consider a
coordination problem where the system response signal is partially observable
by the agents. The authors of [86] consider the implications of the
information rate of a channel on the system stability, where the plants are
collocated with sensors and their up-link channels are lossy. The effect of
time-variant information rate is considered jointly with the potentially
unbounded system disturbances. In particular, the authors have analytically
measured the effect of information rate of the channel on the estimation error
of the system state. This allows to acquire a precise requirement on the
information rate of the plant’s up-link channel depending the magnitude of the
unstable mode of the system. For multidimensional linear systems, the
necessary information rate of the channel is acquired based on the summation
of unstable eigenvalues of the open loop. In [86], however, the joint effect
of communication parameters i.e., latency, reliability and data-rate are not
considered.

The trade-off between the reliability of a communication message and its
latency is studied in [23] for an industrial IoT application. To study this
trade-off, the impact of the age of information that is received by the
plant(s) from the central controller is investigated. This is done by
obtaining the value function of every possible code length through value
iteration algorithm. To be able to run a value iteration algorithm, the
interaction of the central controller with the whole system, including the
plants, is modelled by a semi Markov decision process. Therein, the length of
error correction codes is designed considering the current state of the system
rather than considering just the CSI of the communication channel. In that
sense, the error correction block of the agents are co-designed with their
control policy block.

In [22] the system to be controlled is linear and the up-link channel of the
plant(s) is considered to be noise-free. Whereas, the downlink channel of the
plants is assumed to be an AWGN feedback channel addressing the needs of low-
mobility IIoT applications. The paper obtains a region of stability that
indicates the necessary and sufficient values of communication parameters (i)
information rate as well as (ii) length of error correction code blocks such
that the stability of the system is insured. The paper assumes an ideal
quantization and control policy to be followed across the network.
Accordingly, the quantization as well as the control policy consider the
history of all the communication and feedback signals exchanged between the
plant and the controller. The paper also computes the average lower and upper
bound of the task’s cost function where the bounds are obtained as functions
of communication parameters, information rate and length of error correction
code blocks. The authors of [87] solve a problem very similar to that of [23],
where the difference is that in the former, the up-link channel of the
plant(s) is considered to be noisy.

Although the separability of the estimator of the system’s state from the
controller is studied in a work such as [84], the separability of
communication and control designs remain largely unknown. Authors in [41] have
studied the separability of control design and source coding under mild
conditions. Necessary data rate to guarantee a bounded cost function is
obtained by [19, 18, 88], where communication channels are considered to be
noise-free.

###  IV-B UAV Communications Networks

Unmanned areal vehicles (UAV) play an important role in the development of 5G
and beyond systems due to their flexible and low-cost deployments. The
applications of UAV vary from complementary service to existing infrastructure
to stand-alone service in remote areas or emergency scenarios. Fig. 8 sketches
the diagram of the proposed task-oriented communication framework applied to
the UAV communications networks.

Figure 8: The proposed task-oriented communications framework applied to UAV
networks.

Recently, the adoption of multiple UAVs has received much attention. Compared
to single-UAV system, the main challenge of multi-UAV systems is how to
efficiently coordinate the UAVs’ operations under limited communication
resources. Most of the works on multi-UAV to date, however, assume perfect
communications among the UAVs and only focus on the UAVs’ action policy [89,
90, 91, 92, 93, 94, 95, 96, 97]. In [90], a reinforcement learning-based
sense-and-send framework is proposed for UAV networks. Therein, a BS
communicates with multiple UAVs which sense data and then send it back to the
BS. The objective is to maximize the sensed data sent back to the BS. To
simplify the model, the authors adopt probabilistic sensing model and
orthogonal sub-carriers are assumes, hence there is no interference. The
channel gain is modelled as either LOS and NLOS, depending on BS-UAV distance.
By modeling the system states as MDP, with three steps transmission protocol:
beacon, sensing, and data transmission, a refined action space is proposed to
accelerate the learning performance.

The authors of [91] propose a Deep Q-network in multi-UAV added communications
systems in which multiple UAVs serve ground users. The objective is to
optimize the movements of the UAVs to maximize the system sum-rate, subject to
a constraint on the minimum number of served users. Dueling DQN which uses
neural network and dueling update is employed as the learning solution. In
[92], the authors propose a simultaneous target assignment and path planning
based on multi-agent deep deterministic policy gradient (MADDPG) for multi-UAV
system. The target is to find the shortest path for all the UAV while avoiding
collision of the task assignment. Full knowledge is assumed to be available to
all UAV during the training. The authors of [97] propose a DRL-based algorithm
for multi-UAV networks to optimize the 3D-deployment of the UAVs. The target
is to maximize the quality of experience, defined as a weighted sum of scores
of rate and delay. Q-learning and deep Q-learning is used to model the
problem. However, similar to the previous work, the communication among the
UAVs is not addressed. In [93], the authors study the interference management
of the UAV-aided cellular network, in which the UAVs and the users share the
same frequency bandwidth and talk with the BSs. The objective is to maximize
the average user’s rate via the UAV’s trajectory optimization. The problem is
modeled as a non-cooperative game theory problem with full information. Thus,
the communication among the UAVs is not addressed. The authors of [95]
considers the UAV-aided sense-and-send application in which the UAV, after
sensing the target, compete to access the sub-channel to send the sensed data
to a mobile device. By taking into account all possible UAV’s actions, the
state transition probability is derived, from which a DRL method is proposed
assuming centralized architecture. In [96], the authors study the UAV networks
for maximizing the coverage of an area of interest, dividing into clusters.
Each UAV provides service to one cluster. The goal is to design the UAVs’
navigation policy to maximize the average coverage, as well as fairness among
the clusters. Although modeling the problem as POMDP, it assumes unconstrained
information exchange among the UAVs for free. The authors of [98] investigate
coordinated flight problem in multi-UAV systems in which the UAVs exchange
their local location in order to execute given missions. Therein, an adaptive
binary coding scheme is proposed based on adaptive zooming that adjusts the
quantization level according the moving average of input binary signal.

We note that the works above either do not consider the communications among
the UAVs or consider perfect communications, which is in contrast to practical
scenarios. In [94], the authors propose a RL-based framework for UAV aided
network slicing. Assuming mobile edge computing (MEC) UAV, each UAV serves a
number of tasks in an area of interest. The UAV can choose to perform the jobs
in its region on its own, or offload to a neighboring UAV. Assuming the job
arrival follows the Switched Batch Bernoulli Process, the transition
probability matrix is derived taking into account the states of the region and
queues at the UAVs. Although the communication among the UAVs is considered,
the communication rate is fixed and it does not actively interact with the
learning process.

One of the most important research topics in UAV study is UAV swarms, a
(large) group of UAVs which collaboratively perform some task. The major
challenge in UAV swarms compared to UAV-assisted communication networks lies
in the lack of centralized control due to its very large and highly dynamic
topology. Unlike UAV-aided communications, where the interaction among the
UAVs is aided by a centralized node, e.g., base station, the communication in
UAV swarm is usually done via mesh networks. Due to the large topology, each
UAV can obtain only partial observation of the environment and needs to
cooperate with other UAVs to improve the learning process. An efficient way to
leverage the collaboration in UAV swarms is to employ an interaction graph,
which maintains a set of neighbor nodes for each UAV. This principle is
considered in [99, 100], which study the impact of the communication among the
UAVs to the distributed reinforcement learning task. In particular, they
demonstrate via a rendez-vous problem that allowing more exchanged information
among neighbor UAVs can significantly accelerate the learning process and
results in higher rewards. The authors of [101] analyze the communication
impacts on multi-robot multi-target tracking problem, where each flying robot
can only communicate with its neighbors within its transmission range.
Therein, two learning algorithms are proposed to achieve the agreement among
the robots under limited communication time.

Figure 9: A TPTA system illustrated using the Task-Based Communication
framework.

###  IV-C Tactile Internet

To enhance the degree of immersion of the user in distant communications, it
is known that the communication of haptic information can play a crucial role.
In the well-known scenario of telepresence and teleaction (TPTA), a human user
interacts with a remote environment through: (i) a human system interface,
(ii) a communication link, and (iii) the teleoperator. Such interactions
involve both communications messages and action policies and should be
carefully designed. Fig. 9 presents how the proposed task-oriented
communication framework can be used to model Tactile Internet scenario.

When teleoperation is performed over a communication channel with potential
delays, noise, and uncertainties, it can be shown that achieving good
performance in the teleoperation task can be formulated as a task-oriented
communication problem [102]. Haptic information can be divided in two
different classes. The first class, called kinesthetic information, includes
data related to muscle activation and movements. The second class, called
tactile data, refers to the perception of pressure, texture, and temperature
[103].

Communication of the first category of haptic data, kinesthetic data, involves
the movement of an actuator of the teleoperator in such a way that a
particular task is done with the best possible performance. Some examples for
the tasks that require communication of kinesthetic data can be medical
teleoperations or playing a musical instrument remotely. In the both of the
mentioned examples, achieving a good level of performance in the task is not
equivalent to reducing the distortion between a reference (desired) action
signal and the controlled action process. While a wrong movement of
teleoperator parallel to the surface of a piano keys generates no undesired
musical note, an error in the controlled action process along the axis
vertical to the surface of the piano can generate a wrong musical note. Let us
recall that even wrong movements vertical to surface of the piano, depending
on the location of the actuator and the magnitude of actuation error, may or
may not cause any cost in task. Accordingly, the cost function of the task
cannot be simply characterized by a mean squared error of the action process.
Instead, a task-oriented cost function should be considered to achieve (close
to optimal) performance in the task. Moreover, communication of kinesthetic
data is of one particular sensitivity. Due to the stability requirements of
the control loop on the teleoperator end, signal processing algorithms that
are used at both sending and receiving ends should not cause large algorithmic
delays [104].

On the other hand, task-oriented communication of tactile information, the
second category of haptic information, can also contribute to the improvement
of the system performance. In this case, the cost function of the task is
imposed by the perceptive abilities of an average human. Although there might
be a difference between the communicated tactile signal and the signal
received at the other end of the communication channel, if the difference
between the two signal is beyond the perception of an average human, no cost
should be associated to this communication error [105, 106]. The authors in
[106] utilize the deadband principle to reduce the rate of communication of
haptic information. The deadband principle is understood based on the
perceptive abilities of an average human. According to this principle, the
haptic information $s(t)$ is transmitted only if difference between the last
communicated haptic data $s(t-m)$ and the current available haptic data $s(t)$
is perceptible by the human operator. While it is obvious that widening the
deadband will result in more distortion, this may not affect the performance
of task, which can be measured by the level of preciseness the task is
performed by the user [104]. Accordingly, finding the relationship between the
achieved compression rate of the haptic information and the performance of the
task is considered to be of substantial value [104].

The above-mentioned relationship is studied in [24], under several
assumptions: (i) the uplink channel is noise-free, (ii) the cost function is
considered to be a regularized quadratic function of the error in state of the
system, (iii) the downlink channel noise is an additive white Gaussian noise
and (iv) and the system state is generated by a stochastic linear model.

Figure 10: A Centralized Federated Learning system illustrated using the Task-
Based Communication framework.

###  IV-D Federated Learning

A heuristic is proposed in [60] to reduce the size of communication messages
between the central node and the computing the devices in a distributed
empirical risk minimization task. The task is to optimize a parameterized
model by solving

$\displaystyle\underset{\theta}{\text{argmin}}$
$\displaystyle\frac{1}{n}\sum_{i=1}^{n}l\big{(}\theta,X_{i}\big{)},$ (7)

where $l\big{(}\theta,X_{i}\big{)}$ captures the loss caused at the data point
$X_{i}$ when the the model parameters are chosen to be $\theta$, index of
$X_{i}$ shows a particular training data. The task is done by applying
stochastic gradient descent (SGD) distributively where the gradient of the
cost function at each node $k$ and iteration $t$, given as

$g_{k,t}=\frac{1}{B_{k}}\sum_{i=1}^{B_{k}}\nabla
l\big{(}\theta_{t},X_{i,k}\big{)},$ (8)

which is computed according to the available local training data
$\\{X_{1,k},...,X_{B_{k},k}\\}$ at each node. The nodes, then can communicate
their computed gradients to the master node where the (stochastic) gradient
can be computed. Since the communication between distributed computing nodes
is seen to be a major bottleneck, authors in [60], together with many similar
works [107, 108, 109, 110, 111, 112], introduce a scheme to reduce the size of
communication messages as to enhance the speed of convergence as well as to
overcome communication bandwidth constraints. All these papers are about to
solve very similar problems which can be generalized as

$\displaystyle\underset{Q(x)}{\text{min}}$
$\displaystyle\mathbb{E}_{p(y,x)}\Big{\\{}\mathcal{L}(X_{i},Y_{i})\Big{\\}}$
(9) $\displaystyle\mathtt{s.t.}$ $\displaystyle I\big{(}Y_{i};X_{i}\big{)},$

where $y=Q(x)$ and $Q(x)$ is a quantization function that that defines
equivalence relations on $Supp(X)$ and in this special case, the cost function
$\mathcal{L}(\cdot)$ can be expressed as
$\mathcal{L}_{y|x}(X,Y)=\left\lVert\sum_{i=1}^{n}Y-X\right\rVert_{2}$. The
solution to problem (9) under limited rate of communication channels is
presented in the Appendix. Following the proposed scheme, authors in [60]
report a significant compression ratio while virtually no loss is seen in
optimizing the cost function. The scheme proposed by [60] first finds a
threshold for the magnitude of the gradient vectors, above which the computed
gradient of a node will be considered for communication to the master node.
The gradient vectors with lower size than the threshold, however, will remain
in the node and will be accumulated with the rest of gradients that
will/already have computed by the same node and have not be qualified for
transmission. While in [107, 108, 109], the communication channels are
considered to be rate-limited but error-free, works done in [110, 111, 112]
(partially) consider the effects of the physical layer features of the
communication links on the problem of federated learning.

##  V Challenges and Open Problems

###  V-A Challenges of the Framework

The current framework assumes the availability of the common reward at all
agents at no cost. In practice, the reward has to be send to the agents via
(wireless) communication channels. Therefore, the distribution of rewards
among the agents must be taken into account when designing the communication
strategy in such cases. Another issue is how the training is done. It is well
known that centralized training can provide optimal solution for the multi-
agent system. However, the centralized training architecture is not always
available, especially in dynamic multi-agent systems. This asks for efficient
distributed training design, in which the communications among the agents
during the training phase is crucial and needs to be properly designed. So
far, the communication is optimized based on the codebook and resource
allocation philosophy. While this method is efficient for point-to-point and
small networks, it is difficult to generalize to massive networks or takes
into account latency and privacy constrained aspects. Another challenge in
designing task-oriented communication system is how to properly take into
consideration different reward functions for different agents. In an extreme,
the agents can be competitive rather than collaborative [113]. In these cases,
the communications should be adapted to specify individual target, which in
turns affects both action and communication policies of the agents.

###  V-B Industrial Internet-of-Things Networks

The agents in IIoT systems differ from other applications in two main
features: light computational capability and limited energy budget.
Consequently, energy-efficient communications will be the first design
criteria for IIoT applications. Meanwhile, IIoT systems usually demand timely
decision-making actions and consequently stringent latency, which contradict
to the energy-efficiency target. This trade-off between energy efficiency and
reliability asks for new design method for IIoT networks. Ultra-reliable low
latency communications (URLLC) is a promising solution for designing the
communications in multi-agent IIoT systems. However, the current URLLC design
does not take into account the requirements for the learning process.
Therefore, the URLLC property should be jointly considered with the action
policies with regard to the energy can computation capacities of IIoT nodes.

###  V-C UAV Communications Networks

UAV communications play an essential role in the future communications
networks as not only a stand-alone system in dedicated areas but also
complementary parts of the cellular networks. More specifically, multiple UAVs
can cooperate to provide communications in isolated areas for rescuing or
sensing purposes, or they can aid the macro BSs to enhance handover or provide
ultra-reliable communications to ground users. In either cases, reliable
communications among the UAVs is key to improve the overall performance of the
UAV networks. One challenge in UAV communications networks is how to jointly
design the trajectory of every UAV to reduce collision risk and improve the
system energy efficiency. However, because the UAVs usually operate in dynamic
environment, conventional methods may not be applied due to the lack of proper
system modeling. In fact, the UAVs’ optimal trajectories are difficult, and
sometimes unable to obtain as they depend on the movements and actions of all
the UAVs. Therefore, one efficient way is that the UAV optimizes its
trajectory and action policy while listening to the others, which requires
efficient communications among the UAVs. Efficient communications should cover
both how and what to communicate. On one hand, the UAVs should have to be well
coordinated in accessing the common channel to avoid interference. On the
other hand, each UAV has to determine what message to communicate to other
UAVs in order to maximize the UAVs’ collaboration. As a result, this asks for
a novel design paradigm that optimizes the communications based on specific
semantics.

###  V-D Federated Learning

One of issues in FL is the uneven distribution of data among the edge nodes.
Because the edge nodes. Because the update at the edge nodes are based on
their locally available data, the contribution to the aggregated system
parameters varies from one edge node to others. As a result, always-transmit
policy is no longer the optimal transmission in FL. In fact, a node which does
not have sufficient data usually generates bias gradient parameters. Sending
these parameters to the server does not improve, and sometimes harms the
learning process. This asks for a context-aware transmission policy to tackle
this issue. It is shown in [114] that a proper transmission policy can improve
the FL in terms of both energy efficiency and convergence performance. Another
issue in FL is the different privacy levels among the edge nodes, which
requires unequal-protection transmission and coding designs. Optimizing the
source-channel coding to satisfy the privacy requirements and improve the
learning convergence is usually difficult and requires novel system design
perspectives.

###  V-E Mobile Edge Computing

Mobile edge computing (MEC) will be a key component in the 5GB architecture to
implement the intelligence on the network edge. By being equipped with both
computational and storage capability, MEC nodes are able to determine to
perform requested tasks locally or offloading them to the cloud server. Most
of existing works consider single MEC node that can optimally make task
offloading decision. When applying to multiple cooperative MEC nodes, such
methods, however, no longer render optimal policies. This is because in
multiple MEC agents context, one node’s action can have affects on other
nodes. Furthermore, as the MEC nodes usually share the same communication
medium, e.g., channel bandwidth, it requires proper transmission design to
efficiently mitigate interference among the MEC agents. The major challenge is
how to jointly design the MEC’s local action policy with communications policy
to balance the exploration-exploitation tradeoff. The authors of [115] has
shown the potential of such joint design for IIoT systems, in which edge-
device acts as a machine-type agent (MTA). The MTAs collaboratively learn
optimal policy for channel access and task offloading in multi sub-carrier D2D
environment. At every time slot, each MTA determines to compute the task
locally or off-loads the task to the MEC server. By modeling the state space
including off-loading decisions, channel access status and computation task,
the authors proposed to use multiagent deep deterministic policy gradients
algorithm on actor-critic network at each MTA. then the MTAs exchange their
locally trained model to generalize the global model. The benefit of
multiagent MEC system presented in [115] is based on an assumption that all
the edge nodes can perfectly communicate to each other during the training
phase. In many practical cases, such assumption rarely occurs due to imperfect
communications among the edge nodes and the highly dynamic network topology.
This asks for novel distributed designs of action and communication policies.

##  VI Conclusion

Task-oriented communication has been considered to be a new paradigm for
designing communications strategies for multi-agent cyber-physical systems. In
this article, we have presented a comprehensive review and structures of the
theoretical work across a wide range of research communities. We have then
proposed a general conceptual framework for designing a task-oriented system
and specialized it for the targeted use cases. Furthermore, we have provided a
survey of relevant contributions in four major applications, such as
industrial internet of things, multi-UAV systems, tactile internet and
federated learning. Finally, we have discussed challenges and open issues in
the task-oriented communications design.

## References

  * [1] U. Cisco, “Cisco annual internet report (2018–2023) white paper,” 2020.
  * [2] R. Dobrushin, “General formulation of shannon’s basic theorems of information theory,” _AMS Translations_ , vol. 33, pp. 323–438, 1959.
  * [3] S. Vembu, S. Verdu, and Y. Steinberg, “The source-channel separation theorem revisited,” _IEEE Transactions on Information Theory_ , vol. 41, no. 1, pp. 44–54, 1995.
  * [4] A. Wyner and J. Ziv, “Bounds on the rate-distortion function for stationary sources with memory,” _IEEE Transactions on Information Theory_ , vol. 17, no. 5, pp. 508–513, 1971.
  * [5] M. Sudan, H. Tyagi, and S. Watanabe, “Communication for generating correlation: A unifying survey,” _IEEE Transactions on Information Theory_ , vol. 66, no. 1, pp. 5–37, 2019.
  * [6] P. W. Cuff, H. H. Permuter, and T. M. Cover, “Coordination capacity,” _IEEE Transactions on Information Theory_ , vol. 56, no. 9, pp. 4181–4206, 2010.
  * [7] M. Mylonakis, P. A. Stavrou, and M. Skoglund, “Empirical coordination subject to a fidelity criterion,” in _2019 IEEE Information Theory Workshop (ITW)_. IEEE, 2019, pp. 1–5.
  * [8] A. Sahai and S. Mitter, “The necessity and sufficiency of anytime capacity for stabilization of a linear system over a noisy communication link—part i: Scalar systems,” _IEEE transactions on Information Theory_ , vol. 52, no. 8, pp. 3369–3395, 2006.
  * [9] N. Martins, M. Dahleh, and N. Elia, “Stabilization of uncertain systems in the presence of a stochastic digital link,” _IEEE Trans. Automat. Contr_ , 2004\.
  * [10] R. Gilad-Bachrach, A. Navot, and N. Tishby, “An information theoretic tradeoff between complexity and accuracy,” in _Learning Theory and Kernel Machines_. Springer, 2003, pp. 595–609.
  * [11] R. A. Amjad and B. C. Geiger, “Learning representations for neural network-based classification using the information bottleneck principle,” _IEEE transactions on pattern analysis and machine intelligence_ , 2019.
  * [12] P. Harremoës and N. Tishby, “The information bottleneck revisited or how to choose a good distortion measure,” in _2007 IEEE International Symposium on Information Theory_. IEEE, 2007, pp. 566–570.
  * [13] D. Gondek and T. Hofmann, “Conditional information bottleneck clustering,” in _3rd ieee international conference on data mining, workshop on clustering large data sets_. Citeseer, 2003, pp. 36–42.
  * [14] B. Larrousse, S. Lasaulce, and M. R. Bloch, “Coordination in distributed networks via coded actions with application to power control,” _IEEE Trans. on Information Theory_ , vol. 64, no. 5, pp. 3633–3654, 2018.
  * [15] S. Lasaulce and S. Tarbouriech, “Information constraints in multiple agent problems with iid states,” in _Control subject to computational and communication constraints_. Springer, 2018, pp. 311–323.
  * [16] J. W. Overstreet and A. Tzes, “An internet-based real-time control engineering laboratory,” _IEEE Control Systems Magazine_ , vol. 19, no. 5, pp. 19–34, 1999.
  * [17] B. Aktan, C. A. Bohus, L. A. Crowl, and M. H. Shor, “Distance learning applied to control engineering laboratories,” _IEEE Transactions on education_ , vol. 39, no. 3, pp. 320–326, 1996.
  * [18] S. Tatikonda and S. Mitter, “Control under communication constraints,” _IEEE Transactions on automatic control_ , vol. 49, no. 7, pp. 1056–1068, 2004.
  * [19] G. N. Nair and R. J. Evans, “Stabilizability of stochastic linear systems with finite feedback data rates,” _SIAM Journal on Control and Optimization_ , vol. 43, no. 2, pp. 413–436, 2004.
  * [20] N. C. Martins, M. A. Dahleh, and N. Elia, “Feedback stabilization of uncertain systems in the presence of a direct link,” _IEEE Transactions on Automatic Control_ , vol. 51, no. 3, pp. 438–447, 2006.
  * [21] P. Minero, M. Franceschetti, S. Dey, and G. Nair, “Data rate theorem for stabilization over fading channels,” in _Proc. 45th Ann. Allerton Conf. on Communic., Control and Comput_ , 2007.
  * [22] W. Liu, G. Nair, Y. Li, D. Nesic, B. Vucetic, and H. V. Poor, “On the latency, rate and reliability tradeoff in wireless networked control systems for iiot,” _IEEE Internet of Things Journal_ , 2020.
  * [23] K. Huang, W. Liu, Y. Li, A. Savkin, and B. Vucetic, “Wireless feedback control with variable packet length for industrial iot,” _IEEE Wireless Communications Letters_ , 2020.
  * [24] V. Kostina and B. Hassibi, “Rate-cost tradeoffs in control,” _IEEE Transactions on Automatic Control_ , vol. 64, no. 11, pp. 4525–4540, 2019.
  * [25] A. S. Matveev and A. V. Savkin, _Estimation and control over communication networks_. Springer Science & Business Media, 2009.
  * [26] G. N. Nair, S. Dey, and R. J. Evans, “Infimum data rates for stabilising markov jump linear systems,” in _42nd IEEE International Conference on Decision and Control (IEEE Cat. No. 03CH37475)_ , vol. 2. IEEE, 2003, pp. 1176–1181.
  * [27] Y. Song, J. Yang, M. Zheng, and C. Peng, “Disturbance attenuation for markov jump linear system over an additive white gaussian noise channel,” _International Journal of Control_ , vol. 89, no. 12, pp. 2482–2491, 2016\.
  * [28] H. Zhenglong, S. Yang, Z. Min, J. Li, and T. Yang, “Stabilization of markov jump linear systems over gaussian relay channel,” in _2016 UKACC 11th International Conference on Control (CONTROL)_. IEEE, pp. 1–6.
  * [29] C. Zhang, K. Chen, and G. E. Dullerud, “Stabilization of markovian jump linear systems with limited information-a convex approach,” in _2009 American Control Conference_. IEEE, 2009, pp. 4013–4019.
  * [30] V. S. Borkar, S. K. Mitter, and S. Tatikonda, “Optimal sequential vector quantization of markov sources,” _SIAM journal on control and optimization_ , vol. 40, no. 1, pp. 135–148, 2001.
  * [31] D. P. Bertsekas and D. A. Castanon, “Adaptive aggregation methods for infinite horizon dynamic programming,” _IEEE Transactions on Automatic Control_ , vol. 34, no. 6, pp. 589–598, June 1989.
  * [32] D. Bertsekas, “Biased aggregation, rollout, and enhanced policy improvement for reinforcement learning,” _arXiv preprint arXiv:1910.02426_ , 2019.
  * [33] M. L. Puterman and M. C. Shin, “Modified policy iteration algorithms for discounted markov decision problems,” _Management Science_ , vol. 24, no. 11, pp. 1127–1137, 1978.
  * [34] M. Puterman and M. C. Shin, “Action elimination procedures for modified policy iteration algorithms,” _Operations Research_ , vol. 30, no. 2, pp. 301–318, 1982.
  * [35] M. Voelkel, A.-L. Sachs, and U. W. Thonemann, “An aggregation-based approximate dynamic programming approach for the periodic review model with random yield,” _European Journal of Operational Research_ , vol. 281, no. 2, pp. 286–298, 2020.
  * [36] F. Chatelin and W. L. Miranker, “Acceleration by aggregation of successive approximation methods,” _Linear Algebra and its Applications_ , vol. 43, pp. 17–47, 1982.
  * [37] E. Shafieepoorfard, M. Raginsky, and S. P. Meyn, “Rationally inattentive control of markov processes,” _SIAM Journal on Control and Optimization_ , vol. 54, no. 2, pp. 987–1016, 2016.
  * [38] D. Maity, M. H. Mamduhi, S. Hirche, K. H. Johansson, and J. S. Baras, “Optimal lqg control under delay-dependent costly information,” _IEEE control systems letters_ , vol. 3, no. 1, pp. 102–107, 2018.
  * [39] S. Yüksel and T. Linder, “Optimization and convergence of observation channels in stochastic control,” _SIAM Journal on Control and Optimization_ , vol. 50, no. 2, pp. 864–887, 2012.
  * [40] C. A. Sims, “Implications of rational inattention,” _Journal of monetary Economics_ , vol. 50, no. 3, pp. 665–690, 2003.
  * [41] A. Mostaani, T. X. Vu, S. Chatzinotas, and B. Ottersten, “Task-based information compression for multi-agent communication problems with channel rate constraints,” _arXiv preprint arXiv:2005.14220_ , 2020.
  * [42] A. Mostaani, T. X. Vu, S. Chatzinotas, and B. Ottersten, “State aggregation for multiagent communication over rate-limited channels,” in _GLOBECOM 2020-2020 IEEE Global Communications Conference_. IEEE, 2020, pp. 1–7.
  * [43] A. Mostaani, O. Simeone, S. Chatzinotas, and B. Ottersten, “Learning-based physical layer communications for multiagent collaboration,” in _2019 IEEE Intl. Symp. on Personal, Indoor and Mobile Radio Communications_ , Sep. 2019.
  * [44] J. N. Tsitsiklis and B. Van Roy, “An analysis of temporal-difference learning with function approximation,” _IEEE transactions on automatic control_ , vol. 42, no. 5, pp. 674–690, 1997.
  * [45] D. P. Bertsekas and J. N. Tsitsiklis, “Neuro-dynamic programming: an overview,” in _Proceedings of 1995 34th IEEE Conference on Decision and Control_ , vol. 1. IEEE, 1995, pp. 560–564.
  * [46] M. Riedmiller, “Neural fitted q iteration–first experiences with a data efficient neural reinforcement learning method,” in _European Conference on Machine Learning_. Springer, 2005, pp. 317–328.
  * [47] A. Antos, C. Szepesvári, and R. Munos, “Fitted q-iteration in continuous action-space mdps,” in _Advances in neural information processing systems_ , 2008, pp. 9–16.
  * [48] F. S. Melo, S. P. Meyn, and M. I. Ribeiro, “An analysis of reinforcement learning with function approximation,” in _Proceedings of the 25th international conference on Machine learning_ , 2008, pp. 664–671.
  * [49] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” _arXiv preprint arXiv:1312.5602_ , 2013.
  * [50] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski _et al._ , “Human-level control through deep reinforcement learning,” _nature_ , vol. 518, no. 7540, pp. 529–533, 2015.
  * [51] F. A. Oliehoek, M. T. Spaan, and N. Vlassis, “Optimal and approximate q-value functions for decentralized pomdps,” _Journal of Artificial Intelligence Research_ , vol. 32, pp. 289–353, 2008.
  * [52] F. A. Oliehoek and N. Vlassis, “Q-value functions for decentralized pomdps,” in _Proceedings of the 6th international joint conference on Autonomous agents and multiagent systems_ , 2007, pp. 1–8.
  * [53] F. A. Oliehoek, S. Whiteson, M. T. Spaan _et al._ , “Lossless clustering of histories in decentralized pomdps.” in _AAMAS (1)_ , 2009, pp. 577–584.
  * [54] R. Lowe, Y. Wu, A. Tamar, J. Harb, O. P. Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” in _Proc. Advances in Neural Information Processing Systems_ , Long Beach, 2017, pp. 6382–6393.
  * [55] T. Chen, Y. Sun, and W. Yin, “LASG: Lazily Aggregated Stochastic Gradients for Communication-Efficient Distributed Learning,” _arXiv e-prints_ , p. arXiv:2002.11360, Feb. 2020.
  * [56] Y. Liu, Y. Sun, and W. Yin, “Decentralized learning with lazy and approximate dual gradients,” 2020.
  * [57] D. Abel, D. Hershkowitz, and M. Littman, “Near optimal behavior via approximate state abstraction,” ser. Proceedings of Machine Learning Research, M. F. Balcan and K. Q. Weinberger, Eds., vol. 48, 2016, pp. 2915–2923.
  * [58] L. Li, T. J. Walsh, and M. L. Littman, “Towards a unified theory of state abstraction for mdps,” in _International Symposium on Artificial Intelligence and Mathematics, ISAIM 2006, Fort Lauderdale, Florida, USA, January 4-6, 2006_ , 2006. [Online]. Available: http://anytime.cs.umass.edu/aimath06/proceedings/P21.pdf
  * [59] O. Nachum, S. Gu, H. Lee, and S. Levine, “Near-optimal representation learning for hierarchical reinforcement learning,” _arXiv preprint arXiv:1810.01257_ , 2018.
  * [60] Y. Lin, S. Han, H. Mao, Y. Wang, and W. J. Dally, “Deep gradient compression: Reducing the communication bandwidth for distributed training,” _arXiv preprint arXiv:1712.01887_ , 2017.
  * [61] N. Strom, “Scalable distributed dnn training using commodity gpu cloud computing,” in _Sixteenth Annual Conference of the International Speech Communication Association_ , 2015.
  * [62] A. F. Aji and K. Heafield, “Sparse communication for distributed gradient descent,” _arXiv preprint arXiv:1704.05021_ , 2017.
  * [63] S. Kar, J. M. F. Moura, and H. V. Poor, “${{\cal Q}{\cal D}}$-learning: A collaborative distributed strategy for multi-agent reinforcement learning through ${{\mathsf{m}}consensus}+{{\mathsf{m}}innovations}$,” _IEEE Transactions on Signal Processing_ , vol. 61, no. 7, pp. 1848–1862, April 2013.
  * [64] W. Liu, P. Popovski, Y. Li, and B. Vucetic, “Wireless networked control systems with coding-free data transmission for industrial iot,” _IEEE Internet of Things Journal_ , vol. 7, no. 3, pp. 1788–1801, 2020.
  * [65] A. Feriani and E. Hossain, “Single and multi-agent deep reinforcement learning for ai-enabled wireless networks: A tutorial,” _arXiv preprint arXiv:2011.03615_ , 2020.
  * [66] V. Kostina and S. Verdú, “Lossy joint source-channel coding in the finite blocklength regime,” _IEEE Transactions on Information Theory_ , vol. 59, no. 5, pp. 2545–2575, 2013.
  * [67] M. Grangetto, P. Cosman, and G. Olmo, “Joint source/channel coding and map decoding of arithmetic codes,” _IEEE Transactions on Communications_ , vol. 53, no. 6, pp. 1007–1016, 2005.
  * [68] M. Fresia, F. Perez-Cruz, H. V. Poor, and S. Verdu, “Joint source and channel coding,” _IEEE Signal Processing Magazine_ , vol. 27, no. 6, pp. 104–113, 2010.
  * [69] S. Arimoto, “An algorithm for computing the capacity of arbitrary discrete memoryless channels,” _IEEE Transactions on Information Theory_ , vol. 18, no. 1, pp. 14–20, January 1972.
  * [70] R. Blahut, “Computation of channel capacity and rate-distortion functions,” _IEEE Transactions on Information Theory_ , vol. 18, no. 4, pp. 460–473, 1972\.
  * [71] H. Witsenhausen, “On the structure of real-time source coders,” _Bell System Technical Journal_ , vol. 58, no. 6, pp. 1437–1451, 1979.
  * [72] T. Linder and S. Yüksel, “On optimal zero-delay coding of vector markov sources,” _IEEE Transactions on Information Theory_ , vol. 60, no. 10, pp. 5975–5991, 2014.
  * [73] R. G. Wood, T. Linder, and S. Yüksel, “Optimal zero delay coding of markov sources: Stationary and finite memory codes,” _IEEE Transactions on Information Theory_ , vol. 63, no. 9, pp. 5968–5980, 2017.
  * [74] C. E. Shannon, “A mathematical theory of communication,” _The Bell system technical journal_ , vol. 27, no. 3, pp. 379–423, 1948.
  * [75] S. Vembu, S. Verdu, and Y. Steinberg, “The source-channel separation theorem revisited,” _IEEE Transactions on Information Theory_ , vol. 41, no. 1, pp. 44–54, 1995.
  * [76] M. Gastpar, B. Rimoldi, and M. Vetterli, “To code, or not to code: Lossy source-channel communication revisited,” _IEEE Transactions on Information Theory_ , vol. 49, no. 5, pp. 1147–1158, 2003.
  * [77] T. M. Cover, _Elements of information theory_. John Wiley & Sons, 1999.
  * [78] G. Caire, G. Taricco, and E. Biglieri, “Bit-interleaved coded modulation,” _IEEE transactions on information theory_ , vol. 44, no. 3, pp. 927–946, 1998\.
  * [79] A. J. Goldsmith and S.-G. Chua, “Adaptive coded modulation for fading channels,” _IEEE Transactions on communications_ , vol. 46, no. 5, pp. 595–602, 1998.
  * [80] T. Fehenberger, G. Böcherer, A. Alvarado, and N. Hanik, “Ldpc coded modulation with probabilistic shaping for optical fiber systems,” in _Optical Fiber Communication Conference_. Optical Society of America, 2015, pp. Th2A–23.
  * [81] V. Kostina and S. Verdú, “Lossy joint source-channel coding in the finite blocklength regime,” _IEEE Transactions on Information Theory_ , vol. 59, no. 5, pp. 2545–2575, 2013.
  * [82] J. L. Massey, “Joint source and channel coding,” MASSACHUSETTS INST OF TECH CAMBRIDGE ELECTRONIC SYSTEMS LAB, Tech. Rep., 1977.
  * [83] F. A. Oliehoek, C. Amato _et al._ , _A concise introduction to decentralized POMDPs_. Springer, 2016, vol. 1.
  * [84] L. Schenato, B. Sinopoli, M. Franceschetti, K. Poolla, and S. S. Sastry, “Foundations of control and estimation over lossy networks,” _Proceedings of the IEEE_ , vol. 95, no. 1, pp. 163–187, 2007.
  * [85] W. Liu, P. Popovski, Y. Li, and B. Vucetic, “Real-time wireless networked control systems with coding-free data transmission,” in _2019 IEEE Global Communications Conference (GLOBECOM)_. IEEE, 2019, pp. 1–6.
  * [86] P. Minero, M. Franceschetti, S. Dey, and G. N. Nair, “Data rate theorem for stabilization over time-varying feedback channels,” _IEEE Transactions on Automatic Control_ , vol. 54, no. 2, pp. 243–255, 2009.
  * [87] M. Pezzutto, F. Tramarin, S. Dey, and L. Schenato, “Adaptive transmission rate for lqg control over wi-fi: A cross-layer approach,” _Automatica_ , vol. 119, p. 109092, 2020.
  * [88] A. S. Matveev and A. V. Savkin, “The problem of lqg optimal control via a limited capacity communication channel,” _Systems & control letters_, vol. 53, no. 1, pp. 51–64, 2004.
  * [89] J. Cui, Y. Liu, and A. Nallanathan, “Multi-agent reinforcement learning-based resource allocation for uav networks,” _IEEE Transactions on Wireless Communications_ , vol. 19, no. 2, pp. 729–743, 2020.
  * [90] J. Hu, H. Zhang, and L. Song, “Reinforcement learning for decentralized trajectory design in cellular uav networks with sense-and-send protocol,” _IEEE Internet of Things Journal_ , vol. 6, no. 4, pp. 6177–6189, 2019.
  * [91] Q. Wang, W. Zhang, Y. Liu, and Y. Liu, “Multi-uav dynamic wireless networking with deep reinforcement learning,” _IEEE Communications Letters_ , vol. 23, no. 12, pp. 2243–2246, 2019.
  * [92] H. Qie, D. Shi, T. Shen, X. Xu, Y. Li, and L. Wang, “Joint optimization of multi-uav target assignment and path planning based on multi-agent reinforcement learning,” _IEEE Access_ , vol. 7, pp. 146 264–146 272, 2019.
  * [93] S. Chai and V. K. N. Lau, “Online trajectory and radio resource optimization of cache-enabled uav wireless networks with content and energy recharging,” _IEEE Transactions on Signal Processing_ , vol. 68, pp. 1286–1299, 2020.
  * [94] G. Faraci, C. Grasso, and G. Schembra, “Design of a 5g network slice extension with mec uavs managed with reinforcement learning,” _IEEE Journal on Selected Areas in Communications_ , pp. 1–1, 2020.
  * [95] F. Wu, H. Zhang, J. Wu, and L. Song, “Cellular uav-to-device communications: Trajectory design and mode selection by multi-agent deep reinforcement learning,” _IEEE Transactions on Communications_ , pp. 1–1, 2020.
  * [96] C. H. Liu, X. Ma, X. Gao, and J. Tang, “Distributed energy-efficient multi-uav navigation for long-term communication coverage by deep reinforcement learning,” _IEEE Transactions on Mobile Computing_ , vol. 19, no. 6, pp. 1274–1285, 2020.
  * [97] X. Liu, Y. Liu, and Y. Chen, “Reinforcement learning in multiple-uav networks: Deployment and movement design,” _IEEE Transactions on Vehicular Technology_ , vol. 68, no. 8, pp. 8036–8049, 2019.
  * [98] S. Tomashevich and B. Andrievsky, “Improved adaptive coding procedure for transferring the navigation data between uavs in formation,” in _AIP Conference Proceedings_ , vol. 2046, no. 1. AIP Publishing LLC, 2018, p. 020102.
  * [99] M. Hüttenrauch, A. Sosic, and G. Neumann, “Deep reinforcement learning for swarm systems,” _ArXiv_ , vol. abs/1807.06613, 2019.
  * [100] D. Baldazo, J. Parras, and S. Zazo, “Decentralized multi-agent deep reinforcement learning in swarms of drones for flood monitoring,” in _2019 27th European Signal Processing Conference (EUSIPCO)_ , 2019, pp. 1–5.
  * [101] Y. Sung, A. K. Budhiraja, R. K. Williams, and P. Tokekar, “Distributed assignment with limited communication for multi-robot multi-target tracking,” _Autonomous Robots_ , vol. 44, no. 1, pp. 57–73, 2020.
  * [102] S. K. Sharma, I. Woungang, A. Anpalagan, and S. Chatzinotas, “Toward tactile internet in beyond 5g era: Recent advances, current issues, and future directions,” _IEEE Access_ , vol. 8, pp. 56 948–56 991, 2020.
  * [103] K. Antonakoglou, X. Xu, E. Steinbach, T. Mahmoodi, and M. Dohler, “Toward haptic communications over the 5g tactile internet,” _IEEE Communications Surveys Tutorials_ , vol. 20, no. 4, pp. 3034–3059, 2018.
  * [104] E. Steinbach, S. Hirche, J. Kammerl, I. Vittorias, and R. Chaudhari, “Haptic data compression and communication,” _IEEE Signal Processing Magazine_ , vol. 28, no. 1, pp. 87–96, 2011.
  * [105] C. Shahabi, A. Ortega, and M. R. Kolahdouzan, “A comparison of different haptic compression techniques,” in _Proceedings. IEEE International Conference on Multimedia and Expo_ , vol. 1. IEEE, 2002, pp. 657–660.
  * [106] P. Hinterseer, S. Hirche, S. Chaudhuri, E. Steinbach, and M. Buss, “Perception-based data reduction and transmission of haptic data in telepresence and teleaction systems,” _IEEE Transactions on Signal Processing_ , vol. 56, no. 2, pp. 588–597, 2008.
  * [107] F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu, “1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns,” in _Fifteenth Annual Conference of the International Speech Communication Association_ , 2014.
  * [108] W. Wen, C. Xu, F. Yan, C. Wu, Y. Wang, Y. Chen, and H. Li, “Terngrad: Ternary gradients to reduce communication in distributed deep learning,” in _Advances in neural information processing systems_ , 2017, pp. 1509–1519.
  * [109] S. Zhou, Y. Wu, Z. Ni, X. Zhou, H. Wen, and Y. Zou, “Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients,” _arXiv preprint arXiv:1606.06160_ , 2016.
  * [110] M. M. Amiri, D. Gunduz, S. R. Kulkarni, and H. V. Poor, “Convergence of federated learning over a noisy downlink,” _arXiv preprint arXiv:2008.11141_ , 2020.
  * [111] M. M. Amiri and D. Gündüz, “Machine learning at the wireless edge: Distributed stochastic gradient descent over-the-air,” _IEEE Transactions on Signal Processing_ , vol. 68, pp. 2155–2169, 2020.
  * [112] M. Mohri, G. Sivek, and A. T. Suresh, “Agnostic federated learning,” _arXiv preprint arXiv:1902.00146_ , 2019.
  * [113] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” in _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , ser. NIPS’17. Red Hook, NY, USA: Curran Associates Inc., 2017, p. 6382–6393.
  * [114] V. D. Nguyen, S. K. Sharma, T. X. Vu, S. Chatzinotas, and B. Ottersten, “Efficient federated learning algorithm for resource allocation in wireless iot networks,” _IEEE Internet of Things Journal_ , pp. 1–1, 2020.
  * [115] Z. Cao, P. Zhou, R. Li, S. Huang, and D. Wu, “Multiagent deep reinforcement learning for joint multichannel access and task offloading of mobile-edge computing in industry 4.0,” _IEEE Internet of Things Journal_ , vol. 7, no. 7, pp. 6201–6213, 2020.