iterative and can be highly complex.

As mentioned earlier, another significant benefit of the end-to-end DNN-based
approach is the graceful degradation it provides; that is, the performance,
trained for a specific channel SNR generalizes to other SNRs. This capability
was exploited in [226] to show that DeepJSCC can outperform the separation-
based alternatives with even a greater margin when used over a fading channel,
when channel state information (CSI) is not available. It is later shown in
[242] that, when the CSI is available at the transmitter and the receiver, an
attention mechanism can be used to train a single network that can achieve the
best performance at every SNR.

In [243], it is shown that DeepJSCC can also achieve successive refinability,
that is, the image can be delivered at multiple steps, using gradually more
bandwidth, with minimal loss in performance. This means that receivers can
tune into the transmission until they recover the transmitted image at the
desired quality. In [244], the authors employ adaptive-bandwidth transmission
across features depending on feature entropies. This allows allocating more
bandwidth to more important features, increasing their reconstruction quality.
Bandwidth allocation strategy is learned jointly with the non-linear transform
in an end-to-end fashion, significantly improving the performance of DeepJSCC
in all performance measures (PSNR, MS-SSIM, LPIPS), especially in the large
channel bandwidth regime.

Another challenge in communication systems is to exploit feedback. It was
shown by Shannon that feedback does not increase the channel capacity.
Therefore, in the infinite blocklength regime, it does not help from a JSCC
perspective either. Since separation is optimal in this regime, what matters
for the end-to-end performance is the channel capacity, which the feedback
cannot improve. On the other hand, it is known that feedback helps to improve
the error exponent in channel coding [245, 246]. More interestingly, when
transmitting a Gaussian source signal over a Gaussian channel, it is shown in
[247] that optimality of uncoded transmission shown in [194] only in the case
of matched bandwidth between the source and the channel, extends to arbitrary
bandwidth extensions. In [228], the DeepJSCC scheme is extended to channels
with channel output feedback, and it is shown that feedback can significantly
improve the end-to-end performance. It is shown that the required bandwidth
for image delivery can be reduced by half when variable rate transmission is
allowed and channel feedback is exploited to stop transmission when the
required reconstruction quality is reached at the receiver.

Benefits of DNN-aided JSCC is extended to OFDM channels in [248, 249]. In
[249], the authors employ a double attention mechanism, where one is used to
adaptively allocate power according to qualities of the subchannels, a spatial
attention mechanism is used to map the features to the subchannels to make
sure that the important features are transmitted over the good quality
subchannels. In all these works, the encoder is free to map the input signal
to arbitrary points in the channel input space; that is, the channel inputs
are limited only by an average power constraint, but there is no input
constellation. However, in some practical communication systems, communication
hardware is constrained to a fixed constellation diagram. In [250], a
differentiable quantization approach is used to map channel inputs to points
from a prescribed discrete constellation, and it is shown that the performance
loss compared to DeepJSCC with unconstrained channel inputs is limited as long
as a sufficiently rich constellation can be employed. In [251], JSCC of images
transmitted over binary symmetric and over binary erasure channels is
considered using a variational autoencoder (VAE) assuming a Bernoulli prior.
To overcome the challenges imposed by the non-differentiability of discrete
latent random variables (i.e., the channel inputs), unbiased low-variance
gradient estimation is used, and the model is trained using a lower bound on
the mutual information between the images and their binary representations.

We would like to highlight that, similarly to neural image compression
techniques, an important advantage of DNN-based JSCC approaches, compared to
employing conventional compression and channel coding techniques is that,
these codes can be trained for any desired final fidelity measure, including
various inference tasks, that would not require a complete reconstruction of
the source signal. Indeed, it has been shown that DNN-based JSCC approaches
outperform their conventional counterparts particularly in terms of SSIM and
MS-SSIM performance measure, which are known to better capture the perceived
quality, or semantics, of the transmitted images. Similarly, adversarial
measures can also be used to further improve the perception quality of the
reconstructed images [248].

A semantic communication system for image retrieval over a wireless channel is
considered in [189], which aims to identify the top-$k$ similar images to a
query image in a large dataset of images. The authors proposed directly
mapping the extracted image features to channel inputs through a three-step
training procedure: feature encoder pre-training, followed by JSCC autoencoder
pre-training, and finally, end-to-end training. It is shown that this joint
approach significantly outperforms the separation-based approach that combines
the retrieval-oriented compression scheme mentioned in Section IV-B with a
capacity-achieving channel code. This is despite the fact that a very short
blocklength is considered, and hence, capacity is far from achievable. This
scheme is recently extended in [234]. A semantic rate-distortion theory-based
communication system for multiple image tasks has been investigated in [252],
in which the source image is first restored at the receiver before intelligent
task execution.

The performance of DeepJSCC for image transmission has also been tested and
verified in practical communication systems using a software-defined radio
testbed in [241], which is also exhibited in [253]. The authors in [254]
proposed a real-time semantic testbed based on a visual transformer.

The first work on the JSCC of video signals over wireless channels employing
DNNs is carried out in [255]. In this paper, video signals are divided into
group of pictures (GoPs), similarly to common video compression standards.
Each GoP is directly mapped to channel input symbols of a fixed bandwidth. The
first frame of each GoP is considered as a key frame, and transmitted on its
own using JSCC techniques similar to the one used in [226]. The remaining
frames are transmitted using an interpolation encoder, which encodes the
motion information in that frame and residual information with respect to the
nearest two key frames as reference. Scaled space flow is used to estimate the
motion information using the DNN architecture proposed in [256]. There are two
challenges in the proposed method: First, due to the JSCC for delivering key
frames, the encoder does not know their exact reconstruction at the receiver,
which depends on the noise realization. The authors use a stochastic encoding
method, where the encoder emulates the channel, and generates a reconstruction
of the key frame using the channel statistics. The interpolation is based on
this stochastically generated version of the key frame. Second, the total
bandwidth for the GoP is limited, but in general, one would expect to allocate
more channel bandwidth for frames with more motion content. This is achieved
by reinforcement learning in [255]. The authors show that the proposed learned
bandwidth allocation methodology strictly improves upon equal allocation of
the available bandwidth among the frames. The results in [255] show that the
proposed JSCC technique for video delivery, called DeepWiVe, not only provides
graceful degradation with channel SNR, similarly to DeepJSCC, but also
outperforms state-of-the-art separation-based digital transmission
alternatives combining H.264 or H.265 video encoding with LDPC channel coding
at a specified channel SNR value. These results are promising as they show the
potential advantages of DNN-based JSCC techniques for future augmented/virtual
reality (AR/VR) applications for wireless headsets.

A semantic communication system for image classification has been proposed in
[257], which performs a variational IB (VIB) framework to resolve the
difficulty in mutual information computation of the original IB [89]. The
adopted loss function can be expressed as

$\begin{split}\mathcal{L}_{VIB}\left(\boldsymbol{\phi},\boldsymbol{\theta}\right)&={\mathbf{E}}_{p\left(\boldsymbol{x},\boldsymbol{y}\right)}\\{{\mathbf{E}}_{p_{\boldsymbol{\phi}}\left(\widehat{\boldsymbol{z}}\left|\boldsymbol{x}\right.\right)}\left[-\log\left(q_{\boldsymbol{\theta}}\left(\boldsymbol{y}\left|\widehat{\boldsymbol{z}}\right.\right)\right)\right]\\\
&+\beta
D_{KL}\left(p_{\boldsymbol{\phi}}\left(\widehat{\boldsymbol{z}}\left|\mathbf{x}\right.\right)\parallel
q\left(\widehat{\boldsymbol{z}}\right)\right)\\},\end{split}$ (38)

where $\boldsymbol{x}$ represents the input image, $y$ denotes the target
label, and $\widehat{\boldsymbol{z}}$ is the recovered semantic information at
the receiver. $q\left(\widehat{\boldsymbol{z}}\right)$ and
$q_{\boldsymbol{\theta}}\left(\boldsymbol{y}\left|\widehat{\boldsymbol{z}}\right.\right)$
are two variational distributions to approximate the true distributions of
$p\left(\widehat{\boldsymbol{z}}\right)$ and
$p_{\boldsymbol{\theta}}\left(\boldsymbol{y}\left|\widehat{\boldsymbol{z}}\right.\right)$,
respectively. $\boldsymbol{\phi}$ and $\boldsymbol{\theta}$ are the trainable
parameters at the transmitter and receiver, respectively.
$D_{KL}\left(\cdot\right)$ indicates the Kullback-Leibler divergence.

###  VI-B Distributed Training over Noisy Channels

We can also extend the model training tasks presented in Section IV-A for
rate-limited channels to training over noisy channels. Since model training is
often carried out over many iterations, training among wireless devices
imposes strict delay constraints per iteration. Hence, the conventional
approach of separate model compression and communication would not meet the
desired delay and complexity requirements [1].

The problem of training and delivering a DNN to a remote terminal, called
AirNet, is considered in [258] \- extending the model in Fig. 5 by replacing
the rate-limited error-free link with a noisy wireless channel. The
conventional approach would be to first train a DNN, which is then delivered
reliably over the bandwidth-limited channel. We can either train a low
complexity model, such as MobileNet or ShuffleNet, or first train a larger
model, and then compress it to the level that can be delivered over the
limited capacity wireless link. Here, the size of the delivered model will be
dictated by the available channel capacity, and errors over the channel will
further reduce the accuracy of inference at the decoder side.

An alternative joint training and channel coding approach is considered in
[258], where the trained neural network weights are delivered over the
wireless channel in an analog fashion; that is, they are mapped directly to
the channel inputs. However, given the large size of DNNs, this would require
a very large bandwidth. Moreover, the receiver will recover a noisy version of
the network weights, which varies according to the noise realization. The
authors propose two distinct strategies to remedy these problems. Pruning
[259] is employed to reduce the network size without sacrificing its
performance significantly. The encoder first trains a large-dimensional DNN,
which will be then pruned to the available channel bandwidth. Choosing a large
DNN as an initial point, rather than directly training a DNN of dimension
equal to the available channel bandwidth is motivated by the literature [260],
which shows that pruning a trained large-dimensional DNN generally performs
better than directly training a low-dimensional DNN. The noise problem is
remedied by injecting a certain amount of noise to the network’s weights at
each training iteration, so that the trained network acquires robustness
against channel noise. It is shown in [258] that the analog transmission of
DNN weights achieves better accuracy compared to their digital delivery. A
further unequal error protection strategy is also incorporated by pruning the
network to a size smaller than the available channel bandwidth, and applying
bandwidth expansion to a selected subset of more important weights using
Shannon-Kotelnikov mapping [261].

While the above framework assumes the availability of the dataset at the
encoder, in many practical settings, the encoder may simply have access to the
DNN architecture and weights but no data. Delivering such a network over a
wireless channel is considered in [262], where it is shown that a Bayesian
approach at the receiver when estimating the noisy DNN weights can
significantly improve its performance. The authors assume a Gaussian prior,
and propose a population compensator and a bias compensator to the minimum
mean square error (MMSE) metric.

Another common scenario is when the dataset is distributed across many
wireless devices, which collaborate to train a common model in a federated
manner. When the devices participating in federated learning share a common
wireless medium to the parameter server, this is called federated learning at
the edge (FEEL). FEEL was studied in [263] and [264] for AWGN and fading
channels by first treating the uplink channel from the devices to the
parameter server as a MAC, and the devices communicate at the boundary of the
capacity region. Then, each device, depending on the rate available to it,
reduces the size of its model update by compressing it using the technique
proposed in [265]. With this approach, there is a trade0-off between the
number of devices participating in the model update at each round, and the
accuracy of the updates they can convey to the parameter server. The higher
the number of devices, less wireless channel resources they are allocated, and
the less the accuracy of the updates they transmit to the parameter server. In
[266], the authors studied the trade-off between the energy cost of model
updates and the latency in FEEL over fading channels. A joint wireless
resource allocation problem is formulated in [267] for FEEL over fading
channels to maximize the convergence rate of the underlying learning process.

In [263, 264], an alternative ‘analog’ transmission approach is proposed for
FEEL, by treating the uplink model transmission as a distributed computation
problem over a MAC. Inspired by the optimality of uncoded transmission in
certain distributed computation and JSCC problems over MAC [196, 202], these
papers proposed uncoded and synchronized transmission of local model updates
enables, which enables the the parameter server to directly recover the sum of
the updates from multiple terminals. This ‘over-the-air computation (OAC)’
approach has received significant interest in recent years thanks to its
bandwidth efficiency [263, 264, 268, 269]. Instead of allocating orthogonal
channel resources to the participating devices, they all share the same
bandwidth. Unlike in separate model compression and channel coding, the
accuracy of the resultant computation benefits from more transmitters as the
goal is to recover the sum of their model updates. OAC can also be used in a
fully distributed learning scenario [270, 271], where many computations take
place in parallel. We refer the reader to [272] for a comprehensive overview
of distributed learning techniques over wireless networks.

###  VI-C Solutions to the IB Problem

The IB problem detailed in Section III-E provides a formulation to design
mappings $P_{U|S}$ that allow to extract relevant information within the IB
relevance–complexity region, i.e, the pairs of achievable $(\Delta,R)$, by
solving the IB problem in (30) for different values of $\beta$. However, in
general, this optimization is challenging as it requires computation of mutual
information terms. In this section, we describe how, for a fixed parameter
$\beta$, the optimal solution $P^{\beta,*}_{U|S}$, or an efficient
approximation of it, can be obtained under: (i) known general discrete
memoryless distributions and particular distributions, or particular
distributions such as Gaussian and binary symmetric sources; and (ii) unknown
distributions and only samples are available to design the encoders and
decoders.

Figure 10: Example parametrization of Variational Information Bottleneck using
neural networks.

####  VI-C1 Known Discrete Memoryless Distribution

When the relevant features $V$ and the observation $S$ is discrete and the
joint distribution $P_{S,V}$ is known, the maximizing distributions $P_{U|S}$
in the IB problem in (30), can be efficiently found by an alternating
optimization procedure similar to the expectation-maximization (EM) algorithm
[273] and the standard Blahut–Arimoto (BA) method [274, 275], which is
commonly used in the computation of rate-distortion functions of discrete
memoryless sources. In particular, a solution $P_{U|X}$ to the constrained
optimization problem is determined by the following self-consistent equations,
for all $(u,s,v)\in\mathcal{U}\times\mathcal{S}\times\mathcal{V}$, [89]

$\displaystyle P_{U|S}(u|s)=\frac{P_{U}(u)}{Z(\beta,s)}\exp\Big{(}-\beta
D_{\mathrm{KL}}\Big{(}P_{V|S}(\cdot|s)\|P_{V|U}(\cdot|u)\Big{)}\Big{)}$
$\displaystyle P_{U}(u)=\sum_{s\in
S}P_{S}(s)P_{U|S}(u|S)~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}$ $\displaystyle
P_{V|U}(v|u)=\sum_{s\in S}P_{V|S}(v|s)P_{S|U}(s|u)~{}~{}~{}~{}~{}~{}~{}~{}~{}$
(39)

where $P_{S|U}(s|u)=P_{U|S}(u|s)P_{S}(s)/P_{U}(u)$ and $Z(\beta,s)$ is a
normalization term. It is shown in [89] that alternating iterations of these
equations converges to a solution of the problem for any initial $P_{U|S}$.
However, different to the standard Blahut–Arimoto algorithm for which
convergence to the optimal solution is guaranteed, convergence here may be to
a local optimum only.

####  VI-C2 Unknown Distributions

The main drawback of the solutions presented in the previous section is that
requirement knowing the joint distribution $P_{S,V}$, or at least a good
estimation of it and that iterating (39) can only be performed for sources
with small alphabet (or jointly Gaussian [276, 277, 109]). The Variational IB
(VIB) method was proposed in [98] as a means to obtain approximate solutions
to the IB problem in the case in which the joint distribution is unknown and
only a give training set of $N$ samples $\\{(s_{i},v_{i})\\}_{i=1}^{N}$ is
available or the alphabet is too large. The VIB consists in defining a
variational (lower) bound on the cost of the IB problem in (30), use neural
networks to parameterize this bound and show that, leveraging on the
reparametrization trick [278] its optimization can be performed through
stochastic gradient descendent (SGD). From a task oriented communication
perspective, the VIB approach provides a principled way to generalize the
evidence lower bound (ELBO) and Variational Autoencoders [278] (and its
extension to $\beta$-VAE cost [279]) to scenarios in which the decoder is
interested in recovering the relevant information $V$ for a task that is not
necessarily the observed sample $S$ by maximizing relevance. The idea is to
use the IB principle to train an encoder and decoder, parameterized by DNNs,
which are able to extract the relevant information to forward to a decoder in
charge of reconstructing the relevant information. The resulting architecture
to optimize with an encoder, a latent space, and a decoder parameterized by
Gaussian distributions is shown in Fig. 10. This approach has been used for
task oriented communications also in JSCC scenarios, as a means to extract the
relevant information to transmit over communication noisy channels to perform
a given task at the destination, e.g., [280],[281].

Figure 11: Example parameterization of the distributed variational information
bottleneck method using neural networks for $K=2$ for encoders parameterized
by Gaussian distributions,
$P_{\theta}(u_{k}|s_{k})=\mathcal{N}(s_{k};\boldsymbol{\mu}_{k}^{\theta}(s),\Sigma_{k}^{\theta}(s))$.

More precisely, solving the IB problem in (30) consists in optimizing the IB-
Lagrangian

$\mathcal{L}_{\beta}^{\mathrm{IB}}(P_{U|S}):=I(U;V)-\beta I(U;S).$ (40)

over all $P_{U|S}$ satisfying $U-S-V$. It follows from Gibbs inequality, for a
any $P_{U|S}$ satisfying $U-S-V$, we have the following lower bound on the IB-
Lagrangian,

$\displaystyle\mathcal{L}_{\beta}^{\mathrm{IB}}(P_{U|S})\geq\mathcal{L}^{\mathrm{VIB}}_{\beta}(P_{U|S},Q_{V|U},S_{U})~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}$
(41) $\displaystyle:=\mathrm{E}_{P_{U|S}}\left[\log Q_{V|U}(V|U)\right]-\beta
D_{\mathrm{KL}}(P_{U|S}|S_{U}),$ (42)

where $Q_{V|U}(V|u)$ is a given stochastic map
$Q_{V|U}\>:\>\mathcal{U}\rightarrow[0,1]$ (also referred to as the variational
approximation of $P_{V|U}$ or decoder) and
$S_{U}(u):\mathcal{U}\rightarrow[0,1]$ is a given stochastic map (also
referred to as the variational approximation of $P_{U}$ or prior), and
$D_{\mathrm{KL}}(P_{U|S}|S_{U})$ is the relative entropy between $P_{U|S}$ and
$S_{U}$. The equality holds iff $Q_{V|U}=P_{V|U}$ and $S_{U}=P_{U}$, i.e., the
variational approximations match the true values.

Therefore, optimizing (40) over $P_{U|S}$ is equivalent to optimizing the
variational cost (42) over $P_{U|S}$, $Q_{V|U}$ and $S_{U}$. In the VIB
method, this optimization is done by further parameterizing the encoding and
decoding distributions $P_{U|S}$, $Q_{V|U}$, and $S_{U}$ that are to optimize
using a family of distributions $P_{\theta}(u|s)$, $Q_{\psi}(v|u)$, and
$S_{\varphi}(u)$, whose parameters are determined by DNNs with parameters
$\theta,\phi$, and $\varphi$ respectively. A common example is the family of
multivariate Gaussian distributions [278], which are parameterized by the mean
$\boldsymbol{\mu}^{\theta}$ and covariance matrix $\Sigma^{\theta}$. Given an
observation $X$, the values of
$(\boldsymbol{\mu}^{\theta}(s),\Sigma^{\theta}(s))$ are determined by the
output of the DNN $f_{\theta}$, whose input is $S$, and the corresponding
family member is given by
$P_{\theta}(u|s)=\mathcal{N}(u;\boldsymbol{\mu}^{\theta}(s),\Sigma^{\theta}(s))$.
Another common example are Gumbel-Softmax distibutions [282, 283]).

The bound (42) restricted to family of distributions $P_{\theta}(u|s)$,
$Q_{\psi}(v|u)$, and $S_{\varphi}(u)$ can be approximated using Monte-Carlo
and the training samples $\\{(s_{i},v_{i})\\}_{i=1}^{N}$. To facilitate the
computation of gradients using backpropagation [278], the reparameterization
trick [278] is used to sample from $P_{\theta}(U|S)$. In particular, consider
$P_{\theta}(U|S)$ to belong to a family of distributions that can be sampled
by first sampling a random variable $Z$ with distribution $P_{Z}(z)$,
$z\in\mathcal{Z}$ and then transforming the samples using some function
$g_{\theta}:\mathcal{S}\times\mathcal{Z}\rightarrow\mathcal{U}$ parameterized
by $\theta$, such that $U=g_{\theta}(s,Z)\sim P_{\theta}(U|s)$, e.g. a
Gaussian distribution. The reparametrization trick is used to approximate by
sampling $M$ independent samples $\\{u_{m}\\}_{m=1}^{M}\sim
P_{\theta}(u|s_{i})$ for each training sample $(s_{i},v_{i})$, $i=1,\ldots,N$.
Then, the lower bound on the IB cost can be optimized using optimization
methods such as SGD or ADAM [284] with backpropagation over the the DNN
parameters ${\theta},\phi,\varphi$ as,

$\displaystyle\max_{\boldsymbol{\theta},\boldsymbol{\phi},\boldsymbol{\varphi}}\frac{1}{N}\sum_{i=1}^{N}\mathcal{L}^{\mathrm{emp}}_{\beta,i,M}({\boldsymbol{\theta},\boldsymbol{\phi},\boldsymbol{\varphi}}),$
(43)

where the cost for the $i$-th sample in the training dataset is:

$\displaystyle\mathcal{L}^{\mathrm{emp}}_{\beta,i,M}({\theta,\phi,\varphi})\hskip
144.54pt$ (44) $\displaystyle:=\frac{1}{M}\sum_{m=1}^{M}\Big{[}\log
Q_{\phi}(v_{i}|u_{i,m})-\beta
D_{\mathrm{KL}}(P_{\theta}(U_{i}|s_{i})\|Q_{\varphi}(U_{i}))\Big{)}\Big{]},$

and sampling is performed by using $u_{i,m}=g_{\phi}(s_{i},z_{m})$ with
$\\{z_{m}\\}_{m=1}^{M}$ i.i.d. sampled from $P_{Z}$ for each $(s_{i},v_{i})$
pair.

For inference, let ${\theta}^{*},{\phi}^{*},\varphi^{*}$ be the DNN parameters
obtained in training by solving (43). Inference for a new observation $S$, the
representation $U$ can be obtained by sampling from the encoders
$P_{\theta_{k}^{*}}(U_{k}|S_{k})$ and a soft estimate of the remote source $Y$
can be inferred by sampling from the decoder $Q_{\phi^{*}}(V|U)$. Thus, from a
task oriented communication perspective, $P_{\theta_{k}^{*}}(U_{k}|S_{k})$ is
an encoder trained according to the cost (43) to extract the most relevant
representation $U$ for inference of $V$, and $Q_{\phi^{*}}(V|U)$ is a decoder
that is trained to reconstruct the relevant information $V$ from the
representation $U$ that minimizes the log loss.

Similarly to the steps followed for the variational IB in Section VI-C2,
encoders and decoder performing on the region (34) can be computed by deriving
a variational bound on and parameterizing encoding and decoding distributions
$P_{U_{k}|S_{k}},Q_{U_{k}|V}$ using a family of distributions whose parameters
are determined by DNNs, and optimize it by using the reparameterization trick
[278], Monte Carlo sampling, and stochastic gradient descent (SGD)-type
algorithms. The encoders and decoders parameterized by DNN parameters
$\boldsymbol{\theta},\boldsymbol{\phi},\boldsymbol{\varphi}$ can be optimized
according the distributed IB principles by considering the following empirical
Monte Carlo approximation:

$\displaystyle\max_{\boldsymbol{\theta},\boldsymbol{\phi},\boldsymbol{\varphi}}\frac{1}{n}\sum_{i=1}^{n}\Big{[}\\!\log
Q_{\phi_{\mathcal{K}}}(v_{i}|u_{1,i,j},\ldots,u_{K,i,j})\hskip 36.135pt$ (45)
$\displaystyle+s\sum_{k=1}^{K}\\!\Big{(}\\!\log
Q_{\phi_{k}}(v_{i}|u_{k,i,j})\\!-\\!D_{\mathrm{KL}}(P_{\theta_{k}}(U_{k,i}|s_{k,i})\|Q_{\varphi_{k}}(U_{k,i}))\Big{)}\Big{]},$

where $u_{k,i,j}=g_{\phi_{k}}(s_{k,i},z_{k,j})$ are samples obtained from the
reparameterization trick by sampling from $K$ random variables $P_{Z_{k}}$.
The resulting architecture is shown in Fig. 11. This architecture generalizes
that from autoencoders to the distributed task oriented communication scenario
setup with $K$ encoders.

##  VII Timing as Semantics in Communications

In the previous sections, we have considered goal-oriented communications from
a ‘static’ perspective; that is, we considered the scenario in which a
transmitter observes a certain signal, and wants to deliver it to a receiver
under a certain semantic quality measure, which depends on the underlying
task, but neither the source statistics nor the semantic quality measure
changes over time. Therefore, time did not really factor into our analysis.
However, in many practical systems, particularly those involving control of
cyber-physical systems, timing of messages can be as critical as their
content. A message that arrives too late can be completely useless no matter
how reliably it have been recovered.

In this section, we will deal with a broad class of problems, where the
relevance or value of information related to its timing. A specific sub-class
of these problems relates to the popular idea of age of information (AoI).
However, as we shall argue shortly, AoI is just one piece of the bigger puzzle
when it comes to “timing as semantics” in communications. For a systematic
exposition, we will first summarize key ideas related to the concept of AoI
that will be useful in this discussion. We will then introduce a general real-
time reconstruction problem using a rate-distortion viewpoint where semantic
information is contained in the timing of the source samples. Along the way,
we will explore the connection of age metrics to this rate-distortion
viewpoint.

Figure 12: An illustration of the general setup for Section VII.

The general setup for this section is illustrated in Fig. 12. It consists of a
source modeled as a random process $X(t)$, a transmitter that transmits status
updates to a monitor about the current state of $X(t)$, and the monitor where
the estimated state is denoted by $\hat{X}(t)$. In this setting, the
transmitter needs to take two key decisions during each time slot: (i) whether
to sample $X(t)$ to generate a status update, and (ii) whether to transmit the
status update. Note that it is not always optimal to sample and transmit in
the same time slot when the server is subjected to additional constraints,
such as the energy availability/harvesting constraints. With this background,
we now present some relevant background on AoI.

###  VII-A Age of Information

####  VII-A1 Background and Definition

AoI is a performance metric that quantifies freshness of information at a
monitor about some remote stochastic process $X(t)$ observed by a transmitter
node [285, 286, 287, 288]. Specifically, AoI is defined as the time elapsed
since the last successfully received update packet at the monitor was
generated at the transmitter. Mathematically, it is defined as the following
random process:

$\Delta_{\rm AoI}(t)=t-u(t),$ (46)

where $u(t)$ is the generation time instant of the latest status update
received at the monitor by time $t$. In order to introduce the idea of AoI
concretely, we use Fig. 13, which depicts a realization of AoI at the monitor
as a function of time when the transmitter sends update packets according to a
First-Come-First-Served (FCFS) queuing discipline and only one packet
transmission may occur at any given time. Here, we implicitly model age as a
linear function, which is also the most popular definition of age in the
literature, although more general non-linear age functions have also been
considered. Further, $t_{n}$ and $t^{\prime}_{n}$ denote the generation and
reception time instants of packet $n$ at the transmitter and monitor,
respectively. Therefore, we observe that: i) $X_{n}$ is the inter-arrival time
between packets $n-1$ and $n$, i.e., the time elapsed between the generation
of packets $n-1$ and $n$, ii) $T_{n}$ is the system time of packet $n$, i.e.,
the time elapsed from the generation of packet $n$ at the transmitter until it
is received at the monitor, and iii) AoI is reset to $T_{n}$ at
$t^{\prime}_{n}$ since packet $n$ becomes the latest received update packet at
$t^{\prime}_{n}$, and hence the AoI value at that time instant is the time
passed since the generation of packet $n$, which is $T_{n}$. Under ergodicity,
many key properties of the AoI process, such as the average AoI or the mean
peak AoI, can be studied from its sample functions, such as the one depicted
in Fig. 13. Interested readers are advised to refer to well-known books and
overview articles on this topic, such as [285, 286, 287, 288], for a more
comprehensive introduction to the concept of AoI.

Figure 13: AoI evolution vs. time for $n$ update packets [287].

####  VII-A2 Age of Incorrect Information (AoII)

While AoI is useful in characterizing the freshness of information and has
found applications in a broad range of system settings under a variety of
queuing disciplines [289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299],
it does not explicitly take into account the similarity or discrepancy between
the status of information at the transmitter and the monitor. In particular,
as observed from (46), the AoI always increases as time passes even if the
transmitter and monitor have the same status of information. As an example,
consider $X(t)$ to be a discrete Markov source. If the current state of $X(t)$
matches with that of $\hat{X}(t)$, the monitor has the most updated
information about $X(t)$ irrespective of when the status update was received.
Therefore, in this case AoI is not an accurate measure of information
freshness since it ignores semantic information about the nature of source
$X(t)$. This motivated the introduction of a related metric, termed AoII,
which is defined as [300]:

$\Delta_{\rm AoII}=\left(t-v(t)\right)\mathbf{1}\\{X(t)\neq\hat{X}(t)\\},$
(47)

where $v(t)$ is the last time instant when
$\mathbf{1}\\{X(t)\neq\hat{X}(t)\\}=0$ and $\mathbf{1}(\cdot)$ is an indicator
function. Thus, according to (47), AoII grows with time only when
$X(t)\neq\hat{X}(t)$. Therefore, AoII explicitly includes some semantic
information about the nature of source $X(t)$ that was ignored in the
definition of AoI. In order to make this connection more concrete, we now
discuss the general real-time reconstruction problem from the rate-distortion
viewpoint.

###  VII-B Real-time Reconstruction

We now establish a connection between the well-known age metrics discussed
above and the semantics of information in the context of real-time remote
tracking or reconstruction systems as formulated in [301]. Recall the setup of
Fig. 12, where a transmitter observes source $X(t)$ and sends samples/updates
about that source over time to a monitor. The objective is to reconstruct
$X(t)$ at the monitor using the received samples/updates from the transmitter.
Recall also that the transmitter has two key decisions to take during each
time slot in this general setting: i) whether to sample $X(t)$ or not, and ii)
whether to transmit the available or newly generated sample/update or not.
Further, we consider a semantic-aware objective function (which is usually
referred to as a distortion function), which can be expressed for the cases of
discrete and continuous sources of information as

$\mathcal{S}_{\rm dis}=\underset{T\to\infty}{\rm
lim}{\frac{1}{T}\sum_{t=1}^{T}{c(t)}},$ (48) $\mathcal{S}_{\rm
con}=\underset{T\to\infty}{\rm lim}{\frac{1}{T}\int_{t=0}^{T}{c(t){\rm d}t}},$
(49)

where $c(t)$ is an appropriately chosen cost function. For instance, $c(t)$ in
the discrete case can be either $c(t)=|X(t)-\hat{X}(t)|$,
$c(t)=a(t)\mathbf{1}\\{X(t)\neq\hat{X}(t)\\}$, or
$c(t)=[X(t)-\hat{X}(t)]^{2}$, while the latter is also appropriate for the
continuous case.

When no channel or transmission delays are considered, the above problem can
be thought of as a rate-distortion problem. In the classical rate-distortion
problem, we are interested in the average distortion between the original
process at the transmitter and its reconstruction at the receiver under a
constraint on the number of bits transmitted. Here, on the other hand, ‘real-
time’ operation is considered; that is, the distortion measure does not
tolerate delays, and the process needs to be reconstructed at the receiver in
a real-time fashion. The general objective here is to develop sampling and
transmission policies that minimize the average delay-sensitive distortion. An
extreme case is zero-delay transmission, where the reconstruction at the
receiver at each time depends only on the information received up to that
point. Delay-constrained distortion requirement is naturally motivated by
cyber-physical applications, where the source signal may model the system
state and observations, and the receiver is a controller, which cannot
tolerate the infinite delay requirements of the classical rate-distortion
formulation.

Specific instances of this problem have already been investigated in the
literature. When the transmitter samples and transmits during every time slot,
we obtain a causal compression problem. Real-time compression of a discrete
Markov source under a finite rate constraint is studied in [302]. In this
paper, Witsenhausen showed that if the source is $k$th-order Markov source,
then the transmitted codeword at each time instant can depend only on the last
$k$ source samples and the present state of the receiver’s memory, without
loss of optimality. When the source is memoryless, i.e., $k=1$, then it is
known that the optimal distortion is achieved by the optimal scalar quantizer
(Lloyd-Max) for the source [303].

Many works have also considered the sampling aspect in tracking random
processes under constraints on the sampling rate. In [304, 305, 306], causal
transmission of a discrete random process over a finite time horizon is
considered under transmission cost constraints. While [304] and [305] consider
special decoder and encoder functions, respectively, and optimize the other
one, [306] made no assumptions on the structure of the encoder or the decoder,
and showed that the optimal encoding function is of threshold type and optimal
estimation strategy at the decoder is Kalman-like. The optimal distortion-
transmission rate trade-off is fully characterized in [307] for Gauss-Markov
sources. Continuous-time processes are considered in [308] with a constraint
on the number of samples that can be transmitted. Energy, delay and buffer
constraints are also considered in [309, 310] for delay-limited source
delivery.

All of the aforementioned works consider a perfect communication channel, and
deal mainly with the rate-distortion performance, rate representing either the
usual bits per sample, or the sampling rate. While this can be considered as
the extension of the source coding problems overviewed in Section III-A with a
delay-sensitive distortion measure, in practical systems it is important to
take the effect of the channel into account. Note that, since the channel
transmission also introduces delays, it will have a direct impact on the
achieved distortion.

Optimality conditions for the delay-limited transmission of source samples
over a noisy channel have been studied as early as the 60s [311, 312]. These
works considered discrete channel inputs and outputs. Real-time transmission
of a Markov source over a noisy channel with perfect channel output feedback
is studied in [313]. Perfect channel output feedback in this problem
significantly simplifies the problem as it allows the encoder to track
perfectly the knowledge of the decoder. The more challenging problem of real-
time transmission of a Markov source over a noisy channel without feedback is
studied in [314], and a general framework is introduced to study the
corresponding team decision problems. A similar JSCC problem with real-time
reconstruction constraints is considered in [315], which assumed that the
transmitter can only transmit a certain number of times; that is, [315]
extends the model in [304] to the JSCC scenario.

In [316], the authors consider the real-time transmission of a discrete-time
Markov source over a fading channel, where each transmission is lost with a
probability that depends on the transmission power and the channel state.
Assuming that the channel state is known by both the transmitter and the
receiver instantaneously, the authors characterize the optimal transmission
and estimation strategies for the infinite-horizon average distortion for
vector-valued autoregressive sources. They also show that the optimal
transmission strategy is of threshold-type when the available transmit power
levels are discrete. A similar model is also considered in [317], with the
slight but important difference that the channel state is known
instantaneously only by the receiver, and is fed back to the transmitter with
unit delay. The optimal transmission and estimation schemes are then
characterized for Markov and first-order autoregressive sources, and the
optimality of threshold based transmission is also shown for discrete power
levels.

While these works considered explicit source and channel statistics, and
treated the end-to-end optimization problem as JSCC with a delay-constrained
distortion measure, there has been a significant research interest from the
networking community to treat the physical channel as a random queue [318,
291, 319, 286], and focus on the scheduling aspects rather than source and
channel coding. In most of the early works on AoI, the status updates are
generated as a new packet at the transmitter by an exogeneous process, and the
goal of the transmitter is to choose which packets to transmit to keep the AoI
at the receiver low. The core idea here is that dropping some packets can be
beneficial to reduce the queue waiting time for future status updates.

While AoI is explicitly defined as the measure of performance in these works,
AoI can also be connected to the JSCC framework through a delay-sensitive
distortion measure. As mentioned earlier, transmission of a random Wiener
process over a random delay channel is considered in [301], and the authors
show that the estimation error at the receiver is a function of the AoI. In
[320], the authors consider the mutual information between the real-time
source value and the samples delivered to the receiver, and show that this
mutual information depends on the AoI. These provide operational meaning to
the AoI measure, and connects it closely with cyber-physical systems and
control.

Next, we present three different sampling and transmission policies [301, 16]
that will also highlight subtle connections between the age metrics and
semantics of information.

####  VII-B1 AoI-aware Sampling and Transmission Policy

In this policy, the transmitter samples a new update and sends it to the
receiver once the AoI at the receiver reaches a predefined threshold $A_{\rm
th}$. While this policy takes into account the timeliness of the available
information at the receiver, it is not semantics-aware since it does not
involve the status of information at the source and the receiver in the
process of decision-making. This again highlights the fact that AoI is
agnostic of the source semantics.

####  VII-B2 Source-aware Sampling and Transmission Policy

In this policy, the transmitter samples a new update and sends it to the
receiver when the status of information at the source changes. This policy can
be be thought of as a semantics-aware policy but just from the perspective of
the source signal. Since this does not explicitly account for the discrepancy
of the status at the transmitter and receiver, it could potentially result in
unnecessary transmissions that could have been avoided in a truly semantics-
aware policy. One specific example of such an instance given in [16] is when a
transmission is triggered by a change in the status of the source but the
status update does not reach the monitor because of an erasure. Now, if the
source returns to the previous state in the next time slot, no transmission is
required since there is no discrepancy in the states at the transmitter and
receiver. However, the current policy will still transmit an update since it
does not incorporate end-to-end semantics.

####  VII-B3 Semantics-aware Sampling and Transmission Policy

Not surprisingly, this policy overcomes the drawback of the previous one by
accounting for both the status of the source signal and the status of the
reconstructed signal at the monitor. In particular, the transmitter samples a
new update and sends it to the receiver only if the status of the source
signal is different from that of the reconstructed signal. Here, we would like
to remind the readers that this discrepancy on the status at the transmitter
and the monitor explicitly appeared in the definition of AoII. Therefore, one
can also interpret this policy as follows: the transmitter samples a new
update and sends it to the receiver at the moment when the AoII becomes non-
zero.

Figure 14: Illustration of the effective communication paradigm introduced in
[321].

##  VIII Effective / Pragmatic Communications

In [6], Weaver defines Level C of communications as the effectiveness problem,
which deals with the effect of the communicated symbols on the receiver. At
this level, the communication should have not only a meaning, but also an
impact on the behavior of the receiver. In [321], the authors argue that
communication problems where the receiver takes actions based on the messages
it receives, and the goal is to maximize a long-term reward function, falls
into this category. The reward received following each transmission depends on
the state of the receiver as well as the action it takes. In these problems,
we cannot define a simple fidelity metric as in the preceding rate-distortion
theory based framework. The utility, or fidelity of each transmission depends
on its impact over a long time horizon. Here, as in the previous section, time
plays an important role, and the same message may have different ‘meanings’
depending on the state of the receiver. Therefore, we can consider the action
taken by the receiver after each transmission as the affect of communication
on the receiver, and the state of the receiver as the ‘context’.

You can see in Fig. 14 an illustration of the effective communication paradigm
proposed in [321]. The authors suggest that any single-agent Markov decision
process can be turned into an effective communication problem by considering
two entities, one of them, called the controller, observes the state and the
reward signals, while the actions are taken by the agent, which may or may not
observe its state, but not the rewards. We further assume that there is a
noisy communication channel from the controller to the agent, and the actions
taken by the agent will depend on the signals received from the controller
through this channel. Note that the controller and the agent collaboratively
need to learn the best actions to take to maximize their long term reward,
which can be formulated as a reinforcement learning problem. However, unlike
in conventional formulation, here the learning must be enabled through the
noisy channel. It is shown in [321] that separating the learning problem from
the communication problem will result in a suboptimal performance.

We also note here that both Morris [3] and Weaver [6] conceptualized syntax,
semantics and pragmatics as going from more specific to more general; that is,
pragmatics is the most general of the three. Indeed, the above formulation of
effective/pragmatic communication generalizes both the technical communication
formulation of Shannon (i.e., the source and channel coding theorems), and the
semantic communication as conceptualized in this paper in the form of a JSCC
problem with a prescribed distortion measure. To see this, we can formulate
the channel coding problem as a two-agent guessing game, where the transmitter
observes the message $m\in[M]$ and maps it into a channel codeword $x^{n}$
that belongs to the set $\mathcal{X}^{n}$, which denotes the set of possible
actions of the transmitter. Upon observing the noisy channel output $Y^{n}$,
the receiver takes one of $M$ actions, $\hat{m}\in[M]$, which correspond to
its reconstruction of message $m$. The agents receive a reward of $1$ if
$\hat{m}=m$, and $0$ otherwise. Their goal is to maximize their long-term
average reward, which corresponds to the probability of error, and their joint
policy, which consists of the encoding and decoding functions, form the
channel code. Similarly, if we replace the observation of the encoder with
$s^{m}\in\mathcal{S}^{m}$, its actions with $w\in[M]$, which is observed by
the receiver perfectly, the action set of the receiver with
$\hat{s}^{m}\in\hat{\mathcal{S}}^{m}$, and consider a cost function
$d(s^{m},\hat{s}^{m})$, minimizing the long-term average cost is equivalent to
solving the lossy source coding problem.

It is easy to see that the JSCC problem is also a special case of the above
effective/pragmatic communication framework, where the encoder observes
$s^{m}\in\mathcal{S}^{m}$ and takes an action from set $\mathcal{X}^{n}$,
while the decoder observes $\mathcal{Y}^{n}$ and takes an action from set
$\hat{\mathcal{S}}^{m}$. Minimizing the long-term average distortion, which
can be specified as any of the semantic distortion measures mentioned
throughout this paper, corresponds to solving the corresponding JSCC problem
for the given channel-to-source bandwidth ratio $n/m$. We would like to
emphasize that the proposed effective/pragmatic communication framework
generalizes semantic communication in multiple directions. First, it can allow
memory in the state of the encoder, that is, $s^{m}(t)$ observed in time slot
$t$ can depend on the past state $s^{m}(t-1)$ or states
$s^{m}(1),\ldots,s^{m}(t)$. This would result in the type of problems studied
in Section VII, in which the goal is to enable the receiver to track a
stochastic process available at the transmitter, over a noisy channel. On the
other hand, in the model considered in [321], the actions taken by the
receiver also have an impact on the next state of the transmitter.

It is further shown in [322] that adjusting the communication scheme according
to the state of the agent can result in a better long-term reward, providing
evidence for context-dependent communications.

The effective/pragmatic communication framework can be generalized to multiple
agent scenarios, where the agents communicate with each other over noisy
channels while trying to maximize prescribed reward functions [323]. This
model generalizes also the recently popular models in the machine learning
literature that study the emergence of communication among agents within the
reinforcement learning literature [324, 325, 326, 327]. In these problems all
agents transmit and receive signals in addition to taking actions and
interacting with the environment. Communication serves as a tool to enable
coordination and cooperation among the agents in order to maximize their
accumulated reward, which clearly highlights the pragmatic nature of
communication that goes beyond the transmitted bits over the channel or
reconstructed signals at the receiver; what matters is not which bits are
transmitted, or what the receiver’s estimate of the transmitter’s signal is,
it is the actions taken by the agents.

##  IX Conclusion

In this paper, we have presented a comprehensive overview of the foundations
of semantic- and task-oriented communications as well as practical data-driven
approaches to semantic communication of various information sources including
image, text and video. In particular, we argued that semantics can be
considered in the context of rate-distortion theory, albeit for most practical
information sources it is difficult if not impossible to explicitly state the
corresponding distortion metric. Therefore, we provided an extensive overview
of deep learning aided approaches to communicating practical information
sources over constrained communication channels under various semantic loss
functions. In the case of rate-limited error-free communications, we have
presented results related to remote inference as well as distributed model
training, and exposed the relevance of information theoretic concepts and
techniques such as IB. We emphasized the potential benefits of JSCC when
communicating over noisy channels, both in terms of end-to-end performance and
adaptivity to channel variations. Finally, we put the time dimension into the
framework, and argued that the timeliness of a signal can be considered as yet
another aspect of semantics. Finally, we presented an interpretation of the
effective/ pragmatic communication problem, which involves communication among
agents that take actions to achieve a certain goal. We believe that the
provided framework, which focuses on the communications aspects of semantics
rather than linguistic formulations, and the in-depth overview of the
foundations of semantic- and task-oriented communications will provide
guidelines for the development of more efficient practical solutions for
semantic- and task-oriented communication networks.

## References

  * [1] D. Gunduz, D. B. Kurka, M. Jankowski, M. M. Amiri, E. Ozfatura, and S. Sreekumar, “Communicate to learn at the edge,” _IEEE Communications Magazine_ , vol. 58, no. 12, pp. 14–19, 2020.
  * [2] P. Popovski, O. Simeone, F. Boccardi, D. Gündüz, and O. Sahin, “Semantic-effectiveness filtering and control for post-5G wireless connectivity,” _Journal of Indian Inst. of Science_ , 2020.
  * [3] C. W. Morris, _Foundations of the Theory of Signs_. Chicago, IL: University of Chicago Press, 1938.
  * [4] C. Cherry, _On Human Communication. A Review, a Survey, and a Criticism_. Cambridge, MA: John Wiley & Sons, 1957.
  * [5] N. Chomsky, _Syntactic Structures_. The Hague/Paris: Mouton, 1957.
  * [6] C. E. Shannon and W. Weaver, _The Mathematical Theory of Communication_. Urbana, IL: University of Illinois Press, 1949.
  * [7] C. E. Shannon, “A mathematical theory of communication,” _The Bell System Technical Journal_ , vol. 27, no. 3, pp. 379–423, 1948.
  * [8] N. Wiener, _The Human Use of Human Beings_. Urbana, IL: Houghton Mifflin, 1950.
  * [9] Y. Bar-Hillel and R. Carnap, “Semantic information,” _British Journal for the Philosophy of Science_ , vol. 4, no. 14, pp. 147–157, 1953.
  * [10] C. E. Shannon, “Prediction and entropy of printed english,” _The Bell System Technical Journal_ , vol. 30, no. 1, pp. 50–64, 1951.
  * [11] ——, “Coding theorems for a discrete source with a fidelity criterion,” in _Institute of Radio Engineers, International Convention Record_ , 1959.
  * [12] Q. Lan, D. Wen, Z. Zhang, Q. Zeng, X. Chen, P. Popovski, and K. Huang, “What is semantic communication? a view on conveying meaning in the era of machine intelligence,” _J. Commun. Inform. Netw._ , vol. 6, no. 4, pp. 336–371, 2021\.
  * [13] E. C. Strinati and S. Barbarossa, “6G networks: Beyond shannon towards semantic and goal-oriented communications,” _Computer Networks_ , vol. 190, p. 107930, 2021.
  * [14] M. Kalfa, M. Gok, A. Atalik, B. Tegin, T. M. Duman, and O. Arikan, “Towards goal-oriented semantic signal processing: Applications and future challenges,” _Digital Signal Process._ , vol. 119, p. 103134, 2021.
  * [15] G. Shi, Y. Xiao, Y. Li, and X. Xie, “From semantic communication to semantic-aware networking: Model, architecture, and open problems,” _IEEE Commun. Mag._ , vol. 59, no. 8, pp. 44–50, 2021.
  * [16] M. Kountouris and N. Pappas, “Semantics-empowered communication for networked intelligent systems,” _IEEE Commun. Mag._ , vol. 59, no. 6, pp. 96–102, 2021\.
  * [17] S. Iyer, R. Khanai, D. Torse, R. J. Pandya, K. Rabie, K. Pai, W. U. Khan, and Z. Fadlullah, “A survey on sematic communications for intelligent wireless networks,” _arXiv preprint arXiv:2202.03705_ , 2022.
  * [18] P. Zhang, W. Xu, H. Gao, K. Niu, X. Xu, X. Qin, C. Yuan, Z. Qin, H. Zhao, J. Wei _et al._ , “Toward wisdom-evolutionary and primitive-concise 6G: A new paradigm of semantic communication networks,” _Engineering_ , vol. 8, pp. 60–73, 2022.
  * [19] Z. Qin, X. Tao, J. Lu, and G. Y. Li, “Semantic communications: Principles and challenges,” _arXiv preprint arXiv:2201.01389_ , 2021.
  * [20] R. Carnap, Y. Bar-Hillel _et al._ , _An Outline of A Theory of Semantic Information_. RLE Technical Reports 247, Research Laboratory of Electronics, Massachusetts Institute of Technology., Cambridge MA, Oct. 1952.
  * [21] S. Sequoiah-Grayson and L. Floridi, “Semantic Conceptions of Information,” in _The Stanford Encyclopedia of Philosophy_ , Spring 2022 ed., E. N. Zalta, Ed. Metaphysics Research Lab, Stanford University, 2022.
  * [22] N. J. Venhuizen, M. W. Crocker, and H. Brouwer, “Semantic entropy in language comprehension,” _Entropy_ , vol. 21, no. 12, 2019.
  * [23] I. D. Melamed, “Measuring semantic entropy,” in _Tagging Text with Lexical Semantics: Why, What, and How?_ , 1997.
  * [24] X. Liu, W. Jia, W. Liu, and W. Pedrycz, “Afsse: An interpretable classifier with axiomatic fuzzy set and semantic entropy,” _IEEE Trans. Fuzzy Syst._ , vol. 28, no. 11, pp. 2825–2840, 2020.
  * [25] A. Chattopadhyay, B. D. Haeffele, D. Geman, and R. Vidal, “Quantifying task complexity through generalized information measures,” https://openreview.net/pdf?id=vcKVhY7AZqK, 2020 (accessed 24th Dec. 2021).
  * [26] Q. Wang, Z. Mao, B. Wang, and L. Guo, “Knowledge graph embedding: A survey of approaches and applications,” _IEEE Trans. Knowledge and Data Eng._ , vol. 29, no. 12, pp. 2724–2743, Dec. 2017.
  * [27] M. Nickel, V. Tresp, and H.-P. Kriegel, “A three-way model for collective learning on multi-relational data,” in _Proc. 28th Int. Conf. Mach. Learn._ , 2011, pp. 809–816.
  * [28] A. Bordes, X. Glorot, J. Weston, and Y. Bengio, “A semantic matching energy function for learning with multi-relational data,” _Mach. Learn._ , vol. 94, no. 2, pp. 233–259, 2014.
  * [29] Q. Liu, H. Jiang, A. Evdokimov, Z.-H. Ling, X. Zhu, S. Wei, and Y. Hu, “Probabilistic reasoning via deep learning: Neural association models,” _arXiv preprint arXiv:1603.07704_ , 2016.
  * [30] S. Yang, J. Tian, H. Zhang, J. Yan, H. He, and Y. Jin, “Transms: Knowledge graph embedding for complex relations by multidirectional semantics.” in _IJCAI_ , 2019, pp. 1935–1942.
  * [31] D. Collarana, M. Galkin, I. Traverso-Ribon, C. Lange, M.-E. Vidal, and S. Auer, “Semantic data integration for knowledge graph construction at query time,” in _IEEE 11th International Conference on Semantic Computing (ICSC)_ , 2017, pp. 109–116.
  * [32] K. Zhou, W. X. Zhao, S. Bian, Y. Zhou, J.-R. Wen, and J. Yu, “Improving conversational recommender systems via knowledge graph based semantic fusion,” in _Proceedings of ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2020, pp. 1006–1014.
  * [33] T. Grainger, K. Aljadda, M. Korayem, and A. Smith, “The semantic knowledge graph: A compact, auto-generated model for real-time traversal and ranking of any relationship within a domain,” in _IEEE International Conference on Data Science and Advanced Analytics (DSAA)_ , 2016, pp. 420–429.
  * [34] Z. Liu, C. Xiong, M. Sun, and Z. Liu, “Entity-duet neural ranking: Understanding the role of knowledge graph semantics in neural information retrieval,” _arXiv preprint arXiv:1805.07591_ , 2018.
  * [35] C. Xiong, R. Power, and J. Callan, “Explicit semantic ranking for academic search via knowledge graph embedding,” in _Proceedings of the 26th international conference on world wide web_ , 2017, pp. 1271–1279.
  * [36] S. Purohit, N. Van, and G. Chin, “Semantic property graph for scalable knowledge graph analytics,” in _IEEE International Conference on Big Data (Big Data)_ , 2021, pp. 2672–2677.
  * [37] S. Bhatt, A. Sheth, V. Shalin, and J. Zhao, “Knowledge graph semantic enhancement of input data for improving AI,” _IEEE Internet Computing_ , vol. 24, no. 2, pp. 66–72, Mar. 2020.
  * [38] Z. Qi, Z. Zhang, J. Chen, X. Chen, Y. Xiang, N. Zhang, and Y. Zheng, “Unsupervised knowledge graph alignment by probabilistic reasoning and semantic embedding,” _arXiv preprint arXiv:2105.05596_ , 2021.
  * [39] H. Xiao, M. Huang, and X. Zhu, “Knowledge semantic representation: a generative model for interpretable knowledge graph embedding,” _arXiv preprint arXiv:1608.07685_ , 2016.
  * [40] F. Zhou, Y. Li, X. Zhang, Q. Wu, X. Lei, and R. Q. Hu, “Cognitive semantic communication systems driven by knowledge graph,” _arXiv preprint arXiv:2202.11958_ , 2022.
  * [41] Z. Wang, E. P. Simoncelli, and A. C. Bovik, “Multiscale structural similarity for image quality assessment,” in _The Thrity-Seventh Asilomar Conference on Signals, Systems Computers, 2003_ , vol. 2, Nov. 2003, pp. 1398–1402 Vol.2.
  * [42] Y.-H. Huang, T.-S. Ou, P.-Y. Su, and H. H. Chen, “Perceptual rate-distortion optimization using structural similarity index as quality metric,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 20, no. 11, pp. 1614–1624, 2010.
  * [43] Y. Patel, S. Appalaraju, and R. Manmatha, “Human perceptual evaluations for image compression,” 2019. [Online]. Available: https://arxiv.org/abs/1908.04187
  * [44] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in _CVPR_ , 2018\.
  * [45] Z. Li, S. Qin, and L. Itti, “Visual attention guided bit allocation in video compression,” _Image and Vision Computing_ , vol. 29, no. 1, pp. 1–14, 2011\. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0262885610001083
  * [46] M. T. Khanna, K. Rai, S. Chaudhury, and B. Lall, “Perceptual depth preserving saliency based image compression,” in _Proc. Int’l Conf. on Perception and Machine Intelligence_ , ser. PerMIn, 2015, pp. 218–223.
  * [47] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _Advances in Neural Information Processing Systems_ , Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger, Eds., vol. 27. Curran Associates, Inc., 2014. [Online]. Available: https://proceedings.neurips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf
  * [48] M. Tschannen, E. Agustsson, and M. Lucic, “Deep generative models for distribution-preserving lossy compression,” in _Advances in Neural Information Processing Systems_ , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., vol. 31. Curran Associates, Inc., 2018. [Online]. Available: https://proceedings.neurips.cc/paper/2018/file/801fd8c2a4e79c1d24a40dc735c051ae-Paper.pdf
  * [49] E. Agustsson, M. Tschannen, F. Mentzer, R. Timofte, and L. V. Gool, “Generative adversarial networks for extreme learned image compression,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2019.
  * [50] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein generative adversarial networks,” in _Proceedings of the 34th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, D. Precup and Y. W. Teh, Eds., vol. 70. PMLR, 06–11 Aug 2017, pp. 214–223.
  * [51] S. Nowozin, B. Cseke, and R. Tomioka, “f-gan: Training generative neural samplers using variational divergence minimization,” in _Advances in Neural Information Processing Systems_ , D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, Eds., vol. 29. Curran Associates, Inc., 2016. [Online]. Available: https://proceedings.neurips.cc/paper/2016/file/cedebb6e872f539bef8c3f919874e9d7-Paper.pdf
  * [52] M. Akbari, J. Liang, and J. Han, “Dsslic: Deep semantic segmentation-based layered image compression,” in _ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2019, pp. 2042–2046.
  * [53] K. Mahalingaiah, H. Sharma, P. Kaplish, and I. Cheng, “Semantic learning for image compression (slic),” in _Smart Multimedia_ , T. McDaniel, S. Berretti, I. D. D. Curcio, and A. Basu, Eds. Cham: Springer International Publishing, 2020, pp. 57–66.
  * [54] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing network,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , July 2017.
  * [55] J. Fu, J. Liu, H. Tian, Y. Li, Y. Bao, Z. Fang, and H. Lu, “Dual attention network for scene segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * [56] M. Mirza and S. Osindero, “Conditional generative adversarial nets,” _CoRR_ , vol. abs/1411.1784, 2014. [Online]. Available: http://arxiv.org/abs/1411.1784
  * [57] J. Johnson, R. Krishna, M. Stark, L.-J. Li, D. A. Shamma, M. S. Bernstein, and L. Fei-Fei, “Image retrieval using scene graphs,” in _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015, pp. 3668–3678.
  * [58] X. Chang, P. Ren, P. Xu, Z. Li, X. Chen, and A. G. Hauptmann, “A comprehensive survey of scene graphs: Generation and application,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pp. 1–1, 2021.
  * [59] R. Wang, Z. Wei, P. Li, Q. Zhang, and X. Huang, “Storytelling from an image stream using scene graphs,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 05, pp. 9185–9192, Apr. 2020. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/6455
  * [60] Y. Blau and T. Michaeli, “Rethinking lossy compression: The rate-distortion-perception tradeoff,” in _Proceedings of the 36th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09–15 Jun 2019, pp. 675–685.
  * [61] M. Tschannen, E. Agustsson, and M. Lucic, “Deep generative models for distribution-preserving lossy compression,” in _Advances in Neural Information Processing Systems_ , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., vol. 31. Curran Associates, Inc., 2018. [Online]. Available: https://proceedings.neurips.cc/paper/2018/file/801fd8c2a4e79c1d24a40dc735c051ae-Paper.pdf
  * [62] Z. Wang, G. Wu, H. Sheikh, E. Simoncelli, E.-H. Yang, and A. Bovik, “Quality-aware images,” _IEEE Transactions on Image Processing_ , vol. 15, no. 6, pp. 1680–1689, 2006.
  * [63] L. Theis and A. B. Wagner, “A coding theorem for the rate-distortion-perception function,” in _Neural Compression: From Information Theory to Applications – Workshop @ ICLR 2021_ , 2021. [Online]. Available: https://openreview.net/forum?id=BzUaLGtKecs
  * [64] C. T. Li and A. E. Gamal, “Strong functional representation lemma and applications to coding theorems,” _IEEE Transactions on Information Theory_ , vol. 64, no. 11, pp. 6967–6978, 2018.
  * [65] G. Zhang, J. Qian, J. Chen, and A. Khisti, “Universal rate-distortion-perception representations for lossy compression,” in _Advances in Neural Information Processing Systems_ , M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 11 517–11 529. [Online]. Available: https://proceedings.neurips.cc/paper/2021/file/5fde40544cff0001484ecae2466ce96e-Paper.pdf
  * [66] L. Theis and E. Agustsson, “On the advantages of stochastic encoders,” in _Neural Compression: From Information Theory to Applications – Workshop @ ICLR 2021_ , 2021. [Online]. Available: https://openreview.net/forum?id=FZ0f-znv62
  * [67] A. B. Wagner, “The rate-distortion-perception tradeoff: The role of common randomness,” 2022. [Online]. Available: https://arxiv.org/abs/2202.04147
  * [68] M. Li, J. Klejsa, and W. B. Kleijn, “On distribution preserving quantization,” 2011. [Online]. Available: https://arxiv.org/abs/1108.3728
  * [69] P. Manoranjan, M. Murshed, and L. S. Dooley, “Very low bit-rate video coding,” in _Video Data Management and Information Retrieval_ , S. Deb, Ed. Hershey, PA: IGI Global, 2005, pp. 100–148.
  * [70] B. K. P. Horn and B. G. Schunck, “Determining optical flow,” _Artificial Intelligence_ , vol. 17, no. 1, pp. 185–203, 1981.
  * [71] J. Ostermann, “Object-based analysis-synthesis coding based on the source model of moving rigid 3d objects,” _Signal Processing: Image Communication_ , vol. 6, no. 2, pp. 143–161, 1994. [Online]. Available: https://www.sciencedirect.com/science/article/pii/0923596594900124
  * [72] R. Talluri, K. Oehler, T. Barmon, J. Courtney, A. Das, and J. Liao, “A robust, scalable, object-based video compression technique for very low bit-rate coding,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 7, no. 1, pp. 221–233, 1997.
  * [73] L. Duan, J. Liu, W. Yang, T. Huang, and W. Gao, “Video coding for machines: A paradigm of collaborative compression and intelligent analytics,” _Trans. Image Processing_ , vol. 29, pp. 8680–8695, jan 2020.
  * [74] R. Gbadeyan and C. Joslin, “Object based hybrid video compression,” in _Proce. Int’l Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP,_ , INSTICC. SciTePress, 2021, pp. 785–792.
  * [75] R. Dobrushin and B. Tsybakov, “Information transmission with additional noise,” _IRE Transactions on Information Theory_ , vol. 8, no. 5, pp. 293–304, 1962.
  * [76] J. Liu, W. Zhang, and H. V. Poor, “A rate-distortion framework for characterizing semantic information,” in _IEEE International Symposium on Information Theory (ISIT)_ , 2021, pp. 2894–2899.
  * [77] P. A. Stavrou and M. Kountouris, “A Rate Distortion Approach to Goal-Oriented Communication,” _techrxiv_ , 2 2022. [Online]. Available: https://www.techrxiv.org/articles/preprint/A_Rate_Distortion_Approach_to_Goal-Oriented_Communication/19128026
  * [78] J. Wolf and J. Ziv, “Transmission of noisy information to a noisy receiver with minimum distortion,” _IEEE Transactions on Information Theory_ , vol. 16, no. 4, pp. 406–411, 1970.
  * [79] T. Berger, Z. Zhang, and H. Viswanathan, “The ceo problem [multiterminal source coding],” _IEEE Transactions on Information Theory_ , vol. 42, no. 3, pp. 887–902, 1996.
  * [80] H. Viswanathan and T. Berger, “The quadratic Gaussian CEO problem,” _IEEE Transactions on Information Theory_ , vol. 43, no. 5, pp. 1549–1559, 1997.
  * [81] Y. Oohama, “Rate-distortion theory for Gaussian multiterminal source coding systems with several side informations at the decoder,” _IEEE Transactions on Information Theory_ , vol. 51, no. 7, pp. 2577–2593, 2005.
  * [82] T.-A. Courtade and T. Weissman, “Multiterminal source coding under logarithmic loss,” _IEEE Trans. on Info. Theory_ , vol. 60, pp. 740–761, Jan. 2014\.
  * [83] Y. Ugur, I.-E. Aguerri, and A. Zaidi, “Vector gaussian ceo problem under logarithmic loss,” _IEEE Trans. on Info. Theory. Submitted for publication_ , 2018.
  * [84] R. M. Gray, “Conditional rate-distortion theory,” pp. 6502–2, Oct 1972.
  * [85] A. D. Wyner and J. Ziv, “The rate-distortion function for source coding with side information at the decoder,” _IEEE Trans. Inf. Theory_ , vol. 22, pp. 1–10, Jan. 1976.
  * [86] A. Aaron, E. Setton, and B. Girod, “Towards practical wyner-ziv coding of video,” in _Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429)_ , vol. 3, 2003, pp. III–869.
  * [87] A. Aaron, R. Zhang, and B. Girod, “Wyner-ziv coding of motion video,” in _Conference Record of the Thirty-Sixth Asilomar Conference on Signals, Systems and Computers, 2002._ , vol. 1. IEEE, 2002, pp. 240–244.
  * [88] R. Gray and A. Wyner, “Source coding for a simple network,” _Bell System Technical Journal_ , vol. 53, no. 9, pp. 1681–1721, 1974.
  * [89] N. Tishby, F. Pereira, and W. Bialek, “The information bottleneck method,” in _Proc. 37th Annu. Allerton Conf. Commun., Control, Comput._ , 1999, pp. 368–377.
  * [90] I. Sutskover, S. Shamai, and J. Ziv, “Extremes of information combining,” _IEEE Trans. Inform. Theory_ , vol. 51, no. 04, pp. 1313–1325, 2005.
  * [91] I. Land and J. Huber, “Information combining,” _Foundations and Trends in Commun. and Inform. Theory_ , vol. 03, pp. 227–230, Nov. 2006.
  * [92] Y. Steinberg, “Coding and common reconstruction,” _IEEE Trans. Inf. Theory_ , vol. IT-11, pp. 4995–5010, Nov. 2009.
  * [93] A. D. Wyner, “On source coding with side information at the decoder,” vol. IT-21, pp. 294–300, May 1975.
  * [94] R. Ahlswede and J. Korner, “Source coding with side information and a converse for degraded broadcast channels,” _IEEE Trans. Inf. Theory_ , vol. 21, no. 6, pp. 629–637, November 1975.
  * [95] E. Erkip and T. M. Cover, “The efficiency of investment information,” _IEEE Trans. Info. Theory_ , vol. 44, no. 3, pp. 1026–1040, May 1998.
  * [96] I. Estella Aguerri, A. Zaidi, G. Caire, and S. Shamai Shitz, “On the capacity of cloud radio access networks with oblivious relaying,” _IEEE Transactions on Information Theory_ , vol. 65, no. 7, pp. 4575–4596, 2019.
  * [97] M. Vera, P. Piantanida, and L. R. Vega, “The role of information complexity and randomization in representation learning.” [Online]. Available: https://arxiv.org/abs/1802.05355
  * [98] A. Alemi, I. Fischer, J. Dillon, and K. Murphy, “Deep variational information bottleneck,” in _ICLR_ , 2017. [Online]. Available: https://arxiv.org/abs/1612.00410
  * [99] B. Dai, C. Zhu, B. Guo, and D. Wipf, “Compressing neural networks using the variational information bottleneck,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 1135–1144.
  * [100] A. Zaidi, I. Estella-Aguerri, and S. Shamai, “On the information bottleneck problems: Models, connections, applications and information theoretic views,” _Entropy_ , vol. 22, no. 2, p. 151, 2020.
  * [101] Z. Goldfeld and Y. Polyanskiy, “The information bottleneck problem and its applications in machine learning,” _IEEE Journal on Selected Areas in Information Theory_ , vol. 1, no. 1, pp. 19–38, 2020.
  * [102] Q. Yang, P. Piantanida, and D. Gündüz, “The multi-layer information bottleneck problem,” in _2017 IEEE Information Theory Workshop (ITW)_ , 2017, pp. 404–408.
  * [103] M. Vera, L. Rey Vega, and P. Piantanida, “Collaborative information bottleneck,” _IEEE Transactions on Information Theory_ , vol. 65, no. 2, pp. 787–815, 2019.
  * [104] I. E. Aguerri and A. Zaidi, “Distributed variational representation learning,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 43, no. 1, pp. 120–138, 2021.
  * [105] P. Harremoes and N. Tishby, “The information bottleneck revisited or how to choose a good distortion measure,” in _Proc. IEEE Int. Symp. Information Theory_ , Jun. 2007, pp. 566–570.
  * [106] R.-L. Dobrushin and B. S. Tsybakov, “Information transmission with additional noise,” _IRE Trans. on Info. Theory_ , vol. 85, pp. 293–304, 1962.
  * [107] H.-S. Witsenhausen and A.-D. Wyner, “A conditional entropy bound for a pair of discrete random variables,” _IEEE Trans. on Info. Theory_ , vol. IT-21, pp. 493–501, Sep. 1975.
  * [108] H.-S. Witsenhausen, “Indirect rate distortion problems,” _IEEE Trans. on Info. Theory_ , vol. IT-26, pp. 518–521, Sep. 1980.
  * [109] Y. Ugur, I. Estella-Aguerri, and A. Zaidi, “Vector Gaussian CEO problem under logarithmic loss and applications,” _IEEE Transactions on Information Theory_ , vol. 66, no. 7, pp. 4183–4202, 2020.
  * [110] I. E. Aguerri and A. Zaidi, “Distributed information bottleneck method for discrete and Gaussian sources,” in _Proc. of Int. Zurich Seminar on Information and Communication, IZS_ , Zurich, Switzerland, 2018.
  * [111] T. Berger, “Decentralized estimation and decision theory,” in _IEEE 7th. Spring Workshop on Inf. Theory_ , Mt. Kisco, NY, Sep. 1979.
  * [112] T. S. Han and S. Amari, “Statistical inference under multiterminal data compression,” _IEEE Trans. Information Theory_ , vol. 44, no. 6, pp. 2300–2324, Oct. 1998.
  * [113] Z. Zhang and T. Berger, “Estimation via compressed information,” _IEEE Trans. on Info. Theory_ , vol. 34, no. 2, pp. 198–211, March 1988.
  * [114] T. S. Han and S. Amari, “Parameter estimation with multiterminal data compression,” _IEEE Trans. on Info. Theory_ , vol. 41, no. 6, pp. 1802–1833, Nov 1995.
  * [115] R. Ahlswede and M. V. Burnashev, “On minimax estimation in the presence of side information about remote data,” _The Annals of Statistics_ , vol. 18, no. 1, pp. 141–171, 1990.
  * [116] R. Ahlswede and I. Csiszár, “Hypothesis testing with communication constraints,” _IEEE Trans. Information Theory_ , vol. 32, no. 4, pp. 533–542, Jul. 1986.
  * [117] T. S. Han, “Hypothesis testing with multiterminal data compression,” _IEEE Transactions on Information Theory_ , vol. 33, no. 6, pp. 759–772, 1987\.
  * [118] T. Han and K. Kobayashi, “Exponential-type error probabilities for multiterminal hypothesis testing,” _IEEE Transactions on Information Theory_ , vol. 35, no. 1, pp. 2–14, 1989.
  * [119] H. Shalaby and A. Papamarcou, “Multiterminal detection with zero-rate data compression,” _IEEE Transactions on Information Theory_ , vol. 38, no. 2, pp. 254–267, 1992.
  * [120] J.-F. Chamberland and V. Veeravalli, “Decentralized detection in sensor networks,” _IEEE Transactions on Signal Processing_ , vol. 51, no. 2, pp. 407–416, 2003.
  * [121] M. S. Rahman and A. B. Wagner, “On the optimality of binning for distributed hypothesis testing,” _IEEE Transactions on Information Theory_ , vol. 58, no. 10, pp. 6282–6303, 2012.
  * [122] G. Katz, P. Piantanida, and M. Debbah, “Distributed binary detection with lossy data compression,” _IEEE Trans. Information Theory_ , vol. 63, no. 8, pp. 5207–5227, Mar. 2017.
  * [123] S. Salehkalaibar, M. Wigger, and L. Wang, “Hypothesis testing over the two-hop relay network,” _IEEE Trans. Information Theory_ , vol. 65, no. 7, pp. 4411–4433, Jul. 2019.
  * [124] N. Weinberger and Y. Kochman, “On the reliability function of distributed hypothesis testing under optimal detection,” _IEEE Trans. Information Theory_ , vol. 65, no. 8, pp. 4940–4965, Aug. 2019.
  * [125] P. Escamilla, M. Wigger, and A. Zaidi, “Distributed hypothesis testing: Cooperation and concurrent detection,” _IEEE Transactions on Information Theory_ , vol. 66, no. 12, pp. 7550–7564, 2020.
  * [126] E. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus, “Exploiting linear structure within convolutional networks for efficient evaluation,” in _Proc. Int’l Conf. on Neural Information Processing Systems_ , MA, USA, 2014, pp. 1269–1277.
  * [127] S. Hanson and L. Pratt, “Comparing biases for minimal network construction with back-propagation,” in _Proc. Advances in Neural Information Processing Systems (NIPS)_ , Denver, CO, USA, Nov. 1989.
  * [128] Y. LeCun, J. Denker, and S. Solla, “Optimal brain damage,” in _Proc. Advances in Neural Information Processing Systems (NIPS)_ , Denver, CO, USA, Nov. 1990.
  * [129] B. Hassibi, D. G. Stork, G. Wolff, and T. Watanabe, “Optimal brain surgeon: Extensions and performance comparisons,” in _Proc. Advances in Neural Information Processing Systems (NIPS)_ , San Francisco, CA, USA, Jul. 1993.
  * [130] J. Liu, S. Tripathi, U. Kurup, and M. Shah, “Pruning algorithms to accelerate convolutional neural networks for edge applications: A survey,” _arXiv preprint arXiv:2005.04275_ , 2020.
  * [131] V. Lebedev and V. Lempitsky, “Fast ConvNets using group-wise brain damage,” in _Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Las Vegas, NV, USA, June 2016.
  * [132] W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li, “Learning structured sparsity in deep neural networks,” in _Proc. Advances in Neural Information Processing Systems (NIPS)_ , Barcelona, Spain, Dec. 2016.
  * [133] C. H. Fiesler E, Choudry A, “Weight discretization paradigm for optical neural networks,” in _Optical Interconnections and Networks, Proc. of International Society for Optics and Photonics_ , vol. 1281, 1990, pp. 164–174.
  * [134] W. Balzer, M. Takahashi, J. Ohta, and K. Kyuma, “Weight quantization in boltzmann machines,” _Neural Networks_ , vol. 4, no. 3, pp. 405–409, 1991\. [Online]. Available: https://www.sciencedirect.com/science/article/pii/089360809190077I
  * [135] S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan, “Deep learning with limited numerical precision,” in _Proc. International Conference on Machine Learning (ICML)_ , Lille, France, Jul. 2015.
  * [136] M. Courbariaux, Y. Bengio, and J. P. David, “Training deep neural networks with low precision multiplications,” in _Proc. International Conference on Learning Representations Workshop (ICLR Workshop)_ , San Diego, CA, USA, May. 2015.
  * [137] ——, “Binaryconnect: Training deep neural networks with binary weights during propagations,” in _Proc. Advances in Neural Information Processing Systems (NIPS)_ , Montreal Canada, Dec. 2015.
  * [138] M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Bengio, “Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or -1,” _arXiv preprint arXiv:1602.02830_ , 2016.
  * [139] Y. Choi, M. El-Khamy, and J. Lee, “Towards the limit of network quantization,” in _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_ , 2017. [Online]. Available: https://openreview.net/forum?id=rJ8uNptgl
  * [140] W. Chen, J. T. Wilson, S. Tyree, K. Q. Weinberger, and Y. Chen, “Compressing neural networks with the hashing trick,” in _Proc. International Conference on Machine Learning (ICML)_ , Lille, France, Jul. 2015.
  * [141] Y. Gong, L. Liu, M. Yang, and L. Bourdev, “Compressing deep convolutional networks using vector quantization,” _arXiv preprint arXiv:1412.6115_ , 2014\.
  * [142] S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding,” in _Proc. International Conference on Learning Representations (ICLR)_ , San Juan, Puerto Rico, May. 2016.
  * [143] C. Bucilau, R. Caruana, and A. Niculescu-Mizil, “Model compression,” in _Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , ser. KDD ’06. New York, NY, USA: Association for Computing Machinery, 2006, pp. 535–541. [Online]. Available: https://doi.org/10.1145/1150402.1150464
  * [144] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” 2015, cite arxiv:1503.02531. [Online]. Available: http://arxiv.org/abs/1503.02531
  * [145] F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer, “Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <0.5mb model size,” 2016. [Online]. Available: https://arxiv.org/abs/1602.07360
  * [146] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” 2017. [Online]. Available: http://arxiv.org/abs/1704.04861
  * [147] X. Zhang, X. Zhou, M. Lin, and J. Sun, “Shufflenet: An extremely efficient convolutional neural network for mobile devices,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 6848–6856.
  * [148] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 2261–2269.
  * [149] L. Deng, G. Li, S. Han, L. Shi, and Y. Xie, “Model compression and hardware acceleration for neural networks: A comprehensive survey,” _Proceedings of the IEEE_ , vol. 108, no. 4, pp. 485–532, 2020.
  * [150] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y. Arcas, “Communication-Efficient Learning of Deep Networks from Decentralized Data,” in _Proceedings of the 20th International Conference on Artificial Intelligence and Statistics_ , ser. Proceedings of Machine Learning Research, A. Singh and J. Zhu, Eds., vol. 54. PMLR, 20–22 Apr 2017, pp. 1273–1282. [Online]. Available: https://proceedings.mlr.press/v54/mcmahan17a.html
  * [151] J. Korner and K. Marton, “How to encode the modulo-two sum of binary sources (corresp.),” _IEEE Transactions on Information Theory_ , vol. 25, no. 2, pp. 219–221, Mar 1979.
  * [152] T. Berger, _Multiterminal source coding_. New York, NY, USA: Spring-Verlag, 1977.
  * [153] A. Orlitsky and J. Roche, “Coding for computing,” _IEEE Transactions on Information Theory_ , vol. 47, no. 3, pp. 903–917, 2001.
  * [154] G. Simonyi, “Graph entropy: A survey,” _Combinatorial Optimization_ , vol. 20, pp. 399–441, 1995.
  * [155] H. Witsenhausen, “The zero-error side information problem and chromatic numbers (corresp.),” _IEEE Transactions on Information Theory_ , vol. 22, no. 5, pp. 592–593, 1976.
  * [156] G. Toderici, S. M. O’Malley, S. J. Hwang, D. Vincent, D. Minnen, S. Baluja, M. Covell, and R. Sukthankar, “Variable rate image compression with recurrent neural networks,” _Proc. of the Int. Conf. on Learning Representations (ICLR)_ , 2016.
  * [157] O. Rippel and L. Bourdev, “Real-time adaptive image compression,” in _Proc. Int. Conf. on Machine Learning (ICML)_ , vol. 70, Aug. 2017, pp. 2922–2930.
  * [158] J. Ballé, V. Laparra, and E. P. Simoncelli, “End-to-end optimized image compression,” in _Proc. of Int. Conf. on Learning Representations (ICLR)_ , Apr. 2017, pp. 1–27.
  * [159] D. Minnen, J. Ballé, and G. D. Toderici, “Joint autoregressive and hierarchical priors for learned image compression,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 10 771–10 780.
  * [160] D. Minnen and S. Singh, “Channel-wise autoregressive entropy models for learned image compression,” in _Int. Conf. on Image Compression (ICIP), 2020_ , 2020.
  * [161] E. Agustsson, D. Minnen, N. Johnston, J. Ballé, S. J. Hwang, and G. Toderici, “Scale-space flow for end-to-end optimized video compression,” in _2020 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Jun. 2020, pp. 8500–8509. [Online]. Available: https://ieeexplore.ieee.org/document/9157366/
  * [162] C.-Y. Wu, N. Singhal, and P. Krahenbuhl, “Video Compression through Image Interpolation,” in _2018 European Conference on Computer Vision (ECCV)_ , Sep. 2018, pp. 416–431. [Online]. Available: https://openaccess.thecvf.com/content_ECCV_2018/html/Chao-Yuan_Wu_Video_Compression_through_ECCV_2018_paper.html
  * [163] G. Lu, W. Ouyang, D. Xu, X. Zhang, C. Cai, and Z. Gao, “DVC: an end-to-end deep video compression framework,” in _2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Sep. 2019, pp. 11 006–11 015.
  * [164] O. Rippel, S. Nair, C. Lew, S. Branson, A. G. Anderson, and L. Bourdev, “Learned Video Compression,” pp. 3454–3463, Oct. 2019.
  * [165] A. Djelouah, J. Campos, S. Schaub-Meyer, and C. Schroers, “Neural inter-frame compression for video coding,” in _2019 International Conference on Computer Vision (ICCV)_ , Oct. 2019, pp. 6420–6428.
  * [166] A. Habibian, T. v. Rozendaal, J. M. Tomczak, and T. S. Cohen, “Video compression with rate-distortion autoencoders,” in _2019 International Conference on Computer Vision (ICCV)_ , Oct. 2019, pp. 7033–7042.
  * [167] J. Han, S. Lombardo, C. Schroers, and S. Mandt, “Deep generative video compression,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , vol. 32, Nov. 2019.
  * [168] Z. Hu, G. Lu, and D. Xu, “FVC: A new framework towards deep video compression in feature space,” in _2021 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 1502–1511.
  * [169] B. Liu, Y. Chen, S. Liu, and H.-S. Kim, “Deep learning in latent space for video prediction and compression,” in _2021 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 701–710.
  * [170] S. Kankanahalli, “End-to-end optimized speech coding with deep neural networks,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2018, pp. 2521–2525.
  * [171] W. B. Kleijn, F. S. C. Lim, A. Luebs, J. Skoglund, F. Stimberg, Q. Wang, and T. C. Walters, “Wavenet based low rate speech coding,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE Press, 2018, pp. 676–680.
  * [172] D. Petermann, S. Beack, and M. Kim, “Harp-net: Hyper-autoencoded reconstruction propagation for scalable neural audio coding,” _2021 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)_ , pp. 316–320, 2021.
  * [173] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “Soundstream: An end-to-end neural audio codec,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 30, pp. 495–507, 2022.
  * [174] Y. Yang, S. Mandt, and L. Theis, “An introduction to neural data compression,” 2022. [Online]. Available: https://arxiv.org/abs/2202.06533
  * [175] M. C. Anderson, M. S. Atkins, and J. Vaisey, “Task-oriented lossy compression of magnetic resonance images,” in _Medical Imaging 1996: Image Display_ , ser. Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Y. Kim, Ed., vol. 2707, Apr. 1996, pp. 239–250.
  * [176] L. Pu, M. W. Marcellin, A. Bilgin, and A. Ashok, “Image compression based on task-specific information,” in _2014 IEEE International Conference on Image Processing (ICIP)_ , 2014, pp. 4817–4821.
  * [177] C. Bulla, C. Feldmann, and M. Schink, “Region of interest encoding in video conference systems,” in _MMEDIA 2013_ , 2013.
  * [178] M. Xu, X. Deng, S. Li, and Z. Wang, “Region-of-interest based conversational hevc coding with hierarchical perception model of face,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 8, no. 3, pp. 475–489, 2014.
  * [179] F. A. Andalo, O. A. B. Penatti, and V. Testoni, “Transmitting what matters: Task-oriented video composition and compression,” in _Conference on Graphics, Patterns and Images (SIBGRAPI)_ , 2016, pp. 72–79.
  * [180] M. Minervini and S. A. Tsaftaris, “Classification-aware distortion metric for hevc intra coding,” in _2015 Visual Communications and Image Processing (VCIP)_ , 2015, pp. 1–4.
  * [181] R. Torfason, F. Mentzer, E. Ágústsson, M. Tschannen, R. Timofte, and L. V. Gool, “Towards image understanding from deep compression without decoding,” in _International Conference on Learning Representations_ , 2018. [Online]. Available: https://openreview.net/forum?id=HkXWCMbRW
  * [182] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” _CoRR_ , vol. abs/1512.03385, 2015. [Online]. Available: http://arxiv.org/abs/1512.03385
  * [183] N. Shlezinger and Y. C. Eldar, “Deep task-based quantization,” _Entropy_ , vol. 23, 2021.
  * [184] M. Shohat, G. Tsintsadze, N. Shlezinger, and Y. C. Eldar, “Deep quantization for mimo channel estimation,” in _ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2019, pp. 3912–3916.
  * [185] G. Li, L. Liu, X. Wang, X. Dong, P. Zhao, and X. Feng, “Auto-tuning neural network quantization framework for collaborative inference between the cloud and edge,” in _Artificial Neural Networks and Machine Learning – ICANN 2018_ , V. Kůrková, Y. Manolopoulos, B. Hammer, L. Iliadis, and I. Maglogiannis, Eds. Cham: Springer International Publishing, 2018, pp. 402–411.
  * [186] H. Li, C. Hu, J. Jiang, Z. Wang, Y. Wen, and W. Zhu, “Jalad: Joint accuracy-and latency-aware deep structure decoupling for edge-cloud execution,” in _2018 IEEE 24th International Conference on Parallel and Distributed Systems (ICPADS)_. Los Alamitos, CA, USA: IEEE Computer Society, dec 2018, pp. 671–678. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/PADSW.2018.8645013
  * [187] A. E. Eshratifar, A. Esmaili, and M. Pedram, “Bottlenet: A deep learning architecture for intelligent mobile cloud computing services,” in _2019 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED)_ , 2019, pp. 1–6.
  * [188] Y. Matsubara, R. Yang, M. Levorato, and S. Mandt, “Supervised compression for resource-constrained edge computing systems,” in _2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , 2022, pp. 923–933.
  * [189] M. Jankowski, D. Gunduz, and K. Mikolajczyk, “Wireless image retrieval at the edge,” _IEEE Journal on Selected Areas in Communications_ , vol. 39, no. 1, pp. 89–100, 2021.
  * [190] D. Liu, H. Zhang, and Z. Xiong, “On the classification-distortion-perception tradeoff,” in _Advances in Neural Information Processing Systems_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, Inc., 2019. [Online]. Available: https://proceedings.neurips.cc/paper/2019/file/6c29793a140a811d0c45ce03c1c93a28-Paper.pdf
  * [191] Z. Chen and T. He, “Learning based facial image compression with semantic fidelity metric,” _Neurocomput._ , vol. 338, no. C, pp. 16–25, apr 2019\. [Online]. Available: https://doi.org/10.1016/j.neucom.2019.01.086
  * [192] X. Li, J. Shi, and Z. Chen, “Task-driven semantic coding via reinforcement learning,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 6307–6320, 2021.
  * [193] K. He, G. Gkioxari, P. DollÃ¡r, and R. Girshick, “Mask r-cnn,” in _2017 IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 2980–2988.
  * [194] T. Goblick, “Theoretical limitations on the transmission of data from analog sources,” _IEEE Transactions on Information Theory_ , vol. 11, no. 4, pp. 558–567, 1965.
  * [195] M. Gastpar, B. Rimoldi, and M. Vetterli, “To code, or not to code: lossy source-channel communication revisited,” _IEEE Transactions on Information Theory_ , vol. 49, no. 5, pp. 1147–1158, 2003.
  * [196] T. Cover, A. Gamal, and M. Salehi, “Multiple access channels with arbitrarily correlated sources,” _IEEE Transactions on Information Theory_ , vol. 26, no. 6, pp. 648–657, 1980.
  * [197] C. E. Shannon, “Two-way communication channels,” in _Proc. 4th Berkeley Symposium Math. Stat. and Prob._ , vol. 1, 1961, pp. 611–644.
  * [198] G. Dueck, “A note on the multiple access channel with correlated sources (corresp.),” _IEEE Transactions on Information Theory_ , vol. 27, no. 2, pp. 232–235, 1981.
  * [199] W. Kang and S. Ulukus, “An outer bound for multiple access channels with correlated sources,” in _2006 40th Annual Conference on Information Sciences and Systems_ , 2006, pp. 240–244.
  * [200] J. Nayak, E. Tuncel, and D. Gunduz, “Wyner-ziv coding over broadcast channels: Digital schemes,” _IEEE Transactions on Information Theory_ , vol. 56, no. 4, pp. 1782–1799, 2010.
  * [201] S. Bross, A. Lapidoth, and S. Tinguely, “Broadcasting correlated Gaussians,” _IEEE Trans. Inf. Theory,_ , vol. 56, no. 7, pp. 3057–3068, Jul. 2010.
  * [202] A. Lapidoth and S. Tinguely, “Sending a bivariate Gaussian over a Gaussian MAC,” _IEEE Transactions on Information Theory_ , vol. 56, no. 6, pp. 2714–2752, Jun. 2010.
  * [203] ——, “Sending a bivariate Gaussian source over a Gaussian MAC with feedback,” _IEEE Transactions on Information Theory_ , vol. 56, no. 4, pp. 1852–1864, Apr. 2010.
  * [204] I. Estella Aguerri and D. Gündüz, “Joint source-channel coding with time-varying channel and side-information,” _IEEE Transactions on Information Theory_ , vol. 62, no. 2, pp. 736–753, 2016.
  * [205] S. Sreekumar and D. Gunduz, “Distributed hypothesis testing over discrete memoryless channels,” _IEEE Transactions on Information Theory_ , vol. 66, no. 4, pp. 2044–2066, 2020.
  * [206] H. Shimokawa, T. S. Han, and S. Amari, “Error bound of hypothesis testing with data compression,” in _Proc. IEEE Int. Symp. Inf. Theory_ , Trondheim, Norway, 1994.
  * [207] S. Borade, B. Nakiboğlu, and L. Zheng, “Unequal error protection: An information-theoretic perspective,” _IEEE Trans. on Info. Theory_ , vol. 55, no. 12, pp. 5511–5539, Dec. 2009.
  * [208] P. Minero, S. H. Lim, and Y.-H. Kim, “A unified approach to hybrid coding,” _IEEE Transactions on Information Theory_ , vol. 61, no. 4, pp. 1509–1523, 2015.
  * [209] J.-F. Chamberland and V. Veeravalli, “Asymptotic results for decentralized detection in power constrained wireless sensor networks,” _IEEE Journal on Selected Areas in Communications_ , vol. 22, no. 6, pp. 1007–1015, 2004.
  * [210] R. Niu, B. Chen, and P. Varshney, “Fusion of decisions transmitted over rayleigh fading channels in wireless sensor networks,” _IEEE Transactions on Signal Processing_ , vol. 54, no. 3, pp. 1018–1027, 2006.
  * [211] T. Duman and M. Salehi, “Decentralized detection over multiple-access channels,” _IEEE Transactions on Aerospace and Electronic Systems_ , vol. 34, no. 2, pp. 469–476, 1998.
  * [212] G. Mergen and L. Tong, “Type based estimation over multiaccess channels,” _IEEE Transactions on Signal Processing_ , vol. 54, no. 2, pp. 613–626, 2006\.
  * [213] C.-Z. Lee, L. P. Barnes, and A. Ozgur, “Over-the-air statistical estimation,” _IEEE Journal on Selected Areas in Communications_ , vol. 40, no. 2, pp. 548–561, 2022.
  * [214] J. Modestino and D. Daut, “Combined source-channel coding of images,” _IEEE Transactions on Communications_ , vol. 27, no. 11, pp. 1644–1659, 1979\.
  * [215] C. Moore and J. Gibson, “Self-orthogonal convolutional coding for the dpcm-aqb speech encoder,” _IEEE Transactions on Communications_ , vol. 32, no. 8, pp. 980–982, 1984.
  * [216] N. Farvardin, “A study of vector quantization for noisy channels,” _IEEE Transactions on Information Theory_ , vol. 36, no. 4, pp. 799–809, July 1990.
  * [217] N. Farvardin and V. Vaishampayan, “On the performance and complexity of channel-optimized vector quantizers,” _IEEE Transactions on Information Theory_ , vol. 37, no. 1, pp. 155–160, Jan 1991.
  * [218] M. Skoglund, “Soft decoding for vector quantization over noisy channels with memory,” _IEEE Trans. on Information Theory_ , vol. 45, no. 4, pp. 1293–1307, May 1999.
  * [219] I. Kozintsev and K. Ramchandran, “Robust image transmission over energy-constrained time-varying channels using multiresolution joint source-channel coding,” _IEEE Trans. on Signal Processing_ , vol. 46, no. 4, pp. 1012–1026, Apr. 1998.
  * [220] M. Skoglund, “On channel-constrained vector quantization and index assignment for discrete memoryless channels,” _IEEE Transactions on Information Theory_ , vol. 45, no. 7, pp. 2615–2622, Nov 1999.
  * [221] G. Cheung and A. Zakhor, “Bit allocation for joint source/channel coding of scalable video,” _IEEE Trans. on Image Processing_ , vol. 9, no. 3, pp. 340–356, Mar. 2000.
  * [222] B. Hochwald and K. Zeger, “Tradeoff between source and channel coding,” _IEEE Transactions on Information Theory_ , vol. 43, no. 5, pp. 1412–1424, 1997.
  * [223] S. Jakubczak and D. Katabi, “Softcast: One-size-fits-all wireless video,” _SIGCOMM Comput. Commun. Rev._ , vol. 40, no. 4, pp. 449–450, aug 2010.
  * [224] O. Y. Bursalioglu, G. Caire, and D. Divsalar, “Joint source-channel coding for deep space image transmission using rateless codes,” in _2011 Information Theory and Applications Workshop_ , 2011, pp. 1–10.
  * [225] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning internal representations by error propagation,” California Univ San Diego La Jolla Inst for Cognitive Science, Tech. Rep., 1985.
  * [226] E. Bourtsoulatze, D. Burth Kurka, and D. Gunduz, “Deep joint source-channel coding for wireless image transmission,” _IEEE Transactions on Cognitive Communications and Networking_ , vol. 5, no. 3, pp. 567–579, Sep. 2019.
  * [227] N. Farsad, M. Rao, and A. Goldsmith, “Deep learning for joint source-channel coding of text,” in _2018 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2018, pp. 2326–2330.
  * [228] D. B. Kurka and D. Gunduz, “Deepjscc-f: Deep joint source-channel coding of images with feedback,” _IEEE Journal on Selected Areas in Information Theory_ , vol. 1, no. 1, pp. 178–193, 2020.
  * [229] D. B. Kurka and D. Gunduz, “Bandwidth-agile image transmission with deep joint source-channel coding,” _IEEE Transactions on Wireless Communications_ , vol. 20, no. 12, pp. 8081–8095, 2021.
  * [230] T.-Y. Tung and D. Gündüz, “DeepWiVe: Deep-learning-aided wireless video transmission,” _ArXiv_ , vol. abs/2111.13034, 2021.
  * [231] B. Guler and A. Yener, “Semantic index assignment,” in _2014 IEEE International Conference on Pervasive Computing and Communication Workshops (PERCOM WORKSHOPS)_ , 2014, pp. 431–436.
  * [232] B. Guler, A. Yener, and A. Swami, “The semantic communication game,” _IEEE Transactions on Cognitive Communications and Networking_ , vol. 4, no. 4, pp. 787–802, 2018.
  * [233] Z. Weng and Z. Qin, “Semantic communication systems for speech transmission,” _IEEE Journal on Selected Areas in Communications_ , vol. 39, no. 8, pp. 2434–2444, 2021.
  * [234] H. Xie, Z. Qin, X. Tao, and K. B. Letaief, “Task-oriented multi-user semantic communications,” _arXiv preprint arXiv:2112.10255_ , Dec. 2021.
  * [235] Z. Weng, Z. Qin, and G. Y. Li, “Semantic communications for speech recognition,” in _Proc. IEEE Global Commun. Conf. (GLOBECOM)_ , Madrid, Spain, Dec. 2021.
  * [236] M. Schuster and K. Paliwal, “Bidirectional recurrent neural networks,” _IEEE Trans. Signal Process._ , vol. 45, no. 11, pp. 2673–2681, Nov. 1997\.
  * [237] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks,” in _Proc. Int. Conf. Mach. Learning (ICML)_ , Pittsburgh, USA, Jun. 2006, pp. 369–376.
  * [238] T. Han, Q. Yang, Z. Shi, S. He, and Z. Zhang, “Semantic-aware speech to text transmission with redundancy removal,” _arXiv preprint arXiv:2202.03211_ , Feb. 2022.
  * [239] H. Seo, J. Park, M. Bennis, and M. Debbah, “Semantics-native communication with contextual reasoning,” _arXiv preprint arXiv:2108.05681_ , Aug. 2021\.
  * [240] M. K. Farshbafan, W. Saad, and M. Debbah, “Common language for goal-oriented semantic communications: A curriculum learning framework,” _arXiv preprint arXiv:2111.08051_ , Nov. 2021.
  * [241] D. Burth Kurka and D. Gündüz, “Joint source-channel coding of images with (not very) deep learning,” in _Proc of International Zurich Seminar on Information and Communication (IZS 2020)_. ETH Zurich, 2020, pp. 90–94.
  * [242] J. Xu, B. Ai, W. Chen, A. Yang, P. Sun, and M. Rodrigues, “Wireless image transmission using deep source channel coding with attention modules,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 4, pp. 2315–2328, 2022.
  * [243] D. B. Kurka and D. Gündüz, “Successive refinement of images with deep joint source-channel coding,” in _2019 IEEE 20th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC)_. IEEE, 2019, pp. 1–5.
  * [244] J. Dai, S. Wang, K. Tan, Z. Si, X. Qin, K. Niu, and P. Zhang, “Nonlinear transform source-channel coding for semantic communications,” _IEEE Journal on Selected Areas in Communications_ , vol. 40, no. 8, pp. 2300–2316, 2022\.
  * [245] J. Schalkwijk and T. Kailath, “A coding scheme for additive noise channels with feedback–part I: No bandwidth constraint,” _IEEE Transactions on Information Theory_ , vol. 12, no. 2, pp. 172–182, 1966.
  * [246] R. G. Gallager and B. Nakiboglu, “Variations on a theme by Schalkwijk and Kailath,” _IEEE Transactions on Information Theory_ , vol. 56, no. 1, pp. 6–17, 2010.
  * [247] J. Schalkwijk and L. Bluestein, “Transmission of analog waveforms through channels with feedback (corresp.),” _IEEE Transactions on Information Theory_ , vol. 13, no. 4, pp. 617–619, 1967.
  * [248] M. Yang, C. Bian, and H.-S. Kim, “Ofdm-guided deep joint source channel coding for wireless multipath fading channels,” _IEEE Transactions on Cognitive Communications and Networking_ , vol. 8, no. 2, pp. 584–599, 2022.
  * [249] H. Wu, Y. Shao, K. Mikolajczyk, and D. GÃ¼ndÃ¼z, “Channel-adaptive wireless image transmission with ofdm,” _IEEE Wireless Communications Letters_ , pp. 1–1, 2022.
  * [250] T.-Y. Tung, D. B. Kurka, M. Jankowski, and D. Gunduz, “Deepjscc-q: Constellation constrained deep joint source-channel coding,” _arXiv preprint arXiv:2206.08100_ , 2022. [Online]. Available: https://arxiv.org/abs/2206.08100
  * [251] K. Choi, K. Tatwawadi, A. Grover, T. Weissman, and S. Ermon, “Neural joint source-channel coding,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 1182–1192.
  * [252] F. Liu, W. Tong, Z. Sun, and C. Guo, “Task-oriented semantic communication systems based on extended rate-distortion theory,” _arXiv preprint arXiv:2201.10929_ , Jan. 2022.
  * [253] T.-Y. Tung, D. B. Kurka, and D. Gunduz, “Deepjscc: The future of wireless video transmission,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP): Show and Tell Demonstrations_ , 2020.
  * [254] H. You, L. Dai, S. Kim, and C.-B. Chae, “Demo: Real-time semantic communications with a vision transformer,” in _Proc. IEEE International Conference on Communications_ , 2022.
  * [255] T.-Y. Tung and D. Gunduz, “Deepwive: Deep-learning-aided wireless video transmission,” _IEEE Journal on Selected Areas in Communications_ , to appear, 2022.
  * [256] E. Agustsson, D. Minnen, N. Johnston, J. BallÃ©, S. J. Hwang, and G. Toderici, “Scale-space flow for end-to-end optimized video compression,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 8500–8509.
  * [257] J. Shao, Y. Mao, and J. Zhang, “Learning task-oriented communication for edge inference: An information bottleneck approach,” _IEEE J. Sel. Areas Commun._ , vol. 40, no. 1, pp. 197–211, Nov. 2022.
  * [258] M. Jankowski, D. GÃ¼ndÃ¼z, and K. Mikolajczyk, “Airnet: Neural network transmission over the air,” in _IEEE International Symposium on Information Theory (ISIT)_ , 2022, pp. 2451–2456.
  * [259] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. Graf, “Pruning Filters for Efficient Vonvnets,” in _Int’l Conf. on Learning Repr. (ICLR)_ , 2017.
  * [260] J. Frankle and M. Carbin, “The lottery ticket hypothesis: Finding sparse, trainable neural networks,” in _International Conference on Learning Representations (ICLR)_ , 2019.
  * [261] V. A. Kotelnikov, _The Theory of Optimum Noise Immunity_. McGraw-Hill Book Company, Inc., 1959.
  * [262] Y. Shao, S. C. Liew, and D. Gündüz, “Denoising noisy neural networks: A bayesian approach with compensation,” _arXiv:2105.10699_ , 2021.
  * [263] M. M. Amiri and D. Gunduz, “Machine learning at the wireless edge: Distributed stochastic gradient descent over-the-air,” _IEEE Transactions on Signal Processing_ , vol. 68, pp. 2155–2169, Mar. 2020.
  * [264] ——, “Federated learning over wireless fading channels,” _IEEE Transactions on Wireless Communications_ , vol. 19, no. 5, pp. 3546–3557, May 2020\.
  * [265] F. Sattler, S. Wiedemann, K.-R. MÃ¼ller, and W. Samek, “Sparse binary compression: Towards distributed deep learning with minimal communication,” in _2019 International Joint Conference on Neural Networks (IJCNN)_ , 2019, pp. 1–8.
  * [266] N. H. Tran, W. Bao, A. Zomaya, M. N. H. Nguyen, and C. S. Hong, “Federated learning over wireless networks: Optimization model design and analysis,” in _IEEE INFOCOM 2019 - IEEE Conference on Computer Communications_ , 2019, pp. 1387–1395.
  * [267] M. Chen, Z. Yang, W. Saad, C. Yin, H. V. Poor, and S. Cui, “A joint learning and communications framework for federated learning over wireless networks,” _IEEE Transactions on Wireless Communications_ , vol. 20, no. 1, pp. 269–283, 2021.
  * [268] G. Zhu, Y. Du, D. Gunduz, and K. Huang, “One-bit over-the-air aggregation for communication-efficient federated edge learning: Design and convergence analysis,” _IEEE Transactions on Wireless Communications_ , pp. 1–1, 2020.
  * [269] M. M. Amiri, T. M. Duman, D. Gunduz, S. R. Kulkarni, and H. V. Poor, “Blind federated edge learning,” _IEEE Transactions on Wireless Communications_ , vol. 20, no. 8, pp. 5129–5143, 2021.
  * [270] E. Ozfatura, S. Rini, and D. Gündüz, “Decentralized sgd with over-the-air computation,” in _GLOBECOM 2020 - 2020 IEEE Global Communications Conference_ , 2020, pp. 1–6.
  * [271] H. Xing, O. Simeone, and S. Bi, “Federated learning over wireless device-to-device networks: Algorithms and convergence analysis,” _IEEE Journal on Selected Areas in Communications_ , vol. 39, no. 12, pp. 3723–3741, 2021.
  * [272] M. Chen, D. GÃ¼ndÃ¼z, K. Huang, W. Saad, M. Bennis, A. V. Feljan, and H. V. Poor, “Distributed learning in wireless networks: Recent progress and future challenges,” _IEEE Journal on Selected Areas in Communications_ , vol. 39, no. 12, pp. 3579–3605, 2021.
  * [273] A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihood from incomplete data via the em algorithm,” _Journal of the Royal Statistical Society, Series B_ , vol. 39, no. 1, pp. 1–38, 1977.
  * [274] R. Blahut, “Computation of channel capacity and rate-distortion functions,” _IEEE Transactions on Information Theory_ , vol. 18, no. 4, pp. 460–473, Jul 1972.
  * [275] S. Arimoto, “An algorithm for computing the capacity of arbitrary discrete memoryless channels,” _IEEE Trans. Inf. Theory_ , vol. IT-18, no. 1, pp. 12 – 20, Jan. 1972.
  * [276] A. Globerson and N. Tishby, “On the optimality of the Gaussian information bottleneck curve,” _technical report, Hebrew University_ , 2004.
  * [277] G. Chechik, A. Globerson, N. Tishby, and Y. Weiss, “Information bottleneck for Gaussian variables.” _Journal of Machine Learning Research_ , vol. 6, pp. 165–188, Feb. 2005.
  * [278] D. P. Kingma and M. Welling, “Auto-encoding variational bayes.” _CoRR_ , vol. abs/1312.6114, 2013. [Online]. Available: http://dblp.uni-trier.de/db/journals/corr/corr1312.html#KingmaW13
  * [279] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner, “$\beta$-vae: Learning basic visual concepts with a constrained variational framework,” 2016.
  * [280] J. Shao, Y. Mao, and J. Zhang, “Learning task-oriented communication for edge inference: An information bottleneck approach,” _IEEE Journal on Selected Areas in Communications_ , vol. 40, no. 1, pp. 197–211, 2021.
  * [281] F. Pezone, S. Barbarossa, and P. Di Lorenzo, “Goal-oriented communication for edge learning based on the information bottleneck,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 8832–8836.
  * [282] C. J. Maddison, A. Mnih, and Y. W. Teh, “The concrete distribution: A continuous relaxation of discrete random variables,” _CoRR_ , vol. abs/1611.00712, 2016. [Online]. Available: http://arxiv.org/abs/1611.00712
  * [283] E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with gumbel-softmax,” 2017. [Online]. Available: https://arxiv.org/abs/1611.01144
  * [284] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization.” _CoRR_ , vol. abs/1412.6980, 2014. [Online]. Available: http://dblp.uni-trier.de/db/journals/corr/corr1412.html#KingmaB14
  * [285] R. D. Yates, Y. Sun, D. R. Brown, S. K. Kaul, E. Modiano, and S. Ulukus, “Age of information: An introduction and survey,” _IEEE Journal on Selected Areas in Commun._ , vol. 39, no. 5, pp. 1183–1210, May 2021.
  * [286] A. Kosta, N. Pappas, and V. Angelakis, “Age of information: A new concept, metric, and tool,” _Foundations and Trends in Networking_ , vol. 12, no. 3, pp. 162–259, 2017.
  * [287] M. A. Abd-Elmagid, N. Pappas, and H. S. Dhillon, “On the role of age of information in the Internet of things,” _IEEE Commun. Magazine_ , vol. 57, no. 12, pp. 72–77, Dec. 2019.
  * [288] Y. Sun, I. Kadota, R. Talak, and E. Modiano, “Age of information: A new metric for information freshness,” _Synthesis Lectures on Communication Networks_ , vol. 12, no. 2, pp. 1–224, 2019.
  * [289] R. D. Yates and S. K. Kaul, “The age of information: Real-time status updating by multiple sources,” _IEEE Trans. on Info. Theory_ , vol. 65, no. 3, pp. 1807–1827, Mar. 2019.
  * [290] R. D. Yates, “The age of information in networks: Moments, distributions, and sampling,” _IEEE Trans. on Info. Theory_ , vol. 66, no. 9, pp. 5712–5728, Sept. 2020.
  * [291] A. Soysal and S. Ulukus, “Age of information in g/g/1/1 systems: Age expressions, bounds, special cases, and optimization,” _IEEE Transactions on Information Theory_ , vol. 67, no. 11, pp. 7477–7489, 2021.
  * [292] M. Costa, M. Codreanu, and A. Ephremides, “On the age of information in status update systems with packet management,” _IEEE Trans. on Info. Theory_ , vol. 62, no. 4, pp. 1897–1910, Apr. 2016.
  * [293] C. Kam, S. Kompella, G. D. Nguyen, J. E. Wieselthier, and A. Ephremides, “On the age of information with packet deadlines,” _IEEE Trans. on Info. Theory_ , vol. 64, no. 9, pp. 6419–6428, Sept. 2018.
  * [294] M. Moltafet, M. Leinonen, and M. Codreanu, “On the age of information in multi-source queueing models,” _IEEE Trans. on Commun._ , vol. 68, no. 8, pp. 5003–5017, Aug. 2020.
  * [295] A. Kosta, N. Pappas, A. Ephremides, and V. Angelakis, “Age of information performance of multiaccess strategies with packet management,” _Journal of Commun. and Networks_ , vol. 21, no. 3, pp. 244–255, June 2019.
  * [296] Y. Inoue, H. Masuyama, T. Takine, and T. Tanaka, “A general formula for the stationary distribution of the age of information and its application to single-server queues,” _IEEE Trans. Info. Theory_ , vol. 65, no. 12, pp. 8305–8324, Dec. 2019.
  * [297] J. P. Champati, H. Al-Zubaidy, and J. Gross, “On the distribution of AoI for the GI/GI/1/1 and GI/GI/1/2* systems: Exact expressions and bounds,” in _Proc., IEEE INFOCOM_ , 2019.
  * [298] M. A. Abd-Elmagid and H. S. Dhillon, “Closed-form characterization of the MGF of AoI in energy harvesting status update systems,” IEEE Trans. on Info. Theory, to appear.
  * [299] ——, “Age of information in multi-source updating systems powered by energy harvesting,” _IEEE Journal on Selected Areas in Info. Theory_ , vol. 3, no. 1, pp. 98–112, Mar. 2022.
  * [300] A. Maatouk, S. Kriouile, M. Assaad, and A. Ephremides, “The age of incorrect information: A new performance metric for status updates,” _IEEE/ACM Transactions on Networking_ , vol. 28, no. 5, pp. 2215–2228, 2020.
  * [301] Y. Sun, Y. Polyanskiy, and E. Uysal, “Sampling of the wiener process for remote estimation over a channel with random delay,” _IEEE Transactions on Information Theory_ , vol. 66, no. 2, pp. 1118–1135, 2020.
  * [302] H. S. Witsenhausen, “On the structure of real-time source coders,” _The Bell System Technical Journal_ , vol. 58, no. 6, pp. 1437–1451, 1979.
  * [303] N. Gaarder and D. Slepian, “On optimal finite-state digital transmission systems,” vol. 28, no. 2, pp. 167–186, 1982.
  * [304] O. Imer and T. Basar, “Optimal estimation with limited measurements,” in _Proceedings of the 44th IEEE Conference on Decision and Control_ , 2005, pp. 1029–1034.
  * [305] Y. Xu and J. Hespanha, “Optimal communication logics in networked control systems,” in _2004 43rd IEEE Conference on Decision and Control (CDC) (IEEE Cat. No.04CH37601)_ , vol. 4, 2004, pp. 3527–3532 Vol.4.
  * [306] G. M. Lipsa and N. C. Martins, “Remote state estimation with communication costs for first-order lti systems,” _IEEE Transactions on Automatic Control_ , vol. 56, no. 9, pp. 2013–2025, 2011.
  * [307] J. Chakravorty and A. Mahajan, “Distortion-transmission trade-off in real-time transmission of gauss-markov sources,” in _2015 IEEE International Symposium on Information Theory (ISIT)_ , 2015, pp. 1387–1391.
  * [308] M. Rabi, G. V. Moustakides, and J. S. Baras, “Multiple sampling for estimation on a finite horizon,” in _Proceedings of the 45th IEEE Conference on Decision and Control_ , 2006, pp. 1351–1357.
  * [309] A. Nayyar, T. Basar, D. Teneketzis, and V. V. Veeravalli, “Optimal strategies for communication and remote estimation with an energy harvesting sensor,” _IEEE Transactions on Automatic Control_ , vol. 58, no. 9, pp. 2246–2260, 2013.
  * [310] O. Orhan, D. Gündüz, and E. Erkip, “Source-channel coding under energy, delay, and buffer constraints,” _IEEE Trans. on Wireless Commun._ , vol. 14, no. 7, pp. 3836–3849, 2015.
  * [311] T. Fine, “Properties of an optimum digital system and applications,” _IEEE Transactions on Information Theory_ , vol. 10, no. 4, pp. 287–296, 1964\.
  * [312] B. McMillan, “Communication systems which minimize coding noise,” _The Bell System Technical Journal_ , vol. 48, no. 9, pp. 3091–3112, 1969.
  * [313] J. Walrand and P. Varaiya, “Optimal causal coding - decoding problems,” _IEEE Transactions on Information Theory_ , vol. 29, no. 6, pp. 814–820, 1983\.
  * [314] A. Mahajan and D. Teneketzis, “Optimal design of sequential real-time communication systems,” _IEEE Transactions on Information Theory_ , vol. 55, no. 11, pp. 5317–5338, 2009.
  * [315] X. Gao, E. Akyol, and T. Basar, “Optimal estimation with limited measurements and noisy communication,” in _2015 54th IEEE Conference on Decision and Control (CDC)_ , 2015, pp. 1775–1780.
  * [316] X. Ren, J. Wu, K. H. Johansson, G. Shi, and L. Shi, “Infinite horizon optimal transmission power control for remote state estimation over fading channels,” _IEEE Transactions on Automatic Control_ , vol. 63, no. 1, pp. 85–100, 2018.
  * [317] J. Chakravorty and A. Mahajan, “Remote estimation over a packet-drop channel with markovian state,” _IEEE Transactions on Automatic Control_ , vol. 65, no. 5, pp. 2016–2031, 2020.
  * [318] S. Kaul, R. Yates, and M. Gruteser, “Real-time status: How often should one update?” in _Proc., IEEE INFOCOM_ , 2012.
  * [319] Y.-P. Hsu, E. Modiano, and L. Duan, “Scheduling algorithms for minimizing age of information in wireless broadcast networks with random arrivals,” IEEE Trans. on Mobile Computing, to appear.
  * [320] Y. Sun and B. Cyr, “Information aging through queues: A mutual information perspective,” in _2018 IEEE 19th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC)_ , 2018, pp. 1–5.
  * [321] T.-Y. Tung, S. Kobus, J. P. Roig, and D. Gündüz, “Effective communications: A joint learning and communication framework for multi-agent reinforcement learning over noisy channels,” _IEEE Journal on Selected Areas in Communications_ , vol. 39, no. 8, pp. 2590–2603, 2021.
  * [322] T.-Y. Tung, S. Kobus, and D. Gündüz, “Context-aware effective communications,” in _2021 55th Asilomar Conference on Signals, Systems, and Computers_ , 2021, pp. 334–339.
  * [323] A. Mostaani, O. Simeone, S. Chatzinotas, and B. Ottersten, “Learning-based Physical Layer Communications for Multiagent Collaboration,” in _IEEE Int’l Symp. on Personal, Indoor and Mobile Radio Comms. (PIMRC)_ , Sep. 2019, pp. 1–6.
  * [324] J. N. Foerster, Y. M. Assael, N. de Freitas, and S. Whiteson, “Learning to Communicate with Deep Multi-Agent Reinforcement Learning,” _arXiv:1605.06676 [cs]_ , May 2016, arXiv: 1605.06676.
  * [325] J. Jiang and Z. Lu, “Learning attentional communication for multi-agent cooperation,” in _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , 2018, pp. 7265–7275.
  * [326] N. Jaques, A. Lazaridou, E. Hughes, C. Gulcehre, P. Ortega, D. Strouse, J. Leibo, and N. deFreitas, “Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning,” _arXiv:1810.08647 [cs, stat]_ , Jun. 2019. [Online]. Available: http://arxiv.org/abs/1810.08647
  * [327] A. Das, T. Gervet, J. Romoff, D. Batra, D. Parikh, M. Rabbat, and J. Pineau, “TarMAC: Targeted multi-agent communication,” in _36th Int’l Conf. on Machine Learning_. Long Beach, California: PMLR, Jun 2019, pp. 1538–1546.