# Universal Joint Source-Channel Coding for Modulation-Agnostic Semantic
Communication

Yoon Huh, , Hyowoon Seo, , and Wan Choi Y. Huh and W. Choi are with the
Departement of Electrical and Computer Engineering and the Institute of New
Media and Communications, Seoul National University (SNU), Seoul 08826, Korea
(e-mail: {mnihy621, wanchoi}@snu.ac.kr)H. Seo is with the Department of
Electronics and Communications, Kwangwoon University, Seoul, Korea (e-mail:
hyowoonseo@kw.ac.kr)(_Corresponding Authors: Hyowoon Seo and Wan Choi_)

###### Abstract

From the perspective of joint source-channel coding (JSCC), there has been
significant research on utilizing semantic communication, which inherently
possesses analog characteristics, within digital device environments. However,
a single-model approach that operates modulation-agnostically across various
digital modulation orders has not yet been established. This article presents
the first attempt at such an approach by proposing a universal joint source-
channel coding (uJSCC) system that utilizes a single-model encoder-decoder
pair and trained vector quantization (VQ) codebooks. To support various
modulation orders within a single model, the operation of every neural network
(NN)-based module in the uJSCC system requires the selection of modulation
orders according to signal-to-noise ratio (SNR) boundaries. To address the
challenge of unequal output statistics from shared parameters across NN
layers, we integrate multiple batch normalization (BN) layers, selected based
on modulation order, after each NN layer. This integration occurs with minimal
impact on the overall model size. Through a comprehensive series of
experiments, we validate that this modulation-agnostic semantic communication
framework demonstrates superiority over existing digital semantic
communication approaches in terms of model complexity, communication
efficiency, and task effectiveness.

###### Index Terms:

semantic communication, modulation-agnostic, joint source-channel coding,
vector quantization.

##  I Introduction

We now have access to intelligent services encompassing smart X (e.g., home,
city, vehicle factory, healthcare, etc.) and extended (augmented and virtual)
reality, previously confined to the realm of imagination. However, the
widespread adoption of such services faces a significant obstacle from the
perspective of communication engineering. This challenge pertains to meeting
user demands, often referred to as quality of experience (QoE), by efficiently
and swiftly transmitting vast amounts of data. In response, sixth-generation
(6G) communication has emerged to address these demands, heralding an
anticipated paradigm shift from data-oriented to service (or task)-oriented
communication [1].

A pivotal driver of this transition is semantic communication [2, 3, 4],
evolving in tandem with advancements in machine learning (ML). Semantic
communication targets the resolution of Level B (semantic) and Level C
(effectiveness) communication issues [5, 6], with task-oriented joint source-
channel coding (JSCC) at its core [7, 8, 9, 10]. Unlike traditional
communication systems, which segregate source coding and channel coding
modules, often with some loss of optimality, semantic communication, geared
towards enhancing task performance, underscores an undeniable optimality gap
in such modular separation. Consequently, current research in semantic
communication focuses on harnessing ML capabilities to jointly optimize task
execution, source coding, and channel coding, culminating in the
implementation of neural network (NN)-based JSCC encoder-decoder pairs.

The challenge lies in the nature of NN training reliant on backpropagation,
necessitating the exploration of JSCC techniques to facilitate the exchange of
feature vectors within a continuous vector space between encoders and
decoders. Although the implementation of semantic communication through analog
transceivers might appear straightforward, it faces significant compatibility
issues with the prevailing digital device infrastructure. An alternative
strategy involves encoding the entries of feature vectors at the bit level and
utilizing traditional digital modulation techniques. However, this approach is
often inefficient, diminishing the benefits associated with semantic
communication. Therefore, for effective semantic communication within a
digital context, it is essential to incorporate digital modulation into the
JSCC design considerations.

In the contemporary research landscape, innovative methods have been
developed, notably the application of vector quantization (VQ) to the output
feature vectors of encoders, followed by conventional digital modulation
schemes (e.g., BPSK, 4QAM, 16QAM) or machine learning (ML)-based modulation
utilizing a trained constellation map. These approaches, cited in recent
studies [11, 12, 13], demonstrate robust performance within a specific
modulation order. Yet, the goal of achieving consistent semantic communication
performance across various modulation orders–—a concept termed _modulation-
agnostic semantic communication_ 111Throughout this article, the modulation
techniques discussed all pertain to digital modulation, and modulation-
agnostic semantic communication refers to a digital semantic communication
system designed to transmit or receive information without being constrained
by any specific modulation scheme.—–requires a comprehensive suite of JSCC
encoder-decoder pairs. In general, the optimal modulation order is contingent
upon the channel environment, and maintaining this flexibility is often
inefficient in terms of learning and hardware costs, including memory usage.
Consequently, the pursuit of a universal framework capable of operating
seamlessly across different modulation orders remains crucial for enhancing
efficiency.

In response to the outlined challenges, this article introduces a novel
framework for modulation-agnostic semantic communication, which is anchored in
a single-model universal JSCC scheme that is adept at functioning across
various modulation orders. To realize this, an NN-based uJSCC encoder-decoder
pair in conjunction with a VQ codebook is developed to utilize conventional
digital modulation schemes. Notably, our proposed single-model encoder-decoder
system demonstrates effective communication and task performance while
maintaining reduced model complexity and training overhead compared to other
benchmark approaches. In addition, the VQ codebook is not uniformly designed
but is trained jointly with the uJSCC encoder-decoder, ensuring robust
transmission of the extracted feature vectors.

The proposed framework is validated through a series of experiments in a
semantic communication setting for image transmission and reconstruction
tasks. We compare the proposed uJSCC against benchmark schemes, including
model-efficient JSCC (which leverages a single model trained simultaneously
for all modulation orders) and task-effective JSCC (which independently trains
multiple models for each modulation order). The proposed scheme outperforms
the model-efficient approach and achieves comparable or superior task
performance compared to the task-effective JSCC by employing multiple batch
normalization (BN) modules at each CNN layer. Furthermore, experiments
demonstrate that the proposed method consistently delivers excellent
performance across different encoder-decoder model sizes and with a higher
volume of symbol transmission.

###  I-A Related works

####  I-A1 Digital Semantic Communication

A recent study [11] introduced a deterministic encoder-decoder pair in a
digital semantic communication system using a VQ-variational autoencoder (VQ-
VAE) [14] trained with a straight-through estimator and a vision transformer
[15] backbone. The system enhances robustness against semantic noise through
adversarial training and selective masking of non-critical image patches,
identified via a feature importance module. However, there is a mismatch
between codebook cardinality and modulation order, requiring two 16QAM symbols
to transmit one index from a 256-cardinality codebook.

In [12], the authors proposed a VQ-based JSCC system employing digital
modulation and an information bottleneck (IB) [16] loss function. This loss
function’s hyperparameter regulates the balance between task-relevant
information and coded redundancy, thereby adjusting codebook robustness.
During training, they utilized Gumbel-softmax [17], using the inner product
between feature vectors and codewords as logits. This results in stochastic
encoding, where codeword indices are sampled based on logits. However, despite
the IB-optimized number of encoded bits, the actual transmission did not
reflect this because both codebook cardinality and modulation order were
fixed. Similarly, the encoder in [13] generates complex values, with symbols
sampled based on logits from distances between complex output values and
constellation map symbols. Another approach in [18] involved a stochastic
encoder trained with a mutual information-based loss function, also utilizing
Gumbel-softmax during training. In this system, symbols are directly sampled
from output logits, eliminating the need for a learned codebook. However,
these studies [12, 18, 13] face limitations in adapting to multiple modulation
orders as they require different neural network-based encoder-decoder
parameter pairs for each order.

####  I-A2 Modulation-Agnostic Semantic Communication

In recent work [19], the optimization of modulation order was addressed by
solving a robustness verification problem to achieve a certain robustness
probability, following the training of a semantic encoder and decoder. Another
study [20] optimized the channel coding rate for conventional LDPC channel
code through a similar robustness verification problem. This work utilized the
CROWN method [21] to assess feature importance and measure output perturbation
gaps from pretrained semantic encoders and decoders. As a separate source-
channel coding system was employed, modulation order was aligned with the
coding rate. However, both [19] and [20] did not incorporate digital
modulation in the training of their encoders and decoders; instead, they
applied fixed-bit quantization to analog outputs from the semantic encoder
during inference to facilitate digital modulation, indicating a lack of joint
design with the digital communication framework.

Meanwhile, [22] describes a system where a stochastic encoder produces logits
to sample bits, thereby creating bit sequences. The authors used a binary
symmetric erasure channel in their training scenario, simplifying the process
by assuming a 4QAM context. Following training, the modulation order is
adjusted according to channel SNR, with thresholds established based on
predetermined robustness levels for each bit position. These levels are taken
into account during training as well. However, the rationale behind training
under 4QAM conditions was not clearly articulated, and the determination of
robustness levels and SNR threshold adjustments was heuristic in nature, with
the goal of achieving a smooth performance curve at the cost of increased
complexity in system design.

###  I-B Contributions and Organization

The contributions of this article are summarized as follows:

  * •

This study explores methodologies that enable semantic communication within
digital communication environments. Notably, it proposes a modulation-agnostic
semantic communication framework that incorporates all the digital modulation
orders in training.

  * •

The cornerstone of the proposed framework is a uJSCC scheme that operates
based on a trained encoder-decoder pair and a VQ codebook. This uJSCC scheme
is capable of maintaining semantic communication as the optimal modulation
order adjusts to varying channel conditions. To the best of our knowledge,
this article presents the first instance of using a uJSCC scheme as a
universal model capable of digitalizing semantic communication across multiple
modulation orders.

  * •

Through extensive experimentation, we demonstrate that the proposed uJSCC
surpasses other benchmark schemes–—including model-efficient and task-
effective approaches—–in terms of model and training complexity, as well as
task performance.

  * •

The numerical results illustrate that the proposed model achieves satisfactory
reconstruction quality across all signal-to-noise ratio (SNR) ranges,
utilizing an efficiently trained universal NN structure. Further analysis
confirms that uJSCC can be effectively optimized for larger model sizes and in
scenarios involving the transmission of more symbols, thereby highlighting the
generality of the proposed system with fewer parameters.

The remainder of the article is structured as follows. Section II introduces
the system model and the problem under investigation, along with a discussion
of benchmark schemes. In Section III, we present a modulation-agnostic
semantic communication framework that employs a uJSCC approach, detailing its
implementation, modulation order selection, and training algorithm. Section IV
discusses the extensive experimental results that validate the performance of
the proposed scheme. Finally, conclusions are drawn in Section V.

###  I-C Notations

Vectors and matrices are expressed in bold lower-case and bold upper-case,
respectively. $\|\cdot\|_{2}$ denotes the $l_{2}$ norm of a vector.
$\mathbb{R}$ and $\mathbb{Z}$ represent, respectively, the real number set and
integer set. $[a{:}b]$ is a integer set of $\\{a,a+1,\dots,b\\}$.
$[\cdot]^{T}$ denotes the transpose of a matrix or vector.
$[\mathbf{A}]_{i:j,k:l}$ and $[\mathbf{a}]_{i:j}$ represents, respectively,
the sliced matrix of $\mathbf{A}$ with row indices from $i$ to $j$ and column
indices from $k$ to $l$, and the sliced vector of $\mathbf{a}$ with indices
from $i$ to $j$. $\mathcal{CN}(\boldsymbol{\mu},\boldsymbol{\Sigma})$ is
complex normal distribution with mean vector $\boldsymbol{\mu}$ and covariance
matrix $\boldsymbol{\Sigma}$. $\mathbf{I}_{m}$ is $m\times m$ identity matrix.
$\mathbf{E}[\cdot]$ denotes the expected value of a given random variable.

##  II Preliminaries

This section furnishes preliminary information concerning the system model,
the problem under investigation, and benchmark schemes.

###  II-A System Description

Figure 1: An illustration of a JSCC and VQ-based digital semantic
communication system model with 16QAM.

Consider a point-to-point semantic communication system designed for image
transmission tasks, incorporating JSCC encoder-decoder, vector quantizer-
dequantizer, and modulator-demodulator pairs, as depicted in Fig. 1. For the
sake of clarity and initial exposition, we presume a fixed digital modulation
scheme characterized by order $m$. However, we subsequently investigate the
system’s adaptability to accommodate a spectrum of modulation order options.
The operational structure and functionality of this system are delineated
comprehensively below.

The transmitter encodes a source image $\mathbf{X}\in\mathbb{R}^{C\times
H\times W}$, where $C$, $H$, and $W$ are the number of channels, the height
and width of the source image, respectively, into $N$ feature vectors of $D$
dimensions
$\mathbf{Y}=[\mathbf{y}_{1},\mathbf{y}_{2},\dots,\mathbf{y}_{N}]^{T}\in\mathbb{R}^{N\times
D}$ via an encoder $f_{\boldsymbol{\theta}}$ parameterized by
$\boldsymbol{\theta}$, such that
$\mathbf{Y}=f_{\boldsymbol{\theta}}(\mathbf{X})$. Then, VQ is done utilizing a
codebook with $m$ codewords
$\mathbf{C}=[\mathbf{c}_{1},\mathbf{c}_{2},\dots,\mathbf{c}_{m}]^{T}\in\mathbb{R}^{m\times
D}$ with the minimum distance criterion, i.e., the codeword index $z_{i}$ for
the $i$-th feature vector $\mathbf{y}_{i}$ is chosen as
$z_{i}=\operatorname*{arg\,min}_{j}\|\mathbf{y}_{i}-\mathbf{c}_{j}\|_{2}^{2}$.

Each codeword is mapped one-to-one to a symbol on a given modulation
constellation map
$\mathbb{S}=\\{\mathbf{s}_{1},\mathbf{s}_{2},\dots,\mathbf{s}_{m}\\}$, where
$\mathbf{s}_{i}\in\mathbb{R}^{2}$ and
$\mathbf{E}[\|\mathbf{s}_{i}\|_{2}^{2}]=P$ for all $i\in[1{:}m]$. The index
vector $\mathbf{z}=[z_{1},z_{2},\dots,z_{N}]^{T}\in[1{:}m]^{N}$ is modulated
into a symbol sequence
$\mathbf{s}=[\mathbf{s}_{z_{1}},\mathbf{s}_{z_{2}},\dots,\mathbf{s}_{z_{N}}]\in\mathbb{S}^{N}$
and sent through an additive white Gaussian noise (AWGN) channel. At the
receiver side, the receiver gets the symbol sequence with AWGN noise
$\mathbf{n}\sim\mathcal{CN}(0,\sigma^{2}\mathbf{I}_{N})$ as
$\hat{\mathbf{s}}=\mathbf{s}+\mathbf{n}$, where SNR is calculated as
$\frac{P}{\sigma^{2}}$. Then the receiver detects symbol sequence
$\bar{\mathbf{s}}=[\mathbf{s}_{\hat{z}_{1}},\mathbf{s}_{\hat{z}_{2}},\dots,\mathbf{s}_{\hat{z}_{N}}]\in\mathbb{S}^{N}$,
and dequantization is done from the detected index vector
$\hat{\mathbf{z}}=[\hat{z}_{1},\hat{z}_{2},\dots,\hat{z}_{N}]^{T}\in[1{:}m]^{N}$
utilizing the same codebook $\mathbf{C}$ as the transmitter side, and then
results in
$\hat{\mathbf{Y}}=[\mathbf{c}_{\hat{z}_{1}},\mathbf{c}_{\hat{z}_{2}},\dots,\mathbf{c}_{\hat{z}_{N}}]^{T}\in\mathbb{R}^{N\times
D}$. Finally, the receiver can reconstruct the image
$\hat{\mathbf{X}}\in\mathbb{R}^{C\times H\times W}$ from $\hat{\mathbf{Y}}$
via a decoder $g_{\boldsymbol{\phi}}$, parameterized by $\boldsymbol{\phi}$,
as

$\displaystyle\hat{\mathbf{X}}=g_{\boldsymbol{\phi}}(\hat{\mathbf{Y}}).$ (1)

To optimize the system’s operation, it is essential to train the encoder
parameters $\boldsymbol{\theta}$, decoder parameters $\boldsymbol{\phi}$, and
the codebook $\mathbf{C}$. This optimization is guided by the loss function
proposed in [14], which encompasses three terms:

$\displaystyle\begin{split}\mathcal{L}&=\mathbf{E}_{\mathbf{X}}[\mathsf{e}(\hat{\mathbf{X}},\mathbf{X})+\alpha\mathsf{e}(\hat{\mathbf{Y}},\mathbf{Y}_{\mathsf{d}})+\beta\mathsf{e}(\mathbf{Y},\hat{\mathbf{Y}}_{\mathsf{d}})],\end{split}$
(2)

where $\mathsf{e}(x,y)$ denotes the mean squared error between $x$ and $y$,
and $\mathbf{A}_{\mathsf{d}}$ represents duplicated versions of $\mathbf{A}$.
This duplication is implemented to inhibit the flow of gradients through
$\mathbf{A}$, thereby preventing updates to $\mathbf{A}$ or the NN parameters
involved in computing the matrix. The parameters $\alpha$ and $\beta$ serve as
hyperparameters that regulate the relative significance of the respective loss
components. The role of each component is summarized as follows.

  * •

The first term pertains to the quality of the reconstruction task, albeit
hindered by the inability to apply the chain rule in gradient calculation due
to VQ. Consequently, leveraging the straight-through gradient estimator
between $\mathbf{Y}$ and $\hat{\mathbf{Y}}$, we establish a linkage between
the gradient from $\mathsf{e}(\hat{\mathbf{X}},\mathbf{X})$ to
$\hat{\mathbf{Y}}$ and the gradient from $\mathbf{Y}$ to $\mathbf{X}$. This
term significantly influences the training of both encoder parameter
$\boldsymbol{\theta}$ and decoder parameter $\boldsymbol{\phi}$.

  * •

The second term aims to minimize the disparity between the detected codewords
$\hat{\mathbf{Y}}$ and the duplicated original feature vectors
$\mathbf{Y}_{\mathsf{d}}$, thereby guiding the training of the codebook
$\mathbf{C}$ alone.

  * •

Due to challenges in achieving convergence of the codebook, the third term
serves to regularize the training of the codebook $\mathbf{C}$, which is not
directly related to the update of the codebook but the JSCC encoder. It
achieves this by minimizing the disparity between the original feature vectors
$\mathbf{Y}$ and the duplicated detected codewords
$\hat{\mathbf{Y}}_{\mathsf{d}}$, thereby facilitating the alignment of the
original feature vectors with the nearest codewords in the codebook.

Consequently, the parameters are obtained as

$\displaystyle(\boldsymbol{\theta}^{*},\boldsymbol{\phi}^{*},\mathbf{C}^{*})=\operatorname*{arg\,min}_{\boldsymbol{\theta},\boldsymbol{\phi},\mathbf{C}}\mathcal{L}.$
(3)

During the training process, it is crucial to manage the hyperparameter
$\alpha$ with precision, while $\beta$ usually works well setting to
$0.25\alpha$[14]. If the sum of the second and third terms approaches zero too
rapidly, the decoder will not receive sufficient training due to limited
variation in the input codewords. In this scenario, reducing $\alpha$ is
advisable. Conversely, if the sum of the second and third terms remains
excessively large, increasing $\alpha$ is recommended since this indicates a
substantial vector quantization error in the current codebook.

###  II-B Problem Formulation

Figure 2: The structures of (a) the proposed modulation-agnostic semantic
communication system and (b) conceptually separated universal encoder and
decoder.

In classical digital communication, a desired bit error rate (BER) or data
rate is attained by adjusting the coding scheme or modulation order according
to the channel conditions. Similarly, semantic communication could be
envisioned to achieve optimal task performance by selecting the appropriate
coding scheme or modulation order in response to the channel state.

Hence, the main target of this article is to develop a modulation-agnostic
semantic communication framework, incorporating uJSCC encoder, decoder, and VQ
codebook that support $K$ different modulation orders, i.e.,
$m_{1},m_{2},\dots m_{K}$, where $m_{k}$ represents the $k$-th modulation
order and without loss of generality, $m_{1}<m_{2}<\dots<m_{K}$. In
mathematical expression, let $\mathbf{B}=[b_{1},b_{2},\dots,b_{K-1}]$ be the
SNR boundary to choose modulation order among $K$ types, where

$\displaystyle m=\begin{cases}m_{1},&\text{if}\quad\eta<b_{1},\\\
m_{k},&\text{if}\quad b_{k-1}\leq\eta<b_{k},\\\ m_{K},&\text{if}\quad\eta\geq
b_{K-1},\end{cases}$ (4)

where $k\in[2{:}K-1]$. The designing condition of the SNR boundary will be
discussed in Section III-A.

Assume that both transmitter and receiver are aware of the SNR value $\eta$,
enabling synchronization of the modulation order between the transmitter and
receiver. The following communication process is conducted with respect to
modulation order index $k$ according to the SNR value $\eta$. As illustrated
in Fig. 2, a universal encoder parameterized by $\mathbf{\Theta}$, denoted by
$\mathcal{F}_{\mathbf{\Theta}}$, encodes a source image $\mathbf{X}$ into
feature vectors $\mathbf{Y}^{\scriptscriptstyle(k)}$ given the SNR value
$b_{k-1}\leq\eta<b_{k}$, such that
$\mathbf{Y}^{\scriptscriptstyle(k)}=\mathcal{F}_{\mathbf{\Theta}}(\mathbf{X},\eta)\in\mathbb{R}^{N\times
D_{k}}$ where $D_{k}$ is the dimension of feature vector of the modulation
order $m_{k}$. The encoded feature vectors are quantized via codebook
$\mathbf{C}^{\scriptscriptstyle(k)}$ and modulated into symbols
$\mathbf{s}^{\scriptscriptstyle(k)}$. The codebook for modulation order
$m_{k}$, which should also be trained, can be expressed as

$\displaystyle\mathbf{C}^{\scriptscriptstyle(k)}$
$\displaystyle=\left[\mathbf{c}_{1}^{\scriptscriptstyle(k)},\mathbf{c}_{2}^{\scriptscriptstyle(k)},\dots,\mathbf{c}_{m_{k}}^{\scriptscriptstyle(k)}\right]^{T}\in\mathbb{R}^{m_{k}\times
D_{k}}.$ (5)

At the receiver, the received symbols
$\hat{\mathbf{s}}^{\scriptscriptstyle(k)}$ are detected and dequantized into
$\hat{\mathbf{Y}}^{\scriptscriptstyle(k)}$ via the same codebook
$\mathbf{C}^{\scriptscriptstyle(k)}$. Finally, a universal decoder
parameterized by $\mathbf{\Phi}$, denoted as $\mathcal{G}_{\mathbf{\Phi}}$,
decodes $\hat{\mathbf{Y}}^{\scriptscriptstyle(k)}$ into reconstructed image
$\hat{\mathbf{X}}$ given $\eta$, such that
$\hat{\mathbf{X}}=\mathcal{G}_{\mathbf{\Phi}}(\hat{\mathbf{Y}}^{\scriptscriptstyle(k)},\eta)$.

The objective is to determine an optimal NN structure for the universal
encoder $\mathcal{F}_{\mathbf{\Theta}}$ and universal decoder
$\mathcal{G}_{\mathbf{\Phi}}$, which include data processing paths for $K$
modulation orders, codebook $\mathbf{C}^{\scriptscriptstyle(k)}$ for all
$k\in[1{:}K]$, and the SNR boundary $\mathbf{B}$. When designing this system,
considerations include model complexity, the total number of NN parameters,
training strategies, and task performance.

###  II-C Benchmark Schemes

Before delving into our proposed solution for the formulated problem above, we
first introduce three benchmark schemes for comparison as follows.

####  II-C1 Model-Efficient JSCC

The first and second benchmark schemes we introduce are what we refer to as
the model-efficient (ME) JSCC. To minimize model complexity, this benchmark
scheme utilizes a single encoder-decoder pair, but utilizes distinct VQ
codebooks optimized for each modulation order. Given that higher modulation
orders necessitate larger codeword dimensions, the codeword dimension $D_{k}$
is standardized to $D_{K}$ for all $k\in[1{:}K]$ within this benchmark, such
that $\mathbf{Y}^{\scriptscriptstyle(k)}\in\mathbb{R}^{N\times D_{K}}$,
contrary to the proposed uJSCC. This approach enables a uniform representation
across various modulation orders, simplifying model management and potentially
reducing both training and operational complexities. The model can be trained
using two distinct types of training strategies, details of which will be
elaborated upon in the following.

The first strategy, dubbed _ME 1_, consists of single-stage training. It
jointly trains to optimize the image reconstruction performance of all
modulation orders and obtains a single encoder-decoder pair and $K$ different
codebooks for each modulation order. The loss function for the $k$-th
modulation order can be represented as

$\displaystyle\begin{split}\mathcal{L}_{k}=\mathbf{E}_{\mathbf{X}}[\mathsf{e}(\hat{\mathbf{X}}^{\scriptscriptstyle(k)},\mathbf{X})+\alpha_{k}\mathsf{e}(\hat{\mathbf{Y}}^{\scriptscriptstyle(k)},\mathbf{Y}^{\scriptscriptstyle(k)}_{\mathsf{d}})+\beta_{k}\mathsf{e}(\mathbf{Y}^{\scriptscriptstyle(k)},\hat{\mathbf{Y}}^{\scriptscriptstyle(k)}_{\mathsf{d}})],\end{split}$
(6)

where the computation is done with the SNR value $\eta$ randomly sampled from
SNR range of the $k$-th modulation order. Then, the loss function to jointly
train ME1 can be expressed as

$\displaystyle\mathcal{L}^{\scriptscriptstyle\mathrm{ME_{1}}}=\sum_{k=1}^{K}\lambda_{k}\mathcal{L}_{k},$
(7)

where $\lambda_{k}$ is a hyperparameter to control the importance of loss
terms for each modulation order.

The second strategy, termed ME2, employs a double-stage training process.
Initially, ME2 focuses on optimizing the performance of semantic communication
for the $K$-th modulation order, which is the highest in the sequence. The
loss function for this initial stage is given by:

$\displaystyle\mathcal{L}^{\scriptscriptstyle\mathrm{ME_{2}}}_{\scriptscriptstyle(1)}=\mathcal{L}_{K}.$
(8)

Upon completion of the first stage, the second training stage commences. This
stage is designed to optimize the codebooks
$\mathbf{C}^{\scriptscriptstyle(k)}$ for the remaining modulation orders,
encompassing all $k\in[1{:}K-1]$, while keeping the encoder and decoder
parameters fixed. The loss function for this second stage is defined as:

$\displaystyle\mathcal{L}^{\scriptscriptstyle\mathrm{ME_{2}}}_{\scriptscriptstyle(2)}=\sum_{k=1}^{K-1}\lambda_{k}\mathbf{E}_{\mathbf{X}}[\alpha_{k}\mathsf{e}(\hat{\mathbf{Y}}^{\scriptscriptstyle(k)},\mathbf{Y}^{\scriptscriptstyle(k)}_{\mathsf{d}})].$
(9)

####  II-C2 Task-Effective JSCC

The third benchmark aims to optimize the performance of JSCC for all SNR
ranges. To this end, a set of encoder-decoder pair and codebook for each
modulation order $m_{k}$ should be trained independently with respect to $k$,
different from ME approach and the proposed uJSCC. Then, during inference, a
certain set for $m_{k}$ is switched to be used among $K$ sets, chosen
according to the SNR value $\eta$ and $\mathbf{B}$. Such an approach is dubbed
_task-effective (TE)_ approach. As the codebook cardinality for each
modulation order increases with respect to $k$, we correspondingly scale the
codeword dimensions $D_{k}$ to increase in length, such that
$D_{1}<D_{2}<\dots<D_{K}$. This escalation in codeword dimension is
strategically designed to ensure that the image reconstruction performance is
not compromised by VQ errors.

###  II-D Discussions on Benchmark Schemes

The ME JSCC framework is capable of accommodating $K$ different modulation
orders using a single JSCC encoder-decoder pair, resulting in a parameter
count that is approximately $1/K$ times smaller than that of the TE JSCC. This
significant reduction in parameters translates to a considerably simplified
training process. In terms of training complexity, the ME JSCC strategies
offer a more streamlined procedure compared to the TE JSCC. Specifically, for
the ME1 approach, NN parameters for $K$ different modulation orders are
optimized simultaneously within a single epoch. In contrast, the TE approach
involves sequential training for each modulation order, focusing individually
on the corresponding encoder-decoder pairs. In the ME2 scenario, the initial
training stage aligns with the TE approach, targeting the $K$-th modulation
order. The subsequent stage, however, is less complex as it exclusively
focuses on jointly training the codebooks for the remaining modulation orders.
This bifurcated approach not only simplifies the training process but also
enhances efficiency by concentrating resources on refining the most impactful
elements of the system.

From the perspective of task performance, TE JSCC is optimized to achieve
superior performance across all SNR ranges. However, when considering the ME
approaches, there are distinct limitations and performance boundaries. For the
ME1 approach, its performance is inherently constrained by the use of
parameter sharing across multiple modulation orders, leading to a ceiling
effect dictated by the TE’s capabilities. Furthermore, since the codeword
dimension $D_{k}$ is uniform across all modulation orders and set to $D_{K}$,
the VQ error can be disproportionately large for lower modulation orders,
adversely affecting performance. In the case of ME2, while it may approximate
the best performance for the $K$-th (highest) modulation order due to targeted
optimization, the encoder-decoder pair is not optimized in conjunction with
the codebooks for other modulation orders. Consequently, this decoupling
limits its ability to ensure optimal performance across lower modulation
orders, as the encoder-decoder configuration and the corresponding codebooks
are not fine-tuned to work synergistically for each specific modulation
context.

In summary, the presented solutions fail to simultaneously satisfy model
efficiency and task effectiveness within the framework of modulation-agnostic
semantic communication. Consequently, there is a clear necessity for a novel
and efficient universal system design that not only achieves satisfactory task
performance but also incorporates an appropriate training algorithm.

##  III Modulation-Agnostic Universal Joint Source-Channel Coding

This section presents a novel modulation-agnostic uJSCC scheme for digital
semantic communication. A primary challenge in uJSCC development is the shared
use of CNN parameters across multiple data processing paths for each
modulation order, which can lead to statistical discrepancies and degrade
performance. We address this issue by implementing a switchable batch
normalization layer. The training methodology for the proposed uJSCC system is
also detailed.

###  III-A Universal Joint Source-Channel Coding

TABLE I: Features of uJSCC and benchmark schemes Scheme | # of Enc-Dec | $\boldsymbol{D}_{k}$ | Training | S-BN
---|---|---|---|---
uJSCC | 1 | Increasing order | Single | O
ME1 | 1 | $D_{K}$ | Single | X
ME2 | 1 | $D_{K}$ | Double | X
TE | $K$ | Increasing order | Independent | X

Before proceeding, it is essential to discuss the methodology for setting the
SNR boundaries mentioned previously. In accordance with the 3GPP technical
specification [23], the modulation and coding scheme (MCS) table delineates
channel coding rates and modulation orders to maintain the block error rate
(BLER) below 0.1. Our proposed system utilizes a ML strategy to train the
channel coding scheme across various digital modulation methods and
corresponding SNR ranges. Also note that, in the proposed system, one symbol
corresponds to a single feature vector which contains much more information
than information-theoretic bits in one symbol. Inspired by [23], regarding a
feature vector as the block in the conventional communication system, we
establish SNR boundaries $\mathbf{B}\in\mathbb{Z}^{K-1}$ aiming for a symbol
error rate222SNR boundaries can also be learnable. However, in this paper, we
focus on implementing a universal system that approaches the TE’s perfomance
with a single encoder-decoder pair when all the schemes in this article use
the same SNR bounadaries. (SER) [24] around 0.1. This adjustment adapts
conventional communication standards to the feature vectors in our system,
facilitating joint training of the uJSCC encoder-decoder pair and enhancing
robustness against data compression and channel distortion. For example, with
modulation types such as BPSK, 4QAM, 16QAM, 64QAM, and 256QAM, where $K=5$,
the SNR boundaries would be set at $b_{1}=5$ dB, $b_{2}=12$ dB, $b_{3}=20$ dB,
and $b_{4}=26$ dB, respectively.

The proposed modulation-agnostic semantic communication structure incorporates
a uJSCC encoder $\mathcal{F}_{\mathbf{\Theta}}$, decoder
$\mathcal{G}_{\mathbf{\Phi}}$, and a quantizer/dequantizer based on codebooks
$\mathbf{C}^{\scriptscriptstyle(k)}$ for all $k\in[1{:}K]$ as delineated in
(5). For simplicity, the encoder and decoder are conceptually divided into
inner and outer components. Specifically, as illustrated in Fig. 2, the
encoder $\mathcal{F}_{\mathbf{\Theta}}$ comprises an outer encoder
$f_{\boldsymbol{\theta}_{\mathrm{outer}}}$ and an inner encoder
$f_{\boldsymbol{\theta}_{\mathrm{inner}}}$, while the decoder
$\mathcal{G}_{\mathbf{\Phi}}$ consists of an inner decoder
$g_{\boldsymbol{\phi}_{\mathrm{inner}}}$ and an outer decoder
$g_{\boldsymbol{\phi}_{\mathrm{outer}}}$. Note that the roles of the inner
encoder and decoder are to efficiently process the data, adaptive to the
different required shapes of output and input, respectively, according to a
chosen modulation order, while the roles of the outer encoder and decoder are
preprocessing and postprocessing of the data, respectively, concerned with the
inner coding process. Both the outer encoder and decoder are structured with
multiple CNN layers, and the inner components are each composed of a single
CNN layer. The detailed NN architectures for both the encoder and decoder will
be described in Section IV and features of uJSCC and benchmark schemes are
summarized in TABLE I. Additionally, the operation of the proposed JSCC system
is explained in detail below and summarized in Algorithm 1.

Algorithm 1 Universal Joint Source-Channel Coding

1:Input: $\mathbf{X}$, $\eta$, $\mathbf{B}=[b_{1},b_{2},\dots,b_{K-1}]$,
$\mathcal{F}_{\mathbf{\Theta}}$, $\mathcal{G}_{\mathbf{\Phi}}$,
$\mathbf{C}^{\scriptscriptstyle(k)}$ for $k\in[1{:}K]$

2:Choose $m_{k}$ according to $\eta$ and $\mathbf{B}$ with (4)

3:Encode source image $\mathbf{X}$ into feature vectors
$\mathbf{Y}^{\scriptscriptstyle(k)}$ via $\mathcal{F}_{\mathbf{\Theta}}$
according to (12)

4:Quantize $\mathbf{Y}^{\scriptscriptstyle(k)}$ into codeword indices
$\mathbf{z}^{\scriptscriptstyle(k)}$ using
$\mathbf{C}^{\scriptscriptstyle(k)}$ according to (13)

5:Modulate $\mathbf{z}^{\scriptscriptstyle(k)}$ into symbol sequence
$\mathbf{s}^{\scriptscriptstyle(k)}$

6:Transmit $\mathbf{s}^{\scriptscriptstyle(k)}$ via AWGN channel with $\eta$
as (14)

7:Detect received values into $\bar{\mathbf{s}}^{\scriptscriptstyle(k)}$ and
get $\hat{\mathbf{z}}^{\scriptscriptstyle(k)}$

8:Dequantize $\hat{\mathbf{z}}^{\scriptscriptstyle(k)}$ into
$\hat{\mathbf{Y}}^{\scriptscriptstyle(k)}$ using
$\mathbf{C}^{\scriptscriptstyle(k)}$

9:Reconstruct $\hat{\mathbf{X}}^{\scriptscriptstyle(k)}$ from
$\hat{\mathbf{Y}}^{\scriptscriptstyle(k)}$ via $\mathcal{G}_{\mathbf{\Phi}}$
according to (17)

10:Output: $\hat{\mathbf{X}}^{\scriptscriptstyle(k)}$

The operational mechanism of the proposed uJSCC system is described in detail
as follows. It is presupposed that the channel condition in SNR, denoted as
$\eta$, is known in advance to both the transmitter and receiver, facilitating
the predetermined selection of the modulation order $m_{k}$ as delineated by
(4) and consequently the index $k$. Initially, at the transmitter, the source
image $\mathbf{X}$ is encoded into intermediate feature vectors
$\mathbf{Y}_{\mathrm{inter}}\in\mathbb{R}^{c_{1}\times h_{1}\times w_{1}}$
through the outer encoder, represented by
$f_{\boldsymbol{\theta}_{\mathrm{outer}}}$:

$\displaystyle\mathbf{Y}_{\mathrm{inter}}=f_{\boldsymbol{\theta}_{\mathrm{outer}}}(\mathbf{X},k).$
(10)

The intermediate vectors are subsequently processed by the inner encoder
$f_{\boldsymbol{\theta}_{\mathrm{inner}}}$ tailored for the $k$-th modulation
order. The inner encoder transforms these vectors into reshaped feature
vectors
$\mathbf{Y}^{\scriptscriptstyle(k)}=\left[\mathbf{y}_{1}^{\scriptscriptstyle(k)},\mathbf{y}_{2}^{\scriptscriptstyle(k)},\dots,\mathbf{y}_{N}^{\scriptscriptstyle(k)}\right]^{T}$,
appropriate for $k\in[1{:}K]$:

$\displaystyle\mathbf{Y}^{\scriptscriptstyle(k)}=f_{\boldsymbol{\theta}_{\mathrm{inner}}}(\mathbf{Y}_{\mathrm{inter}},k),$
(11)

where $\boldsymbol{\theta}_{\mathrm{inner}}$ denotes the NN parameters of the
inner encoder. This encoder features a single CNN layer and operates with
computational efficiency to yield $\mathbf{Y}^{\scriptscriptstyle(k)}$ in the
desired configuration of $N\times D_{k}$. This process avoids simply slicing
an output tensor of dimension $N\times D_{K}$ and instead utilizes the
capabilities of the inner encoder to adapt the dimensions effectively for each
specific modulation order, contrary to the ME approach.

Let
$\boldsymbol{\theta}_{\mathrm{inner}}^{\scriptscriptstyle\mathrm{CNN}}\in\mathbb{R}^{D_{K}\times
c_{1}\times f\times f}$ represent the tensor of CNN parameters in
$\boldsymbol{\theta}_{\mathrm{inner}}$, where $f\times f$ denotes the kernel
size. This tensor comprises $D_{K}$ convolutional filters, each of shape
$c_{1}\times f\times f$, which are applied to a zero-padded input tensor using
striding techniques. Depending on the chosen modulation order $m_{k}$, only
the first $D_{k}$ filters, denoted as
$\left[\boldsymbol{\theta}_{\mathrm{inner}}^{\scriptscriptstyle\mathrm{CNN}}\right]_{1:D_{k},:,:,:}$,
are utilized by the CNN layer of the inner encoder for encoding purposes. As
$k$ increases, so does the dimension of the feature vector for each modulation
order, with $D_{1}<D_{2}<\dots<D_{K}$, resulting in reduced computational load
for lower modulation orders due to the utilization of fewer parameters.
Consequently, the inner encoder requires $2ND_{k}c_{1}f^{2}$ floating-point
operations (FLOPs) for encoding, in contrast to $2ND_{K}c_{1}f^{2}$ FLOPs when
all filters are used, which is $D_{K}/D_{k}$ times more. This sophisticated
encoding process achieves the same data processing efficacy as an encoder
specifically designed for the $k$-th modulation order, consistent with the TE
approach. The universal encoder $\mathcal{F}_{\mathbf{\Theta}}$ can be
mathematically expressed as follows:

$\displaystyle\mathcal{F}_{\mathbf{\Theta}}(\mathbf{X},\eta)=f_{\boldsymbol{\theta}_{\mathrm{inner}}}(f_{\boldsymbol{\theta}_{\mathrm{outer}}}(\mathbf{X},k),k).$
(12)

Subsequently, VQ is performed using a codebook
$\mathbf{C}^{\scriptscriptstyle(k)}$, which should also be optimized jointly
with the encoder and decoder during training, based on the minimum distance
criterion. For each $i$-th feature vector
$\mathbf{y}_{i}^{\scriptscriptstyle(k)}$, a symbol index
$z_{i}^{\scriptscriptstyle(k)}$ is selected according to the following
equation:

$\displaystyle
z_{i}^{\scriptscriptstyle(k)}=\operatorname*{arg\,min}_{j}\|\mathbf{y}_{i}^{\scriptscriptstyle(k)}-\mathbf{c}_{j}^{\scriptscriptstyle(k)}\|_{2}^{2}.$
(13)

Each codeword $\mathbf{c}_{j}^{\scriptscriptstyle(k)}$ corresponds one-to-one
to a symbol $\mathbf{s}_{j}^{\scriptscriptstyle(k)}$ indexed on the
corresponding constellation map $\mathbb{S}^{\scriptscriptstyle(k)}$. An index
vector
$\mathbf{z}^{\scriptscriptstyle(k)}=\left[z_{1}^{\scriptscriptstyle(k)},z_{2}^{\scriptscriptstyle(k)},\dots,z_{N}^{\scriptscriptstyle(k)}\right]^{T}$
is then modulated into a symbol sequence
$\mathbf{s}^{\scriptscriptstyle(k)}\in\left(\mathbb{S}^{\scriptscriptstyle(k)}\right)^{N}$,
which is subsequently transmitted through an additive white Gaussian noise
(AWGN) channel.

Upon receiving the symbol sequence altered by AWGN noise

$\displaystyle\hat{\mathbf{s}}^{\scriptscriptstyle(k)}=\mathbf{s}^{\scriptscriptstyle(k)}+\mathbf{n},$
(14)

the receiver detects it into
$\bar{\mathbf{s}}^{\scriptscriptstyle(k)}\in\left(\mathbb{S}^{\scriptscriptstyle(k)}\right)^{N}$.
Subsequently, vector dequantization is performed using the detected index
vector
$\hat{\mathbf{z}}^{\scriptscriptstyle(k)}=\left[\hat{z}_{1}^{\scriptscriptstyle(k)},\hat{z}_{2}^{\scriptscriptstyle(k)},\dots,\hat{z}_{N}^{\scriptscriptstyle(k)}\right]^{T}$
and the same codebook $\mathbf{C}^{\scriptscriptstyle(k)}$ employed at the
transmitter, yielding the received feature vectors
$\hat{\mathbf{Y}}^{\scriptscriptstyle(k)}$. These vectors are then decoded by
the inner decoder $g_{\boldsymbol{\phi}_{\mathrm{inner}}}$ into recovered
intermediate feature vectors
$\hat{\mathbf{Y}}_{\mathrm{inter}}\in\mathbb{R}^{c_{1}\times h_{1}\times
w_{1}}$, formulated as:

$\displaystyle\hat{\mathbf{Y}}_{\mathrm{inter}}=g_{\boldsymbol{\phi}_{\mathrm{inner}}}(\hat{\mathbf{Y}}^{\scriptscriptstyle(k)},k),$
(15)

where $\boldsymbol{\phi}_{\mathrm{inner}}$ denotes the NN parameters of the
inner decoder. This decoder, consisting of a single transpose CNN layer,
efficiently processes $\hat{\mathbf{Y}}^{\scriptscriptstyle(k)}$ without
requiring zero-padding to conform to the shape $N\times D_{K}$, contrary to
the ME approach.

Let
$\boldsymbol{\phi}_{\mathrm{inner}}^{\scriptscriptstyle\mathrm{CNN}}\in\mathbb{R}^{D_{K}\times
c_{1}\times f\times f}$ denote the tensor of CNN parameters in
$\boldsymbol{\phi}_{\mathrm{inner}}$, where $f\times f$ represents the kernel
size. This tensor comprises $D_{K}$ transpose convolution filters, each of
shape $c_{1}\times f\times f$, while the $i$-th filter is applied to the
$i$-th channel of the input tensor via the Kronecker product, followed by
striding, summation, and edge cutting for size adjustment. Similarly to the
encoder, the inner decoder employs only the first $D_{k}$ filters, denoted as
$\left[\boldsymbol{\phi}_{\mathrm{inner}}^{\scriptscriptstyle\mathrm{CNN}}\right]_{1:D_{k},:,:,:}$,
for decoding, corresponding to the modulation order $m_{k}$. This selective
filter usage necessitates $2ND_{k}c_{1}f^{2}$ FLOPs for lower modulation
orders, in contrast to $2ND_{K}c_{1}f^{2}$ FLOPs required if all filters were
utilized. This optimized decoding process mirrors the efficiency of a decoder
specifically tailored for the $k$-th modulation order, consistent with the TE
approach.

Finally, the receiver reconstructs the image
$\hat{\mathbf{X}}^{\scriptscriptstyle(k)}$ from
$\hat{\mathbf{Y}}^{\mathrm{inter}}$ using the outer decoder
$g_{\boldsymbol{\phi}_{\mathrm{outer}}}$, described as:

$\displaystyle\hat{\mathbf{X}}^{\scriptscriptstyle(k)}=g_{\boldsymbol{\phi}_{\mathrm{outer}}}(\hat{\mathbf{Y}}_{\mathrm{inter}},k),$
(16)

where $\boldsymbol{\phi}_{\mathrm{outer}}$ are the NN parameters of the outer
decoder. The overall operation of the universal decoder
$\mathcal{G}_{\mathbf{\Phi}}$ is encapsulated by:

$\displaystyle\mathcal{G}_{\mathbf{\Phi}}(\hat{\mathbf{Y}},\eta)=g_{\boldsymbol{\phi}_{\mathrm{outer}}}(g_{\boldsymbol{\phi}_{\mathrm{inner}}}(\hat{\mathbf{Y}}^{\scriptscriptstyle(k)},k),k).$
(17)

###  III-B Switchable Batch Normalization Layer

Figure 3: An illustration of switchable batch normalization layer with five
paths.

Batch normalization (BN) layers [25] significantly facilitate the training of
NN models and enhance their performance by standardizing the statistics of an
output tensor from each NN layer. This ensures that subsequent layers receive
inputs with consistent statistical properties during each forward pass.

Let $\mathbf{T}^{(n,i)}$ denote the $i$-th channel of the output tensor from
the $n$-th NN layer, and let $t_{j}^{(n,i)}$ represent its $j$-th element,
both pertaining to a single mini-batch. The BN layer first normalizes each
$t_{j}^{(n,i)}$ into a normalized value $\hat{t}_{j}^{(n,i)}$ using the mini-
batch mean $\mu^{(n,i)}$ and standard deviation $\sigma^{(n,i)}$. The
normalized values are then scaled and biased to produce the batch-normalized
value $\tilde{t}_{j}^{(n,i)}$, as described by the following equations:

$\displaystyle\mu^{(n,i)}$
$\displaystyle=\mathbf{E}_{j}\left[t_{j}^{(n,i)}\right],$ (18)
$\displaystyle\sigma^{(n,i)}$
$\displaystyle=\sqrt{\mathbf{E}_{j}\left[\left(t_{j}^{(n,i)}-\mu^{(n,i)}\right)^{2}\right]},$
(19) $\displaystyle\hat{t}_{j}^{(n,i)}$
$\displaystyle=\frac{t_{j}^{(n,i)}-\mu^{(n,i)}}{\sigma^{(n,i)}+\epsilon},$
(20) $\displaystyle\tilde{t}_{j}^{(n,i)}$
$\displaystyle=\gamma^{(n,i)}\hat{t}_{j}^{(n,i)}+\beta^{(n,i)},$ (21)

where $\gamma^{(n,i)}$ and $\beta^{(n,i)}$ are the parameters learned for
rescaling and bias adjustment, respectively, and $\epsilon$ is a small
constant for numerical stability.

During inference, where batch-level computations are absent, the mean
$\mu^{(n,i)}$ and standard deviation $\sigma^{(n,i)}$ from the training phase
are replaced with running estimates $\bar{\mu}^{(n,i)}$ and
$\bar{\sigma}^{(n,i)}$, calculated via a moving average:

$\displaystyle\hat{t}_{j}^{(n,i)}$
$\displaystyle=\frac{t_{j}^{(n,i)}-\bar{\mu}^{(n,i)}}{\bar{\sigma}^{(n,i)}+\epsilon},$
(22) $\displaystyle\tilde{t}_{j}^{(n,i)}$
$\displaystyle=\gamma^{(n,i)}\hat{t}_{j}^{(n,i)}+\beta^{(n,i)},$ (23)

Simplified expressions during inference are:

$\displaystyle\tilde{\gamma}^{(n,i)}$
$\displaystyle=\frac{\gamma^{(n,i)}}{\bar{\sigma}^{(n,i)}+\epsilon},$ (24)
$\displaystyle\tilde{\beta}^{(n,i)}$
$\displaystyle=\beta^{(n,i)}-\tilde{\gamma}^{(n,i)}\bar{\mu}^{(n,i)},$ (25)
$\displaystyle\tilde{t}_{j}^{(n,i)}$
$\displaystyle=\tilde{\gamma}^{(n,i)}t_{j}^{(n,i)}+\tilde{\beta}^{(n,i)},$
(26)

which require only two adjusted variables per channel.

The architecture of the proposed model integrates a BN layer following each
CNN layer. Given that the proposed uJSCC model’s input tensor size for the
inner decoder $g_{\boldsymbol{\phi}_{\mathrm{inner}}}$ varies with each
modulation order, it is essential to manage the differing statistics of the
output tensor across modulation orders. To accommodate the $K$ different
modulation orders in uJSCC, we employ $K$ specific BN layers, each configured
to manage the statistics for its respective data processing path, thus
maintaining optimal task performance despite modulation-agnostic
characteristics, which is the biggest difference with the ME approach.
Furthermore, this variability in statistics impacts other CNN layers,
necessitating $K$ switchable BN (S-BN) layers post each CNN layer, selected
based on the modulation order $m_{k}$. This multi-layered S-BN concept,
referred to in [26], ensures model efficiency given that the parameter count
of S-BN is minimal compared to the total model parameters. For example, with
$K=5$, as illustrated in Fig. 3,
$\mathrm{S{-}BN}(d_{1},d_{2},d_{3},d_{4},d_{5})$ comprises five BN layers,
each corresponding to a different dimension
$\\{d_{1},d_{2},d_{3},d_{4},d_{5}\\}$.

###  III-C Training Strategy

The training procedure for the uJSCC system is conducted jointly across all
modulation orders. This method optimizes the universal encoder parameters
$\mathbf{\Theta}=\\{\boldsymbol{\theta}_{\mathrm{outer}},\boldsymbol{\theta}_{\mathrm{inner}}\\}$,
universal decoder parameters
$\mathbf{\Phi}=\\{\boldsymbol{\phi}_{\mathrm{outer}},\boldsymbol{\phi}_{\mathrm{inner}}\\}$,
and codebooks $\mathbf{C}^{\scriptscriptstyle(k)}$ for $k\in[1{:}K]$. The
overall training process is outlined below and summarized in Algorithm 2.

Algorithm 2 uJSCC Training Algorithm

1:Input: $\mathcal{D}$,
$\mathbf{B}_{\textrm{train}}=[b_{0},b_{1},\dots,b_{K}]$, ($\alpha_{k}$,
$\beta_{k}$, $\lambda_{k}$) for $k\in[1{:}K]$, $I_{\textrm{max}}$

2:Initialize:
$\mathbf{\Theta}=\\{\boldsymbol{\theta}_{\mathrm{outer}},\boldsymbol{\theta}_{\mathrm{inner}}\\}$,
$\mathbf{\Phi}=\\{\boldsymbol{\phi}_{\mathrm{outer}},\boldsymbol{\phi}_{\mathrm{inner}}\\}$,
$\mathbf{C}^{\scriptscriptstyle(k)}$ for $k\in[1{:}K]$

3:$\boldsymbol{\varphi}\leftarrow\\{\mathbf{\Theta},\mathbf{\Phi},\mathbf{C}^{(1)},\mathbf{C}^{(2)},\dots,\mathbf{C}^{\scriptscriptstyle(K)}\\}$

4:$i=0$$\triangleright$ Iteration index

5:while $i<I_{\textrm{max}}$ or early stopping criterion is not met do

6: $\mathbf{J}=[~{}~{}]$$\triangleright$ Empty array

7: for $k=1{:}K$ do

8: Sample $\eta$ such that $b_{k-1}\leq\eta<b_{k}$

9: Append $\eta$ into $\mathbf{J}$

10: end for

11: $\mathcal{L}_{\text{total}}=0$$\triangleright$ Clear loss value

12: for $\eta$ in $\mathbf{J}$ do

13: Choose $m_{k}$ according to $\eta$ and $\mathbf{B}_{\textrm{train}}$ with
(4)

14: Compute $\mathcal{L}_{k}$ via Algorithm 1 and (6)

15:
$\mathcal{L}_{\text{total}}\leftarrow\mathcal{L}_{\text{total}}+\lambda_{k}\mathcal{L}_{k}$

16: end for

17: Update $\boldsymbol{\varphi}$ via Adam[27] optimizer using
$\mathcal{L}_{\text{total}}$ of (27)

18: $i\leftarrow i+1$

19:end while

20:Output: $\boldsymbol{\varphi}$

The training utilizes a dataset $\mathcal{D}$, with hyperparameters such as
$\alpha_{k}$, $\beta_{k}$, and $\lambda_{k}$, and an SNR boundary
$\mathbf{B}_{\textrm{train}}$, which includes upper and lower bounds added to
$\mathbf{B}$ for training conditions. All parameters are initialized before
training begins. For each training cycle, one SNR value per modulation order
for all $k\in[1{:}K]$ is sampled. The corresponding loss value
$\mathcal{L}_{k}$ is computed for each SNR using Algorithm 1 according to the
modulation loss function (6). The overall system is trained jointly, and the
total loss is computed as:

$\displaystyle\mathcal{L}_{\text{total}}=\sum_{k=1}^{K}\lambda_{k}\mathcal{L}_{k},$
(27)

where $\lambda_{k}$ weighs the importance of the loss terms for each
modulation order. Parameter updates are performed using the Adam optimizer
[27] based on $\mathcal{L}_{\text{total}}$. The training process continues
iteratively until the maximum number of iterations is reached or an early
stopping criterion is met, ensuring optimal parameter optimization across the
system.

TABLE II: NN structure (“/” is used to separately describe cases $N=256$ and $N=1024$) Module | Layers | Output size
---|---|---
Outer encoder | $\mathrm{Conv}(3,c_{1},5,1,2)+\mathrm{S{-}BN}(c_{1},c_{1},c_{1},c_{1},c_{1})+\mathrm{ReLU}$ | $c_{1}\times 32\times 32$
$\mathrm{Conv}(c_{1},c_{2},5,1,2)+\mathrm{S{-}BN}(c_{2},c_{2},c_{2},c_{2},c_{2})+\mathrm{ReLU}$ | $c_{2}\times 32\times 32$
$\mathrm{Res}(c_{2})+\mathrm{ReLU}$ | $c_{2}\times 32\times 32$
$\mathrm{Conv}(c_{2},c_{1},2,2,0)~{}/~{}\mathrm{Conv}(c_{2},c_{1},5,1,2)+\mathrm{S{-}BN}(c_{1},c_{1},c_{1},c_{1},c_{1})+\mathrm{ReLU}$ | $c_{1}\times 16\times 16~{}/~{}c_{1}\times 32\times 32$
$\mathrm{Res}(c_{1})+\mathrm{ReLU}$ | $c_{1}\times 16\times 16~{}/~{}c_{1}\times 32\times 32$
Inner encoder | $\mathrm{Conv}(c_{1},D_{5},5,1,2)+\mathrm{S{-}BN}(D_{1},D_{2},D_{3},D_{4},D_{5})+\mathrm{Tanh}+\mathrm{Reshape}$ | $256\times D_{k}~{}/~{}1024\times D_{k}$
Inner decoder | $\mathrm{Reshape}+\mathrm{T{-}Conv}(D_{5},c_{1},5,1,2)+\mathrm{S{-}BN}(c_{1},c_{1},c_{1},c_{1},c_{1})+\mathrm{ReLU}$ | $c_{1}\times 16\times 16~{}/~{}c_{1}\times 32\times 32$
Outer decoder | $\mathrm{T{-}Res}(c_{1})+\mathrm{ReLU}$ | $c_{1}\times 16\times 16~{}/~{}c_{1}\times 32\times 32$
$\mathrm{T{-}Conv}(c_{1},c_{2},2,2,0)~{}/~{}\mathrm{T{-}Conv}(c_{1},c_{2},5,1,2)+\mathrm{S{-}BN}(c_{2},c_{2},c_{2},c_{2},c_{2})+\mathrm{ReLU}$ | $c_{2}\times 32\times 32$
$\mathrm{T{-}Res}(c_{2})+\mathrm{ReLU}$ | $c_{2}\times 32\times 32$
$\mathrm{T{-}Conv}(c_{2},c_{1},5,1,2)+\mathrm{S{-}BN}(c_{1},c_{1},c_{1},c_{1},c_{1})+\mathrm{ReLU}$ | $c_{1}\times 32\times 32$
$\mathrm{T{-}Conv}(c_{1},3,5,1,2)+\mathrm{S{-}BN}(3,3,3,3,3)+\mathrm{Tanh}$ | $3\times 32\times 32$
Res(c) | $\mathrm{Conv}(c,c,3,1,1)+\mathrm{S{-}BN}(c,c,c,c,c)+\mathrm{ReLU}$ | $c\times h\times w$
$\mathrm{Conv}(c,c,1,1,0)+\mathrm{S{-}BN}(c,c,c,c,c)$ | $c\times h\times w$
T-Res(c) | $\mathrm{T{-}Conv}(c,c,1,1,0)+\mathrm{S{-}BN}(c,c,c,c,c)+\mathrm{ReLU}$ | $c\times h\times w$
$\mathrm{T{-}Conv}(c,c,3,1,1)+\mathrm{S{-}BN}(c,c,c,c,c)$ | $c\times h\times w$

##  IV Experimental Results

This section evaluates the uJSCC scheme’s performance in an AWGN channel,
assuming perfect channel knowledge at both the transmitter and receiver.
Benchmark schemes ME1, ME2, and TE JSCC are assessed, as detailed in Section
II-C. The task of image reconstruction is quantified using peak SNR (PSNR) and
the structural similarity index measure (SSIM) [28], with modulation schemes
of BPSK, 4QAM, 16QAM, 64QAM, and 256QAM across modulation orders $K=5$, where
$m_{1}=2$, $m_{2}=4$, $m_{3}=16$, $m_{4}=64$, and $m_{5}=256$.

The NN structure for the proposed model, for sizes $N=256$ and $N=1024$, is
detailed in TABLE II. Below, the notation used within this table is clarified:

  * •

$\mathrm{Conv}(c_{in},c_{out},f,s,p)$: Represents a 2D convolutional layer
with input channel $c_{in}$, output channel $c_{out}$, kernel size $f$, stride
$s$, and padding $p$.

  * •

$\mathrm{T{-}Conv}(c_{in},c_{out},f,s,p)$: A 2D transpose convolutional layer,
parameterized similarly to the $\mathrm{Conv}$ layer.

  * •

$\mathrm{Res}(c)$ and $\mathrm{T{-}Res}(c)$: Indicate a residual layer [29]
and transpose residual layer, respectively, each preserving the input data’s
dimensions.

  * •

$\mathrm{Reshape}$: Used for reordering dimensions and altering dimensions via
merging or splitting.

  * •

$\mathrm{ReLU}$ and $\mathrm{Tanh}$: Activation functions using rectified
linear unit and hyperbolic tangent functions, respectively.

Training employs Algorithm 2 with SNR boundary
$\mathbf{B}_{\textrm{train}}=[0,5,12,20,26,30]$. CIFAR-10 dataset[30] usage
includes 50,000 training, with a 20% validation subset, and 10,000 test
images. Training parameters include an initial learning rate of 0.001, halved
every 20 epochs, a batch size of 64, and a maximum of $I_{\mathrm{max}}=400$
epochs. Hyperparameters $\alpha$ and $\lambda$ are noted in TABLE III, with
$\beta$ fixed at $0.25\alpha$.

Three simulation configurations are established as follows:

  * •

Basic (or B): This setup configures the number of symbols to $N=256$. It
features codeword dimensions $(D_{1},D_{2},D_{3},D_{4},D_{5})=(2,4,8,12,16)$
and NN channel counts $(c_{1},c_{2})=(32,64)$.

  * •

Large (or L): Maintaining the same number of symbols, $N=256$, this
configuration increases the codeword dimensions to
$(D_{1},D_{2},D_{3},D_{4},D_{5})=(4,8,16,24,32)$ and adjusts the channel
counts to $(c_{1},c_{2})=(64,32)$. This effectively doubles the codeword
dimensions and reconfigures the channel numbers relative to the Basic
configuration.

  * •

More Symbols (or MS): This configuration also sets the symbol count at
$N=1024$ and retains the codeword dimensions and channel numbers of the Basic
setup: $(D_{1},D_{2},D_{3},D_{4},D_{5})=(2,4,8,12,16)$ and
$(c_{1},c_{2})=(32,64)$. However, it increases the total number of transmitted
symbols by fourfold compared to the Basic configuration.

Each configuration aims to evaluate the performance of the system under
varying structural complexities and transmission capacities.

TABLE III: Hyperparameters for training ($\alpha$ and $\lambda$ for each modulation are written sequentially in increasing modulation order. For ME2, the first stage is for 256QAM and the second stage is for the rest of the modulation orders. Note that multiple $\lambda$-s are for joint training method.) Model | Hyper-parameters
---|---
Setting | Scheme | $\alpha$ | $\lambda$
B | uJSCC | 3, 1.5, 1, 0.7, 0.5 | 1, 1, 1, 1, 1
ME1 | 5, 4, 3, 2, 1.5 | 1, 1, 1, 4, 16
ME2 | l | 1 | -
2 | 5, 3, 2, 1 | 1, 1, 1, 1
TE | 4, 1, 1, 1, 1 | -
uJSCC $((c_{1},c_{2})=(48,96))$ | 3, 1.5, 1, 0.7, 0.5 | 1, 1, 1, 4, 16
uJSCC $((c_{1},c_{2})=(64,128))$ | 3, 2, 1, 0.7, 0.5 | 1, 1, 1, 4, 16
L | uJSCC | 3, 2, 1, 0.7, 0.5 | 1, 1, 2, 4, 16
ME1 | 5, 4, 3, 2, 1.5 | 1, 1, 1, 4, 16
ME2 | 1 | 1 | -
2 | 5, 3, 2, 1 | 1, 1, 1, 1
5SN | 4, 2, 1, 1, 1 | -
MS | uJSCC | 3, 2, 1, 0.7, 0.5 | 1, 1, 1, 4, 16
ME1 | 5, 4, 3, 2, 1.5 | 1, 1, 1, 4, 16
ME2 | 1 | 1 | -
2 | 5, 3, 2, 1 | 1, 1, 1, 1
TE | 1, 1, 1, 1, 1 | -

###  IV-A Model Efficiency of uJSCC

TABLE IV: Number of parameters in NN models (The ratio of BN parameters is written in percentage. The ratio of the total parameters compared to uJSCC is written in multiple scales.) Model | Number of parameters
---|---
Setting | Scheme | BN (%) | Total
B | uJSCC | 6514 (2.52%) | 258098
ME | 1318 (0.54%) | 252902 ($\times 0.98$)
TE | 6514 (0.52%) | 1203634 ($\times 4.66$)
uJSCC $((c_{1},c_{2})=(48,96))$ | 9714 (1.76%) | 552978 ($\times 2.14$)
uJSCC $((c_{1},c_{2})=(64,128))$ | 12914 (1.35%) | 958450 ($\times 3.71$)
L | uJSCC | 12998 (1.29%) | 1009734
ME | 2630 (0.26%) | 999366 ($\times 0.99$)
TE | 12998 (0.27%) | 4753478 ($\times 4.71$)
MS | uJSCC | 6514 (1.89%) | 344114
ME | 1318 (0.39%) | 338918 ($\times 0.98$)
TE | 6514 (0.40%) | 1633714 ($\times 4.75$)
TABLE V: FLOPs and training epochs of NN models (FLOPs are written in “Encoder FLOPs/Decoder FLOPs” format and the number of epochs of TE is summed right side.) Model | Measurement
---|---
Setting | Scheme | Modulation | FLOPs (G) | Epochs
B | uJSCC | BPSK | 0.205/0.205 | 178
4QAM | 0.205/0.206
16QAM | 0.207/0.207
64QAM | 0.209/0.209
256QAM | 0.210/0.210
TE | BPSK | 0.205/0.205 | 41 | 473
4QAM | 0.205/0.206 | 44
16QAM | 0.207/0.207 | 93
64QAM | 0.209/0.209 | 143
256QAM | 0.210/0.210 | 152
L | uJSCC | BPSK | 0.807/0.807 | 178
4QAM | 0.811/0.811
16QAM | 0.817/0.817
64QAM | 0.824/0.824
256QAM | 0.830/0.830
TE | BPSK | 0.807/0.807 | 34 | 441
4QAM | 0.811/0.811 | 32
16QAM | 0.817/0.817 | 65
64QAM | 0.824/0.824 | 125
256QAM | 0.830/0.830 | 185
MS | uJSCC | BPSK | 0.324/0.324 | 141
4QAM | 0.327/0.327
16QAM | 0.334/0.334
64QAM | 0.340/0.340
256QAM | 0.347/0.347
TE | BPSK | 0.324/0.324 | 64 | 513
4QAM | 0.327/0.327 | 93
16QAM | 0.334/0.334 | 100
64QAM | 0.340/0.340 | 149
256QAM | 0.347/0.347 | 107

TABLE IV shows that BN layer parameters account for only 1-2% of total model
parameters, indicating that S-BN layers in uJSCC minimally impact overall
model size, with ME1 and ME2 comprising about 98-99% of uJSCC’s parameters.
Conversely, TE’s parameter count is nearly fivefold higher due to its five
separate JSCC encoder-decoder pairs for each modulation order, underscoring
the proposed model’s parameter efficiency.

TABLE V details the FLOPs for the encoder and decoder during forward pass,
primarily affected by the CNN layers, with nearly identical figures for
uJSCC’s universal encoder and decoder. FLOPs for uJSCC are on par with TE,
indicating comparable operational efficiency across all of the modulation
orders. uJSCC, employing parameter sharing and joint training across
modulation orders, requires far fewer training epochs than TE, which trains
each modulation order separately. This efficient training approach of uJSCC
leads to fewer epochs especially as TE’s training epochs increase with higher
modulation orders due to greater codebook cardinality and longer codeword
dimensions, highlighting uJSCC’s efficiency in training complexity.

###  IV-B uJSCC vs. Benchmarks under Basic Setting

(a) SSIM vs. SNR (b) PSNR vs. SNR

Figure 4: Performance comparison between uJSCC and benchmark schemes under
Basic setting.

(a) Original image (b) uJSCC (c) TE (d) ME1 (e) ME2

Figure 5: Original image and reconstructed images for BPSK, 4QAM, and 16QAM
(Performance is numerically written in “MSE/PSNR/SSIM” format.) under Basic
setting.

(a) SSIM vs. SNR (b) PSNR vs. SNR

Figure 6: Performance comparison between TE and uJSCC with different number of
channels ($N=256$, $(D_{1},D_{2},D_{3},D_{4},D_{5})=(2,4,8,12,16)$).

Fig. 4 demonstrates that uJSCC, with only a single universal encoder and
decoder, surpasses benchmark schemes in low SNR environments when transmitting
256 symbols using BPSK and 4QAM. This highlights the advantage of semantic
communication where uJSCC, benefiting from joint training and parameter
sharing across multiple modulation orders, enhances image reconstruction in
lower modulation orders using the capabilities of higher orders. ME1 lacks
S-BN layers, impairing its ability to equalize output tensor statistics across
CNN layers for each modulation order. For higher orders like 64QAM and 256QAM,
TE and ME2 excel, with ME2 matching TE’s performance initially trained for
256QAM, yet showing weaker performance in lower orders due to suboptimal
training across remaining codebooks.

Fig. 5 displays reconstructed images from various models for BPSK, 4QAM, and
16QAM. The results for 64QAM and 256QAM are omitted as all models, except ME1,
demonstrate satisfactory image quality. uJSCC clearly outperforms other models
in lower modulation orders like BPSK and 4QAM. For 16QAM, uJSCC and TE show
comparable and superior image quality over ME1 and ME2, affirming uJSCC’s
advantage in low SNR settings through effective semantic communication.

Fig. 6 assesses image reconstruction between TE and uJSCC with varying channel
counts, $c_{1}$ and $c_{2}$. By augmenting the number of parameters while
maintaining codeword dimensions $D_{k}$ for $k\in[1{:}K]$, uJSCC approaches
TE’s performance. Despite an increase in model size, as noted in TABLE IV,
uJSCC retains fewer parameters than TE, indicating that enhancing model
parameters can narrow the performance gap while remaining more parameter-
efficient.

###  IV-C uJSCC vs. Benchmarks under Large Setting

(a) SSIM vs. SNR (b) PSNR vs. SNR

Figure 7: Performance comparison between uJSSS vs. benchmark schemes under
Large setting.

Fig. 7 depicts the image reconstruction performance of uJSCC and benchmark
schemes using an enlarged model. This setup exhibits a trend similar to the
Basic configuration. According to the SSIM metric, uJSCC generally outperforms
except in the BPSK range, where ME1 leads narrowly but shows weaker
performance across other modulation orders. Notably, uJSCC significantly
surpasses other models in the 4QAM range, crucial for the visual quality of
the reconstructed image.

In terms of the PSNR metric, uJSCC excels in the low SNR zones like BPSK and
4QAM and closely matches TE’s performance in higher modulation orders, despite
TE having approximately five times more parameters. The slight PSNR shortfall
in the high SNR region is not detrimental as uJSCC nearly reaches peak
performance in terms of SSIM, which is more reflective of the structural
integrity of the reconstructed image. Given that ME1 underperforms compared to
uJSCC in the BPSK range, both SSIM and PSNR metrics are crucial for a
comprehensive evaluation of the image reconstruction quality. This simulation
confirms that a larger uJSCC model, when implemented with the devised training
strategy, can achieve superior or competitive reconstruction across all SNR
ranges with reduced training complexity relative to TE.

###  IV-D uJSCC vs. Benchmarks under More Symbols Setting

(a) SSIM vs. SNR (b) PSNR vs. SNR

Figure 8: Performance comparison between uJSCC vs. benchmark schemes under
More Symbols setting.

(a) Original image (b) uJSCC (c) TE (d) ME1 (e) ME2

Figure 9: Original image and reconstructed images for BPSK, 4QAM, and 16QAM
(Performance is numerically written in “MSE/PSNR/SSIM” format.) under More
Symbols setting.

Fig. 8 evaluates the image reconstruction performance of uJSCC and benchmark
schemes with a transmission of 1024 symbols, i.e., $N=1024$. Transmitting more
symbols enhances overall performance compared to the Basic setting. TE excels
across all SNR ranges, as its encoder-decoder pairs are specifically trained
for individual modulation orders at corresponding SNR levels. However, uJSCC
closely rivals TE, particularly matching its performance in the high SNR
region with 64QAM and 256QAM, as measured by SSIM. ME1 consistently
underperforms across all SNR ranges, while ME2 performs poorly in low SNR
settings with BPSK and 4QAM but well in higher SNR areas. This discrepancy
highlights the challenge of optimizing a model across all SNR ranges through
joint training, which is not the case for uJSCC. The effectiveness of the
proposed NN structure, particularly with increased codeword dimension order
and the integration of S-BN layers, alongside the training algorithm, is
demonstrated in achieving high-quality outcomes for the downstream task. uJSCC
maintains moderate to high performance across a broad SNR spectrum, supporting
multiple modulation orders with a single universal model.

Fig. 9 presents the reconstructed images from each model for BPSK, 4QAM, and
16QAM when transmitting 1024 symbols. As higher modulation orders like 64QAM
and 256QAM generally yield satisfactory results, the focus is on lower orders.
ME2, optimized only for 256QAM, shows compromised image quality in lower
modulation orders despite the high symbol count. ME1 exhibits better
performance in low SNR regions, showcasing the strength of joint training.
Nonetheless, as uJSCC mirrors TE’s optimal performance, it delivers superior
image quality in low SNR regions. This simulation confirms that uJSCC, with a
robustly designed NN structure and training strategy, can be optimized to
maintain satisfactory task quality across all SNR ranges when transmitting a
higher number of symbols.

##  V Conclusion

In this paper, we introduced a uJSCC framework that facilitates modulation-
agnostic digital semantic communication. The data processing path within uJSCC
mimics a traditional JSCC encoder-decoder pair tailored to a specific
modulation order. Utilizing parameter sharing, which enhances model
efficiency, the output statistics of each CNN layer are regulated by S-BN
layers tailored for each modulation order to standardize the output statistics
across different data processing paths. Through our proposed joint training
strategy, experimental results demonstrate that uJSCC, with significantly
fewer parameters and reduced training epochs, either matches or closely
approaches the task performance of traditional encodings as evaluated by
metrics such as SSIM, PSNR, and the quality of reconstructed images for each
modulation order. Furthermore, we have validated the versatility of uJSCC in
larger models and with different numbers of transmitted symbols. Future
research may explore optimizing the number of transmitted symbols or the
dimensions of codewords for each modulation order. This study aims to
illuminate the practical applicability of JSCC-based semantic communication
using digital communication devices in real-world scenarios.

## References

  * [1] Y. Shi, Y. Zhou, D. Wen, Y. Wu, C. Jiang, and K. B. Letaief, “Task-oriented communications for 6g: Vision, principles, and technologies,” _IEEE Wireless Communications_ , vol. 30, no. 3, pp. 78–85, 2023.
  * [2] D. Gündüz, Z. Qin, I. E. Aguerri, H. S. Dhillon, Z. Yang, A. Yener, K. K. Wong, and C.-B. Chae, “Beyond transmitting bits: Context, semantics, and task-oriented communications,” _IEEE Journal on Selected Areas in Communications_ , vol. 41, no. 1, pp. 5–41, 2022.
  * [3] H. Seo, J. Park, M. Bennis, and M. Debbah, “Semantics-native communication via contextual reasoning,” _IEEE Transactions on Cognitive Communications and Networking_ , 2023.
  * [4] H. Seo, Y. Kang, M. Bennis, and W. Choi, “Bayesian inverse contextual reasoning for heterogeneous semantics- native communication,” _IEEE Transactions on Communications_ , vol. 72, no. 2, pp. 830–844, 2024.
  * [5] W. Weaver, “Recent contributions to the mathematical theory of communication,” _ETC: a review of general semantics_ , pp. 261–281, 1953.
  * [6] X. Luo, H.-H. Chen, and Q. Guo, “Semantic communications: Overview, open issues, and future research directions,” _IEEE Wireless Communications_ , vol. 29, no. 1, pp. 210–219, 2022.
  * [7] N. Farsad, M. Rao, and A. Goldsmith, “Deep learning for joint source-channel coding of text,” in _2018 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2018, pp. 2326–2330.
  * [8] E. Bourtsoulatze, D. B. Kurka, and D. Gündüz, “Deep joint source-channel coding for wireless image transmission,” _IEEE Transactions on Cognitive Communications and Networking_ , vol. 5, no. 3, pp. 567–579, 2019.
  * [9] H. Xie, Z. Qin, G. Y. Li, and B.-H. Juang, “Deep learning enabled semantic communication systems,” _IEEE Transactions on Signal Processing_ , vol. 69, pp. 2663–2675, 2021.
  * [10] J. Xu, B. Ai, W. Chen, A. Yang, P. Sun, and M. Rodrigues, “Wireless image transmission using deep source channel coding with attention modules,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 4, pp. 2315–2328, 2021.
  * [11] Q. Hu, G. Zhang, Z. Qin, Y. Cai, G. Yu, and G. Y. Li, “Robust semantic communications with masked vq-vae enabled codebook,” _IEEE Transactions on Wireless Communications_ , 2023.
  * [12] S. Xie, S. Ma, M. Ding, Y. Shi, M. Tang, and Y. Wu, “Robust information bottleneck for task-oriented communication with digital modulation,” _IEEE Journal on Selected Areas in Communications_ , 2023.
  * [13] T.-Y. Tung, D. B. Kurka, M. Jankowski, and D. Gündüz, “Deepjscc-q: Constellation constrained deep joint source-channel coding,” _IEEE Journal on Selected Areas in Information Theory_ , 2022.
  * [14] A. Van Den Oord, O. Vinyals _et al._ , “Neural discrete representation learning,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [15] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [16] N. Tishby, F. C. Pereira, and W. Bialek, “The information bottleneck method,” _arXiv preprint physics/0004057_ , 2000.
  * [17] E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with gumbel-softmax,” _arXiv preprint arXiv:1611.01144_ , 2016.
  * [18] Y. Bo, Y. Duan, S. Shao, and M. Tao, “Joint coding-modulation for digital semantic communications via variational autoencoder,” _IEEE Transactions on Communications_ , 2024.
  * [19] H. Gao, G. Yu, and Y. Cai, “Adaptive modulation and retransmission scheme for semantic communication systems,” _IEEE Transactions on Cognitive Communications and Networking_ , 2023.
  * [20] Y. He, G. Yu, and Y. Cai, “Rate-adaptive coding mechanism for semantic communications with multi-modal data,” _IEEE Transactions on Communications_ , 2023.
  * [21] H. Zhang, T.-W. Weng, P.-Y. Chen, C.-J. Hsieh, and L. Daniel, “Efficient neural network robustness certification with general activation functions,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [22] J. Park, Y. Oh, S. Kim, and Y.-S. Jeon, “Joint source-channel coding for channel-adaptive digital semantic communications,” _arXiv preprint arXiv:2311.08146_ , 2023.
  * [23] 3GPP, “NR; Physical layer procedures for data,” 3rd Generation Partnership Project (3GPP), Technical Specification (TS) 38.214, March 2024, version 18.2.0.
  * [24] J. G. Proakis and M. Salehi, _Fundamentals of communication systems_. Pearson Education India, 2007.
  * [25] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _International conference on machine learning_. pmlr, 2015, pp. 448–456.
  * [26] J. Yu, L. Yang, N. Xu, J. Yang, and T. Huang, “Slimmable neural networks,” _arXiv preprint arXiv:1812.08928_ , 2018.
  * [27] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [28] A. Hore and D. Ziou, “Image quality metrics: Psnr vs. ssim,” in _2010 20th international conference on pattern recognition_. IEEE, 2010, pp. 2366–2369.
  * [29] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [30] A. Krizhevsky, G. Hinton _et al._ , “Learning multiple layers of features from tiny images,” 2009.