# Integrating Pre-Trained Language Model with Physical Layer Communications

Ju-Hyung Lee Dong-Ho Lee Joohan Lee Jay Pujara  J.-H. Lee and J. Lee are with
Ming Hsieh Department of Electrical and Computer Engineering, University of
Southern California, Los Angeles, USA (Emails: {juhyung.lee,
joohanl}@usc.edu).D.-H. Lee and J. Pujara are with Information Science
Institute, University of Southern California, Marina Del Rey, USA (Emails:
{dongho.lee, jpujara}@usc.edu).

###### Abstract

The burgeoning field of on-device AI communication, where devices exchange
information directly through embedded foundation models, such as language
models (LMs), requires robust, efficient, and generalizable communication
frameworks. However, integrating these frameworks with existing wireless
systems and effectively managing noise and bit errors pose significant
challenges. In this work, we introduce a practical on-device AI communication
framework, integrated with physical layer (PHY) communication functions,
demonstrated through its performance on a link-level simulator. Our framework
incorporates end-to-end training with channel noise to enhance resilience,
incorporates vector quantized variational autoencoders (VQ-VAE) for efficient
and robust communication, and utilizes pre-trained encoder-decoder
transformers for improved generalization capabilities. Simulations, across
various communication scenarios, reveal that our framework achieves a $50\%$
reduction in transmission size while demonstrating substantial generalization
ability and noise robustness under standardized 3GPP channel models.

###### Index Terms:

Physical layer communications, language model, VQ-VAE, natural language
processing (NLP), link-level simulation.

AWGN

    additive white Gaussian noise
BER

    bit error rate
SER

    symbol error rate
DL

    deep learning
GPU

    graphic processing unit
LoS

    line-of-sight
NLoS

    non-LOS
MIMO

    multiple-input multiple-output
MLP

    multilayer perceptron
MSE

    mean squared error
NN

    neural network
QAM

    quadrature amplitude modulation
SNR

    signal-to-noise ratio
FEC

    forward error correction
LDPC

    low-density parity-check
TX

    transmitter
RX

    receiver
DNN

    deep neural network
CE

    cross entropy
i.i.d.

    independent and identically distributed
AI

    artificial intelligence
ML

    machine learning
LLM

    large language model
LM

    language model
OFDM

    orthogonal frequency-division multiplexing
OOD

    out-of-domain
OOV

    out-of-vocabulary
NLP

    natural language processing
PHY

    physical layer
E2E

    end-to-end
VQ-VAE

    vector quantised variational auto encoder
HARQ

    hybrid automatic repeat request
ARQ

    automatic repeat request
JSCC

    joint source-and-channel coding
CDL

    clustered delay line

##  I Introduction

The increasing capabilities of mobile devices and the advancements in large
language model (LLM), particularly foundation models, have paved the way for a
new era of on-device artificial intelligence (AI).111Throughout this paper, we
will use the term “on-device AI” and “on-device AI/ language model (LM)”
interchangeably to refer to a foundation model or a language model deployed at
both transmitter (TX) side and receiver (RX) side in a distributed manner.
This paradigm shift allows devices to possess their own AI capabilities,
enabling tasks such as real-time translation, personalized recommendations,
and even autonomous driving, all while ensuring privacy and efficiency [1, 2,
3, 4].

On-device AI communication enables devices equipped with advanced foundation
models like LMs to directly exchange information, forming a network of
distributed intelligence. This distributed AI-to-AI communication requires
efficient and accurate transmission of information between devices. The
”encoded representation” generated by the AI/LLM at the TX side (called _AI-
Src-Enc_) needs to be reliably transmitted and accurately interpreted as
”decoding input” by the AI/LLM at the RX side (called _AI-Src-Dec_) [5].

However, seamlessly integrating this novel technology with existing
communication infrastructure presents a significant challenge. Maintaining
backward compatibility is a crucial requirement in wireless communication
systems, allowing devices designed for older networks (e.g., 4G and 5G) to
operate seamlessly with newer technologies (e.g., 6G). This interoperability
ensures a smooth transition between networks, such as handover and roaming,
enhancing user experience. For on-device AI communication, ensuring backward
compatibility is still essential, enabling existing communication
infrastructure to support this new paradigm without requiring major
modifications. Thus, it is required to integrate the on-device AI model
application into wireless communication systems, ensuring they adhere to
existing legacy protocols for seamless interoperability [6].

Furthermore, effective on-device AI communication demands several key
qualities: (1) Robustness: The system must be resilient to communication
errors (e.g., bit error) and noise (e.g., channel fading) inherent in dynamic
wireless environments; (2) Efficiency: The system must efficiently compress
and transmit data while maintaining accuracy and fidelity; and (3)
Generalization Capability: The system should be able to handle diverse inputs
and adapt to various communication scenarios. Meeting these requirements
presents another challenge, requiring novel approaches in system design and
optimization.

###  I-A Related Works

Semantic communication. Semantic communication, which focuses on effectively
transmitting semantic (meaningful) information rather than merely receiving
individual symbols or bits [7], is the most relevant research direction for
distributed AI-to-AI communications. In order to accomplish this, semantic
communication systems leverage neural network (NN) at both the TX and RX to
extract and decode the semantic information. Semantic communication research
can be classified based on the types of data they utilize, including image [8,
9, 10], video [11, 12, 13], and speech/text data [14, 15, 16]. These works
pioneered the inclusion of AI in source coding within transceivers, optimizing
joint source-and-channel coding (JSCC) under specific noisy channels and data
distributions.

However, such semantic communication framework poses significant challenges
for practical application in on-device AI communication. Firstly, it demands
major modifications to existing legacy communication protocols (e.g., 5G-NR
protocol), distinguishing it from practical distributed AI-to-AI communication
systems. JSCC necessitates that both channel coding and communication modules
be modified and trainable, requiring major modifications in existing legacy
communication protocols that are impractical in conventional wireless
communication systems.

Secondly, much of the existing research assumes ideal channel conditions
(e.g., additive white Gaussian noise (AWGN)) and bases analysis on information
theory and symbol-level transmission. This overlooks the complexities of non-
ideal (realistic) channel conditions and the necessity for link (or
system)-level analysis and practical bit-level transmission.

Advancing the pioneering efforts in semantic communication towards practical
application in on-device AI communication necessitates two critical
components: (1) _Reliable_ communication systems need to seamlessly operate
with existing infrastructure. This typically involves interacting with various
physical-layer communication function modules like mappers, forward error
correction (FEC), and orthogonal frequency-division multiplexing (OFDM), each
serving a distinct purpose. Therefore, it becomes necessary to integrate AI-
Src-Enc and AI-Src-Dec within the transceiver system, with the operational
parameters of modules compatible with, or analogous to, those in 5G-NR
physical-layer modules. (2) _Efficient_ communication systems should be able
to compress the output of AI-Src-Enc and accurately decode the input of AI-
Src-Dec. Developing an effective compression approach is crucial for reducing
the size of semantic messages while retaining their essential information.

Fig. 1 shows the difference between semantic communication and on-device AI
communication systems.

((a)) Semantic communication system.

((b)) On-device AI/LM communication system.

Figure 1: Overview of semantic communication systems and on-device AI/LM
communication system.

Pre-Trained Language Model. Recent studies on text semantic communication
exploit transformer architectures [17] to extract semantics at the TX and
recover the original information at the RX [18, 19]. However, such frameworks
may have the following challenges: (i) training an end-to-end (E2E) semantic
communication pipeline requires a huge computational effort due to the many
randomly initialized parameters of semantic encoder/decoder to be trained on;
(ii) difficulty in handling out-of-vocabulary (OOV) since they only use a set
of whitespace-separated tokens in the training data. _General-purpose_
communication systems should be able to effectively handle any data, including
out-of-domain (OOD) data that lies outside the training data used to develop
the framework. This necessitates using pre-trained language models with
superior generalizability, allowing them to perform well even when
encountering unforeseen data variations.

Pre-trained language models, which have been extensively trained on vast web
datasets, are revolutionizing the field of natural language processing (NLP)
[20, 21]. These models, often referred to as “foundation models”, possess the
ability to effectively generalize even to data where they have not
specifically been trained for [2]. Their capabilities are a result of
leveraging transfer learning and scaling techniques. Transfer learning
involves utilizing knowledge acquired from one task and applying it to another
[22]. In the context of deep learning, pre-training is the prevailing approach
to transfer learning, where a model is initially trained on a surrogate task
and subsequently fine-tuned to adapt to the downstream task of interest.

###  I-B Contributions

To design and evaluate a practical and widely applicable on-device AI
communication framework, we focus on three key questions:

Q1:1

How can we integrate a pre-trained language model in practical physical layer
(PHY) communication systems and evaluate its reliability in realistic network
scenario?

Q2:2

How can we efficiently compress the output of the AI-Src-Enc and accurately
decode the noisy input of the RX AI-Src-Dec?

Q3:3

How can we leverage the knowledge gained from a large dataset to build a
general-purpose encoder/decoder for system-wide application?

Our main contributions, which address these challenges, are summarized as
follows:

  * •

We integrate a language model with the link-level simulator, NVIDIA Sionna
[23], incorporating 5G-NR PHY communication functions (e.g., Polar channel
coding and QAM mapper). We evaluate its performance on a channel that contains
not only noise but also delay dispersion (e.g., 3GPP CDL-family channel),
seeing the efficacy of on-device AI communication. Furthermore, we propose a
_noise-tuning_ method to optimize its reliability. This addresses Q1.

  * •

Converting on-device AI information (e.g., output of AI-Src-Enc and input of
AI-Src-Dec) into bit-level information often increases data load.
Additionally, even minor bit errors, induced by channel noise, can greatly
hinder communication. Addressing this, we propose a novel approach that
leverages vector quantization techniques [24] for efficient compression and
decoding of on-device AI information. This approach transforms high-
dimensional vectors into discrete data via a codebook, significantly reducing
transmission overhead even while mitigating the impact of bit errors (i.e.,
robustness against channel noise). This addresses Q2.

  * •

We incorporate encoder-decoder transformers into E2E on-device AI
communication systems, initializing them with pre-trained weights. This
approach reduces the reliance on channel-specific and data distribution-
specific optimization, thus enhancing the system’s generalizability; This
addresses Q3. In particular, we employ a pre-trained encoder-decoder
transformer (BART [25]) to initialize the parameters of AI-Src-Enc/Dec so that
the pipeline itself requires little or no computational effort and use a pre-
trained tokenizer to effectively handle OOV so that our pipeline can be
generalized to any other text.

  * •

We release source code for the experiments to promote reproducible ML research
in wireless communication.222https://github.com/abman23/on-device-ai-comm

###  I-C Paper Organization and Notation

This paper is organized as follows: Sec.II provides the background information
on two key concepts: _on-device AI communication_ and _ vector quantised
variational auto encoder (VQ-VAE)_ After presenting a detailed system model
for the on-device AI communication systems in Sec.III, Sec. IV provides
comparison results and ablation studies to assess the contribution of each of
our proposed approaches, with a specific emphasis on the accuracy and
efficiency of text transmission. Then, we also present an in-depth analysis of
our proposed _noise-tuning_ methods, followed by concluding remarks in Sec.
VI.

Notation. Random variables are denoted by capital italic font, e.g., $X,Y$,
with realizations $x,y$, respectively. $I(X;Y)$, $p(y|x)$ and $p(x,y)$
represent the mutual information, conditional probability, and joint
probability distribution of the two random variables $X$ and $Y$. Multivariate
random variables are represented with capital bold font, e.g.,
$\mathbf{Y}=[Y_{0},Y_{1}]^{T}$. Vectors are represented using a lowercase bold
font, e.g., $\mathbf{y}$. We use $\mathbb{R}^{D\times 1}$ to represent the
$D$-dimensional space of real-valued vectors. We also use $\|\cdot\|$ to
denote the $L^{2}$-norm, which is an Euclidean norm.

##  II Background

###  II-A On-device AI (Distributed AI-to-AI) Communication

On-device AI (distributed AI-to-AI) communication refers to a paradigm where
AI models, e.g., LM, are embedded locally within devices and communicate with
each other over a network. This decentralized approach allows devices to
exchange complex, context-rich information, leveraging the capabilities of AI
models to facilitate personalized and intelligent interactions without relying
on centralized servers.

In the process of AI-to-AI communication using via LM, the TX side initially
encodes the input sequence. This encoded data is then sent across through the
physical layer to the RX side, where it is decoded back into its original
form. To enable this, several key components are required. Both AI-Src-Enc at
the TX and AI-Src-Dec at the RX must employ a shared tokenizer and embedder.
These tools play a pivotal role in accurately encoding and decoding the data.
Initially, the tokenizer divides the input sequence into discrete tokens.
Subsequently, each token is transformed into an embedding representation
through the embedder. This series of mapped embeddings is then input into the
encoder to generate a transmittable representation. At the RX end, to
reconstruct the original sequence from this representation, the same embedder
is used to deduce the token index, which is then converted back to the
corresponding token using the tokenizer.

The transmittable representation itself is a high-dimensional vector made up
of floating point numbers. For its transmission via the physical layer, this
vector needs to be converted into a form of bit-level information, suitable
for digital transmission. However, these floating-point numbers are
particularly susceptible to bit-flips, which can occur due to noise within the
physical layer as mentioned in Q2. For instance, a 32-bit floating-point
number representing 1.0 could change to infinity with just one bit-flip (bit
error) in the second position, leading to a significant decline in
transmission accuracy [26].

###  II-B Vector Quantized Variational Autoencoders (VQ-VAE)

VQ-VAE are a version of variational autoencoders [27], distinguished by their
use of vector quantization techniques to effectively encode information into
discrete latent representations [24]. At the core, an autoencoder is a neural
network that identifies latent spaces, which are complex and non-linear
functions derived from the data. This neural network architecture is divided
into encoder and decoder. The encoder’s role is to process and transform the
input data into a latent vector representation. Following this, the decoder is
tasked with the accurate reconstruction of the original data, using only the
latent vector representation provided by the encoder. Unlike traditional
approaches where this latent representation is continuous [27], the VQ-VAE
creates discrete latent representation by incorporating a discrete codebook.
This codebook, forming an array of vector entries each assigned a specific
index, is utilized to discretize the latent space within the autoencoder. By
implementing this strategy, the VQ-VAE facilitates the learning of a discrete
latent space that effectively captures key features of the data. The resultant
latent representation in this model is a sequence of integer indices
corresponding to the codebook entries, which not only allows for more
efficient data compression but also maintains a level of reconstruction
quality that is comparable to traditional methods.

Figure 2: Framework overview of on-device AI/LM PHY communication systems
integrated with pre-trained language model. This framework incorporates a
link-level simulator to realistically emulate bit-level transmission within
PHY communication systems. At the transmission (TX) end, the symbol stream
${\bf\it s}$ undergoes AI-Src-Enc $S_{\mathrm{\mathbf{E}}}(\cdot)$, vector
quantization $Q_{\mathrm{\mathbf{E}}}(\cdot;Z)$, and channel encoder
$C_{\mathrm{\mathbf{E}}}(\cdot)$ to produce ${\bf\it x}$. Conversely, at the
receiver (RX) end, the received signal $y$ is channel-decoded, vector-
dequantized, and semantically decoded to recover the symbol ${\bf\it\hat{s}}$.
We evaluate the system in three different criteria: lexical similarity,
semantic similarity, and the compression rate.

##  III On-device AI Communication Systems

###  III-A Problem Description

Consider a sentence ${\bf\it s}$ that maps to symbol stream ${\bf\it x}$:

${{\bf\it
x}}=C_{\mathrm{\mathbf{E}}}\left(Q_{\mathrm{\mathbf{E}}}\left(S_{\mathrm{\mathbf{E}}}\left({\bf\it
s}\right);Z\right)\right),$ (1)

where $C_{\mathrm{\mathbf{E}}}(\cdot)$, $Q_{\mathrm{\mathbf{E}}}(\cdot;Z)$,
and $S_{\mathrm{\mathbf{E}}}(\cdot)$ represent the channel encoder, vector
quantization function which maps semantic features into discrete indices with
the codebook $Z$, and the AI-Src-Enc, respectively. This symbol stream passes
through a physical channel, $h$, with noise in the RF front end of a RX; which
is expressed by the received signal, ${\bf\it y}$:

${{\bf\it y}}=h{{\bf\it x}}+{{\bf\it n}},$ (2)

Here, the encoded signal by TX propagates over the channel; RX receives the
attenuated signal with ${{\bf\it n}}\sim{\cal
CN}\left({0,\sigma_{n}^{2}}\right)$. Then, ${\bf\it y}$ is decoded at the RX
to estimate the sentence ${\bf\it\hat{s}}$:

${\bf\it\hat{s}}=S_{\mathrm{\mathbf{D}}}\left(Q_{\mathrm{\mathbf{D}}}\left(C_{\mathrm{\mathbf{D}}}\left({\bf\it
y}\right);Z\right)\right),$ (3)

where $S_{\mathrm{\mathbf{D}}}(\cdot)$ and $C_{\mathrm{\mathbf{D}}}(\cdot)$
represent the AI-Src-Dec and the channel decoder, respectively, and
$Q_{\mathrm{\mathbf{D}}}(\cdot;Z)$ represents the vector dequantization
function which recovers received discrete indices into semantic features with
the same codebook $Z$.

On-device AI/LM = (New) Lossy Source Coding. While conventional (lossy) source
coding focuses on compressing the source input by ensuring statistical
similarity between the original and reconstructed signals (${\bf\it s}$ and
$\hat{{\bf\it s}}$), AI-Src-Enc takes a different approach. The system aims to
minimize lexical errors (i.e., lexical similarity) and semantic errors (i.e.,
semantic similarity) while also reducing the number of bits/symbols retrieved
from AI-Src-Enc, thereby achieving compression. In particular, it prioritizes
compressing the embedding size (i.e., the dimensionality of $\mathbf{r}$)
while maintaining, or even enhancing, both lexical and semantic similarities
between the original and reconstructed data.

While leveraging advanced traditional PHY communication system’s techniques
(e.g., multiple-input multiple-output (MIMO), OFDM, channel coding, etc.)
which prioritize achieving low bit error rate (BER) (or symbol error rate
(SER)), on-device AI/LM communication system focuses more on preserving the
meaning between the original and reconstructed data to ensure successful
distributed AI-to-AI (inter-AI) transmission. To this end, we design and
evaluate our E2E on-device AI communication system within the context of bit-
level PHY communication transmission to see its E2E performance in realistic
scenarios.333 In this study, our primary focus is on PHY layer functions, and
we do not address MAC (or higher) layer functions, like automatic repeat
request (ARQ) and hybrid automatic repeat request (HARQ), which aim to ensure
error-free transmission. It is important to note that in certain network
scenarios—where transmitted data are latency-sensitive and cannot afford the
delay caused by HARQ, or in broadcast situations where HARQ is not applicable,
such as in URLLC or LEO satellite networks—the retransmission process is
bypassed.  Our rationale for this system is to preserve the meaning and
context of the data, even in the presence of realistic channel noise and bit
errors.

###  III-B System Model

The architecture of the communication system is illustrated in Fig. 2, where
the TX consists of $C_{\mathrm{\mathbf{E}}}(\cdot)$,
$Q_{\mathrm{\mathbf{E}}}(\cdot;Z)$, and $S_{\mathrm{\mathbf{E}}}(\cdot)$ and
RX consists of $C_{\mathrm{\mathbf{D}}}(\cdot)$,
$Q_{\mathrm{\mathbf{D}}}(\cdot;Z)$, and $S_{\mathrm{\mathbf{D}}}(\cdot)$,
while both TX and RX share the codebook $Z$.

On the TX end, the symbol stream ${\bf\it s}$ is initially encoded into a
vector representation ${\bf\it r}$ that encapsulates its semantic information,
using the AI-Src-Enc $S_{\mathrm{\mathbf{E}}}(\cdot)$. This vector is then
mapped to discrete indices ${\bf\it t}$ within the codebook $Z$ through a
vector quantization technique $Q_{\mathrm{\mathbf{E}}}(\cdot;Z)$. Following
this, this discrete indices ${\bf\it t}$ are processed by the channel encoder
$C_{\mathrm{\mathbf{E}}}(\cdot)$, transforming them into ${\bf\it x}$ with
added redundancy, this step ensures reliable detection and correction of bit
errors.

Inversely, on the RX side, a channel decoder $C_{\mathrm{\mathbf{D}}}(\cdot)$
first decodes the received signal; vector dequantization technique
$Q_{\mathrm{\mathbf{D}}}(\cdot;Z)$ recovers the discrete indices
${\bf\it\hat{t}}$ into vector representation ${\bf\it\hat{r}}$ using the
codebook $Z$; and then the AI-Src-Dec $S_{\mathrm{\mathbf{D}}}(\cdot)$ decodes
the signal and recovers the symbol ${\bf\it\hat{s}}$.

AI-Source-Encoder and AI-Source-Decoder. Both the AI-Src-Enc
$S_{\mathrm{\mathbf{E}}}(\cdot)$ and AI-Src-Dec
$S_{\mathrm{\mathbf{D}}}(\cdot)$ are constructed from a series of $6$
transformer layers [17]. Each of these layers integrates a self-attention
mechanism, positional encoding, and a densely connected layer, fortified by
residual connections. Notably, while they share a similar foundational
architecture, the specific manner in which the self-attention operates
diverges between the encoder and decoder components. This distinction in the
self-attention mechanism ensures specialized processing tailored to the unique
demands of both encoding and decoding tasks. The encoder employs a fully-
visible self-attention strategy, granting the model the capability to focus on
any token of the input while the decoder leverages an auto-regressive self-
attention mechanism, limiting the model’s attention solely to previous
outputs, which is more appropriate for the streaming paradigm in which the
decoder operates. These architectures can be pre-trained on a large scale
corpus by corrupting documents and computing the cross entropy loss between
the decoder’s output and the original document to learn the model
generalizable knowledge [25].

Here, we employ such pre-trained checkpoints (BART-base [25]) and use the
encoder and decoder weights to initialize the weights of
$S_{\mathrm{\mathbf{E}}}(\cdot)$ and $S_{\mathrm{\mathbf{D}}}(\cdot)$
respectively. For a more detailed understanding of the en/decoder’s operation,
both $S_{\mathrm{\mathbf{E}}}(\cdot)$ and $S_{\mathrm{\mathbf{D}}}(\cdot)$
share the pre-trained embedding $\mathcal{E}$ and the pre-trained tokenizer
$\mathcal{T}$. Once the sentence ${\bf\it s}$ is given to
$S_{\mathrm{\mathbf{E}}}(\cdot)$, $\mathcal{T}$ tokenizes ${\bf\it s}$ into
tokens ${\bf\it s}_{t}=[s_{t_{1}},s_{t_{2}},...s_{t_{n}}]$ and maps each token
to embedding ${\bf\it s}_{e}=[s_{e_{1}},s_{e_{2}},...s_{e_{n}}]$ by
$\mathcal{E}$. Then, $S_{\mathrm{\mathbf{E}}}(\cdot)$ encodes ${\bf\it s}_{e}$
into hidden states ${\bf\it r}=[r_{1},r_{2},...r_{n}]\in\mathbb{R}^{n\times
d_{r}}$ through multiple transformer layers, and passes it to channel encoder
$C_{\mathrm{\mathbf{E}}}(\cdot)$. In this context, $n$ denotes the total
number of tokens, while $d_{r}$ specifies the dimension of the feature
representation of each token.

After transmission is complete and the hidden states ${\bf\it\hat{r}}$ are
retrieved from the channel decoder $C_{\mathrm{\mathbf{D}}}(\cdot)$ and
provided to $S_{\mathrm{\mathbf{D}}}(\cdot)$, the semantic decoder
$S_{\mathrm{\mathbf{D}}}({\bf\it\hat{r}})$ establishes the conditional
distribution
$p_{\theta_{S_{\mathrm{\mathbf{D}}}}}\left({\bf\it\hat{s}}_{i}\mid{\bf\it\hat{s}}_{0:i-1},{\bf\it\hat{r}}\right)$
and auto-regressively samples words from this distribution for each index.

Figure 3: Architecture of AI-Source-Encoder (AI-Src-Enc) and Quantizer

Vector Quantization (VQ-VAE). Our framework employs vector quantization to
transform continuous feature representations into discrete indices, enabling
efficient data transmission. The core of this process is a discrete codebook

$Z=\left\\{\boldsymbol{z}_{k}\right\\}_{k=1}^{K}\in\mathbb{R}^{K\times
d_{z}},$ (4)

where $K$ denotes the codebook size and $d_{z}$ the dimension of each code
vector.

At the TX end, the continuous feature vector ${\bf\it r}\in\mathbb{R}^{n\times
d_{r}}$, generated by the AI-Src-Enc $S_{\mathrm{\mathbf{E}}}(\cdot)$, is
quantized into discrete indices ${\bf\it t}\in\mathbb{R}^{n\times d_{z}}$
using $Z$. This quantization aims for $d_{z}<d_{r}$ to ensure transmission
efficiency. Specifically, each feature vector $r_{i}\in{\bf\it r}$ is divided
into segments $r_{i_{j}}$, each with dimension $d_{z}$. For instance, a
feature vector $r_{i}$ with dimension 768 and a codebook dimension of 256 can
be segmented into three parts ($[r_{i_{1}},r_{i_{2}},r_{i_{3}}]$), each of
dimension 256. The quantizer $Q_{\mathrm{\mathbf{E}}}(\cdot;Z)$ maps each
segment $r_{i_{j}}$ to the closest codebook entry $\boldsymbol{z}_{k}\in Z$,
forming the discrete representation $t_{i}\in{\bf\it t}$:

$t_{i}=Q_{\mathrm{\mathbf{E}}}(r_{i};Z)=\left(\underset{\boldsymbol{z}_{k}\in
Z}{\operatorname{argmin}}\left\|r_{i_{j}}-\boldsymbol{z}_{k}\right\|^{2}\right)\in\mathbb{R}^{\zeta\times
d_{z}},$ (5)

where $\zeta$ is the length of the encoded sequence. In the given example,
$\zeta$ equals 3.

Conversely, at the RX end, the dequantizer $Q_{\mathrm{\mathbf{D}}}(\cdot;Z)$
converts the received discrete indices $\hat{t}_{i}$ back into continuous
vectors $\hat{r}_{i}$ by directly referencing $Z$, reconstructing the feature
representation $\hat{r}_{i}$ from $\hat{t}_{i}$.

Channel En/Decoder. The channel encoder $C_{\mathrm{\mathbf{E}}}(\cdot)$ and
decoder $C_{\mathrm{\mathbf{D}}}(\cdot)$ incorporate various PHY communication
functions, such as Polar code, QAM mapper, OFDM, MIMO, selected for their
alignment with 5G-NR standards. This approach ensures that our framework
mirrors real-world communication scenarios while acknowledging its partial
compliance with 5G-NR protocols.

Firstly, the input bit for $C_{\mathrm{\mathbf{E}}}(\cdot)$ is grouped with
other bits to form a codeword. Polar codes, a form of linear block error
correction codes, are considered one of the channel coding schemes in 5G-NR,
where low complexity error correction is available [28]. It adds redundancy to
the codeword with a certain coderate $\rho$. This helps detect and correct bit
errors introduced during transmission. The codeword is then further processed
and segmented for transmission.

Each segmented codeword is mapped to a specific point on a constellation
diagram, such as our considered QAM. Higher QAM schemes offer more bits per
symbol (better efficiency) but are more susceptible to noise. The chosen QAM
symbol is then represented by a combination of amplitude and phase variations
in a carrier signal. The number of bits per symbol used is denoted by $m$ and
the mapper takes as input a message $u\in\\{0,\dots,2^{m}-1\\}$

The modulated symbols are divided into subcarriers across a wide spectrum.
This distributes the signal energy, making it more resilient to frequency-
selective fading channels. Each user (or subframe) can be modulated
independently, enabling adaptive bit rate and channel equalization by using
the received modulation and coding scheme (MCS) index. The MCS index is
dynamically selected and can be adjusted on a per-subframe basis, based on the
latest channel quality information (CQI).

Data is transmitted simultaneously leveraging multiple antennas, i.e., MIMO,
to exploit spatial diversity or multiplexing. This creates multiple
independent channels, increasing reliability, data rate, or both. The
processed signal is transmitted over the air. At the receiver, the signal is
received by multiple antennas. The received signal is then demodulated (QAM
decoding) and demultiplexed (OFDM decoding). Channel coding algorithms (Polar
decoding) are applied to correct any errors introduced during transmission.
Finally, the decoded bits are reassembled into the original data.

Channel. In the on-device AI communication, the wireless propagation channel,
henceforth simply referred to as _channel_ , introduces various types of noise
and impairments, which can significantly affect the transmitted signal. These
impacts include, but are not limited to, path loss, fading, Doppler shift, and
AWGN.

3GPP has defined standardized models that are used for the simulation and
testing of various wireless system concepts. These channel models describe
time variation, delay dispersion, angular dispersion (at both link ends) and
amplitude characteristics. We particularly leverage the standardized 3GPP
clustered delay line (CDL)-family channel models, such as CDL-A, CDL-B, and
CDL-C [29]. It is worth noting that while these CDL models are useful for
testing, particularly for link-level simulation, they often diverge
significantly from real-world channels.

###  III-C Train Objectives

Our framework is jointly trained using two loss functions:
$\mathcal{L}_{\mathrm{CE}}$ and $\mathcal{L}_{\mathrm{VQVAE}}$. The
$\mathcal{L}_{\mathrm{CE}}$ minimizes the discrepancy between input sentence
${\bf\it s}$ at the TX end and its predicted sentence ${\bf\it\hat{s}}$ at the
RX end, employing the cross entropy loss:

$\begin{split}\mathcal{L}_{\mathrm{CE}}(\hat{\mathbf{s}},\mathbf{s})=-\sum_{i=1}\nolimits
p\left(\mathbf{s}_{t_{i}}\right)\log\left(p\left(\hat{\mathbf{s}}_{t_{i}}\right)\right)+\\\
\left(1-p\left(\mathbf{s}_{t_{i}}\right)\right)\log\left(1-p\left(\hat{\mathbf{s}}_{t_{i}}\right)\right),\end{split}$
(6)

where $p\left(\mathbf{s}_{t_{i}}\right)$ is the true distribution in which the
correct token at the $i$-th index has a probability of 1, and all other tokens
have a probability of 0. On the other hand,
$p\left(\hat{\mathbf{s}}_{t_{i}}\right)$ denotes the predicted probability
distribution over all the possible tokens for the $i$-th index.
$\mathcal{L}_{\mathrm{VQVAE}}$ aims to jointly train codebook $Z$ and the
framework in end-to-end as follows:

$\begin{array}[]{r}\mathcal{L}_{\mathrm{VQVAE}}=\left\|\operatorname{sg}[{\bf\it
r}]-{\bf\it t}\right\|_{2}^{2}+\beta\left\|\operatorname{sg}[{\bf\it
t}]-{\bf\it r}\right\|_{2}^{2}.\end{array}$ (7)

where $\operatorname{sg}[\cdot]$ represents the stop-gradient operation,
ensuring no gradient is passed through and treating it as a constant that
doesn’t update. Our framework is trained by combining the two aforementioned
loss functions as follows:

$\begin{array}[]{r}\mathcal{L}=\mathcal{L}_{\mathrm{CE}}+\mathcal{L}_{\mathrm{VQVAE}}.\end{array}$
(8)

In this training process, the trainable parameters include AI-Src-Enc
$S_{\mathrm{\mathbf{E}}}(\cdot)$, AI-Src-Dec $S_{\mathrm{\mathbf{D}}}(\cdot)$,
its embedding $\mathcal{E}$, quantizer $Q_{\mathrm{\mathbf{E}}}(\cdot;Z)$,
dequantizer $Q_{\mathrm{\mathbf{D}}}(\cdot;Z)$, and its codebook $Z$.

###  III-D Optimization Techniques

Our system employs three key optimization techniques to enhance performance:

Noise-Tuning. We optimize robustness against communication noise by adaptively
tuning AI-Src-Enc and AI-Src-Dec to bit (or symbol) errors or channel noise.
During fine-tuning, we expose them to simulated channel impairments (e.g.,
under the 3GPP CDL-A channel) alongside the channel encoder/decoder. This
simple fine-tuning, without modifying any PHY module, significantly improves
reliability across diverse channel/communication conditions.

Codebook. To achieve efficient data compression, we utilize a VQ-VAE. It
learns a codebook of discrete vectors capturing the essence of the AI-Src-
Enc’s continuous representations. These codebook vectors, significantly
smaller than the originals, are transmitted, reducing bandwidth demands.
Interestingly, it also mitigates the impact of bit errors during transmission
as its representation learns the bit (or symbol) error pattern - potentially
due to it learning the bit (or symbol) error pattern during pre-training. Upon
reception, the VQ-VAE decodes these indices back into high-dimensional
vectors. This decoding utilizes the shared latent space (i.e., $Z$) learned
during pre-training, ensuring effective recovery of the semantic content
within the compressed representation. This method balances accuracy,
compression, and inference complexity through the adjustable compression rate
(and embedding size).

Pre-Training. To reduce the need for extensive application-specific training
data, we leverage pre-trained models like BART-base. This pre-training equips
the system with a broad base of generalizable knowledge, enhancing
generalization performance.

##  IV Experiments

###  IV-A Dataset

Train Dataset. Theoretically, for training purposes, any text can be utilized.
We can set both the input ${\bf\it s}$ and the output ${\bf\it\hat{s}}$ to be
identical for each sentence. This approach aims to train the framework to
ensure the faithful transmission of sentences without any modifications,
aligning with the primary objective of the system. However, following the
precedent set by earlier studies in text semantic communication [18, 19], we
utilize the European Parliament dataset [30] (called _EuroParl_) which is
extracted from the proceedings of the European Parliament.

Test Dataset. For the evaluation, it is important to evaluate the
generalization efficacy of framework on out-of-distribution data, particularly
on sentences or even tokens not encountered during training. To evaluate the
generalizability, we randomly sample 1K sentences from the image-caption
dataset _Flickr_ [31]. The token distribution in this dataset differs from our
training data, a result of reporting bias. This bias emerges because people
tend to report what interests them (e.g., parliamentary discussions) rather
than typical and general facts (e.g., describing an image).

###  IV-B Implementation Details

TABLE I: Type (or parameter) for channel en/decoder. Type (or Parameter) | Value
---|---
_Channel En/Decoder_ |
Channel coding | Polar code
Coderate ($\rho$) | $0.5$
# of coded bits | 960
# of information bits | 480
Mapper | QAM
# of bits per symbol ($m$) | $4$
# of OFDM symbols | 14
# of TX$\times$RX antennas | 2$\times$2, 1$\times$1
Direction | Downlink
Carrier frequency | 2.6 [GHz]
Subcarrier spacing | 15 [kHz]
FFT size (# of subcarrier) | 72
Channel (train) | CDL-A, Rayleigh
Channel (test) | CDL-$\\{$A$\sim$D$\\}$, Rayleigh
_AI-Source-En/Decoder_ |
Pre-trained LM ($\mathcal{T}$, $\mathcal{E}$, $S_{\mathrm{\mathbf{E}}}(\cdot)$, $S_{\mathrm{\mathbf{D}}}(\cdot)$) | BART-base
embedding dimension ($d_{r}$) | 768
_Codebook_ |
Dimension of each code ($d_{z}$) | 2
Codebook size ($K$) | 1024

Our entire framework is built using the Keras framework [32]. For modeling
various channel structures, we utilize the Sionna library [33]. We employ the
Huggingface library [34] to load the pre-trained weights of the BART-base
model, which serves as the foundation for initializing the embedding
$\mathcal{E}$, tokenizer $\mathcal{T}$, AI-Src-Enc
$S_{\mathrm{\mathbf{E}}}(\cdot)$ and the AI-Src-Dec
$S_{\mathrm{\mathbf{D}}}(\cdot)$, respectively. $\mathcal{T}$, pre-trained
embedding $\mathcal{E}$ In the fine-tuning phase, the Adam optimizer [35] is
employed, with the sequence length capped at 64 tokens, and the training has
been done for 3 epochs. We conduct a grid search to find the best learning
rate from the set [3e-4, 1e-4, 5e-5, 2e-5, 1e-5] and the batch size from [2,
4, 8], using the development dataset to determine the optimal parameters. All
of our experiments are run on an RTX 2080-Ti using 32-bit floating-point
precision.

Detailed system parameters are summarized in Table I.

###  IV-C Evaluation

In traditional communication systems, the performance of information
transmission is evaluated by measuring the accuracy of transmitting individual
bits (0s and 1s), as reflected by the Bit Error Rate (BER), or by assessing
the fidelity of conveying symbols, which are collections of bits, denoted by
the Symbol Error Rate (SER).

In contrast, on-device AI communication prioritizes the transmission of
meaningful content444This objective aligns with those of other research in the
field of semantic communication., focusing on the efficient utilization of
bandwidth for more effective data transfer. To accurately evaluate the
performance of on-device AI communication systems, we utilize three specific
metrics: lexical similarity (e.g., BLEU [36]), semantic similarity (e.g.,
SBERT [37]) between ${\bf\it s}$ and ${\bf\it\hat{s}}$, and the compression
rate of $\rho$.

Framework | PHY Comm. | Transmission | Pre-Training | Noise-Tuning |  EuroParl (In-dist.) |  Flickr (Out-dist.)
---|---|---|---|---|---|---
BLEU-3 | BLEU-4 | SBERT | BLEU-3 | BLEU-4 | SBERT
DeepSC [14] | ✗ (symbol-level) | ✗ | ✗ | ✗ | 0.6329 | 0.5802 | 0.7669 | 0.1878 | 0.1111 | 0.3737
Seq2Seq-SC [26] | ✓ (bit-level) | Tanh | ✓ | ✗ | 0.6428 | 0.5732 | 0.8142 | 0.5522 | 0.4795 | 0.7910
Ours ($1\times 1$) | ✓ (bit-level) | VQ-VAE | ✓ | ✓ | 0.9959 | 0.9946 | 0.9959 | 0.9714 | 0.9682 | 0.9998

TABLE II: Performance comparison study. Each baseline is trained with EuroParl
training set and evaluated using both the EuroParl and Flickr test sets
(Channel = Rayleigh, EbN0 $=7$ [dB]). The EuroParl test set assesses the in-
distribution performance, whereas the Flickr test set evaluates the out-of-
distribution performance.

BLEU. The BLEU score [36], initially developed for evaluating machine
translation, quantifies the correspondence between the $n$-grams of a
generated sentence ${\bf\it\hat{s}}$ and those in a reference sentence
${\bf\it s}$ [18, 19]. Here, $n$-grams mean a collection of $n$ successive
words in a sentence. BLEU score involves two key factors: (1) $n$-gram-based
precision of the generated sentence ${\bf\it\hat{s}}$ and the reference
sentence ${\bf\it s}$ as follow:

$p_{n}=\frac{\sum_{n\text{-gram}\in{\bf\it\hat{s}}}\operatorname{Count}_{\text{clip}}(n\text{-gram})}{\sum_{n\text{-gram}\in{\bf\it\hat{s}}}\operatorname{Count}(n\text{-gram})}.$
(9)

Here, $\operatorname{Count}_{\text{clip}}(n\text{-gram})$ represents the
number of $n$-grams from the generated sentence ${\bf\it\hat{s}}$ that are
found in a reference sentence ${\bf\it s}$ while
$\operatorname{Count}(n\text{-gram})$ represents the total number of $n$-grams
in the generated sentence ${\bf\it\hat{s}}$.

However, simply counting identical $n$-grams can lead to an overestimation.
Consider the case where the reference sentence is “I am a boy” and the
generated sentence is “a a a a”. In this scenario, the unigram precision
$p_{1}$ would erroneously be computed as 1, as each occurrence of the unigram
“a” is found in the reference sentence.

To address this issue, $\operatorname{Count}{\text{clip}}(n\text{-gram})$ is
employed to cap the count at the highest frequency observed in the reference
sentence ${\bf\it s}$, thereby preventing overcounting. In the given example,
$\operatorname{Count}{\text{clip}}(\text{``a''})$ is adjusted to 1 instead of
4, reflecting the maximum occurrence of the unigram “a” in the reference
sentence ${\bf\it s}$; (2) A brevity penalty is used to mitigate the influence
of sentence length, preventing the overfitting to the sentence length. This
penalty comes into play when the length of generated sentence
${\bf\it\hat{s}}$ is shorter than the sentence length of the reference
sentence ${\bf\it s}$ as follow:

$\mathrm{BP}=\begin{cases}1,&\text{ if }|{\bf\it\hat{s}}|>|{\bf\it s}|\\\
e^{(1-|{\bf\it s}|/|{\bf\it\hat{s}}|)}.&\text{ if
}|{\bf\it\hat{s}}|\leq|{\bf\it s}|\end{cases}$ (10)

The BLEU score for $n$-grams, represented as BLEU-$n$, is calculated as the
product of this brevity penalty and the exponential of the precision score for
$n$-grams $\exp\left(\log p_{n}\right)$. The overall BELU score is the brevity
penalty multiplied by the exponential of the weighted sum of the log precision
scores for different $n$-grams, represented as
$\exp\left(\sum_{n=1}^{N}w_{n}\log p_{n}\right)$, where $w_{n}$ denotes the
weight of the $n$-gram. Here, BLEU-$n$ refers to the BLEU score considering
only $n$-grams, and the general BLEU score is a weighted average of BLEU-1,
BLEU-2, BLEU-3, and BLEU-4. Higher-order $n$-grams (i.e., longer $n$-grams)
are indicative of the fluency and grammatical accuracy of the generated
sentence ${\bf\it\hat{s}}$.

SBERT. Despite a low lexical overlap between the sentences ${\bf\it s}$ and
${\bf\it\hat{s}}$, their semantic content may be closely aligned. For example,
the words “child” and “children” bear a close semantic relationship, yet a
BLEU score based on lexical matching would not recognize this and would rate
the similarity as zero. To compute such semantic similarity, sentences can be
represented into vector embeddings through an embedding model denoted as
$\boldsymbol{M}$. The degree of semantic similarity can then be quantified by
computing the cosine similarity between these vector representations as
follow:

$\operatorname{match}(\hat{\mathbf{s}},\mathbf{s})=\frac{\boldsymbol{M}(\mathbf{s})\cdot\boldsymbol{M}(\hat{\mathbf{s}})^{T}}{\left\|\boldsymbol{M}(\mathbf{s})\right\|\left\|\boldsymbol{M}(\hat{\mathbf{s}})\right\|}.$
(11)

Existing research in semantic communication often employs BERT [38] as the
embedding model $\boldsymbol{M}$ to encode sentences. To derive sentence
embedding, they typically use methods like average pooling or extract the
embedding from the first token (i.e., CLS pooling). These embeddings are then
used to calculate cosine similarity [18, 19]. Nonetheless, such pre-trained
models, when not fine-tuned for the specific task of semantic textual
similarity, may not effectively capture the true semantic nuances of
sentences. This is attributed to the anisotropic nature of the embedding
space, which can lead to embeddings that do not properly represent the
semantic variance across sentences [39]. Here, we use SBERT [37], which is
fine-tuned on semantic textual similarity tasks, to encode the sentence
embedding.

Compression Efficiency. General (lossy) source coding in conventional systems
emphasizes compact representation of information to enhance
transmission/spectral efficiency, often at the expense of minor data loss.
This contrasts with on-device AI communication, where the exchange of
efficient representations prioritizes semantic fidelity over mere data volume
reduction. The aim here is to transmit the essence and context of messages
with minimal semantic distortion.

The compression rate in on-device AI communication, therefore, assesses how
effectively AI-Src-Enc output is compacted. It is quantified by the ratio of
the sizes of AI-Src-Enc output ($\mathbf{r}$) and channel encoder input
($\mathbf{t}$), described as:

$\operatorname{compress}(\mathbf{r},\mathbf{t})=\frac{\operatorname{size}(\mathbf{r})}{\operatorname{size}(\mathbf{t})},$
(12)

where $\operatorname{size}(\mathbf{x})$ denotes the size of vector
$\mathbf{x}$.

To evaluate how compression impacts the conveyance of a message’s meaning, we
define compression efficiency ($\mathcal{E}_{\text{compress}}$) as a metric
that examines the balance between the compression rate and semantic
similarity:

$\mathcal{E}_{\text{compress}}\triangleq\frac{\operatorname{match}(\hat{\mathbf{s}},\mathbf{s};\rho)}{\log_{2}{(\operatorname{compress}(\mathbf{r},\mathbf{t}))}},$
(13)

where $\operatorname{match}(\cdot,\cdot;\rho)$ assesses the SBERT for a given
$\rho$ compress rate.

##  V Experimental Results

Transmit ${\bf\it s}$ | A man pushes a bicycle along the beach as the sun sets behind him.
---|---
Baseline | Receive $\hat{{\bf\it s}}$
DeepSC [14] | a man between a pity along the bureau as the danish sets behind him.
Seq2Seq-SC [26] | A man pushes a bicycle along the beach as sun sets behind him.
Ours ($1\times 1$) | A man pushes a bicycle along the beach as the sun sets behind him.

TABLE III: Received output examples of baselines and ours (Channel = Rayleigh,
EbN0 $=7$ [dB]).

((a)) BLEU-4 (EuroParl)

((b)) SBERT (EuroParl)

((c)) BLEU-4 (Flickr)

((d)) SBERT (Flickr)

Figure 4: Performance comparison with existing frameworks under different EbN0
[dB] (Channel = Rayleigh)

###  V-A Setup/Baseline

We primarily evaluate our proposed on-device AI/LLM communication frameworks
(“Ours”) from four distinct viewpoints:

PHY Communication. PHY communication conduct bit-level transmission by
leveraging specific PHY functions, e.g., Polar coding, QAM mapper, OFDM. On
the other hand, existing semantic communication works, exemplified by systems
like DeepSC [18]), conduct symbol-level transmission, not considering the
practical PHY communication functions.

Pre-Training Utilization. (1) w/o pre-training utilizes a transformer
architecture for the AI-Src-Enc and AI-Src-Dec, starting with random
initialization; while (2) w/ pre-training begins with pre-trained model
weights to initialize the AI-Src-Enc and AI-Src-Dec.

Noise-Tuning Utilization. (1) w/o noise-tuning fine-tunes the framework in the
absence of any simulated channel noise; while (2) w/ noise-tuning incorporates
simulated channel noise, inducing bit errors, into the fine-tuning process. In
particular, the systems is noise-tuned (fine-tuned) under the CDL family of
channel models and bit-level PHY communication setup (see details in Table I)
and energy per-bit to noise ratio (EbN0) $=5\sim 15$ [dB].

Codebook Utilization. In PHY communication, it is necessary to transform the
vector output of the AI-Src-Enc into a discrete bit-level representation. One
straightforward method is to directly map each floating-point number in the
vector to its bit-level equivalent (e.g., 768-dimensional vector results in
768$\times$32 bits). However, this method is highly susceptible to channel
noise, where even a single bit error can significantly distort the vector
representation (e.g., _floating-point arithmetic error_). To mitigate this
issue, we explore and compare various techniques that are more robust to
noise. (1) Direct encodes the floating-point numbers of ${\bf\it r}$ straight
into bits for transmission; (2) Tanh also transforms the floating-point
numbers of ${\bf\it r}$ into bits for sending but applies a hyperbolic tangent
activation function to adjust for the cases when bit errors from channel noise
push the received representation ${\bf\it\hat{r}}$ beyond certain range (e.g.,
$[-1,1]$). By doing so, it realigns the out-of-bound values back into the
$[-1,1]$ range, and has been used in Seq2Seq-SC [26]; (3) VQ-VAE employs
vector quantization along with a codebook $Z$ to encode vector representation
${\bf\it r}$ into a set of discrete codebook indices ${\bf\it t}$ for
transmission.

###  V-B Comparative Study

We begin by evaluating the overall efficacy of our proposed framework against
earlier works in semantic communication. Each baseline is trained using the
EuroParl training set and evaluated on both the EuroParl and Flickr test sets.
There is a significant overlap in vocabulary between the EuroParl training and
test sets, while the overlap is minimal between the EuroParl training set and
the Flickr test set. This setup allows for an assessment of the framework’s
generalizability. For a fair comparison, all baseline models, including the
baseline DeepSC which focuses on symbol-level transmission, are evaluated
under identical conditions of $1\times 1$ SISO and Rayleigh channel throughout
the comparative study.

Table II illustrates the performance comparison under a Rayleigh Channel with
EbN0 $=7$ [dB]. Meanwhile, Table III offers qualitative examples demonstrating
the transmission accuracy of each methodology, and Fig. 4 highlights the
robustness of each approach.

Combination | Codebook | Pre-Training | Noise-Tuning |  EuroParl (In-dist.) |  Flickr (Out-dist.)
---|---|---|---|---|---
BLEU-3 | BLEU-4 | SBERT | BLEU-3 | BLEU-4 | SBERT
Ours | VQ-VAE | ✓ | ✓ | 0.9918 | 0.9888 | 0.9944 | 0.9296 | 0.9178 | 0.9959
w/o pre-training ($3$ epochs) | VQ-VAE | ✗ | ✓ | 0.0690 | 0.0393 | 0.3571 | 0.0002 | 0.0000 | 0.0080
w/o pre-training ($10$ epochs) | VQ-VAE | ✗ | ✓ | 0.6136 | 0.5376 | 0.7277 | 0.0573 | 0.0246 | 0.0933
w/o noise-tuning | VQ-VAE | ✓ | ✗ | 0.9515 | 0.9325 | 0.9804 | 0.9147 | 0.8907 | 0.9787

TABLE IV: Performance variance w/ and w/o pre-training and noise-tuning
(Channel = CDL-A, EbN0 $=4$ [dB]).

Tables II and III demonstrate that our framework significantly outperforms
both DeepSC and Seq2Seq-SC. A key observation from Table II is the substantial
decline in DeepSC’s performance on the Flickr dataset, whereas the performance
reduction for Seq2Seq-SC on Flickr is less severe in comparison. This
difference can be mainly due to the use of pre-training which exhibits
generalization capability for OOV \- that will be elaborated on a later
section.

Fig. 4 also demonstrates the superior noise robustness of our framework
compared to Seq2Seq-SC. An interesting point in Fig. 4 is the gradual increase
in DeepSC’s trend compared to the sharp rise in Seq2Seq-SC and our framework.
This primarily stems from whether the transmission takes place through bit-
level transmission via practical PHY communication. Transmitting through the
PHY process information in bit-level, which requires converting the floating-
point vector into bits, where bit errors caused by noise can lead to
substantial changes in the information (called floating-point arithmetic
error). Therefore, in conditions of substantial noise, such as when EbN0 $<5$
[dB], transmitting vector representations through the PHY becomes challenging.
Conversely, DeepSC transmits representations based on symbol-level, where
noise is directly added into the continuous vector representation, while not
considering the practical issues in bit-level operation. This results in such
a gradual increase in its performance trend as it does not experience any
floating-point error.

((a)) BLEU-4 (In-dist.)

((b)) BLEU-4 (Out-dist.)

((c)) BLEU-4 (Out-dist.)

((d)) BLEU-4 (Out-dist.)

((e)) SBERT (In-dist.)

((f)) SBERT (Out-dist.)

((g)) SBERT (Out-dist.)

((h)) SBERT (Out-dist.)

Figure 5: Impact of noise-tuning (w/o Noise-Tuning vs. w/ Noise-Tuning). The
CDL-A channel assesses the in-distribution performance, whereas the CDL-B, -C,
and -D evaluate the out-of-distribution performance.

###  V-C Ablation Study

In this part of the discussion, we explore the impact of each component within
our optimized E2E on-device AI communication framework and examine their
respective roles. Table IV shows how performance varies with and without the
use of pre-training and noise-tuning. Additionally, Table VI shows the
performance differences when using different transmission approaches including
Naive, Tanh and VQ-VAE, showing the impact of the use of Codebook.

####  V-C1 Impact of Pre-Training

As shown in Table IV, w/o pre-training leads to a considerable performance
drop. After increasing the number of training epochs, the framework shows
improvement on in-distribution data that shares a substantial vocabulary and
distribution overlap with the training data. However, its performance on other
datasets with minimal overlap with the training data remains close to zero.
This indicates that pre-training plays a crucial role in the framework’s
ability to generalize - which allows this framework to be usable across a wide
range of applications.

####  V-C2 Impact of Noise-Tuning

As shown in Table IV, w/o noise-tuning leads relatively small performance
drop. However, it provides an intriguing insight that LM and VQ-VAE (i.e., AI-
Scr-Enc and AI-Src-Dec) are somewhat able to learn how to handle bit (or
symbol) errors during their communication with another end.

EbNo ${[\mathrm{dB}]}$ | Noise-Tuning | Test | BER | BLEU | SBERT
---|---|---|---|---|---
(CDL-A) | (Channel)
8pt. $2$ [dB] | ✗ | CDL-A | $2.124\times 10^{-1}$ | 0.5801 | 0.8477
| ✓ | CDL-A |  | 0.7278 | 0.9147
| ✗ | CDL-B | $2.332\times 10^{-1}$ | 0.4588 | 0.7734
| ✓ | CDL-B | 0.6125 | 0.8500
| ✗ | CDL-C | $2.419\times 10^{-1}$ | 0.4072 | 0.7359
| ✓ | CDL-C | 0.5394 | 0.7988
8pt. $3$ [dB] | ✗ | CDL-A | $1.641\times 10^{-1}$ | 0.7931 | 0.9481
| ✓ | CDL-A |  | 0.8893 | 0.9872
| ✗ | CDL-B | $1.844\times 10^{-1}$ | 0.7312 | 0.9224
| ✓ | CDL-B | 0.8441 | 0.9691
| ✗ | CDL-C | $1.934\times 10^{-1}$ | 0.6858 | 0.9023
| ✓ | CDL-C | 0.8260 | 0.9619

TABLE V: Impact of noise-tuning (w/o Noise-Tuning vs. w/ Noise-Tuning).

Case |  Codebook | Pre-train | Noise-tuning |  EuroParl (In-dist.) |  Flickr (Out-dist.)
---|---|---|---|---|---
BLEU-3 | BLEU-4 | SBERT | BLEU-3 | BLEU-4 | SBERT
Naive Transmission |  ✗ (Naive) | ✓ | ✓ | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000
Tanh Transmission |  ✗ (Tanh) | ✓ | ✓ | 0.3660 | 0.2759 | 0.6691 | 0.1636 | 0.0983 | 0.5094
VQ-VAE Transmission (Ours) |  ✓ (VQ-VAE) | ✓ | ✓ | 0.9918 | 0.9888 | 0.9944 | 0.9296 | 0.9178 | 0.9959

TABLE VI: Performance variance with different transmission approaches (Channel
= CDL-A, EbN0 $=4$ [dB]).

To assess the impact of noise-tuning in more detail, we present Table V-C2 and
Fig. 5. These results compare the performance of AI-Src-Enc and AI-Src-Dec
trained with and without noise-tuning (specifically, the CDL-A channel at EbN0
of $5\sim 15$ [dB]) in four different communication conditions by testing
those in CDL-A, B, C, and D channels. The performance difference showcases the
impact of incorporating realistic communication errors into the fine-tuning
process, indicating that AI-Src-Enc and AI-Src-Dec can learn and partially
mitigate bit error patterns induced by noisy channel encountered during link-
level PHY communication.555 We conjecture that the AI-Src-Enc and AI-Src-Dec
(e.g., LM and VQ-VAE) learn bit error pattern of a pre-determined
constellation in mapper/demapper. When symbol error occurs, neighbor symbols
likely are detected, not far-distant symbols. During noise-tuning, such
intelligent en/decoder adaptively modifies its $\mathcal{E}$ and $\mathcal{Z}$
to be more robust to bit errors. However, how the AI-Src-Enc and AI-Src-Dec
learn and overcome communication errors is still an open question that we
defer to our future work. Overall, the AI-Src-Enc/AI-Src-Dec archives EbN0
gain up to $\sim 1$ [dB] by utilizing the noise-tuning.

An important challenge for on-device AI communication is their robustness to
test-time distributional shifts that naturally occur when the test environment
no longer matches the conditions in a training environment. This is present in
wireless communication systems, where the propagation conditions may change
whenever the user is moving. Thus, an important question is whether the on-
device AI communication can retain its performance, both from the same
communication condition in training (in-distribution) and other communication
conditions (out-of-distribution). To this end, we also test its performance in
CDL-B, -C, and -D channels, whose condition is not seen in training, in Table
V-C2 and Figs. 5.666 CDL-A channels encompass both line-of-sight (LoS) and
non-LOS (NLoS) conditions, offering a more diverse representation of wireless
channel propagation. In contrast, CDL-B and CDL-C channels are predominantly
NLoS. CDL-D channels, on the other hand, are characterized by LoS.  CDL-B and
CDL-C results indicate that noise-tuning enhances the robustness even in
channels not seen in training, while the CDL-D (Figs. 5(d) and 5(h)) case
highlights that there is still room for improvement for the noise-tuning to be
more channel-agnostic.

((a)) w/ vs. w/o VQ-VAE (BLEU)

((b)) w/ vs. w/o VQ-VAE (SBERT)

((c)) Compress vs. Accuracy (BLEU)

((d)) Compress vs. Accuracy (SBERT)

((e)) Codebook size vs. Accuracy (BLEU)

Figure 6: Impact of codebook. Figs. 6(a) and 6(b) compares w/o Codebook and w/
Codebook. Fig. 6(c) and 6(d) show the tradeoff between compression and
accuracy. Fig. 6(e) shows a tradeoff between codebook size and accuracy.

Figure 7: Compression efficiency over compression rate

####  V-C3 Impact of Codebook

Table VI shows that the VQ-VAE transmission significantly outperforms other
methods, achieving near-flawless results under the CDL-A channel at an EbN0
$=4$ [dB]. This demonstrates the effectiveness of using a discrete codebook
for vector representation quantization in enhancing both accuracy and
transmission robustness.

Figs. 6(a) and 6(b) further illustrate the impact of the codebook (VQ-VAE)
over different EbN0 [dB]. In the scenario where a codebook is not used, the
shape for transmitting information is an $n\times d_{r}$, where each element
is a floating point vector. Thus, the total size of the transmitted
information $x$ becomes $n\times d_{r}\times 32$ bits, considering that each
floating-point vector is 32 bits. Conversely, when a codebook is used when
$d_{z}=2$, the size of transmission is reduced to $n\times(d_{r}/2)$. In this
case, each point in the transmission is an integer index. The total size of
the transmitted information $x$ becomes $n\times(d_{r}/2)\times 32$, with the
32 bits now representing the size of each integer. It is worth noting that
using integer datatype is notably robust against rounding errors and precision
loss; additionally, it allows for using smaller data types, such as 16-bit
integers. Remarkably, even with a $50\%$ compression of the output of AI-Src-
Enc, with-codebook consistently outperforms without-codebook, achieving EbN0
gains up to $\sim 6$ [dB]. This suggests the potential of the encoded message
from the AI-Src-Enc to be effectively compressed without losing its accuracy.

Tradeoff: Compression (and Codebook size) vs. Accuracy. The codebook
dimensions ($d_{z}$) and its size ($K$) in AI-Src-Enc/Dec impact the
performance of the on-device AI communication systems. Figs. 6(c) and 6(d)
show how $d_{z}$ affects performance, examining increments through
$\\{2,4,8,16,32,64\\}$. Increasing $d_{z}$ enhances compression but may reduce
transmission accuracy, as compressing a data vector of size $d_{z}$ into a
single index increases vulnerability to transmission errors—a single index
error can impact the entire $d_{z}$.

Fig. 6(e) shows the relationship between $K$ and the performance. Smaller $K$
reduces computational demands, yet it might cause an information loss during
quantization, which can adversely affect the fidelity of the reconstructed
representation. On the other hand, increasing $K$ provides a more fine-grained
resolution and greater accuracy. However, this comes with increased
computational cost due to the cost of training and inference overhead, and it
may even hinder the convergence of training.

Additionally, Fig. 7 the relationship between compression efficiency
($\mathcal{E}_{\text{compress}}$) and compression rate, identifying a Pareto
optimal point between the two. Therefore, finding the right balance and
designing an appropriate compression rate and codebook size is crucial for
achieving a desirable balance between accuracy and efficiency. Thus,
optimizing the compression rate (and codebook size) for AI-Src-Enc/Dec is
crucial for striking an optimal balance between accuracy and computational
efficiency.

##  VI Conclusions

In this study, we integrated the BART pre-trained LM with the NVIDIA SIONNA
link-level simulator to simulate on-device AI communication within a 5G-NR
framework. The performance of on-device communication was assessed in bit-
level transmission across various PHY communication setups, including 3GPP
CDL-family channels, MIMO, OFDM, and Polar code.

Our analysis yielded several key insights from three optimization techniques:
(1) Integration of pre-trained language models significantly enhances the
generalization capabilities of the on-device AI communication system; (2)
Noise-tuning results in an approximate EbN0 gain of around $1$ [dB],
effectively improving performance in both familiar (CDL-A channel) and new
(CDL-B$\sim$D channels) communication conditions; (3) Codebook utilization
leads to an EbN0 gain of up to approximately $6$ [dB], accompanied by a $50\%$
reduction in AI-Src-Enc output size. Even with a compression rate of $8$
(utilizing only $12.5\%$ of the compressed information), the system maintains
reasonable performance levels.

The implementation strategy involves two stages:

  * •

_Training Stage (Offline)_ : The AI-Src-En/Dec embedding, along with the
codebook, are trained by minimizing the loss function described in Sec. III-C.
This computationally intensive stage occurs offline, leveraging powerful
resources.

  * •

_Execution Stage (On-Device)_ : The on-device system employs the pre-trained
AI-Src-Enc/Dec and codebooks, requiring only inference-level computations.

Limitations. While our proposed framework showcases the performance of on-
device AI communication under a bit-level transmission with a link-level PHY
communication setup, there are notable limitations that should be addressed to
achieve full compliance with 5G-NR communication systems. Specifically, beyond
the PHY, the medium access control (MAC) or network layers typically employs
automatic repeat request (ARQ) or hybrid ARQ (HARQ) protocols for packet error
correction. The absence of consideration for these higher-layer protocols in
our study limits the assessment of the practical performance of our framework.
Incorporating ARQ or HARQ mechanisms into our simulation setup would provide a
more comprehensive evaluation of the on-device AI communication framework’s
efficacy in real-world communication scenarios.

Open Questions. Experimental results in Sec. IV-C highlight that on-device AI
can improve communication performance through its predictive and contextual
learning abilities. While integrating channel encoding/decoding with AI-Src-
Enc/Dec, such as through JSCC or AI-native PHY/MAC, could unlock the full
potential, this faces challenges like compatibility with 4G-LTE and 5G-NR,
regulatory issues, and cost implications.

Therefore, employing AI as a complementary tool while keeping or carefully
optimizing existing PHY or higher-layer functions presents a more feasible
approach. Future research could focus on: (1) finding the best noise-tuning
conditions without altering PHY functions; (2) exploring how language models
can effectively counteract noisy conditions; (3) improving specific PHY
functions to boost AI communication; (4) determining if AI/LM can also learn
packet error patterns to ease the HARQ (or ARQ) process.

Acknowledgements: This research was partly funded by the National Science
Foundation (NSF). We express our deep gratitude to Prof. Andreas F. Molisch
for his invaluable feedback and multifaceted support throughout this work. We
also thank Thomas Choi for his insightful discussions during the initial
stages of our research.

## References

  * [1] S. Dhar, J. Guo, J. J. Liu, S. Tripathi, U. Kurup, and M. Shah, “A survey of on-device machine learning: An algorithms and learning theory perspective,” _ACM Trans. Internet Things_ , vol. 2, no. 3, Jul. 2021.
  * [2] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill _et al._ , “On the opportunities and risks of foundation models,” _arXiv preprint arXiv:2108.07258_ , 2022.
  * [3] S. Geng, S. Liu, and et al., “Recommendation as language processing (RLP): A unified pretrain, personalized prompt & predict paradigm (P5),” in _Proc. ACM Conf. on Recommender Systems_ , New York, NY, USA, 2022, p. 299–315.
  * [4] C. Cui, Y. Ma, and et al., “A survey on multimodal large language models for autonomous driving,” _arXiv preprint arXiv:2311.12320_ , 2023.
  * [5] G. Delétang, A. Ruoss, P.-A. Duquenne, E. Catt, T. Genewein, C. Mattern, J. Grau-Moya, L. K. Wenliang, M. Aitchison, L. Orseau, M. Hutter, and J. Veness, “Language modeling is compression,” _arXiv preprint arXiv:2309.10668_ , 2023.
  * [6] X. Lin, “An overview of 5G advanced evolution in 3GPP release 18,” _IEEE Commun. Standards Mag._ , vol. 6, no. 3, pp. 77–83, 2022.
  * [7] Z. Qin, X. Tao, J. Lu, W. Tong, and G. Y. Li, “Semantic communications: Principles and challenges,” _arXiv preprint arXiv:2201.01389_ , 2021.
  * [8] E. Bourtsoulatze, D. B. Kurka, and D. Gündüz, “Deep joint source-channel coding for wireless image transmission,” _IEEE Trans. on Cognitive Communications and Networking_ , vol. 5, no. 3, pp. 567–579, 2019.
  * [9] J. Shao, Y. Mao, and J. Zhang, “Learning task-oriented communication for edge inference: An information bottleneck approach,” _IEEE J. Sel. Areas Commun._ , vol. 40, no. 1, pp. 197–211, 2021.
  * [10] D. Huang, F. Gao, X. Tao, Q. Du, and J. Lu, “Toward semantic communications: Deep learning-based image semantic coding,” _IEEE J. Sel. Areas Commun._ , vol. 41, no. 1, pp. 55–71, 2022.
  * [11] T.-Y. Tung and D. Gündüz, “DeepWiVe: Deep-learning-aided wireless video transmission,” _IEEE J. Sel. Areas Commun._ , vol. 40, no. 9, pp. 2570–2583, 2022.
  * [12] S. Wang, J. Dai, Z. Liang, K. Niu, Z. Si, C. Dong, X. Qin, and P. Zhang, “Wireless deep video semantic transmission,” _IEEE J. Sel. Areas Commun._ , vol. 41, no. 1, pp. 214–229, 2022.
  * [13] P. Jiang, C.-K. Wen, S. Jin, and G. Y. Li, “Wireless semantic communications for video conferencing,” _IEEE J. Sel. Areas Commun._ , vol. 41, no. 1, pp. 230–244, 2022.
  * [14] H. Xie, Z. Qin, G. Y. Li, and B.-H. Juang, “Deep learning enabled semantic communication systems,” _IEEE Trans. on Signal Processing_ , vol. 69, pp. 2663–2675, 2021.
  * [15] T. Han, Q. Yang, Z. Shi, S. He, and Z. Zhang, “Semantic-preserved communication system for highly efficient speech transmission,” _IEEE J. Sel. Areas Commun._ , vol. 41, no. 1, pp. 245–259, 2022.
  * [16] Z. Weng, Z. Qin, X. Tao, C. Pan, G. Liu, and G. Y. Li, “Deep learning enabled semantic communications with speech recognition and synthesis,” _IEEE Trans. on Wireless Communications_ , 2023.
  * [17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems (NIPS)_ , vol. 30, 2017.
  * [18] H. Xie, Z. Qin, G. Y. Li, and B.-H. Juang, “Deep learning enabled semantic communication systems,” _IEEE Trans. on Signal Process._ , vol. 69, pp. 2663–2675, 2021.
  * [19] H. Hu, X. Zhu, F. Zhou, W. Wu, R. Q. Hu, and H. Zhu, “One-to-many semantic communication systems: Design, implementation, performance evaluation,” _IEEE Commun. Lett._ , 2022.
  * [20] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _Advances in neural information processing systems (NIPS)_ , vol. 33, pp. 1877–1901, 2020.
  * [21] A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso _et al._ , “Beyond the imitation game: Quantifying and extrapolating the capabilities of language models,” _arXiv preprint arXiv:2206.04615_ , 2022.
  * [22] S. Thrun, “Lifelong learning algorithms,” in _Learning to learn_. Springer, 1998, pp. 181–209.
  * [23] J. Hoydis, S. Cammerer, F. A. Aoudia, A. Vem, N. Binder, G. Marcus, and A. Keller, “Sionna: An open-source library for next-generation physical layer research,” _arXiv preprint arXiv:2203.11854_ , 2022.
  * [24] A. Van Den Oord, O. Vinyals _et al._ , “Neural discrete representation learning,” _Advances in neural information processing systems (NIPS)_ , vol. 30, 2017.
  * [25] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” in _Proc. the Association for Computational Linguistics_ , Online, Jul. 2020, pp. 7871–7880.
  * [26] J.-H. Lee, D.-H. Lee, E. Sheen, T. Choi, and J. Pujara, “Seq2seq-sc: End-to-end semantic communication systems with pre-trained language model,” _Asilomar Conference on Signals, Systems, and Computers_ , 2023.
  * [27] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” _arXiv preprint arXiv:1312.6114_ , 2013.
  * [28] 3GPP TS 38.212 v17.3.0, “NR; multiplexing and channel coding,” Sep. 2022.
  * [29] 3GPP, “Study on channel model for frequencies from 0.5 to 100 GHz,” _Tech. Rep. 38.901 (V17.0.0)_ , Mar. 2022.
  * [30] P. Koehn, “Europarl: A parallel corpus for statistical machine translation,” in _Proc. Machine Translation Summit X: Papers_ , Phuket, Thailand, Sep. 2005, pp. 79–86.
  * [31] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier, “From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,” _Trans. of the Association for Computational Linguistics_ , vol. 2, pp. 67–78, 2014.
  * [32] F. Chollet _et al._ , “Keras,” https://keras.io, 2015.
  * [33] J. Hoydis, S. Cammerer, F. Ait Aoudia, A. Vem, N. Binder, G. Marcus, and A. Keller, “Sionna: An open-source library for next-generation physical layer research,” _arXiv:2203.11854 [cs.IT]_ , Mar. 2022.
  * [34] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz _et al._ , “Transformers: State-of-the-art natural language processing,” in _Proc. Empirical Methods in Natural Language Processing_ , 2020, pp. 38–45.
  * [35] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _International Conference on Learning Representations (ICLR)_ , San Diega, CA, USA, 2015.
  * [36] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: a method for automatic evaluation of machine translation,” in _Proc. the Annual Meeting of the Association for Computational Linguistics_ , Philadelphia, Pennsylvania, USA, 2002, pp. 311–318.
  * [37] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using Siamese BERT-networks,” in _Proc. Empirical Methods in Natural Language Processing_ , Hong Kong, China, Nov. 2019, pp. 3982–3992.
  * [38] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _Proc. Association for Computational Linguistics_ , Minneapolis, Minnesota, 2019, pp. 4171–4186.
  * [39] B. Li, H. Zhou, J. He, M. Wang, Y. Yang, and L. Li, “On the sentence embeddings from pre-trained language models,” in _Proc. the 2020 Conf. on Empirical Methods in Natural Language Processing_ , Online, Nov. 2020, pp. 9119–9130.

  *[LLM]: large language model
  *[AI]: artificial intelligence
  *[LM]: language model
  *[TX]: transmitter
  *[RX]: receiver
  *[NN]: neural network
  *[JSCC]: joint source-and-channel coding
  *[AWGN]: additive white Gaussian noise
  *[FEC]: forward error correction
  *[OFDM]: orthogonal frequency-division multiplexing
  *[E2E]: end-to-end
  *[OOV]: out-of-vocabulary
  *[OOD]: out-of-domain
  *[NLP]: natural language processing
  *[PHY]: physical layer
  *[VQ-VAE]: vector quantised variational auto encoder
  *[MIMO]: multiple-input multiple-output
  *[BER]: bit error rate
  *[SER]: symbol error rate
  *[ARQ]: automatic repeat request
  *[HARQ]: hybrid automatic repeat request
  *[CDL]: clustered delay line
  *[LoS]: line-of-sight
  *[NLoS]: non-LOS