#  LightCode: Light Analytical and Neural Codes for Channels with Feedback

Sravan Kumar Ankireddy Correspondence to: Sravan Ankireddy
<EMAIL_ADDRESS>University of Texas at Austin  Krishna Narayanan
Texas A&M University, College Station  Hyeji Kim  University of Texas at
Austin

###### Abstract

The design of reliable and efficient codes for channels with feedback remains
a longstanding challenge in communication theory. While significant
improvements have been achieved by leveraging deep learning techniques, neural
codes often suffer from high computational costs, a lack of interpretability,
and limited practicality in resource-constrained settings.

We focus on designing low-complexity coding schemes that are interpretable and
more suitable for communication systems. We advance both analytical and neural
codes. First, we demonstrate that PowerBlast, an analytical coding scheme
inspired by Schalkwijk-Kailath (SK) and Gallager-Nakibŏglu (GN) schemes,
achieves notable reliability improvements over both SK and GN schemes,
outperforming neural codes in high signal-to-noise ratio (SNR) regions. Next,
to enhance reliability in low-SNR regions, we propose LightCode, a lightweight
neural code that achieves state-of-the-art reliability while using a fraction
of memory and compute compared to existing deep-learning-based codes. Finally,
we systematically analyze the learned codes, establishing connections between
LightCode and PowerBlast, identifying components crucial for performance, and
providing interpretation aided by linear regression analysis.

###### Index Terms:

Channels with Feedback, Deep Learning, Channel Coding, Feedback Coding, Finite
Block Length Coding

##  I Introduction

Shannon introduced the concept of feedback channel in [1]. In a channel with
feedback, the transmitter cooperates with the receiver to improve the
probability of successful transmission by utilizing the feedback from the
receiver. In [1], Shannon assumes a perfect noiseless feedback channel with
unit delay and demonstrates that the availability of feedback at the
transmitter does not change the capacity of the resultant forward channel for
memoryless channels. Interestingly, while the capacity remains the same,
significant improvements in terms of error exponents can be achieved with the
help of feedback.

One of the seminal schemes in the noiseless feedback setting has been provided
by Shalkwijk and Kailath in [2, 3] (SK) using a simple linear encoding scheme
resulting in a doubly exponential error exponent for finite block lengths. In
[4], a two-phase variant of the SK scheme, which we refer to as Gallager-
Nakibŏglu (GN), was discussed that can lead to further exponential decay of
the error, provided a sufficiently good SNR is available for the forward
channel. The SK scheme has been enhanced in various ways. The Modulo-SK scheme
[5] and schemes by Chance-Love [6] and Mishra et al [7], extends the SK scheme
for noisy feedback, while compressed error correction (CEC) [8] and
accumulative iterative code (AIC) [9] focus on reducing channel use through
continuous error vector compression, using noiseless feedback.

While guaranteeing impressive error exponents, SK and other analytical coding
schemes have not been adapted to practical communication systems, as the
improvement in performance does not justify the cost incurred in terms of
high-numerical precision, increase in the amount of feedback, and large number
of rounds of communication. Because of this, analytical feedback coding
schemes in practice are limited to automatic repeat request (ARQ) and hybrid-
ARQ (HARQ) retransmission schemes, where the receiver provides simple one-bit
feedback to indicate success (acknowledgment/ACK) or failure (negative
acknowledgment/NACK). Extending ARQ to multi-bit feedback is an interesting
research topic; for example, compressed error hybrid ARQ (CE-HARQ) [10] and
Griffin et al. [11] propose using full feedback from the receiver to
iteratively improve the error vector at the receiver.

Recent advances in deep learning revived the interest in coding for channels
with feedback by leveraging the expressive power [12] of deep neural networks.
Several works proposed deep learning approaches to improve the performance of
channel codes, ranging from augmenting analytical decoders with learnable
parameters [13, 14, 15, 16] to creating novel neural network architectures
based neural encoders and decoders [17, 18] and improving the code design
using sequential models [19, 20]. For channels with feedback, deep learning
techniques were used to design new encoding and decoding schemes that take
advantage of the high-capacity feedback channel. Deepcode [21] modeled both
encoder and decoder functions as recurrent neural networks (RNNs) to process a
bit stream in a sequential manner to directly minimize the end-to-end
transmission error over an AWGN channel. Several works further explored this
idea of using learning-based approaches for modeling the encoding and decoding
operations, which can be broadly classified into the RNN family of codes [22,
23, 24] and the transformer family of codes [25, 26], discussed in detail in
Section V. The current state-of-the-art is generalized block attention
feedback (GBAF) [26], which uses self-attention and transformer architecture
to perform block coding.

While these state-of-the-art deep-learning-based feedback codes provide
tremendous improvements in BLER performance, they also come with significant
memory and computational costs that may not be supported by the next
generation of communication transceivers that operate with limited onboard
resources. Therefore, it is desirable to develop lightweight codes that use
simple schemes while providing desirable performance. To accomplish a reduced
complexity coding scheme, we explore two different directions in this work.
First, we review and understand the existing analytical feedback coding
schemes to identify the limitations and propose a new feedback coding scheme
that provides non-trivial performance improvements over the existing schemes
for channels with passive, noiseless feedback. Next, we propose a lightweight
deep-learning-based feedback coding scheme that can greatly reduce the
complexity compared to neural block-feedback coding schemes by limiting
ourselves to a symbol-by-symbol scheme instead of block coding schemes.

Our main contributions are summarized as follows:

  * •

We provide a comprehensive review of the existing analytical and deep-
learning-based coding schemes for channels with feedback and identify the
limitations of existing approaches (Section III and Section V).

  * •

We propose PowerBlast, an analytical coding scheme that noticeably improves
the performance over Schalkwijk-Kailath and two-phase Gallager-Nakibŏglu
schemes (Section IV) and exhibits reliability comparable to state-of-the-art
deep-learning-based coding schemes in regions of high-SNR (Section VII).

  * •

We propose LightCode, a lightweight neural coding scheme that achieves a
performance superior to current state-of-the-art feedback coding schemes using
$10\times$ fewer parameters and computational complexity (Section VI, Section
VII), while maintaining interpretability.

  * •

We analyze the representations learned by the LightCode encoder using linear
regression and draw comparisons with PowerBlast. We also demonstrate that the
relation between encoded symbols and the feedback is highly non-linear in the
final rounds, underscoring the need for deep-learning-based coding scheme
(Section VIII).

##  II System Model

We consider a feedback coding scheme with an AWGN forward channel and an AWGN
feedback channel with noisy passive feedback. The goal is to transmit a
message vector $\mathbf{u}\in\\{0,1\\}^{K}$ of length $K$ to the receiver
using a total of $D$ channel uses, resulting in an overall coding rate of
$R=\nicefrac{{K}}{{D}}$. In this work, we consider symbol-by-symbol coding,
where the entire block of $K$ bits will be mapped to one symbol. Hence, the
terms block and symbol can be used interchangeably. Based on this, the number
of rounds of communication for a rate $\nicefrac{{K}}{{D}}$ code is $D$.

Figure 1: Illustration of the $i^{\text{th}}$ round of communication for
channels with feedback. The encoder takes as input the message bits
$\mathbf{u}$ and the encoder output from previous rounds $x^{(i-1)}$,
concatenated with the feedback from previous rounds $\tilde{y}^{(i-1)}$, to
compute $x_{i}$.

In the first round, the transmitter encodes the message block
$\mathbf{u}\in\\{0,1\\}^{K}$ to a real-valued output $x_{1}\in\mathbb{R}$ and
transmits using the forward AWGN channel as

$\displaystyle x_{1}$ $\displaystyle=\phi(\mathbf{u}),$ (1) $\displaystyle
y_{1}$ $\displaystyle=x_{1}+n_{1},$ (2)

where $n_{1}\sim\mathcal{N}(0,\sigma^{2}_{ff})\in\mathbb{R}$ denotes the
feedforward noise. The receiver then sends the noisy received symbol as
feedback using the feedback channel as

$\tilde{y}_{1}=y_{1}+\tilde{n}_{1},$ (3)

where $\tilde{n}_{1}\sim\mathcal{N}(0,\sigma_{fb}^{2})\in\mathbb{R}$ denotes
the feedback noise.

At round $i>1$, the encoder $\phi$ computes the output $x_{i}$ using the input
message bits $\mathbf{u}$ and the encoder outputs from previous $i-1$ rounds
$x^{(i-1)}=\\{x_{1},\dots,x_{i-1}\\}$ and the feedback from previous $i-1$
rounds $\tilde{y}^{(i-1)}=\\{\tilde{y}_{1},\dots,\tilde{y}_{i-1}\\}$ as

$x_{i}=\phi\left(\mathbf{u},x_{1},\dots,x_{i-1},\tilde{y}_{1},\dots,\tilde{y}_{i-1}\right).$
(4)

At the end of $D$ rounds of communication, the decoder $\psi$ estimates the
transmitted message vector $\hat{\mathbf{u}}$ using the received symbols from
all $D$ rounds $\\{y_{1},y_{2},\dots,y_{D}\\}$ as

$\hat{\mathbf{u}}=\psi\left(y_{1},y_{2},\dots,y_{D}\right),$ (5)

where $\hat{\mathbf{u}}\in\\{0,1\\}^{K}$. The objective is to design an
encoder-decoder pair $\\{\phi,\psi\\}$ that minimizes the probability of error
$\mathbb{P}\\{\mathbf{u}\neq\hat{\mathbf{u}}\\}$ for a given number of rounds
$D$, under a sum power constraint of
$\sum_{i=1}^{D}\mathbb{E}[|x_{i}|^{2}]=D$.

##  III Analytical Coding Schemes for Channels with Noiseless Feedback

In this section, we review analytical coding schemes for channels with
noiseless feedback i.e., $\sigma_{fb}^{2}=0$. We begin by reviewing the
celebrated Schalkwijk-Kailath coding scheme [2], one of the seminal works in
coding for channels with noiseless feedback. We then proceed to review a less
widely-known scheme by Gallager-Nakibŏglu [4]. This scheme is similar to the
SK scheme for the first $D-1$ rounds. Still, it deviates greatly in the last
round, tailoring for the transmission of discrete messages, often
significantly improving the performance in one round of communication. The
Gallager-Nakibŏglu (GN) scheme is typically not considered as a baseline as it
exhibits a worse error rate compared to SK. However, in the next section, we
introduce a novel analytical coding scheme, building on the SK and GN schemes.
This new scheme achieves a significantly lower error rate than either the SK
or GN schemes and is often comparable to highly complex neural coding schemes
in high-SNR regions, defying the conventional belief that neural coding
schemes are much more reliable than analytical ones.

###  III-A Schalkwijk-Kailath coding scheme

SK scheme considers the problem of transmitting a fixed number of bits on an
AWGN forward channel and a noiseless feedback channel. The transmission begins
by mapping $K$ bits of information symbols to a single $2^{K}$-ary pulse-
amplitude modulation (PAM) symbol $\Theta$ from the constellation

$\Theta\in\\{\pm 1\eta,\pm 2\eta,\dots,\pm(2^{K}-1)\eta\\},$ (6)

where $\eta=\sqrt{\nicefrac{{3}}{{(2^{2K}-1}})}$ is the scaling factor to
ensure unit power normalization of the PAM constellation. We note that even
though a block of information is transmitted since all the $K$ bits of
information are mapped to one PAM symbol, the SK scheme at its core can be
considered a symbol-by-symbol feedback scheme. We now describe the SK coding
scheme in detail.

In the first round, the uncoded PAM symbol is transmitted after accounting for
power constraint $P$ i.e., $x_{1}=\sqrt{P}\Theta$ as

$y_{1}=x_{1}+n_{1},$ (7)

where $n_{1}\in\mathcal{N}(0,\sigma_{ff}^{2})$ is the noise in the forward
AWGN channel. The received symbol $y_{1}$ is then sent back to the transmitter
noiselessly. For the second round, the transmitter first computes the
receiver’s estimate of the transmitted symbol based on $y_{1}$ as
$\hat{\Theta}_{1}=\frac{y_{1}}{1+\sigma^{2}/P}$ using linear minimum mean
square error (LMMSE) estimator. In the second round, after receiving $y_{1}$
as feedback, the transmitter sends the scaled version of the error in the
estimate from the previous round, i.e.,
$\epsilon_{1}=\hat{\Theta}_{1}-\Theta$. This process continues for the
remaining rounds. In other words, starting from round $i=2$, the goal of the
transmitter is to transmit the error in estimate from the previous round,
$\epsilon_{i-1}=\hat{\Theta}_{i-1}-\Theta$, after scaling appropriately to
satisfy the power constraint. The complete algorithm is described in Alg. 1.

Error analysis.  It is shown in [5] that the probability of error for rate
$\nicefrac{{K}}{{D}}$ SK scheme is given by

$p_{SK}=2(1-2^{-K})Q\left(\sqrt{\frac{3S(1+S)^{D-1}}{2^{2K}-1}}\right),$ (8)

where $S$ is the SNR of the forward AWGN channel on a linear scale.

Input: Message symbol $\Theta$, number of rounds $D$, noise variance
$\sigma^{2}$

Round 1: Tx: Power normalization: $x_{1}=\sqrt{P}\Theta$;

Forward channel: $y_{1}=x_{1}+n_{1}$;

Rx: LMMSE estimate of transmit symbol
$\hat{\Theta}_{1}=\frac{y_{1}}{1+\sigma^{2}/P}$ ;

/* Tx sends the error in estimate $\hat{\Theta}_{1}-\Theta$ over the next
$D-1$ rounds */

while  _$2\leq i\leq D$_ do

Tx: Compute the error in estimate of previous round
${\epsilon}_{i-1}=\hat{\Theta}_{i-1}-\Theta$ ;

Power normalization: $x_{i}=\frac{\sqrt{P}}{\sigma_{i-1}}{\epsilon}_{i-1}$,
where $\sigma^{2}_{i-1}=\mathbb{E}[\epsilon^{2}_{i-1}]$;

Forward channel: $y_{i}=x_{i}+n_{i}$;

Rx: LMMSE estimate of transmit symbol
$\hat{\epsilon}_{i-1}=\frac{\sqrt{P}\sigma_{i-1}}{P+\sigma^{2}}y_{i}$ ;

Update the estimate of $\Theta$ as:
$\hat{\Theta}_{i}=\hat{\Theta}_{i-1}-\hat{\epsilon}_{i-1}$

end while

Decoding: Map $\hat{\Theta}_{D}$ to the closest symbol in the $2^{K}$ PAM
constellation.

Algorithm 1 Schalkwijk-Kailath (SK) coding scheme

###  III-B Gallager-Nakibŏglu coding scheme

In [4], Gallager-Nakibŏglu proposed a two-phase scheme that builds on the
Elias scheme, which is also closely related to the SK scheme. While the SK
scheme considers the problem of transmitting a discrete symbol, Elias [27]
studied the problem of transmitting a Gaussian random variable
$U\in\mathcal{N}(0,\sigma^{2})$. The strategy for forward and feedback
transmissions is similar to SK, where a scaled version of the error in the
LMMSE estimate is transmitted in every round $i>1$. The main difference
between the SK and Elias schemes lies in that the SK scheme aims to refine the
message itself, while the Elias scheme aims to refine the noise added in the
very first transmission.

GN scheme operates in two phases and relies on the assumption that after a
sufficiently large number of rounds of the Elias scheme, the effective SNR for
the forward channel shall be adequately high for the noise variance to be
considered small compared to the distance between the symbols in the PAM
constellation. In such a high-SNR regime, a strategy superior to the Elias
scheme can be implemented by taking advantage of the discrete nature of the
signal. Instead of transmitting the original error vector with respect to
$2^{K}$-ary PAM, the integer difference in the PAM index of the estimate and
the true PAM symbol is transmitted. We refer to this as discrete-symbol
scheme. This method results in an error exponent that decreases with an
exponential order, which increases linearly with the number of rounds. For
this work, we assume that the high-SNR region is realized in the final round
of communication, which is valid according to the 2-phase strategy described
in [4].

We now describe the GN scheme in detail. The first round of GN is simply
uncoded PAM, except for the power allocation. In [4], it is shown the optimal
power distribution across $D$ rounds is attained by choosing $P_{1}$ and
$P_{2}$ such that $P_{1}+(D-1)P_{2}=DP$ and $P_{1}=P_{2}+1$ where $P_{1}$ is
the power constraint in round 1 and $P_{2}$ is the power constraint in
remaining rounds. Hence, the transmission in round 1 is given by

$y_{1}=\sqrt{P_{1}}\Theta+n_{1}.$ (9)

For the remaining rounds, the goal of the transmitter is to send the noise
$n_{1}$ to the receiver as in the Elias scheme, where the LMMSE estimate at
the receiver is improved iteratively, and a maximum likelihood (ML) detection
is used at the end of $D-1$ rounds of communication to map the estimate to the
original PAM constellation.

Finally, for the last round, GN uses a discrete-symbol scheme suitable for the
high-SNR region by transmitting the error in the PAM index $U$. We follow the
assumption from [4] that the high-SNR region guarantees that
$U\in\\{-1,0,1\\}$ with high probability. In [4], an ML decoder is used for
the ease of analysis across multiple rounds of discrete symbol scheme, which
is sub-optimal. However, since we are assuming only one round of the discrete-
symbol scheme in this work, we assume a maximum-a-posteriori (MAP) decoder,
which is optimal for the performance. Hence, the final round of GN can be
viewed as MAP detection on a constellation $\\{-1,0,1\\}$ with probability
distribution
$\\{\nicefrac{{p_{GN1}}}{{2}},1-p_{GN1},\nicefrac{{p_{GN1}}}{{2}}\\}$ where
$p_{GN1}$ is the probability of error after $D-1$ rounds. The complete
algorithm is described in Alg. 4 in Appendix Section A.

Error analysis.  The probability of error for a rate $\nicefrac{{K}}{{D}}$
Gallager-Nakibŏglu scheme can be computed in two phases. The first phase can
be analyzed as 1 round of uncoded PAM followed by $D-2$ rounds of Elias
scheme, for which it is shown in [4] that the probability of error is given by

$p_{GN1}=2(1-2^{-K})Q\left(\sqrt{\frac{3(1+S-\frac{1}{D-1})^{D-1}}{2^{2K}-1}}\right),$
(10)

where $S$ is the SNR of the forward AWGN channel on a linear scale.

In the final round, the discrete integer difference between the PAM index of
the decoded message $\hat{m}$ and the index of the true message $m$ is
transmitted, where $(\hat{m}-m)\in\\{-1,0,1\\}$ with probability distribution
$\\{\nicefrac{{p_{GN1}}}{{2}},1-p_{GN1},\nicefrac{{p_{GN1}}}{{2}}\\}$ based on
the assumption from [4]. The probability of error in decoding using a MAP
decoder can thus be computed as

$p_{GN}=2(1-p_{GN1})Q\left(\gamma\sqrt{S}\right)+p_{GN1}Q\left(\left(\frac{1}{p_{GN1}}-\gamma\right)\sqrt{S}\right),$
(11)

where $S$ is the SNR of the forward AWGN channel in linear scale and $\gamma$
is the detection threshold given by

$\gamma=\frac{1}{2\sqrt{p_{GN1}}}+\frac{\sqrt{p_{GN1}}}{S}\log\left(\frac{2(1-p_{GN1})}{p_{GN1}}\right),$
(12)

where $S$ is the SNR of the forward AWGN channel in linear scale.

##  IV Proposed analytical coding scheme: PowerBlast

In this section, we propose PowerBlast, a hybrid 2-phase scheme, to
iteratively refine the LMMSE estimate of the PAM symbol at the receiver and
shift to a discrete symbol scheme that takes advantage of the sparsity of the
error in the estimate of the PAM index in the final round. Through theoretical
analysis and empirical results, we demonstrate that the performance of
PowerBlast is better than both SK and GN, in several regimes of interest.

For a rate $\nicefrac{{K}}{{D}}$ code, PowerBlast begins by mapping $K$ bits
of information to a $2^{K}$ PAM constellation and transmits the symbols on the
forward AWGN channel. The receiver performs an LMMSE estimate and sends the
estimate as feedback to the transmitter through the noiseless feedback
channel. This continues for the first $D-1$ rounds. In the final round,
PowerBlast uses a discrete symbol strategy to send the error in the PAM index
of the estimate. The complete algorithm is described in Alg. 2.

Input: Message symbol $\Theta$, number of rounds $D$, noise variance
$\sigma_{ff}^{2}$

Round 1: Tx: Power normalization: $x_{1}=\sqrt{P}\Theta$;

Forward channel: $y_{1}=x_{1}+n_{1}$;

Rx: LMMSE estimate of transmit symbol
$\hat{\Theta}_{1}=\frac{y_{1}}{1+\sigma_{ff}^{2}/P}$ ;

/* Tx sends the error in estimate $\hat{\Theta}_{1}-\Theta$ over the next
$D-2$ rounds */

while  _$2\leq i\leq D-1$_ do

Tx: Compute the error in estimate of previous round
${\epsilon}_{i-1}=\hat{\Theta}_{i-1}-\Theta$ ;

Power normalization: $x_{i}=\frac{\sqrt{P}}{\sigma_{i-1}}{\epsilon}_{i-1}$,
where $\sigma^{2}_{i-1}=\mathbb{E}[\epsilon^{2}_{i-1}]$;

Forward channel: $y_{i}=x_{i}+n_{i}$;

Rx: LMMSE estimate of transmit symbol
$\hat{\epsilon}_{i}=\frac{\sqrt{P}\sigma_{i-1}}{P+\sigma_{ff}^{2}}y_{i}$ ;

Update the estimate of $\Theta$ as:
$\hat{\Theta}_{i}=\hat{\Theta}_{i-1}-\hat{\epsilon}_{i-1}$

end while

Decoding after round $D-1$: Map
$\hat{\Theta}_{D-1}=\frac{\hat{x_{1}}}{\sqrt{P_{1}}\eta}$ to the closest
symbol in the $2^{K}$ PAM constellation.

/* High-SNR scheme for final round */

Round D: Tx: Compute the difference in PAM indices $U=\hat{m}-m$;

Power normalization: $x_{D}=\frac{\sqrt{P}U}{\sigma_{ff}}$; where
$U=\hat{m}-m$;

Transmit: $y_{D}=x_{D}+n_{D}$;

Final Decoding: Use ML decoder to detect $\hat{U}$ and detect original PAM
signal using $\hat{U}$.

Algorithm 2 PowerBlast coding scheme

We note that PowerBlast can be seen as a combination of SK and GN. The key
difference between the SK scheme and the initial rounds of the GN scheme lies
in the choice of information transmitted in the second round. In SK, the
difference between $\Theta$ and the LMMSE estimate $\hat{\Theta}$ is
transmitted, whereas in GN, the noise in the channel realization from round 1,
$\mathbf{n_{1}}$, is transmitted. Hence, PowerBlast can be interpreted as
$D-1$ rounds of SK scheme followed by GN scheme for the final round, resulting
in a non-negligible improvement in performance. The error analysis for the
PowerBlast scheme is provided below.

######  Theorem 1 (Error Analysis).

The probability of error for rate $\nicefrac{{K}}{{D}}$ PowerBlast scheme is
given by

$p_{PB}=2(1-p_{SK})Q\left(\gamma\sqrt{S}\right)+p_{SK}Q\left(\left(\frac{1}{p_{SK}}-\gamma\right)\sqrt{S}\right)$
(13)

where

$p_{SK}=2(1-2^{-K})Q\left(\sqrt{\frac{3S(1+S)^{D-2}}{2^{2K}-1}}\right)$ (14)

and $\gamma$ denotes the detection threshold and $S$ denotes the SNR of the
forward AWGN channel on a linear scale, and $Q()$ is the standard Q-function.

###### Proof.

We begin by computing the probability of error for phase 1, which consists of
$D-1$ rounds of SK scheme. For ease of analysis, we follow the same assumption
of [5], where the estimator after round one is assumed to be the minimum
variance unbiased estimator instead of LMMSE implying,

$\hat{\Theta}_{1}=\frac{y_{1}}{\sqrt{P}}.$ (15)

Further, based on Alg. 1, we can view the effective channel corresponding to
rounds 1 to $D-1$ as one round with effective SNR $S.1+S)^{D-2}$, where
$S=\nicefrac{{P}}{{\sigma_{ff}^{2}}}$ is SNR for the forward channel in linear
scale. Further, it is straightforward to show that the probability of error
incurred in transmitting a symbol from unit power normalized $2^{K}$ PAM,
using an optimal detector, is

$\displaystyle p_{e}$
$\displaystyle=2(1-2^{-K})Q\left(\frac{\sqrt{P}\eta}{\sigma}\right)$
$\displaystyle=2(1-2^{-K})Q\left(\sqrt{\frac{3S}{2^{2K}-1}}\right),$

where $S$ is the forward channel SNR in linear scale and $Q()$ is the
Q-function. Hence, the effective probability of error after $D-1$ rounds of SK
is

$p_{SK}=2(1-2^{-K})Q\left(\sqrt{\frac{3S(1+S)^{D-2}}{2^{2K}-1}}\right).$ (16)

We now proceed to compute the probability of error for phase two, which is
transmitting the difference in integer index over an AWGN channel. The key
assumption for the high-SNR region is that the noise variance is small enough
(compared to the effective forward SNR) that any errors beyond decoding to
adjacent PAM symbols i.e., errors beyond 1 index difference are essentially
negligible [4]. Hence, the problem can be rephrased as finding the probability
of error to communicate a symbol from the constellation $\\{-1,0,1\\}$, with
probability distribution
$\\{\nicefrac{{p_{SK}}}{{2}},1-p_{SK},\nicefrac{{p_{SK}}}{{2}}\\}$ using MAP
decoder. This can be computed as

$p_{PB}=2(1-p_{SK})Q\left(\gamma\sqrt{S}\right)+p_{SK}Q\left(\left(\frac{1}{p_{SK}}-\gamma\right)\sqrt{S}\right),$
(17)

∎

where $\gamma$ is the detection threshold and $S$ is the SNR of the forward
AWGN channel on a linear scale.

PowerBlast vs. SK and GN schemes. The performance comparison between the error
expressions for a general SNR and rate is not straightforward (as they are
represented in terms of Q functions). Instead, we limit our comparison to the
canonical settings considered in the recent literature on channels with
feedback (e.g., [26]), for rates $\nicefrac{{3}}{{6}}$ and
$\nicefrac{{3}}{{9}}$ and plot the BLER performance in Fig. 2; we observe
significant gains of PowerBlast compared to both SK and GN in terms of the
BLER performance.

Figure 2:  By combining SK and discrete-symbol strategy of GN scheme,
PowerBlast noticeably improves the BLER performance upon both SK and GN
schemes. A coding rate of $R=\nicefrac{{3}}{{6}}$ and $R=\nicefrac{{3}}{{9}}$
are considered in (a) and (b) respectively, for a blocklength of $K=3$.

From the results so far, we have demonstrated that PowerBlast provides the
best performance among the analytically tractable solutions for channels with
noiseless feedback that are recently considered in the literature (to
demonstrate the reliability of deep-learning-based coding schemes [26]). In
the coming sections, we consider deep-learning-based coding schemes such as
GBAF and show that, surprisingly, the analytical PowerBlast coding scheme
often delivers competing performance. Nevertheless, PowerBlast still falls
short when the SNR is very low; thus, we investigate conditions under which
deep-learning-based schemes provide the highest gain. Finally, we propose
LightCode, a lightweight neural coding scheme that achieves state-of-the-art
performance with very low complexity.

##  V Deep Learning based Coding schemes

In this section, we review the current state of the learning-based codes for
channels with feedback, closely analyze the generalized block attention
feedback (GBAF) [26], discuss the shortcomings, and provide motivation for a
new learning-based coding scheme we introduce in the next section.

###  V-A RNN families

deep-learning-based algorithms are capable of modeling complex input-output
relations. One of the key challenges in designing codes for channels with
feedback is accurately computing the next transmission that minimizes the
probability of decoding error, conditioned on the feedback from previous
rounds. Existing classical coding schemes employ linear estimators at the
receivers and model the dependencies in a linear fashion at the transmitter,
which is sub-optimal. Instead, the sequential nature of the feedback from
previous can be better leveraged by using deep-learning architectures such as
RNNs that are tailored for processing sequential data. Deepcode [21]
demonstrated this advantage in modeling the dependencies across bits and
rounds using bi-directional gated recurrent units (GRUs). Several follow-up
works investigated the use of other RNN-based architectures such as long
short-term memory (LSTMs) [22, 23] and, more recently, Robust Coding [24] was
proposed, which combined attention mechanism with bi-directional GRUs to
optimize the symbol-by-symbol code design for noisy feedback channels across
the rounds. Despite the promising performance, one disadvantage of using RNN-
based architecture is the necessity to store the hidden states of the model,
which are typically of much higher dimensions than the inputs and demand a lot
of memory. Additionally, the sequential and iterative nature of encoding and
decoding in RNNs can result in significant latency in the system.

Figure 3:  (Left)Architecture for GBAF: The positional encoding and
transformer encoding modules are used for block coding to encourage the mixing
of symbols across the positions. (Right): Using a symbol-by-symbol scheme,
LightCode significantly reduces the complexity of encoding and achieves more
than $10$x reduction in the number of parameters. On the left, we see the
architecture for GBAF [26], and on the right, we see the architecture for
LightCode (ours).

###  V-B Transformer families

In another line of work, self-attention-based transformer architectures have
been explored for designing neural feedback codes. AttentionCode [25]
introduced the idea of replacing RNN-based architecture with pure attention-
based models, resulting in better alignment between a symbol and the
corresponding feedback. More recently, GBAF [26] introduced the idea of
performing block coding across the codeblock, in addition to coding across the
rounds, resulting in orders of magnitude improvement in performance at
extremely low SNRs.

As illustrated in Fig. 3, the encoding of GBAF is performed across the rounds
by causally concatenating the message and feedback symbols from previous
rounds and using a series of feature extractors and multi-layer perception
(MLP) modules. Additionally, positional encoding (PE) and a self-attention-
based transformer encoder layer are deployed to encourage the mixing of
information across the symbols within a codeblock, leading to block coding.
More specifically, a block of $L$ bits is divided into $l$ sub-blocks of $K$
bits each and first encoded independently using a feature extractor. Next, PE
and transformer encoder modules perform cross-symbol coding across the $l$
symbols. Finally, an MLP module is used to encode each symbol. A similar
architecture is used at the decoder.

For concreteness, in [26], GBAF considers a block size $K=3$ and codeblock
length of $L=51$ and performs a block coding across $l=17$ symbols. This is
the present state-of-the-art in performance, achieving a BLER of $7\times
10^{-10}$ at SNR $-1.0$ dB for rate $\nicefrac{{3}}{{9}}$.

###  V-C Important open problems

While providing impressive performance in reliability, transformer
architecture is computationally expensive. Further, it is well known that the
transformer architecture does not scale well to larger sequences. The self-
attention mechanism imposes a compute complexity of $O(n^{2})$ during training
and $O(n)$ during inference, even after using KV cache, with respect to input
length $n$. Moreover, as the blocklength scales, a memory complexity of
$O(n^{2})$ prohibits training the algorithm from using a large batch size,
which is crucial for attaining a good performance for any deep-learning
models.

An alternative approach would be to design a symbol-by-symbol coding scheme
that can be scaled to any blocklength $L$ by encoding $K$ bits at a time
independently. Within this context, an important question is: what is the
extent of performance degradation compared to neural block coding schemes like
GBAF? Moreover, it is necessary to formulate a novel and simplified
architecture tailored to symbol-by-symbol processing. Finally, through
streamlining the architecture and confining to symbol-by-symbol schemes, can
we analyze and interpret the codes learned by deep learning models, discerning
the reasons behind their notably superior performance compared to analytical
counterparts?

In the coming sections, we answer all these questions. In Section VI, we
systematically present LightCode, a lightweight symbol-by-symbol neural coding
scheme, which achieves state-of-the-art BLER performance (Section VII).
Further, we analyze GBAF to study the efficacy of block coding by performing a
systematic ablation study and also analyze LightCode to identify the crucial
components necessary for achieving ultra-low BLER in Section VIII.

##  VI Proposed neural coding scheme: LightCode

Our goal is to design lightweight neural codes without the need for block
coding i.e., we limit ourselves to symbol-by-symbol schemes suitable for both
noiseless and noisy feedback settings. To this end, we design a lightweight
deep-learning-based scheme with $10\times$ fewer parameters compared to
existing schemes. Surprisingly, this low-complex solution achieves a
performance superior to current state-of-the-art deep-learning-based block-
coding schemes. We now present the architecture and training choices that were
crucial in achieving this new state-of-the-art BLER performance.

###  VI-A LightCode: Architecture

Our architecture. We split the design of the encoder-decoder architecture into
two parts: the feature extractor and the MLP module, as illustrated in Fig. 3
(right). The choice of the feature extractor plays a crucial role in
determining the downstream performance. A higher complexity feature extractor
can provide a better performance but might not be desirable for practical
applications. For LightCode, after experimenting with various choices, we
found that the design illustrated in Fig. 4 gave the optimal trade-off in
complexity vs BLER performance. Specifically, we consider a hidden dimension
of $d=32$ and 3 hidden layers. At the final layer, we use a skip connection to
aggregate the outputs of layer 1 and layer 3 to project down to a latent
dimension of $d=16$. The motivation for this architecture comes from our prior
work on the design of deep-learning-based non-linear polar codes, DeepPolar
[18], where skip connections have proven to be crucial to ensure good
performance. Finally, the output of the feature extractor will be passed to an
MLP module to perform encoding or decoding. For the encoder, we choose a $2$
layer MLP, and for the decoder, we choose a $1$ layer MLP to transform the
features into encoded representations or decoder outputs, respectively.

Our architecture vs. RNNs. A popular choice for modeling the cross-round
relation in feedback coding has been the RNN family of architectures. The
sequential nature of the data makes it suitable for RNNs, GRUs, LSTMs, and
other similar architectures. While these architectures provided impressive
performance, a significant drawback is the necessity to store the hidden
states from previous encoding steps, which is a high-dimensional latent that
requires a lot of memory. Hence, a feed-forward architecture, which does not
need a hidden state, is a better choice for resource-constrained scenarios.

Our architecture vs. transformers. While transformer models overcome the issue
of storing hidden states by using positional encoding and self-attention, they
also come with great computational complexity. Moreover, recent transformer-
based feedback coding schemes proposed block coding using self-attention,
which requires $O(n^{2})$ memory with respect to codeblock length. Hence, a
feed-forward architecture to perform a symbol-by-symbol coding scheme is more
practical.

By carefully designing the architecture suitable for a symbol-by-symbol
scheme, we achieve a lightweight design with more than $10\times$ reduction in
the number of parameters compared to the RNN family of schemes such as Robust
Coding and transformer family of block coding schemes such as GBAF. Further,
we show in Section VIII-C that this reduction in complexity provides up to
$171\times$ higher decoding throughput compared to Robust Coding and up to
$10\times$ higher throughput compared to GBAF. We now describe the encoding
and decoding procedure in detail below.

Encoding. At round $i$, the encoder takes as input the original message and
any available feedback from previous $i-1$ rounds to compute $x_{i}$. Further,
after every round, a power reallocation is done across the rounds, similar to
Deepcode [21]. This is also based on the theoretical justification that
allocating more power to the initial rounds results in an optimal performance
[4]. The resulting encoding process at round $i$ is

$x_{i}=\alpha_{i}\phi(\mathbf{u},\tilde{y}_{1},\tilde{y}_{2},\dots,\tilde{y}_{i-1}),$
(18)

where $\tilde{y}_{i}=x_{i}+n_{i}+\tilde{n}_{i}$ is the feedback from the
previous round and $\alpha_{i}$ is the scaling factor to ensure sum power
constraint across the rounds $\sum_{i=1}^{D}\alpha_{i}^{2}=D$.

Decoding. At the end of $D$ rounds of transmission, the decoder uses all the
received symbols and estimates the original message $\hat{\mathbf{u}}$ as

$\hat{\mathbf{u}}=\psi(y_{1},y_{2},\dots,y_{D}),$ (19)

where $y_{i}=x_{i}+n_{i}$ is the noisy received symbol in round $i$.

Here, $\phi$ and $\psi$ are the encoder and decoder neural networks based on
the architecture in Fig. 3. Using this simple symbol-by-symbol scheme,
LightCode archives a performance similar to that of GBAF. It turns out that
the computationally intensive self-attention mechanism and the cross-symbol
coding are not adding considerable value to GBAF, as will be evidenced by our
ablation studies in Section VIII-A.

Choice of feature vectors. Empirically, we observed that the choice of input
to the feature extractor has a noticeable effect on the BLER performance based
on the feedback channel. For noiseless feedback, the input features
$(\mathbf{u},y_{1},y_{2},\dots,y_{i})$ worked the best. Further, as will be
evident from discussions in Section VII-C, the input features
$(\mathbf{u},x_{1},x_{2},\dots,x_{i},n_{1}+\tilde{n}_{1},n_{2}+\tilde{n}_{2},\dots
n_{i}+\tilde{n}_{i})$ gave the best performance for noisy feedback. Apart from
this difference, the rest of the architecture remains the same for noiseless
and noisy feedback scenarios.

Figure 4: Feature extractor design for LightCode.

Support for multiple rates. One of the limitations of existing symbol-by-
symbol coding schemes is the inability to serve a variety of rates i.e., only
rates of the form $\nicefrac{{1}}{{D}},D\in\mathbb{Z^{+}}$ are supported. In
order to overcome this, instead of processing 1 bit at a time, LightCode
encodes a block of $K$ bits into 1 symbol and communicates this symbol to the
receiver over $D$ rounds. By independently varying $K$ and $D$ any rate of the
form $\nicefrac{{K}}{{D}}$ where, $K,D\in\mathbb{Z^{+}}$ can be supported,
making the scheme more flexible, while simultaneously reducing the overall
latency of the communication by a factor of $K$. The results for multiple
rates for a block length of $K=3$ are discussed in Table II.

###  VI-B Training

Our primary region of interest is rate $\nicefrac{{3}}{{9}}$ code at SNR
$-1.0$ dB, where our target BLER is $\approx 10^{-9}$. This is the current
state-of-the-art performance by GBAF [26]. In order to achieve such an
extremely low error rate, it is important to reliably train and simulate large
amounts of samples. Inspired by the high-SNR variant of Gallager-Nakibŏglu
[4], we hypothesize that in regions of high-SNR or low errors, a significant
benefit to the forward transmission arises from the power reallocation to
symbols with non-zero errors. This can only be realized during the training by
considering a large batch and enforcing the power constraint per batch.
Moreover, it is well understood that deep-learning models generalize better
when trained with large batch sizes. Hence, we consider an extremely large
batch size of $10^{5}$. While this is significantly larger compared to GBAF,
which has a batch size of 8192, it is still a smaller number of symbols,
considering that GBAF contains 17 symbols per codeblock, whereas our scheme
consists of only 1 symbol per codeblock. Further, it is possible for us to
train with such a large batch size because of the small number of parameters
compared to other deep-learning-based coding schemes. For a fair comparison,
we follow a training methodology similar to that of GBAF, as explained below.

Input: Encoder model $\phi$, Decoder model $\psi$, Block length $K$, number of
rounds $D$, noise variance $\sigma_{ff}^{2}$, batch size $B$, number of epochs
$E$, learning rate lr

for  _$i\leq E$_ do

Generate batch of random binary vectors $\mathbf{u}\in\\{0,1\\}^{K\times B}$

for  _$i\leq D$_ do

$x_{i}=\phi\left(\mathbf{u},y_{1},y_{2},\dots,y_{i-1}\right)$ ;

/* Encoding at round $i$ */

$y_{i}=x_{i}+n_{i}$

end for

$\mathbf{p_{u}}=\psi(y_{1},y_{2},\dots,y_{D})$ ;

/* Decoding after $D$ rounds */

Compute the NLL loss
$\frac{1}{B}\sum_{j=1}^{B}L_{\text{NLL}}(\mathbf{c_{j}},\mathbf{p_{u_{j}}})$,
where $\mathbf{c_{j}}$ is the class index corresponding to $j^{\text{th}}$
message vector $\mathbf{u_{j}}$ and $\mathbf{p_{u_{j}}}$ is class probability
vector after the SoftMax layer.

Clip the gradients to $0.5$.

Update model parameters for $\phi$ and $\psi$ using AdamW optimizer with
learning rate lr.

Update the learning rate using LambdaLR.

end for

Algorithm 3 Training LightCode

We use AdamW optimizer, a stochastic optimization method that modifies the
typical implementation of weight decay in Adam by decoupling weight decay from
the gradient update. We initialize the learning to $10^{-3}$ and use a
LambdaLR scheduler with a weight decay of $0.01$. Additionally, we clip all
the gradient values to $0.5$ for numerical stability. Starting with randomly
initialized weights, we train the encoder and decoder models simultaneously on
$1.2\times 10^{5}$ batches. Each input symbol corresponds to $K$ bits,
resulting in a $2^{K}$ category classification problem for the decoder.
Accordingly, we use cross entropy (CE) loss to measure the performance as

$L_{\text{CE}}=\frac{1}{B}\sum_{i=1}^{B}\left(\sum_{j=0}^{2^{K}-1}c_{ij}\log
p_{ij}\right),$

where $B$ is the batch size, $2^{K}$ is the number of classes , $c_{ij}$ is
the true class probability and $p_{ij}$ is the predicted probability of the
$j^{\text{th}}$ class, for the $i^{\text{th}}$ sample.

The hyperparameters used for training the rate $R=\nicefrac{{3}}{{9}}$ code
are listed in Table I.

Hyperparameter | Value
---|---
Encoder training SNR | -1.0 dB
Decoder training SNR | -1.0 dB
Mini batch size ($B$) | 100,000
Total epochs ($E$) | 120
Batches per epoch | 1000
Initial learning rate (lr) | $10^{-3}$
Optimizer | AdamW
TABLE I: Hyperparameters used in model training for rate
$\nicefrac{{3}}{{9}}$, SNR -1.0 LightCode.

Once the training is complete, we compute the mean power and standard
deviation for the encoded data after every round for a large number of
samples, $O(10^{6})$, to reliably estimate the mean and standard deviation
corresponding to encoder outputs to be used during inference. This is crucial
in enforcing the power constraint in expectation. The algorithm for training
LightCode is described in detail in Algorithm 3.

##  VII Main Results

We begin by comparing the performance of LightCode with the current state-of-
the-art in deep-learning-based coding schemes for noiseless passive feedback
setting, GBAF [26] and other schemes, demonstrated in Fig. 5. Next, we discuss
a method for extending LightCode to moderate block-length regimes of up to
$L=300$ and study the performance.

Finally, we look at deep-learning-based coding schemes for noisy feedback
settings and how LightCode can be extended to this setting with minimal
changes. We then proceed to compare the BLER performance for the same
configuration as before but with a feedback SNR of $20$ dB.

###  VII-A LightCode and PowerBlast vs. existing neural codes for noiseless
feedback

In this section, we evaluate the performance of LightCode and PowerBlast and
compare them against several existing analytical and deep-learning feedback
schemes. For concreteness, we consider the canonical setting of rate
$R=\nicefrac{{3}}{{9}}$ with block length $K=3$ and a number of rounds $D=9$
transmitted on AWGN forward channel and noiseless feedback.

Baselines. Our primary comparison is against GBAF [26], which is the current
state-of-the-art for the noiseless passive feedback setting. Additionally, we
consider Deepcode [21], DEFC [22], DRFC [23], Attentioncode [25], and Robust
Coding [21]. Additionally, for completeness as well as to understand the
relative gains of deep-learning-based coding schemes, we also compare the
performance against Schalkwijk-Kailath [2], Gallager-Nakibŏglu [4] and
PowerBlast.

In Fig. 5, we compare the BLER performance of rate $\nicefrac{{3}}{{9}}$
coding schemes. LightCode consistently outperforms the existing deep-learning-
based schemes, including GBAF, while utilizing
$<\nicefrac{{1}}{{10}}^{\text{th}}$ the number of parameters. Moreover, these
results indicate that PowerBlast surpasses the performance of all existing
schemes, including LightCode, when an adequately high signal-to-noise ratio is
provided, which is $-0.5$ dB for rate $\nicefrac{{3}}{{9}}$. However, at lower
SNRs, the performance of deep-learning codes is significantly better.

Figure 5: Performance comparison of LightCode with GBAF and other existing
neural feedback codes for rate $\nicefrac{{3}}{{9}}$ and noiseless feedback.
LightCode achieves superior BLER performance compared to GBAF while utilizing
$<\nicefrac{{1}}{{10}}^{\text{th}}$ the number of parameters.

Multiple rates. The coding rate of LightCode is determined by two factors,
block length $K$ and number of rounds of communication $D$. We now keep the
block length constant and vary $D$ to compare the performance against GBAF
across multiple rates, as shown in Table II. LightCode consistently
outperforms GBAF across a range of rates
$\\{\nicefrac{{3}}{{8}},\nicefrac{{3}}{{7}},\nicefrac{{3}}{{6}},\nicefrac{{3}}{{5}}\\}$,
while utilizing a fraction of the number of parameters.

SNR (dB) | Rate | GBAF | PowerBlast | LightCode
---|---|---|---|---
$-1.0$ | $3/9$ | $7\times 10^{-10}$ | $2.8\times 10^{-4}$ | $\mathbf{4.5\times 10^{-10}}$
$0.0$ | $3/8$ | $6.1\times 10^{-8}$ | $8.8\times 10^{-7}$ | $\mathbf{5.1\times 10^{-9}}$
$1.0$ | $3/7$ | $7.5\times 10^{-8}$ | $1.0\times 10^{-8}$ | $\mathbf{1.0\times 10^{-8}}$
$2.0$ | $3/6$ | $1.5\times 10^{-6}$ | $1.5\times 10^{-8}$ | $\mathbf{8.3\times 10^{-7}}$
$3.0$ | $3/5$ | $8.7\times 10^{-7}$ | $1.9\times 10^{-4}$ | $\mathbf{2.7\times 10^{-7}}$
TABLE II: BLER performance comparison of LightCode with GBAF with PowerBlast
for different rates. LightCode consistently performs better than GBAF while
using $<\nicefrac{{1}}{{10}}^{\text{th}}$ the number of parameters.

###  VII-B Performance in moderate blocklength regime

Using a code designed for $K=3$, it is straightforward to encode larger blocks
of length $L$ by treating the message as a packet of $l=\nicefrac{{L}}{{K}}$
independent symbols. Hence, the packet error rate (PER) for a packet of size
$l$ can be considered BLER for blocklength $L$. This modular approach to
encoding is motivated by two key observations. The first is from the ablation
studies in Section VIII-A, which demonstrate that explicit block coding is not
beneficial in this regime and does not scale well. Second is the increase in
computational complexity of both encoder and decoder modules associated with
block coding. Results from Fig. 6 illustrate that LightCode scales gracefully
with blocklength by independently encoding information in sets of $K=3$,
keeping the complexity constant. Further, we also note that the bit error rate
(BER) remains constant regardless of the blocklength $L$.

Figure 6: deep-learning-based coding schemes scale gracefully with the length,
providing a flexible way to code for any length without an increase in
computational complexity.

###  VII-C Coding for Channels with Noisy Feedback

Finally, we now consider the case of channels with noisy feedback i.e.,
$\sigma_{fb}^{2}>0$. While the assumption of noiseless feedback is easier to
study, most practical feedback channels suffer from noise even when the
receiver sending the feedback operates at high power. Because of this
limitation, both SK and GN schemes do not perform well. To address this, [6]
introduces a linear feedback scheme that is implemented as an inner code to a
concatenated code, which was found to be optimal within the linear family of
codes [28]. More recently, [7] proposed using dynamic programming to improve
the performance, which turns out to be a generalized version of SK. However,
despite these improvements, the linearity of these schemes severely limits the
performance that can be achieved.

Recalling the architecture of deep-learning-based schemes introduced in
Section V, one of the interesting properties of these codes is their
robustness to noise in the feedback channel. By taking advantage of the
general architecture, it is straightforward to inject noise into the feedback
channel during training to make the encoder-decoder robust. Hence, learning-
based schemes can help design a practically realizable class of codes that can
trained purely in a data-driven fashion and can provide gains under both
noiseless and noisy feedback settings.

Figure 7: Performance comparison of LightCode with GBAF and other existing
neural feedback codes for rate $\nicefrac{{3}}{{9}}$ and noisy feedback with
SNR = $20$ dB. LightCode achieves BLER performance comparable to GBAF while
utilizing only $\nicefrac{{1}}{{10}}^{\text{th}}$ the number of parameters.

For training LightCode on noisy feedback channels, empirically, we found that
selecting the input features as the previous encoder outputs $x_{1}$ and the
cumulative noise $n_{i}+\tilde{n}_{i}$ separately works better than directly
passing the feedback $\tilde{y_{i}}$ from previous rounds, which was the
choice for noiseless feedback. Except for the change in input feature, the
rest of the architecture and training details remain the same as the noiseless
feedback setting. In Fig. 7, we compare the performance of rate
$\nicefrac{{3}}{{9}}$ LightCode against existing schemes, for a feedback SNR
of $20$ dB. We see that LightCode exhibits similar performance compared to
GBAF while still utilizing only a fraction of the number of parameters.

##  VIII Analysis

As evident from results in Section VII, symbol-by-symbol neural schemes can
achieve performance comparable to block coding schemes such as GBAF.
Consequently, we examine GBAF and LightCode in greater detail to
systematically analyze their architecture, training process, and performance,
with the aim of identifying the key factors contributing to their performance.
Furthermore, we compare the memory and computational complexity of LightCode
with existing schemes to quantify the gains. Finally, we provide an
interpretation of the encoded representations of LightCode.

###  VIII-A Ablation studies on GBAF

We investigate the contribution of the self-attention mechanism to the
performance of GBAF by performing an ablation study to understand the compute
vs performance trade-off better. First, the self-attention mechanism is
disabled, and it is observed that this has no significant effect on the BLER
performance. Next, both self-attention and positional encoding blocks are
disabled, and the performance remains approximately the same. These results,
provided in Table III, demonstrate a surprising observation that the self-
attention and PE modules, which are responsible for cross-symbol block coding,
contribute only marginally to the performance of GBAF. This brings into
question the value of block encoding and motivates us to find simpler designs
that perform only symbol-by-symbol coding while providing competing
performance.

SNR (dB) | GBAF | GBAF (no attn) | GBAF (no attn, no PE)
---|---|---|---
-1.5 | 9.8e-5 | 7.5e-5 | 2.6e-5
-1.0 | 2.6e-9 | 1.2e-9 | 2.7e-9
0.0 | 5.4e-10 | 6.6e-10 | 8.1e-10
TABLE III: Effect of block coding on performance of GBAF for rate
$\nicefrac{{3}}{{9}}$ code with noiseless feedback. Positional encoding and
self-attention modules have no noticeable effect on BLER performance.

###  VIII-B Scaling laws: batch size

In section Section VII, we have demonstrated that it is possible to achieve
performance comparable to GBAF with fewer parameters and lower compute
complexity. We also hypothesized in Section VI-B that it is important to train
the encoder using a very large batch size in order to accurately capture the
statistics of the distribution so that the available power can be optimally
allocated to the symbols with error at the receiver while reducing the power
to the remaining symbols in a batch. In deep learning literature, it is well
known that increasing the batch size can noticeably improve the performance of
the neural network model [29]. To better understand the effect of batch size
on LightCode, we perform a systematic study by training LightCode with
different batch sizes. As noted in Fig. 8, at a batch size of $1.5\times
10^{3}$, LightCode has a BLER of $3.7\times 10^{-9}$, similar to that of GBAF
(w/o BU). However, the performance drops to $4\times 10^{-10}$, exceeding
GBAF, when the batch size increases to $5\times 10^{4}$ and beyond.

######  Remark 1.

Using a lightweight network with a small number of parameters makes it
feasible to train with very large batch size, resulting in a significantly
lower error rate.

Figure 8: BLER performance of LightCode with with respect to training batch
size for rate $\nicefrac{{3}}{{9}}$ code at SNR -1.0 dB with noiseless
feedback. Performance improves significantly with respect to batch size,
surpassing the performance of GBAF at batch size $>5\times 10^{4}$.

###  VIII-C Complexity and Throughput

Scheme | # parameters
---|---
GBAF [26] | $9.1\times 10^{4}$
Robust Coding [24] | $8.6\times 10^{4}$
LightCode (ours) | $\mathbf{7.3\times 10^{3}}$
TABLE IV: Rate $\nicefrac{{3}}{{9}}$ LightCode requires
$<\nicefrac{{1}}{{10}}^{\text{th}}$ the number of parameters compared to GBAF
and Robust Coding.

We compare the total number of parameters in the encoder and decoder for a
rate $\nicefrac{{3}}{{9}}$ LightCode against GBAF and Robust Coding within
Table IV. LightCode reduces the parameter count by more than $10\times$
compared to GBAF and Robust Coding, resulting in significant savings in
memory.

Scheme | Enc (CPU) | Dec (CPU) | Enc (GPU) | Dec (GPU)
---|---|---|---|---
GBAF [26] | $1.6\times 10^{5}$ | $1.7\times 10^{6}$ | $4.4\times 10^{8}$ | $3.3\times 10^{9}$
Robust Coding [24] | $2.0\times 10^{5}$ | $7.6\times 10^{4}$ | $4.1\times 10^{8}$ | $2.4\times 10^{9}$
LightCode (ours) | $\mathbf{1.5\times 10^{6}}$ | $\mathbf{1.3\times 10^{7}}$ | $\mathbf{2.9\times 10^{9}}$ | $\mathbf{3.1\times 10^{10}}$
TABLE V: Throughput (symbols/sec) comparison. Rate $\nicefrac{{3}}{{9}}$
LightCode achieves up to $10\times$ higher decoding throughput compared to
GBAF and up to $171\times$ higher decoding throughput in CPU mode compared to
Robust Coding.

To compare the throughputs, we measure the inference time in CPU mode on a AMD
Ryzen Threadripper PRO 5975WX 32-Cores processor as well as in GPU mode using
an NVIDIA GeForce RTX 4090, using a batch size of $10^{5}-10^{7}$. LightCode
provides up to $10\times$ higher throughput compared to GBAF. Further,
LightCode provides up to $171\times$ higher decoding throughput in CPU mode
and up to $10\times$ higher throughput in GPU mode compared to Robust Coding,
providing significant gains in latency, as shown in Table V.

###  VIII-D Interpretation of LightCode

By limiting to a symbol-by-symbol strategy, LightCode allows for better
interpretability and analysis of the learned encoder representations compared
to block coding schemes such as GBAF. By analyzing the power allocation and
relation between encoder output and feedback from previous rounds, we draw
connections between LightCode and PowerBlast.

####  VIII-D1 Power distribution

A key contributing factor to the superior performance of PowerBlast compared
to SK is the discrete-symbol scheme in the final round of communication. In
the high-SNR regime, we are only interested in the error in the PAM index of
the decoded symbol with respect to the original symbol; this will result in a
sparse distribution where most of the samples are 0. Thus, a majority of the
available power is naturally allocated to the symbol locations with non-zero
error. Surprisingly, we find similar behavior for LightCode towards the final
rounds of communication where the error is sparse.

To test this hypothesis, we choose a moderately sparse error regime for ease
of analysis. In Fig. 9, we plot the power distribution of the encoder output
in round $7$ for rate $\nicefrac{{3}}{{9}}$ code at SNR $-1.0$ dB and
noiseless feedback by randomly sampling $50$ symbols. On the X-axis, we see
the sample number, and on the Y-axis, we plot the magnitude of error in the
integer PAM index of the estimate after round $6$ and compare it against the
magnitude of encoder output in round $7$. Note that a difference in index of 1
corresponds to a magnitude of 2 in the un-normalized PAM from Eqn. 6. It is
evident from Fig. 9 that the highest power is allocated to the symbols with
error in the estimate.

Figure 9: LightCode allocates more power to the symbols with error in estimate
from the previous round, improving the overall probability of decoding.

####  VIII-D2 Interpreting the encoder

In Fig. 10, we plot the output of the encoder in round 2 with respect to the
feedback in round 1. The encoder is approximately transmitting a linearly
scaled version of noise from round 1. Interestingly, for the symbols on the
boundary of the constellation embeddings, the encoder does not need to
transmit any data when the noise favors the ground truth, unlike in the SK
scheme, where all noise needs to be corrected. For instance, consider
$\Theta_{1}$, mapped to the symbol at the boundary on the left, where a
negative noise in round 1 is favored. And $\Theta_{8}$ is mapped to the symbol
at the boundary on the right, where a positive noise in round 1 is favored. In
such scenarios, the transmit power saved here can be reallocated to other
symbols, improving the overall decoding error, which is only possible because
of the non-linear activation functions.

Figure 10: Output of encoder in round 2 ($x_{2}$) shows that the encoder is
directly compensating for the noise experienced in round 1 ($n_{1}$).  Figure
11: Comparison of linear approximation (Y-axis) vs true encoder output
(X-axis). As the rounds progress, the linearity of the relation between
encoder output and the feedback from previous rounds breaks, making it harder
for analytical schemes to perform well.

Visualization of the encoder output beyond round 2 is difficult as the number
of inputs to the encoder increases linearly with the number of rounds of
communication. Alternatively, we test the linear dependency between $x_{i+1}$
and $\\{x_{1},\dots,x_{i},n_{1},\dots,n_{i}\\}$ by performing a linear
regression as

$\hat{x}_{i+1}=\sum_{j=1}^{i}\alpha_{j}x_{j}+\beta_{j}n_{j}+c,$ (20)

where $\alpha_{j}$ and $\beta_{j}$ are regression coefficients and $c$ is the
intercept found using LinearRegression toolbox in sklearn over $10^{6}$
samples. To make the analysis tractable, we build one linear regression model
for each symbol in the PAM constellation. In Fig. 11, we plot the predicted
output using linear regression vs. true encoder output when the input PAM
symbol index is $\Theta=1$, for rounds $j=2$ to $9$. It is evident that as the
round number increases, the relation becomes highly non-linear, and hence,
classical schemes such as SK and GN as other linear schemes [6, 7] fail to
model these dependencies.

##  IX Conclusion

In this work, we address the problem of designing lightweight coding schemes
for channels with feedback. First, we propose an analytical scheme,
PowerBlast, that can be viewed as a combination of Schalkwijk-Kailath and
Gallager-Nakibŏglu schemes. Using a hybrid strategy and taking advantage of
the discrete nature of the signal in the final round, PowerBlast noticeably
outperforms both SK and GN, providing a performance that competes with current
deep learning schemes in regions of high-SNR.

Next, we propose a lightweight deep-learning-based scheme, LightCode, that can
achieve state-of-the-art BLER performance while using less than
$\nicefrac{{1}}{{10}}^{\text{th}}$ the parameters and compute complexity
compared to existing deep-learning-based schemes. By limiting to a symbol-by-
symbol strategy and carefully designing the feature extractor using skip
connections, combined with a training strategy that uses a very large batch
size of $10^{5}$, LightCode achieves a BLER up to $\sim 10^{-10}$.

Additionally, with the help of systematic ablation studies, we have
demonstrated that the self-attention module in the transformer-based GBAF code
has very little effect on the BLER performance, demonstrating the limited
benefit of block coding in this regime.

Further, we interpret the LightCode to show that power distribution in the
sparse error regime of LightCode is similar to that of PowerBlast, where a
majority of the available power is allocated to the symbols with error.
Finally, we also perform a linear regression on the encoder outputs and the
feedback from previous rounds. Our findings show that although the early
stages of communication exhibit a linear relationship, it becomes non-linear
towards the later stages, underscoring the importance of employing deep-
learning-based non-linear coding techniques to attain optimal performance in
regions of extremely low SNR.

## References

  * [1] C. Shannon, “The zero error capacity of a noisy channel,” _IRE Transactions on Information Theory_ , vol. 2, no. 3, pp. 8–19, 1956.
  * [2] J. Schalkwijk and T. Kailath, “A coding scheme for additive noise channels with feedback–i: No bandwidth constraint,” _IEEE Transactions on Information Theory_ , vol. 12, no. 2, pp. 172–182, 1966.
  * [3] J. Schalkwijk, “A coding scheme for additive noise channels with feedback–ii: Band-limited signals,” _IEEE Transactions on Information Theory_ , vol. 12, no. 2, pp. 183–189, 1966.
  * [4] R. G. Gallager and B. Nakiboğlu, “Variations on a theme by schalkwijk and kailath,” _IEEE Transactions on Information Theory_ , vol. 56, no. 1, pp. 6–17, 2009.
  * [5] A. Ben-Yishai and O. Shayevitz, “Interactive schemes for the awgn channel with noisy feedback,” _IEEE Transactions on Information Theory_ , vol. 63, no. 4, pp. 2409–2427, 2017.
  * [6] Z. Chance and D. J. Love, “Concatenated coding for the awgn channel with noisy feedback,” _IEEE Transactions on Information Theory_ , vol. 57, no. 10, pp. 6633–6649, 2011.
  * [7] R. Mishra, D. Vasal, and H. Kim, “Linear coding for awgn channels with noisy output feedback via dynamic programming,” _IEEE Transactions on Information Theory_ , 2023.
  * [8] J. M. Ooi and G. W. Wornell, “Fast iterative coding techniques for feedback channels,” _IEEE Transactions on Information Theory_ , vol. 44, no. 7, pp. 2960–2976, 1998.
  * [9] A. G. Perotti, B. M. Popovic, and A. R. Safavi, “Accumulative iterative codes based on feedback,” _arXiv preprint arXiv:2106.07415_ , 2021.
  * [10] S. K. Ankireddy, S. A. Hebbar, Y. Jiang, P. Viswanath, and H. Kim, “Compressed error harq: Feedback communication on noise-asymmetric channels,” in _2023 IEEE International Symposium on Information Theory (ISIT)_. IEEE, 2023, pp. 1160–1165.
  * [11] J. Griffin, P. Yuan, P. Popovski, K. R. Duffy, and M. Médard, “Code at the receiver, decode at the sender: Grand with feedback,” in _2023 IEEE Information Theory Workshop (ITW)_. IEEE, 2023, pp. 341–346.
  * [12] M. Raghu, B. Poole, J. Kleinberg, S. Ganguli, and J. Sohl-Dickstein, “On the expressive power of deep neural networks,” in _international conference on machine learning_. PMLR, 2017, pp. 2847–2854.
  * [13] E. Nachmani, Y. Be’ery, and D. Burshtein, “Learning to decode linear codes using deep learning,” in _2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton)_. IEEE, 2016, pp. 341–346.
  * [14] E. Nachmani, E. Marciano, L. Lugosch, W. J. Gross, D. Burshtein, and Y. Be’ery, “Deep learning methods for improved decoding of linear codes,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 12, no. 1, pp. 119–131, 2018.
  * [15] S. K. Ankireddy and H. Kim, “Interpreting neural min-sum decoders,” in _ICC 2023-IEEE International Conference on Communications_. IEEE, 2023, pp. 6645–6651.
  * [16] S. A. Hebbar, R. K. Mishra, S. K. Ankireddy, A. V. Makkuva, H. Kim, and P. Viswanath, “Tinyturbo: Efficient turbo decoders on edge,” in _2022 IEEE International Symposium on Information Theory (ISIT)_. IEEE, 2022, pp. 2797–2802.
  * [17] S. A. Hebbar, V. V. Nadkarni, A. V. Makkuva, S. Bhat, S. Oh, and P. Viswanath, “Crisp: Curriculum based sequential neural decoders for polar code family,” in _International Conference on Machine Learning_. PMLR, 2023, pp. 12 823–12 845.
  * [18] S. A. Hebbar, S. K. Ankireddy, H. Kim, S. Oh, and P. Viswanath, “Deeppolar: Inventing nonlinear large-kernel polar codes via deep learning,” _arXiv preprint arXiv:2402.08864_ , 2024.
  * [19] Y. Li, Z. Chen, G. Liu, Y.-C. Wu, and K.-K. Wong, “Learning to construct nested polar codes: An attention-based set-to-element model,” _IEEE Communications Letters_ , vol. 25, no. 12, pp. 3898–3902, 2021.
  * [20] S. K. Ankireddy, S. A. Hebbar, H. Wan, J. Cho, and C. Zhang, “Nested construction of polar codes via transformers,” _arXiv preprint arXiv:2401.17188_ , 2024.
  * [21] H. Kim, Y. Jiang, S. Kannan, S. Oh, and P. Viswanath, “Deepcode: Feedback codes via deep learning,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [22] A. R. Safavi, A. G. Perotti, B. M. Popovic, M. B. Mashhadi, and D. Gunduz, “Deep extended feedback codes,” _arXiv preprint arXiv:2105.01365_ , 2021.
  * [23] M. B. Mashhadi, D. Gunduz, A. Perotti, and B. Popovic, “Drf codes: Deep snr-robust feedback codes,” _arXiv preprint arXiv:2112.11789_ , 2021.
  * [24] J. Kim, T. Kim, D. Love, and C. Brinton, “Robust non-linear feedback coding via power-constrained deep learning,” _arXiv preprint arXiv:2304.13178_ , 2023.
  * [25] Y. Shao, E. Ozfatura, A. Perotti, B. Popovic, and D. Gündüz, “Attentioncode: Ultra-reliable feedback codes for short-packet communications,” _IEEE Transactions on Communications_ , 2023.
  * [26] E. Ozfatura, Y. Shao, A. G. Perotti, B. M. Popović, and D. Gündüz, “All you need is feedback: Communication with block attention feedback codes,” _IEEE Journal on Selected Areas in Information Theory_ , vol. 3, no. 3, pp. 587–602, 2022.
  * [27] P. Elias, “Channel capacity without coding,” in _Lectures on Communication System Theory_ , E. Baghdady, Ed. New York: McGraw Hill, 1961, quarterly progress report, MIT Research Laboratory of Electronics, Oct 15 1956.
  * [28] M. Agrawal, D. J. Love, and V. Balakrishnan, “An iteratively optimized linear coding scheme for correlated gaussian channels with noisy feedback,” in _2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton)_. IEEE, 2011, pp. 1012–1018.
  * [29] Y. Bahri, E. Dyer, J. Kaplan, J. Lee, and U. Sharma, “Explaining neural scaling laws,” _arXiv preprint arXiv:2102.06701_ , 2021.

##  Appendix A Algorithm for Gallager-Nakibŏglu

Input: Message symbol $\Theta$, number of rounds $D$, noise variance
$\sigma_{ff}^{2}$

Round 1: Tx: Power normalization: $X_{1}=\sqrt{P_{1}}\Theta$;

Transmit: $Y_{1}=X_{1}+N_{1}$;

Rx: Send $Y_{1}$ as feedback to Tx ;

/* Tx sends the noise $U_{2}=N_{1}$ in round 2 */

Round 2: Tx: Power normalization:
$X_{2}=\frac{\sqrt{P_{2}}U_{2}}{\sigma_{2}}$; where $U_{2}=N_{1}$ and
$\sigma_{2}=\sigma_{ff}$;

Transmit: $Y_{2}=X_{2}+N_{2}$;

Rx: Compute the LMMSE estimate of transmit symbol
$\mathbb{E}[U_{2}|Y_{2}]=\frac{\sigma_{2}\sqrt{P_{2}}Y_{2}}{1+P_{2}}$;

Update the estimate as $\hat{X_{1}}=X_{1}-\mathbb{E}[U_{2}|Y_{2}]$

/* Tx sends the error in estimate $\epsilon_{2}=\mathbb{E}[U_{2}|Y_{2}]-U_{2}$
over the next $D-2$ rounds */

while  _$3\leq i\leq D-1$_ do

Tx: Compute the error in estimate of previous round
$U_{i}=\mathbb{E}[U_{i-1}|Y_{i-1}]-U_{i-1}$ ;

Power normalization: $X_{i}=\frac{\sqrt{P_{2}}U_{i}}{\sigma_{i}}$, where
$\sigma^{2}_{i}=\frac{\sigma^{2}_{i-1}}{1+S_{i-1}}$;

Transmit: $Y_{i}=X_{i}+N_{i}$;

Rx: Send feedback as LMMSE estimate of transmit symbol
$\mathbb{E}[U_{i}|Y_{i}]=\frac{\sigma_{i}\sqrt{P_{2}}Y_{i}}{1+P_{2}}$ ;

Update the estimate as
$\hat{X_{1}}=X_{1}-\sum_{j=2}^{j=i}\mathbb{E}[U_{j}|Y_{j}]$

end while

Decoding after round $D-1$: Map
$\hat{\Theta}_{D-1}=\frac{\hat{X_{1}}}{\sqrt{P_{1}}\eta}$ to the closest
symbol in the $2^{K}$ PAM constellation.

/* High-SNR scheme for rounds $D$ */

Round D: Tx: Compute the difference in PAM indices $U_{D}=\hat{m}-m$;

Power normalization: $X_{D}=\frac{\sqrt{P_{2}}U_{D}}{\sigma_{D}}$; where
$U_{D}=\hat{m}-m$ and $\sigma_{D}=\sigma_{ff}$;

Transmit: $Y_{D}=X_{D}+N_{D}$;

Final Decoding: Use ML decoder to detect $\hat{U}$ and detect original PAM
signal using $\hat{U}$.

Algorithm 4 Gallager-Nakibŏglu coding scheme