# Low-Complexity Soft-Decision Detection for Combating DFE Burst Errors in
IM/DD Links

Kaiquan Wu, , Gabriele Liga, , Jamal Riani, , and Alex Alvarado This work is
supported by the Netherlands Organisation for Scientific Research via the VIDI
Grant ICONIC (project No. 15685). The work of Alex Alvarado is supported by
the European Research Council (ERC) under the European Union’s Horizon 2020
research and innovation programme (grant agreement No. 757791).Kaiquan Wu,
Gabriele Liga, and Alex Alvarado are with the Information and Communication
Theory Lab, Signal Processing Systems Group, Department of Electrical
Engineering, Eindhoven University of Technology, Eindhoven 5600 MB, The
Netherlands (e-mails: {k.wu, g.liga, a.alvarado}@tue.nl).Jamal Riani is with
Marvell Technology, Santa Clara, CA, United States (e-mail:
rjamal@marvell.com).

###### Abstract

The deployment of non-binary pulse amplitude modulation (PAM) and soft
decision (SD)-forward error correction (FEC) in future intensity-modulation
(IM)/direct-detection (DD) links is inevitable. However, high-speed IM/DD
links suffer from inter-symbol interference (ISI) due to bandwidth-limited
hardware. Traditional approaches to mitigate the effects of ISI are filters
and trellis-based algorithms targeting symbol-wise maximum a posteriori (MAP)
detection. The former approach includes decision-feedback equalizer (DFE), and
the latter includes Max-Log-MAP (MLM) and soft-output Viterbi algorithm
(SOVA). Although DFE is easy to implement, it introduces error propagation
(EP). Such burst errors distort the log-likelihood ratios (LLRs) required by
SD-FEC, causing performance degradation. On the other hand, MLM and SOVA
provide near-optimum performance, but their complexity is very high for high-
order PAM. In this paper, we consider a one-tap partial response channel
model, which is relevant for high-speed IM/DD links. We propose to combine DFE
with either MLM or SOVA in a low-complexity architecture. The key idea is to
allow MLM or SOVA to detect only $3$ typical DFE symbol errors, and use the
detected error information to generate LLRs in a modified demapper. The
proposed structure enables a tradeoff between complexity and performance: (i)
the complexity of MLM or SOVA is reduced and (ii) the decoding penalty due to
EP is mitigated. Compared to SOVA detection, the proposed scheme can achieve a
significant complexity reduction of up to $94\%$ for PAM-$8$ transmission.
Simulation and experimental results show that the resulting SNR loss is
roughly $0.3\sim 0.4$ dB for PAM-$4$, and becomes marginal $0.18$ dB for
PAM-$8$.

###### Index Terms:

Direct detection, decision-feedback equalizer, intensity modulation, MAP
algorithm, pulse amplitude modulation, soft-decision decoding, soft-output
Viterbi algorithm.

##  I Introduction

The growing demands for optical high-speed data traffic stemming from
applications such as media services, and cloud computing, has driven data
center interconnect (DCI) networks to move forward towards 800Gb/s and beyond.
On top of the rate increases, the requirements of low power consumption, low
complexity, low cost, and low latency are also extremely important [1, 2].
Intensity modulation (IM) and direct detection (DD) links use only a single
photodiode (PD) to receive optical signals, allowing short-reach systems to be
highly cost-effective. However, to support large-bandwidth signal
transmission, bandwidth-limited transceiver components used in these systems
cause severe inter-symbol interference (ISI). For achieving high speeds and
overcoming ISI simultaneously, an effective coded modulation (CM) and
equalization scheme is highly desired in IM/DD links.

Spectral-efficient modulation formats and soft decision (SD)-forward error
correction (FEC) are two indispensable elements for CM in future IM/DD links.
Nonreturn to zero (NRZ) modulation is becoming insufficient to support the
growing data rates. To go beyond the $1$ bit/symbol limit of NRZ, high-order
$M$-ary amplitude modulation (PAM) such as PAM-$4$ and PAM-$8$ [3, Sec. III]
is needed. Recently, an SD concatenated code has been adopted as the optical
interconnect FEC in 200 Gb/s PAM-$4$ serial intra DCI applications [4]. As a
consequence of the increased number of symbol levels, the noise sensitivity is
worsened. FEC is a key enabler for dealing with the worsened noise
sensitivity. Indeed, FEC is expected to have an overhead between $7\%$ and
$15$% for the sake of complexity and power consumption, and provide a net
coding gain of around $10$ dB [2, Sec. IV-C]. In pursuit of a higher symbol
rate, next-generation DCIs are likely to resort to powerful SD-FEC for the
additive white Gaussian noise (AWGN) channel. However, the potential of SD-FEC
under a memoryless AWGN assumption is yet to be fully exploited due to the
performance penalty caused by strong ISI [5, 6].

Equalization techniques aim at removing the ISI from the received symbols. A
decision-feedback equalizer (DFE) has a simple structure and is effective at
canceling post-cursor ISI [7]. The performance of DFE is limited by a
phenomenon known as error propagation (EP). Due to the feedback nature of DFE,
one symbol decision error causes erroneous subsequent decisions, resulting in
burst errors. The statistics of burst errors are usually analyzed by Markov
models that describe the evolution of the DFE decisions. For the purpose of
post-FEC bit error rate (BER) estimates, several Markov models have been
proposed [8, 7, 9, 10]. The analysis becomes complex when multi-tap DFE [9] or
residual ISI [10] is considered.

DFE burst errors degrade the error-correcting performance of FEC decoders. For
systems using NRZ and hard decision (HD)-FEC, the decoding penalty due to
burst errors can be reduced by simple techniques such as bit interleaving or
precoding based on differential and modulo operations [11, 12]. These two
techniques have been adopted in the IEEE 802.3ca standard [13]. However, when
PAM-$M$ and SD-FEC are used, their effectiveness becomes questionable. For
example, the error-correcting performance of SD-FEC is sensitive to the
accuracy of the input log-likelihood ratios (LLRs). LLRs are often computed
using a memoryless AWGN assumption, which is incorrect in the presence of
burst errors [5, 14]. In addition, interleaving might cause an unacceptably
high latency, and precoding prevents traditional demapper from accurate LLR
computation due to the modulo operation error [15, Sec. III-A], making them
less attractive.

To decrease the impact of burst errors, the so-called weighted decision
feedback equalizer (WDFE) was proposed for wireless communications in [16,
17]. WDFE exploits the reliability of the symbol’s HD and weighs them in the
feedback loop. However, a rigorous mathematical derivation of the WDFE
reliability is not available. The values are currently computed via heuristic
reliability functions [18]. Nevertheless, WDFE has been recently studied for
short-reach IM/DD links with HD-FEC in [19, 20]. Significant performance
improvements were reported in experiments. A performance evaluation of WDFE
for SD decoding is missing.

Compared to DFE, trellis-based detection algorithms are the most effective
approach to overcoming ISI. In general, for PAM-$M$ and a channel with memory
$L$, such algorithms perform calculations on a fully-connected trellis
constructed on a $M^{L}$-state machine. Maximum likelihood sequence detection
(MLSD), which finds the most probable path on the trellis given the received
symbols, can be implemented using the Viterbi algorithm (VA). Since we target
IM/DD systems using SD-FEC, a _symbol-wise_ soft output is necessary for the
subsequent bit-wise LLRs. Hence, MLSD is not applicable, but instead, symbol-
wise maximum a posteriori (MAP) detection [21] is needed. Practical MAP
detection algorithms include the Max–Log–MAP (MLM) [22] and the soft-output
Viterbi algorithm (SOVA) [23, 24, 25]. MLM and SOVA achieve near-optimum
performance but at the cost of high complexity due to a large number of states
($M^{L}$). Various state pruning methods have been proposed for MLSD [26, 27,
28, 29].

In this paper, to reduce the penalty of _linear_ ISI on SD decoding in short-
reach IM/DD systems, we propose to combine DFE with a trellis-based detection
algorithm (MLM or SOVA). This paper extends our work in [30] by achieving more
accurate computation of LLRs. In the proposed scheme, MLM and SOVA are
designed to detect only $3$ decision error states resulting from DFE. Then, a
state demapper is proposed to merge the outputs of DFE and the detection
algorithm to yield LLRs. The numerical and experimental results show that the
proposed scheme has low complexity and mitigates the impact of DFE burst
errors on the SD-FEC decoding.

The remainder of the paper is organized as follows. Sec. II presents a brief
review of trellis-based detection algorithms, followed by DFE and its EP.
Then, the finite-state machine for $3$ typical DFE errors is introduced in
Sec. III. MLM and SOVA for the DFE-$3$ state detection and the state demapper
are introduced in the same section. The performance of the proposed scheme is
investigated in Sec. IV. Finally, conclusions are drawn in Sec. V.

_Notation_ : In this paper, the probability of a discrete random variable is
denoted by $P(\cdot)$. The probability density function (PDF) is denoted by
$p(\cdot)$. The variance of a variable is denoted by
$\mathrm{Var}\left[\cdot\right]$. Sequences are denoted by boldface letters,
e.g., $\boldsymbol{y}$, and if necessary sub/superscripts are used for the
boundary, e.g.,
$\boldsymbol{y}_{a}^{b}\triangleq[y_{a},y_{a+1},\ldots,y_{b}]$. Given values
of mean $\mu$ and variance $\sigma^{2}$, Gaussian distribution is
$\mathcal{N}(\mu,\sigma^{2})$. To denote the log version of the corresponding
variable, we use $\tilde{\cdot}\triangleq\log(\cdot)$.

##  II Preliminaries

###  II-A Partial Response Channel

In this paper, we assume that an ideal feedforward equalizer (FFE) cancels all
pre-cursor ISI in the IM/DD link. We concentrate on a single-tap post-cursor
ISI because the impact of multiple taps is considered to be negligible, as
previously done in [14, 6]. The resulting effective channel is a one-tap
partial response (PR) channel, as depicted in the green frame in Fig. 1. At
time instant $i$ ($i=1,2,\ldots,N$), given the transmitted symbol $x_{i}$, the
received symbol $y_{i}$ is written as

$y_{i}=x_{i}+hx_{i-1}+n_{i},$ (1)

where $h$ indicates the magnitude of the interference, and $n_{i}$ stands for
the AWGN. Given the noise variance $\sigma^{2}$, we have
$n_{i}\sim\mathcal{N}(0,\sigma^{2})$. As we are targeting short-reach IM/DD
systems without amplifiers, bipolar PAM-$M$ is considered [31]. The symbol is
drawn from a real set $\mathcal{X}$ defined as $\mathcal{X}\triangleq\\{\pm
1\Delta,\pm 3\Delta,\ldots,\pm(M-1)\Delta\\}$ for every $m=\log_{2}M$ bits.
The considered PR channel is illustrated in the dashed frame of Fig. 1(b),
where $D$ represents a delay element. We assume that the information bits are
encoded/decoded using a binary SD-FEC code, and the coded bits
$[b_{i,1},b_{i,2},\ldots,b_{i,m}]$ are mapped into independent and uniformly
distributed symbol $x_{i}\in\mathcal{X}$ with the binary reflected Gray
labeling.

###  II-B Decision-Feedback Equalizer and Error Propagation

(a) DFE and AWGN demapper. (b) MLM/SOVA working with PAM-$M$ states.

Figure 1: Block diagrams of PAM-$M$ transmission in the one-tap PR channel
with different _traditional_ detection schemes. In (a), interleaving and
precoding (dashed blocks) can be alternatively employed in the DFE-based
scheme.  Figure 2: Structure of feedback filter in DFE. The HD at time instant
$i$ is used to remove the ISI at instant $i+1$.

The feedback structure of DFE for the PR channel transmission is shown in Fig.
1(a). In addition to DFE, bit-interleaving or precoding, depicted in dashed
blocks in Fig. 1(a), can be employed to alleviate the EP. The idea behind DFE
is to use the HDs on previous symbols to remove their contribution to ISI from
the present received symbol. A one-tap DFE is shown in Fig. 2. Given the
received symbol $y_{i}$ and the previous HD $\hat{x}_{i-1}$, the DFE-equalized
symbol $\overline{y}_{i}$ is computed as

$\overline{y}_{i}=y_{i}-h\hat{x}_{i-1}.$ (2)

In view of (1) and (2), the ISI is removed if $\hat{x}_{i-1}=x_{i-1}$.

The HD $\hat{x}_{i}$ is obtained based on $\overline{y}_{i}$ by slicers with
thresholds $\\{0,\pm 2\Delta,\ldots,\pm(M-2)\Delta\\}$ located halfway between
the symbols from $\mathcal{X}$. For example, the HD $\hat{x}_{i}$ for PAM-$4$
is

$\hat{x}_{i}=\mathcal{F}(\overline{y}_{i})=\left\\{\begin{IEEEeqnarraybox}[]{[}][c]{l^{\prime}s}-3\Delta,&for$\overline{y}_i<-2\Delta$\\\
-1\Delta,&for$-2\Delta\leq\overline{y}_i<0$\\\
+1\Delta,&for$0\leq\overline{y}_i<+2\Delta$\\\
+3\Delta,&for$\overline{y}_i\geq+2\Delta$\end{IEEEeqnarraybox}\right..$ (3)

The procedures in (2)-(3) repeat to remove ISI from the next received symbol
$y_{i+1}$.

Given the equalized symbols
$\overline{\boldsymbol{y}}=[\overline{y}_{1},\overline{y}_{2},\ldots,\overline{y}_{N}]$,
a traditional AWGN demapper treats the PR channel and DFE together as an AWGN
channel. Therefore, the LLRs are computed as if all ISI has been removed and
only AWGN remains. Given the equalized symbol $\overline{y}_{i}$, the
resulting _mismatched_ LLR $L_{i,j}$ is given by

$L_{i,j}=\log\frac{\sum_{x\in\mathcal{X}^{1}_{j}}p(\overline{y}_{i}|x_{i}=x)}{\sum_{x\in\mathcal{X}^{0}_{j}}p(\overline{y}_{i}|x_{i}=x)},$
(4)

where $p(\overline{y}_{i}|x_{i}=x)$ is assumed to follow
$\overline{y}_{i}\sim\mathcal{N}(x,\sigma^{2})$, and
$\mathcal{X}^{b}_{j}\subset\mathcal{X}$ are the sets of PAM-$M$ symbols
labeled by bit $b_{i,j}\in\\{0,1\\}$ at bit position $j$. Fig. 1(a) shows the
scheme using DFE and the AWGN demapper to generate bit-wise LLRs for SD-FEC
decoding.

The EP of DFE comes from the inherent feedback structure. It can be seen from
(2) that the removal of ISI for the current equalized symbol
$\overline{y}_{i}$ depends on a correct prior HD $\hat{x}_{i-1}$. Let the
error be defined as $e_{i-1}\triangleq x_{i-1}-\hat{x}_{i-1}$. For PAM-$M$
with the minimum euclidean distance $d=2\Delta$, there are $(2M-1)$ possible
errors, i.e., $e_{i-1}\in\\{0,\pm d,\pm 2d,\ldots,\pm(M-1)d\\}$. At relatively
high signal-to-noise ratios (SNRs) and small $h$, most errors occur between
adjacent symbols. Therefore, the most likely errors occurring are
$e_{i-1}\in\\{0,\pm d\\}$ considered instead. After the subtraction in (2),
these $3$ errors lead to $3$ different _biased states_ $s_{i}$ that are
possibly imposed on $\overline{y}_{i}$, which are written as

$s_{i}\triangleq\left\\{\begin{IEEEeqnarraybox}[]{[}][c]{l^{\prime}s}l,&if$e_{i-1}=-d$\\\
c,&if$e_{i-1}=0$\\\ r,&if$e_{i-1}=+d$\end{IEEEeqnarraybox}\right..$ (5)

In (5), $s_{i}\in\mathcal{S}\triangleq\\{l,c,r\\}$ stands for a set of left-
biased, center (unbiased), and right-biased states, resp. Fig. 3 illustrates
the association between the DFE errors and the biased states by displaying the
PDF of $p(\overline{y}_{i}|x_{i},s_{i})$. Suppose $s_{i-1}=c$ and
$x_{i-1}=+1\Delta$, Fig. 3(a) shows that the two most relevant symbol errors
are $e_{i-1}=-d$ and $d$. When $x_{i}=+1\Delta$, these errors lead to the left
and the right-biased observations of $\overline{y}_{i}$ ($s_{i}=l,r$) resp.,
as shown in Fig. 3(b). From now on, we call $l,c$ and $r$ as “DFE-$3$ states”,
and define a function $\omega(s)$ as

$\omega(s_{i})\triangleq\left\\{\begin{IEEEeqnarraybox}[]{[}][c]{l^{\prime}s}-d,&if$s_i=l$\\\
0,&if$s_i=c$\\\ +d,&if$s_i=r$\end{IEEEeqnarraybox}\right..$ (6)

Using (5) and (6), (2) can be alternatively written as

$\overline{y}_{i}=x_{i}+n_{i}+h\omega(s_{i})=\hat{x}_{i}+n_{i}+h\omega(s_{i})+\omega(s_{i+1}).$
(7)

Figure 3: An illustration of the association between the symbol errors
$e_{i-1}$ and the biased state $s_{i}$ via the conditional probabilities
$p(\overline{y}_{i}|x_{i},s_{i})$. Here we assume $s_{i-1}=c$ and
$x_{i-1}=x_{i}=+1\Delta$. The colored areas represent $P(e_{i}|x_{i},s_{i})$.
The thick vertical lines represent the slicers for decision.

In $l$ or $r$ state, the subsequent HD is based on a biased channel
observation $\overline{y}_{i}$ and thus becomes more prone to an error. For
the example in Fig. 3 (b), when $x_{i}=+1\Delta$ and $s_{i}=l$, the PDF is
shifted to the left by $hd$. Compared to the center state $c$, with the
decision threshold $0$, $\hat{x}_{i}$ has a higher chance to be $-1\Delta$ at
state $l$, as indicated by the areas of $e_{i-1}=-d$ and $e_{i}=-d$ in Fig. 3
(a) and (b). If error $e_{i}=-d$ occurs, consequently, the subsequent state is
directed to the opposite biased state, i.e., $s_{i}=l\rightarrow s_{i+1}=r$,
which affects again the following decisions. The EP ends only when a correct
decision is made in a biased state.

###  II-C Weighted DFE

WDFE mitigates the EP of DFE via an improved decision rule that incorporates
the reliability information of HD [16, 17]. Compared to (2), the WDFE-
equalized symbol $\overline{y}_{i}$ is computed as

$\overline{y}_{i}=y_{i}-h\tilde{y}_{i-1},$ (8)

where $\tilde{y}_{i-1}$ is a “hybrid” decision based on the previous HD
$\hat{x}_{i-1}$ and the equalized symbol $\overline{y}_{i-1}$, i.e.,

$\tilde{y}_{i-1}=f(\eta_{i-1})\hat{x}_{i-1}+\left[1-f(\eta_{i-1})\right]\overline{y}_{i-1}.$
(9)

In (9), $\eta_{i-1}$ is the reliability estimate of $\hat{x}_{i-1}$, and is
used in function $f(\cdot)$ to produce a weighting factor. The values of
$\eta_{i-1}$ and $f(\eta_{i-1})$ range from $0$ to $1$.

The drawback of WDFE is that the reliability computation is heuristic and thus
inaccurate [17]. $\eta_{i}$ is defined as

$\eta_{i}\triangleq\left\\{\begin{IEEEeqnarraybox}[]{[}][c]{l^{\prime}s}1&if$|\overline{y}_{i}|\geq(M-1)\Delta$\\\
1-\frac{|\hat{x}_{i}-\overline{y}_{i}|}{\Delta},&else\end{IEEEeqnarraybox}\right..$
(10)

A popular choice of reliability function $f(\eta_{i})$ is the sigmoid
nonlinear function defined as [17, 18, 20]

$f(\eta_{i})\triangleq\frac{1}{2}\left(\frac{1-\exp\left[-a(\eta_{i}/b-1)\right]}{1+\exp\left[-a(\eta_{i}/b-1)\right]}\right),$
(11)

where $a$ ($a>0$) and $b$ ($0\leq b\leq 1$) are user-defined compression
parameters. When $b=0$, $f(\eta_{i})=1$ and (8) is equivalent to (2), and thus
WDFE acts as DFE.

###  II-D Symbol-by-Symbol Soft-Output Detection

In this section, we give a high-level overview of traditional MLM and SOVA for
PR channel detection. Their receiver structure is shown in Fig. 1(b). The
specific principles of MLM and SOVA will be discussed in detail in Sec. III
(using a different finite state machine model).

In the presence of ISI, MAP is the optimal rule for symbol-wise detection, in
the sense that the a posteriori probability (APP) $P(x_{i}|\boldsymbol{y})$
for each transmitted symbol is maximized. Upon receiving the symbol vector
$\boldsymbol{y}=[y_{1},y_{2},\ldots,y_{N}]$, at time instant $i$, the MAP
detector evaluates the symbol LLRs $\Lambda_{i}(x)$ for $x\in\mathcal{X}$.
Without loss of generality, we consider the symbol $+1\Delta$ as the
reference. In this case $\Lambda_{i}(+1\Delta)=0$, and

$\Lambda_{i}(x)\triangleq\log\left.\frac{P(x_{i}=x|\boldsymbol{y})}{P(x_{i}=+1\Delta|\boldsymbol{y})}\right.,x\in\mathcal{X}.$
(12)

For the binary SD-FEC decoding, symbol LLRs $\Lambda_{i}(x)$ must be
translated into bit-wise LLRs $L_{i,j}$ ($j=1,2,\ldots,m$). Assuming equally
likely bits, such translation is achieved according to

$L_{i,j}=\log\frac{\sum_{x\in\mathcal{X}^{1}_{j}}\exp{\Lambda_{i}(x)}}{\sum_{x\in\mathcal{X}^{0}_{j}}\exp{\Lambda_{i}(x)}}.$
(13)

To obtain the symbol LLRs in (12), a finite-state machine that characterizes
all possible ISI (states), as well as the transitions among them, is used. For
the channel in (1), the state at time instant $i$ is the transmitted symbol
$x_{i-1}$, which contributes to the ISI $hx_{i-1}$ imposed on symbol $x_{i}$.
By unfolding the sates in the time dimension, a fully-connected trellis is
obtained, where each path represents a possible transmitted symbol sequence
$\boldsymbol{x}=[x_{1},x_{2},\ldots,x_{N}]$. An illustration of the state
machine and the corresponding trellis for PAM-$4$ is shown in Fig. 4.

(a) Finite-state machine of PAM-$4$.  (b) Trellis of PAM-$4$.

Figure 4: The finite-state machine (a) and the trellis (b) of PAM-$4$ for the
one-tap partial response channel in (1). In (a), we note that the inputs for
only a subset of edges are labeled.

There are two popular approaches for obtaining the symbol LLRs in (12). One is
BCJR to directly compute the APP $P(x_{i}|\boldsymbol{y})$ by performing two
recursive computations in the forward and backward directions on the trellis.
The implementation of BCJR comes in two flavors, i.e., Log-MAP (LM) and MLM.
As will be shown in Sec. III-C, the difference between LM and MLM is that the
APP in the log domain is computed either exactly (in LM), or approximately in
(MLM). The other approach is to estimate the difference between two APPs in
(12), which is used by SOVA. As an augmented version of VA, SOVA performs an
additional traceback procedure w.r.t. VA to estimate the symbol LLRs within a
length-$\delta$ window. The path metric differences between the surviving path
and the nonsurviving paths are the results of their different symbol
decisions, which are roughly proportional to the reliability of the decisions.
When the traceback window $\delta$ is infinitely long, SOVA is equivalent to
MLM, as proved in [25].

Figure 5: Block diagrams of PAM-$M$ transmission in the one-tap PR channel
with the _proposed_ MLM/SOVA detection working with DFE-3 states.

In terms of implementation, both MLM and SOVA extensively rely on compare and
select (CS) operations. Nonetheless, the difference between SOVA and MLM,
mainly lies in their backward search on the trellis. SOVA traceback within a
limited window, while MLM performs backward recursion on the whole frame of
symbols, making them possess distinct advantages. In general, MLM offers
better performance, while SOVA is more friendly for hardware implementation
[32]. For a comparative study between MLM and SOVA, readers are referred to
[33]. The choice between SOVA or MLM often depends on the application under
consideration.

##  III Soft-Output Trellis-Based Detection Algorithms of Reduced DFE-3 States

In this section, we propose a trellis-based soft detection scheme for the
PAM-$M$ one-tap PR channels, whose general structure is shown in Fig. 5.
Instead of producing the symbol LLRs as expressed in (12), the proposed scheme
focuses on the LLRs for the $3$ DFE biased states in (5). We first determine
the APPs and the LLRs of interests. Then, the corresponding state machine and
trellis visualized in Fig. 6 will be discussed. Based on this trellis, MLM or
SOVA are enabled to detect the DFE-$3$ states. Finally, a state demapper is
proposed to process the output of MLM or SOVA prior to the SD decoder.

###  III-A Target APP and LLR of DFE-3 States

The target APP of the biased states for MAP detection is
$P(s_{i}|\overline{\boldsymbol{y}})$. This probability can be expressed as

$P(s_{i}\\!=\\!s^{\prime}|\overline{\boldsymbol{y}})=\frac{\sum\limits_{s\in\mathcal{S}}p(s_{i}\\!=\\!s^{\prime},s_{i+1}\\!=\\!s,\overline{\boldsymbol{y}})}{\sum\limits_{s^{\prime}\in\mathcal{S}}\sum\limits_{s\in\mathcal{S}}p(s_{i}\\!=\\!s^{\prime},s_{i+1}\\!=\\!s,\overline{\boldsymbol{y}})}.$
(14)

For clarity, we denote the joint probability
$p_{i}(s^{\prime},s,\overline{\boldsymbol{y}})\triangleq
p(s_{i}=s^{\prime},s_{i+1}=s,\overline{\boldsymbol{y}})$ and the symbol vector
$\overline{\boldsymbol{y}}_{i}^{j}\triangleq[\overline{y}_{i},\overline{y}_{i+1},\ldots,\overline{y}_{j}]$.
Then, $p_{i}(s^{\prime},s,\overline{\boldsymbol{y}})$ is factored by the chain
rule and Markov properties as [34, Sec. 4.5.3]

$p_{i}(s^{\prime},s,\overline{\boldsymbol{y}})=\alpha_{i}(s^{\prime})\gamma_{i+1}(s^{\prime},s)\beta_{i+1}(s),$
(15)

where

$\displaystyle\alpha_{i}(s^{\prime})\triangleq
p\left(s_{i}=s^{\prime},\overline{\boldsymbol{y}}_{1}^{i-1}\right),$ (16)
$\displaystyle\gamma_{i+1}(s^{\prime},s)\triangleq
p(s_{i+1}=s,\overline{y}_{i}|s_{i}=s^{\prime}),$ (17)
$\displaystyle\beta_{i+1}(s)\triangleq
p\left(\overline{\boldsymbol{y}}_{i+1}^{N}|s_{i+1}=s\right).$ (18)

The values of $\gamma_{i+1}(s^{\prime},s)$ in (17) are the probabilities that
characterize the transition behavior within the DFE-3 states. The values of
$\alpha_{i}(s^{\prime})$ and $\beta_{i+1}(s)$ are computed through forward and
backward recursive inference [21].

The last step (see Fig. 5) is to calculate the LLRs of the DFE-3 states. These
LLRs are denoted by $\Gamma_{i}(s^{\prime})$, and defined w.r.t. the center
state $c$. Hence, $\Gamma_{i}(c)=0$ and

$\Gamma_{i}(s^{\prime})\\!\triangleq\\!\log\frac{P(s_{i}\\!=\\!s^{\prime}|\overline{\boldsymbol{y}})}{P(s_{i}\\!=\\!c|\overline{\boldsymbol{y}})}\\!=\\!\log\frac{p(s_{i}\\!=\\!s^{\prime},\overline{\boldsymbol{y}})}{p(s_{i}\\!=\\!c,\overline{\boldsymbol{y}})},s^{\prime}\\!\in\\!\\{l,r\\},$
(19)

where the expression on the right-hand side follows from the fact that
$p(s_{i}\\!=\\!s^{\prime}|\overline{\boldsymbol{y}})=p(s_{i}\\!=\\!s^{\prime},\overline{\boldsymbol{y}})/p(\overline{\boldsymbol{y}})$,
and we obtain $p(s_{i}\\!=\\!s^{\prime},\overline{\boldsymbol{y}})$ by using
the total probability law on $p_{i}(s^{\prime},s,\overline{\boldsymbol{y}})$
in (15), i.e.,
$p(s_{i}=s^{\prime},\overline{\boldsymbol{y}})=\sum_{s\in\mathcal{S}}p(s_{i}\\!=\\!s^{\prime},s_{i+1}\\!=\\!s,\overline{\boldsymbol{y}})$.

###  III-B DFE-$3$ States and the Transition Probability $\gamma$

(a) Finite-state machine of DFE-$3$.  (b) Trellis of DFE-$3$.  (c) PAM-$4$
trellis masked by DFE-$3$ trellis.

Figure 6: The finite-state machine (a) and trellis (b) of $3$ DFE biased
states. (c) shows an example section of DFE-$3$ trellis given certain HD input
on top of PAM-$4$ trellis. The blue/red loops indicate error-free and error
propagation of the DFE output, resp. The dotted/dashed lines indicate
transitions that cannot occur (for the two outermost PAM symbols).

The state machine and the trellis proposed for the DFE-$3$ states are depicted
in Fig. 6. For the state machine in Fig. 6(a), each edge labeled with the
transition probability $\gamma_{i+1}(s^{\prime},s)$. There are two major
transition loops that characterize the DFE behavior. The transition indicated
by the blue loop implies an error-free DFE output, while the EP is represented
by the red loop between states $l$ and $r$. As explained in Sec. II-B, the
self-transitions seldom occur for the biased states, i.e., $\gamma_{i+1}(l,l)$
and $\gamma_{i+1}(r,r)$, and hence are not considered in EP. Fig. 6(b) depicts
the corresponding DFE-$3$ trellis.

Fig. 6(c) compares DFE-$3$ trellis with PAM-$4$ trellis. Given a certain input
of HD $\hat{x}$, the corresponding section of the DFE-$3$ trellis is put on
top of the PAM-$4$ trellis in Fig. 4(b). The center state $c$ falls upon the
HD, and the disconnection to $l$ takes place when $c$ happens to be on the
leftmost symbol $-3\Delta$. We conclude that the proposed DFE-$3$ state
machine leads to a simpler analysis, because (i) the number of states
decreases from $M$ to $3$ and (ii) part of transitions are possibly
disconnected and thus leading to a complexity reduction (indicated by the
dotted and dashed edges).

In the following, we explain the computation of $\gamma_{i+1}(s^{\prime},s)$
with or without disconnections. Given an equalized symbol $\overline{y}_{i}$,
$\hat{x}_{i}=\mathcal{F}(\overline{y}_{i})$ is known, and the next state
$s_{i+1}=s$ is determined by $x_{i}-\hat{x}_{i}=\omega(s)$. Hence,
$\gamma_{i+1}(s^{\prime},s)$ in (17) is equivalent to

$\displaystyle\gamma_{i+1}(s^{\prime},s)=p(\omega(s),\overline{y}_{i}|s_{i}=s^{\prime})$
$\displaystyle=$ $\displaystyle
P(x_{i}=\hat{x}_{i}+\omega(s))p(\overline{y}_{i}|s_{i}=s^{\prime},x_{i}=\hat{x}_{i}+\omega(s)).$
(20)

When $\hat{x}_{i}$ is one of the two outermost symbols, i.e.,
$\hat{x}_{i}\in\\{\pm(M-1)\Delta\\}$, the transition to either $s=l$ or $r$ is
impossible. This is due to the fact that the border symbols have only one
neighboring symbol. As a result, one of the HD errors is avoided. For example,
when $\hat{x}_{i}=-(M-1)\Delta$, the transition probability to $l$ is
$\gamma_{i+1}(s^{\prime},l)=0$, because
$x_{i}=\hat{x}_{i}+\omega(l)=-(M+1)\Delta\notin\mathcal{X}$ causes
$P(x_{i}=\hat{x}_{i}+\omega(s))=0$. The probability of “crossing out” a
transition because of this bordering effect is $2/M$ for PAM-$M$. Such
disconnection can facilitate end of burst errors and also save the computation
for transition probability.

When $\hat{x}_{i}$ is an inner symbol, i.e., $\hat{x}_{i}\in\\{\pm 1\Delta,\pm
3\Delta,\ldots,\pm(M-3)\Delta\\}$, $x_{i}=\hat{x}_{i}+\omega(s)\in\mathcal{X}$
$\forall s=l$, $c$ and $r$. In this case, $P(x_{i}=\hat{x}_{i}+\omega(s))=1/M$
in (III-B) for equiprobable PAM-$M$. Therefore, we only need to compute
$p(\overline{y}_{i}|s_{i}=s^{\prime},x_{i}=\hat{x}_{i}+\omega(s))$. Due to the
AWGN $n_{i}$ in $\overline{y}_{i}$ (see (7)), we have
$\overline{y}_{i}\sim\mathcal{N}(x_{i}+h\omega(s_{i}),\sigma^{2})$. Hence, in
view of (III-B), $\gamma_{i+1}(s^{\prime},s)$ is computed as

$\gamma_{i\\!+\\!1}(s^{\prime}\\!,\\!s)\\!\\!=\\!\\!\frac{1}{\sqrt{2\pi}\\!\sigma\\!M}\\!\exp\\!\\!\left(\\!\frac{\\!-\\!\left(\overline{y}_{i}\\!-\\!\hat{x}_{i}\\!-\\!h\omega(s^{\prime})\\!-\\!\omega(s)\right)^{2}}{2\sigma^{2}}\right).$
(21)

Figure 7: The state transition probabilities
$\gamma_{i\\!+\\!1}(s^{\prime}\\!,\\!s)$ for $s_{i}=c$, $l$, and $r$ are shown
in the top, center, and bottom sub-figures, resp. In each sub-figure, suppose
$\overline{y}_{i}=-0.6\Delta$ (implicitly $\hat{x}_{i}=-1\Delta$) and $h=0.7$,
$\gamma_{i\\!+\\!1}(s^{\prime}\\!,\\!s)$ for the transition to states
$s_{i+1}=c$, $l$ and $r$ are indicated by the three markers, resp. The thick
vertical lines represent the slicers for decision.

######  Example 1 ($\gamma_{i\\!+\\!1}(s^{\prime}\\!,\\!s)$ for PAM-$4$)

Fig. 7 shows $\gamma_{i\\!+\\!1}(s^{\prime}\\!,\\!s)$ for PAM-$4$. Each sub-
figure corresponds to a case of state $s_{i}=c,l,r$ from top to bottom, and
consists of $4$ distributions for PAM-$4$ symbols. Specifically, when
$\overline{y}_{i}=-0.6\Delta$ (indicated by the dashed vertical line), the HD
$\hat{x}_{i}=\mathcal{F}(\overline{y}_{i})=-1\Delta$ is an inner symbol, and
hence all transitions are possible. The transitions to $s_{i+1}=l,c,r$ states
occur when $x_{i}=-3\Delta$, $x_{i}=-1\Delta$, and $x_{i}=+1\Delta$, resp.
Therefore, $\gamma_{i\\!+\\!1}(s^{\prime}\\!,\\!s)$ are the crossing points
between the dashed line and the corresponding distributions for different
$x_{i}$ (depicted in black).

Typically, $\gamma_{i\\!+\\!1}$ in (21) is referred to as “branch metric”. In
the log domain, the argument of the exponential function in (21) denoted as
$\tilde{\gamma}_{i+1}(s^{\prime},s)$ becomes the branch metric. Algorithm 1
shows how $\tilde{\gamma}_{i+1}(s^{\prime},s)$ is computed. Given the DFE
equalized symbol $\overline{y}_{i}$, $\tilde{\gamma}_{i+1}(s^{\prime},s)$ is
always computed for transitions to $c$, and possibly skipped for transitions
to $l$ and $r$. Note that the factor $1/\sqrt{2\pi}M\sigma$ in (21) is
omitted, because when eventually computing the DFE-$3$ LLRs as shown in (19),
it will appear as a common factor in the denominator/numerator and thus will
be canceled out.

###  III-C MLM for DFE-3 States Detection

Input: DFE-equalized symbol $\overline{y}_{i}$, hard decision $\hat{x}_{i}$,
noise variance $\sigma^{2}$, modulation order $M$, biased state set
$\mathcal{S}$, and tap coefficient $h$

Output: Branch metrics $[\tilde{\gamma}_{i+1}(s^{\prime},s)]$, for
$s^{\prime},s\in\mathcal{S}$

1

// Initialize

2 $\tilde{\gamma}_{i+1}(s^{\prime},s)=-\infty$, $\forall
s^{\prime},s\in{\mathcal{S}}$

// For transitions to state $c$

3 for  _$s^{\prime}\in{\mathcal{S}}$_ do

4
$\tilde{\gamma}_{i+1}(s^{\prime},c)=-(\overline{y}_{i}-\hat{x}_{i}-h\omega(s^{\prime}))^{2}/(2\sigma^{2})$

5 end for

6

// For transitions to state $l$

7 if  _$\hat{x}_{i}\neq+(M-1)\Delta$ _ then

8 for  _$s^{\prime}\in{\mathcal{S}}$_ do

9
$\tilde{\gamma}_{i+1}(s^{\prime},l)=-(\overline{y}_{i}-\hat{x}_{i}-h\omega(s^{\prime})+d)^{2}/(2\sigma^{2})$

10 end for

11

12 end if

13

// For transitions to state $r$

14 if  _$\hat{x}_{i}\neq-(M-1)\Delta$ _ then

15 for  _$s^{\prime}\in{\mathcal{S}}$_ do

16
$\tilde{\gamma}_{i+1}(s^{\prime},r)=-(\overline{y}_{i}-\hat{x}_{i}-h\omega(s^{\prime})-d)^{2}/(2\sigma^{2})$

17 end for

18

19 end if

20

return _$[\tilde{\gamma}_{i+1}(s^{\prime},s)]$ _

Algorithm 1 Branch Metric Computation

Knowing $\gamma_{i+1}(s^{\prime},s)$ for $s^{\prime},s\in\mathcal{S}$,
$\alpha_{i+1}(s)$ is computed recursively in the forward direction via

$\alpha_{i+1}(s)=\sum_{s^{\prime}\in\mathcal{S}}\alpha_{i}(s^{\prime})\gamma_{i+1}(s^{\prime},s).$
(22)

Similarly, the backward recursion is done for $\beta_{i}(s^{\prime})$ as

$\beta_{i}(s^{\prime})=\sum_{{s}\in\mathcal{S}}\gamma_{i+1}(s^{\prime},s)\beta_{i+1}(s).$
(23)

Without prior knowledge, the values of $\beta_{N+1}(s)$ and $\alpha_{1}(s)$
are initialized as
$[\alpha_{1}(l),\alpha_{1}(c),\alpha_{1}(r)]\\!=\\![\beta_{N\\!+\\!1}(l),\beta_{N\\!+\\!1}(c),\beta_{N\\!+\\!1}(r)]=[0,\\!1,\\!0]$.

Input: DFE-equalized symbols
$\overline{\boldsymbol{y}}=[\overline{y}_{1},\overline{y}_{2},...,\overline{y}_{N}]$,
hard decisions
$\hat{\boldsymbol{x}}=[\hat{x}_{1},\hat{x}_{2},...,\hat{x}_{N}]$, noise
variance $\sigma^{2}$, modulation order $M$, biased state set $\mathcal{S}$,
and tap coefficient $h$

Output: DFE-3 LLRs
$\boldsymbol{\Gamma}=[\Gamma_{1}(s^{\prime}),\Gamma_{2}(s^{\prime}),...,\Gamma_{N}(s^{\prime})]$
for $s^{\prime}\in\mathcal{S}$

1

// Initialize

2
$[\tilde{\alpha}_{1}(l),\tilde{\alpha}_{1}(c),\tilde{\alpha}_{1}(r)]=[-\infty,0,-\infty]$

3
$[\tilde{\beta}_{N+1}(l),\tilde{\beta}_{N+1}(c),\tilde{\beta}_{N+1}(r)]=[-\infty,0,-\infty]$

4

// Compute branch metric $\tilde{\gamma}_{i+1}(s^{\prime},s)$

5 for  _$i=1,2,...,N$_ do

6 Algorithm 1

7 end for

8

// Forward recursion for $\tilde{\alpha}_{i+1}(s)$ in (26)

9 for  _$i=1,2,...,N$_ do

10 for  _$s\in{\mathcal{S}}$_ do

11
$\tilde{\alpha}_{i+1}(s)=\max\limits_{s^{\prime}\in{\mathcal{S}}}\left[\tilde{\alpha}_{i}(s^{\prime})+\tilde{\gamma}_{i+1}(s^{\prime},s)\right]$

12 end for

13

14 end for

15

// Backward recursion for $\tilde{\beta}_{i}(s^{\prime})$ in (27)

16 for  _$i=N,N-1,...,1$_ do

17 for  _$s^{\prime}\in{\mathcal{S}}$_ do

18
$\tilde{\beta}_{i}(s^{\prime})=\max\limits_{s\in{\mathcal{S}}}\left[\tilde{\beta}_{i+1}(s)+\tilde{\gamma}_{i+1}(s^{\prime},s)\right]$

19 end for

20

// Compute $\tilde{p}(s_{i}=s^{\prime},\boldsymbol{y})$ in (28)

21 for  _$s^{\prime}\in{\mathcal{S}}$_ do

22
$\tilde{p}(s_{i}=s^{\prime},\boldsymbol{y})=\max\limits_{s\in{\mathcal{S}}}\left[\tilde{\alpha}_{i}(s^{\prime})+\tilde{\gamma}_{i+1}(s^{\prime},s)+\tilde{\beta}_{i+1}(s)\right]$

23 end for

24

// Compute DFE-3 LLRs $\Gamma_{i}(s^{\prime})$ in (19)

25 for  _$s^{\prime}\in\\{l,r\\}$_ do

26
$\Gamma_{i}(s^{\prime})=\tilde{p}(s_{i}=s^{\prime},\boldsymbol{y})-\tilde{p}_{i}(s_{i}=c,\boldsymbol{y})$

27 end for

28

29 end for

30

return _$\boldsymbol{\Gamma}$_

Algorithm 2 DFE-3 MLM

In practice, (22) and (23) are implemented in the log domain, which leads to
the LM algorithm [Sec. 2.4][33], i.e.,

$\tilde{\alpha}_{i+1}(s)=\log\left(\sum_{s^{\prime}\in\mathcal{S}}\exp\left(\tilde{\alpha}_{i}(s^{\prime})+\tilde{\gamma}_{i+1}(s^{\prime},s)\right)\right),$
(24)

and

$\tilde{\beta}_{i}(s^{\prime})=\log\left(\sum_{s\in\mathcal{S}}\exp\left(\tilde{\gamma}_{i+1}(s^{\prime},s)+\tilde{\beta}_{i+1}(s)\right)\right).$
(25)

Note that the Log-Sum-Exp in (24) and (25) can be replaced by the Jacobian
logarithm for ease of implementation. Alternatively, one can apply Max-Log
(ML) approximation to the Log-Sum-Exp, resulting in MLM. Hence, the recursions
are achieved in MLM as

$\tilde{\alpha}_{i+1}(s)=\max\limits_{s^{\prime}\in{\mathcal{S}}}\left[\tilde{\alpha}_{i}(s^{\prime})+\tilde{\gamma}_{i+1}(s^{\prime},s)\right],$
(26)

and

$\tilde{\beta}_{i}(s^{\prime})=\max\limits_{s\in{\mathcal{S}}}\left[\tilde{\gamma}_{i+1}(s^{\prime},s)+\tilde{\beta}_{i+1}(s)\right].$
(27)

Therefore, in view of (15), MLM computes
$\tilde{p}(s_{i},\overline{\boldsymbol{y}})$ as

$\displaystyle\tilde{p}(s_{i}=s^{\prime},\overline{\boldsymbol{y}})=\max\limits_{s\in{\mathcal{S}}}\left[\tilde{\alpha}_{i}(s^{\prime})+\tilde{\gamma}_{i+1}(s^{\prime},s)+\tilde{\beta}_{i+1}(s)\right].$
(28)

The final step in DFE-$3$ MLM is to compute the LLRs of biased states
$\Gamma_{i}(s^{\prime})$ (see Fig. 5) based on (19). The DFE-$3$ MLM algorithm
is summarized in Algorithm 2. After initialization, it calls Algorithm 1 to
obtain the branch metrics. Then, $3$ major loops achieve the two recursions
and the update of the joint probabilities
$\tilde{p}(s_{i}=s^{\prime},\overline{\boldsymbol{y}})$, resp.

###  III-D SOVA for DFE-$3$ States Detection

Input: DFE-equalized symbols
$\overline{\boldsymbol{y}}=[\overline{y}_{1},\overline{y}_{2},...,\overline{y}_{N}]$,
hard decisions
$\hat{\boldsymbol{x}}=[\hat{x}_{1},\hat{x}_{2},...,\hat{x}_{N}]$, noise
variance $\sigma^{2}$, modulation order $M$, biased state set $\mathcal{S}$,
traceback length $\delta$, and tap coefficient $h$

Output: DFE-3 LLRs
$\boldsymbol{\Gamma}=[\Gamma_{1}(s^{\prime}),\Gamma_{2}(s^{\prime}),...,\Gamma_{N}(s^{\prime})]$
for $s^{\prime}\in\mathcal{S}$

1

// Initialize

2
$[\tilde{\xi}_{1}(l),\tilde{\xi}_{1}(c),\tilde{\xi}_{1}(r)]=[\infty,0,\infty]$

3

4for  _$i=1,2,...,N$_ do

5

// Compute branch metric $\tilde{\gamma}_{i+1}(s^{\prime},s)$

6 Algorithm 1

7 for  _$s\in{\mathcal{S}}$_ do

8

// Compute path metric $\tilde{\xi}_{i+1}(s)$ in (29)

9
$\tilde{\xi}_{i+1}(s)=\min\limits_{s^{\prime}\in\mathcal{S}}\left[\tilde{\xi}_{i}(s^{\prime})-\tilde{\gamma}_{i+1}(s^{\prime},s)\right]$

10

// Update current $\Gamma^{s}_{i}(s^{\prime})$ in (30)

11 for  _$s^{\prime}\in{\mathcal{S}}$_ do

12
$\Delta(s^{\prime},s)=\tilde{\xi}_{i}(s^{\prime})-\tilde{\gamma}_{i+1}(s^{\prime},s)-\tilde{\xi}_{i+1}(s)$

13 $\Gamma^{s}_{i}(s^{\prime})=\Delta(s^{\prime},s)$

14 end for

15

// Traceback past $\Gamma^{s}_{j}(s^{\prime})$ in (31)

16 for  _$j=i-1,i-2,...,i-\delta$_ do

17 for  _$s^{\prime\prime}\in{\mathcal{S}}$_ do

18
$\Gamma^{s}_{j}(s^{\prime\prime})=\min\limits_{s^{\prime}\in\mathcal{S}}[\Gamma^{s^{\prime}}_{j}(s^{\prime\prime})+\Delta(s^{\prime},s)]$

19 end for

20

21 end for

22

23 end for

24

25 end for

26

27$s^{\star}=\mathop{\mathrm{argmin}}\limits_{s\in\mathcal{S}}\left(\tilde{\xi}_{N+1}(s)\right)$

28 for  _$i=1,2,...,N$_ do

29 for  _$s^{\prime}\in\\{l,r\\}$_ do

// Estimate DFE-3 LLRs $\Gamma_{i}(s^{\prime})$ with the smallest path metric

30
$\Gamma_{i}(s^{\prime})\approx\Gamma^{s^{\star}}_{i}(s^{\prime})-\Gamma^{s^{\star}}_{i}(c)$

31 end for

32

33 end for

34

35return _$\boldsymbol{\Gamma}$_

Algorithm 3 DFE-3 SOVA

For the DFE-$3$ trellis in Fig. 6, at each time, every state is connected with
the previous states through edges. The path with the smallest path metric is
chosen as the surviving path for the present terminating state. Let the path
metric of the surviving path that terminates in $s_{i+1}=s$ be denoted as
$\tilde{\xi}_{i+1}(s)$. SOVA calculates $\tilde{\xi}_{i+1}(s)$ in the same way
as MLM calculates $\tilde{\alpha}_{i+1}(s)$ in (26), but with opposite signs
for indicating the difference between trellis branches, i.e.,

$\tilde{\xi}_{i+1}(s)=\min\limits_{s^{\prime}\in\mathcal{S}}\left[\tilde{\xi}_{i}(s^{\prime})-\tilde{\gamma}_{i+1}(s^{\prime},s)\right].$
(29)

Similarly, the initialization of $\tilde{\xi}_{1}(s)$ is done as that of
$\tilde{\alpha}_{1}(s)$ in Algorithm 2, except with the opposite sign, i.e.,
$[\tilde{\alpha}_{1}(l),\tilde{\alpha}_{1}(c),\tilde{\alpha}_{1}(r)]\\!=[-\infty,\\!0,\\!-\infty]$.

Then, for the terminating state $s_{i+1}=s$, SOVA approximates the reliability
of state $s_{i}=s^{\prime}$, i.e. $\Gamma^{s}_{i}(s^{\prime})$, to be the
cumulative path metric differences between the competing paths and the current
surviving path, i.e.,

$\Gamma^{s}_{i}(s^{\prime})=\Delta(s^{\prime},s)\triangleq\underbrace{\tilde{\xi}_{i}(s^{\prime})-\tilde{\gamma}_{i+1}(s^{\prime},s)}_{\text{competing
path}}-\underbrace{\tilde{\xi}_{i+1}(s)}_{\text{surviving path}}.$ (30)

It can be seen that $\Gamma^{s}_{i}(s^{\prime})$ consists of $3$ sets of
reliability of $s_{i}=s^{\prime}$, each accounts for a terminating state
$s_{i+1}=s$.

The past reliabilities were computed based on the metric differences from the
previous surviving path, which is not necessarily the final surviving path.
Since the present surviving path has been extended, these past DFE-3 LLRs need
an update. SOVA looks back to update a limited range of the recent DFE-3 LLRs.
The update is done via [24, Eq. (6)]

$\Gamma^{s}_{j}(s^{\prime\prime})=\min\limits_{s^{\prime}\in\mathcal{S}}[\Gamma^{s^{\prime}}_{j}(s^{\prime\prime})+\Delta(s^{\prime},s)].$
(31)

Eventually, when reaching the end of the trellis, we have $3$ surviving paths
for each terminating state. The final surviving path is the one giving the
smallest path metric, whose terminating state is denoted as $s^{\star}$. Then,
the DFE-$3$ LLRs $\Gamma_{i}(s^{\prime})$ are determined based on the
$\Gamma^{s^{\star}}_{i}(s^{\prime})$ belonging to the final surviving path.
The DFE-$3$ SOVA is given in Algorithm 3. Note that Algorithm 3 is written
such that it outputs the DFE-3 state LLRs for the whole frame. However, to
achieve low latency, it is possible to output LLRs continuously once they are
out of the traceback window.

###  III-E DFE State Soft Demapping

The proposed state demapper111The idea of the state demapper originates from
our previous work in [30]. However, the state demapper in [30] uses
$P(s_{i}=s|\overline{\boldsymbol{y}}_{1}^{i-1})$ instead of
$P(s_{i}=s|\overline{\boldsymbol{y}})$.
$P(s_{i}=s|\overline{\boldsymbol{y}}_{1}^{i-1})$ is obtained by normalizing
$\alpha_{i}(s^{\prime})$ in (16). computes bit-wise LLRs $L_{i,j}$ after MLM
or SOVA, as shown in Fig. 5. The reliabilities of DFE-$3$ states are used as
weighting factors to account for the bias imposed on the $\overline{y}_{i}$.
As a result, the LLR accuracy is improved and thus benefits the SD-FEC
decoding. The state demapper _approximates_ the bit-wise LLR in (13) with the
DFE-3 state information, i.e.,

$\displaystyle
L^{S}_{i,j}\\!\triangleq\\!\log\frac{\sum\limits_{s^{\prime}\in\mathcal{S}}\sum\limits_{x\in\mathcal{X}^{1}_{j}}\\!p(\overline{y}_{i}|s_{i}\\!=\\!s^{\prime}\\!,\\!x_{i}\\!=\\!x)\\!P(s_{i}\\!=\\!s^{\prime}|\overline{\boldsymbol{y}})}{\sum\limits_{s^{\prime}\in\mathcal{S}}\sum\limits_{x\in\mathcal{X}^{0}_{j}}p(\overline{y}_{i}|s_{i}\\!=\\!s^{\prime}\\!,\\!x_{i}\\!=\\!x)\\!P(s_{i}\\!=\\!s^{\prime}|\overline{\boldsymbol{y}})},$
(32)

where $p(\overline{y}_{i}|s_{i},x_{i})$ has been discussed in Sec.III-B. In
(13), the effects of all biased states (HD errors) have been implicitly taken
into account in $\Lambda_{i}(x)$ from the PAM-$M$ trellis. By contrast, in
(32), only the information of $3$ important biased states
$P(s_{i}\\!=\\!s^{\prime}|\overline{\boldsymbol{y}})$ are used to compensate
the effects.

In practice, given $\overline{y}_{i}$ and $\Gamma_{i}(s^{\prime})$, the LLRs
are calculated with the ML approximation as

$\displaystyle L^{S}_{i,j}=$
$\displaystyle\max\limits_{s^{\prime}\in\mathcal{S}}\left[\Gamma_{i}(s^{\prime})\\!-\\!\frac{\left.\min\limits_{x\in\mathcal{X}_{k}^{1}}\left[(\overline{y}_{i}\\!-\\!x\\!-\\!h\omega(s^{\prime}))^{2}\right]\right.}{2\sigma^{2}}\right]$
$\displaystyle\\!-\\!\max\limits_{s^{\prime}\in\mathcal{S}}\left[\Gamma_{i}(s^{\prime})\\!-\\!\frac{\left.\min\limits_{x\in\mathcal{X}_{k}^{0}}\left[(\overline{y}_{i}\\!-\\!x\\!-\\!h\omega(s^{\prime}))^{2}\right]\right.}{2\sigma^{2}}\right].$
(33)

The derivation of (III-E) is given in Appendix A.

(a) PAM-$4$ at SNR of $15$ dB.  (b) PAM-$8$ at SNR of $20.5$ dB.

Figure 8: Pre-FEC BER vs. the factor $b$ for multiple candidate factors $a$.

##  IV Results

Figure 9: Simulation results for PAM-$4$ and PAM-$8$ transmissions: (a) Pre-
FEC BER vs. SNR; (b) Post-FEC BER vs. SNR.

###  IV-A Simulation Setup

Based on the effective PR channel estimated in short-reach DCI applications,
we consider $h=0.7$ in (1) to simulate the transmission of PAM-$4$ and
PAM-$8$. The LDPC code from IEEE 802.3ca standard with codeword length of
$17664$ and code rate of $R_{c}\approx 0.83$ is employed. The SD decoder
performs belief propagation (BP) with $6$ iterations. Transmission in the AWGN
channel is simulated and used as a baseline comparison. For each considered
scheme, at least $500$ codewords are transmitted and at least $10^{4}$ bit
errors are accumulated. To make a fair comparison between the AWGN and the PR
channel transmissions, assuming unit signal power, the SNR is defined as
$(1+h^{2})/\sigma^{2}$ for the PR channels [35, Eq. (52)], and $1/\sigma^{2}$
for the AWGN channels (since $h=0$)222Note that in [30], we used the AWGN SNR
($1/\sigma^{2}$) for the PR channel..

We focus on the performance comparison between the traditional and the
proposed schemes. The proposed schemes in Fig. 5 are referred to as _DFE- $3$
LM/MLM/SOVA_. They require DFE, one of the aforementioned trellis detection
algorithms, and the state demapper. The traditional schemes have been depicted
previously in Fig. 1 and include

  * •

_DFE_ : DFE plus AWGN demapper

  * •

_WDFE_ : WDFE plus AWGN demapper

  * •

_DFE/WDFE Int._ : _DFE_ or _WDFE_ scheme plus a random bit-interleaver (the
same interleaver permutation is used for every 4 consecutive codewords).

  * •

_DFE Prec._ : _DFE_ in combination with precoding.

  * •

_PAM- $M$ LM/MLM/SOVA_: Trellis-based detection algorithms plus bit-wise
demapper.

For PAM-$M$/DFE-$3$ SOVA, we set traceback length $\delta=10$. In addition, to
indicate the SD decoding gains, DFE or MLSD with HD decoding will be
presented. Note that for the HD decoding of precoding, DFE, and MLSD, the
demapper uses a lookup table to map each one of the $M$ possible de-precoding
outputs into $m$ constant bit-wise LLRs. This table of LLR values is
numerically optimized by estimating the crossover probabilities of the
equivalent symmetric channel [36, Ch. 3.4]. Regarding WDFE, the compression
parameters $a$ and $b$ of the sigmoid function in (11) are optimized from a
set of candidate values to yield the lowest pre-FEC BER [20]. As shown in Fig.
8, ($a=3$, $b=0.25$) and ($a=3$, $b=0.3$) are chosen for PAM-$4$/$8$
transmissions, resp.

###  IV-B Simulation Results

Firstly, the equalizing capability of different schemes is demonstrated in
Fig. 9(a) via pre-FEC BER vs. SNR performance. The results for PAM-$4$ and
PAM-$8$ transmissions are in two clusters. Within the PAM-$4$ cluster, non-
trellis-based and trellis-based detection schemes are distinguishable. Among
the trellis-based schemes in the left inset, PAM-$4$ LM/MLM/SOVA (dashed
lines) correct the most symbol errors, leaving a gap of $0.41$ dB w.r.t. the
DFE-$3$ counterparts (solid lines). The DFE-$3$ schemes are followed by DFE
with precoding, with a $0.44$ dB penalty. Weighting the decision reliability
improves the DFE performance by $0.3$ dB, allowing WDFE to approach the
performance of DFE with precoding. Interleaving only breaks the correlation
between errors, and thus, the curves for DFE with and without interleaving
overlap.

Comparing the results of PAM-$8$ to PAM-$4$, the following changes are
observed: (i) higher SNR is required to achieve the same BER due to increased
noise sensitivity. (ii) Since PAM-$8$ cause severer EP than PAM-$4$, weighting
and precoding improves the pre-FEC BER of DFE more significantly under PAM-$8$
transmission than PAM-$4$, especially for precoding. (iii) Among the trellis-
based schemes, the difference between DFE-$3$ and PAM-$8$ schemes is notably
smaller. For example, the right inset shows that PAM-$8$ MLM provides $0.23$
dB gain over DFE-$3$ MLM, while this number for PAM-$4$ is $0.41$ dB. The
performance penalty caused by state pruning reduces as the modulation order
increases from PAM-$4$ to PAM-$8$, which has also been previously reported in
[28, Fig. 8]. The intuition behind this reduced penalty is that the
performance of the trellis-based schemes deteriorates for an increased number
of states due to inherent quasi-catastrophic EP [37, Sec. IV-A], and thus,
PAM-$8$ suffers from more degradation than DFE-$3$. (iv) SOVA (green curves)
is unable to approach the performance of LM and MLM, regardless of PAM-$8$ or
DFE-$3$ states. The performance degradation of SOVA implies that using
$\delta=10$ is insufficient for PAM-$8$. The reason is that on a large
trellis, the paths diverge more easily, and it hampers the accurate update of
symbol LLRs within a short window. (v) DFE with precoding outperforms DFE-$3$
SOVA, due to the fact that burst errors become longer and also appear more
frequently for PAM-$8$ transmission.

Fig. 9(b) shows end-to-end results of post-FEC BER. Large gaps, around $2$ to
$3$ dB, are observed between the AWGN capacity (blue vertical lines) and the
performances in the AWGN channel (solid blue lines) and the PR channel (other
lines). Among the non-trellis-based schemes, WDFE exhibits the best
performance. For DFE and WDFE, interleaving yields slight performance
improvements. Although precoding provides good pre-FEC BER as has been shown
in Fig. 9(a), its post-FEC BER is penalized due to the use of LLR lookup
table, and this HD decoding penalty even causes the worst BER results in the
PAM-$4$ transmission. However, in the presence of severer EP induced by
PAM-$8$, such penalty is compensated more by the increased precoding gains in
eliminating burst errors. Nonetheless, the trellis-based schemes can still
offer about $1.5$ dB additional gains over precoding. Compared to the DFE HD
curves, it can be seen taht using SD decoding on DFE brings $0.56$ dB and
$0.58$ dB gains for PAM-$4$/$8$, resp.

Furthermore, Fig. 9(b) shows that the trellis-based schemes outperform WDFE
schemes by more than $0.8$ dB for both PAM-$4$ and PAM-$8$ transmissions.
Comparing the two insets of the trellis-based schemes, we observe that the
penalty due to the use of DFE-$3$ states decreases to around $0.3$ dB for the
PAM-$8$ transmission. Especially for PAM-$8$ SOVA (green lines), the right
inset shows that the SNR is reduced by $0.18$ dB compared to DFE-$3$ SOVA (at
a BER of $10^{-6}$). We also see that for PAM-$8$ transmission, the gap
between LM and MLM is $0.07$ dB, which is wider than that for PAM-$4$
transmission, yet still very small. The widened gap is intuitively explained
by the fact that the ML used in MLM is more accurate for PAM-$4$ than PAM-$8$
modulation format [38, Corollary 1]. The soft coding gains for VA based
algorithms is obtained by comparing the results of MLSD and PAM-$M$ SOVA,
which are $1.18$ dB and $0.96$ dB for PAM-$4$/$8$, resp.

Figure 10:  Post-FEC BER vs. NGMI of the considered schemes for the simulated
PAM-$M$ ($M=4$ or $8$) transmissions.  Figure 11: Block diagrams of PAM-$4$
IM/DD experiment and offline processing. The dashed pink blocks indicate
optional DSPs, depending on the considered schemes. ECL: external cavity
laser; DAC: digital-to-analog converter; MZM: Mach-Zehnder modulator; SSMF:
standard single-mode fiber; VOA: variable optical attenuator; PD: photo diode;
OSC: oscilloscope.

Fig. 10 compares the LLR-based normalized generalized mutual information
(NGMI) [39, Sec. III-F] with the post-FEC BER of various schemes. The schemes
using HD decoding are excluded. Depending on the considered schemes, the post-
FEC BER is achieved at different NGMI than that of the AWGN channel, since
LLRs are mismatched to different extents [39, Sec. II-B]. The modulations
PAM-$4$ and PAM-$8$ have no impacts on the results, whose curves for each
scheme overlap each other. Specifically, all the trellis-based schemes,
achieve the same BER at the almost identical NGMI higher than that of AWGN
channel. By contrast, at the same BER, DFE produces LLRs with less NGMI than
trellis-based schemes, and similar results have also been reported in [6].
Compared to DFE, WDFE improves the LLR quality and thus increases the NGMI. In
addition, applying interleaver to DFE and WDFE does not change the LLR values
but breaks their correlations, and thus lower BER is achieved at the same
NGMI.

###  IV-C PAM-$4$ Experimental Results

The performances of the trellis-based schemes have been evaluated in an
experiment for PAM-$4$ transmission333Part of these results were presented in
ECOC 2023, Glasgow, Scotland.. The offline processing and experimental setup
are depicted in Fig. 11. The experiment in Fig. 11 targets short-reach 100G
serial links, operating at a rate of 53.125 GBaud and covering a fiber
transmission distance of 500 m. The memory size of the DSP chip for storing
received symbols is 16 Kbytes. In addition, the testbed suffers from high
insertion loss caused by lossy/long electrical traces (e.g., between DAC and
driver) and bandwidth limitation. At the receiver, noise whitening and FFE are
performed. For the resulting effective channel, the first tap coefficient
dominates with a peak magnitude of $0.66$, while the other taps have
significantly smaller magnitude $<0.02$. Therefore, the effective channel can
be approximated as a one-tap PR channel as in (1) with $h=0.66$, where the
discrepancy mainly lies in the residual ISI due to the use of finite taps.

In the offline generation, $2^{15}$ bits from a pseudo-random bit sequence
(PRBS) are mapped to PAM-$4$ symbols for transmission. The received symbols
are stored for offline evaluation. The offline evaluation of the pre/post-FEC
BER is based on an emulation approach [40, 41]. In the upper left corner of
Fig. 11, scrambling (achieved by XOR operations) and re-encoding are conducted
to convert the PRBS-generated codeword into a real codeword. For each
considered scheme, at least $500$ codewords are transmitted. Given the limited
memory size, at most $7525$ codewords can be emulated by applying a sliding
window to the stored symbols. Considering only single tap coefficient
$h=0.66$, the noise variance after addressing the one-tap ISI is estimated in
the electrical domain as
$\sigma^{2}=\mathrm{Var}\left[y_{i}-x_{i}-hx_{i-1}\right]$. To sweep different
SNR values, noise loading is used as shown in Fig. 11, where extra offline-
generated noise $\tilde{n}_{i}$ is added.

Compared to the numerical analysis in the previous section, the traditional
schemes under consideration are changed. First, to address the effect of
residual ISI, the multi-tap DFE and the EP-free DFE are performed. The EP-free
DFE is a genie-aided approach, in which case a DFE always makes correct
decisions such that all ISI covered by the feedback loops have been removed
[42]. Second, the performance of WDFE will not be presented. The reason is
that the numerical optimization shows that within our experimental setup, the
optimal performance of WDFE is achieved only when it approaches DFE with $b=0$
(see (11)). This phenomenon has also been reported in [18].

Fig. 12(a) shows the pre-FEC BER vs. SNR performance. The 1000-tap EP-free DFE
exhibits the best pre-FEC BER. However, with the penalty of EP, the
performance improvement by increasing the DFE taps becomes marginal [6].
Considerable gains are only observed with a large number of taps, as
illustrated by the $50$-tap and the $1000$-tap DFE in Fig. 12(a). Compared to
the simulated PAM-$4$ transmission results in Fig. 9, due to the presence of
residual ISI, error floors are observed for DFE schemes. The advantage of
precoding in correcting burst errors is fully leveraged, and thus DFE with
precoding even outperforms trellis-based schemes at SNR lower than $15.2$ dB.
Regarding the trellis-based schemes, using the PAM-$4$ trellis corrects more
errors than the DFE-$3$ trellis. Surprisingly, LM exhibits worse performance
than MLM and SOVA for both trellises. We conjecture that LM is more sensitive
than MLM and SOVA to the presence of residual ISI, because LM follows more
strictly the AWGN assumption on the additive noise. The fact that LM is more
complex, but ends up with worse performance, makes LM less attractive in real
IM/DD systems than MLM and SOVA.

Figure 12:  PAM-4 experimental results of: (a) Pre-FEC BER vs. SNR; (b) Post-
FEC BER vs. SNR. Figure 13:  Post-FEC BER vs. NGMI of the considered schemes
for the PAM-$4$ experimental results.

Fig. 12(b) shows the end-to-end results of post-FEC BER. The 1000-tap EP-free
DFE outperforms other schemes at least by $0.97$ dB. However, with the penalty
of EP, the one/multi-tap DFEs exhibit poor performances. Naturally, the one-
tap DFE using HD decoding provides the highest BER. Even if the tap number is
increased to 1000, DFE cannot reach the BER of $10^{-5}$. In addition,
interleaving brings negligible performance improvement. Although DFE with
precoding and MLSD yield good pre-FEC BER, as shown in Fig. 12(a), its post-
FEC BER is limited due to the use of discrete LLRs for HD decoding. On the
other hand, Fig. 12(b) shows that the trellis-based solutions enables better
SD decoding performances. Their BER curves are grouped into two clusters,
representing the use of PAM-$4$ and the DFE-$3$ states, resp. MLM slightly
outperforms LM, and a gain of $0.09$ dB is observed within the PAM-$4$
cluster. The smallest gap between the DFE-$3$ and the PAM-$4$ clusters is
$0.39$ dB. In particular, using the DFE-$3$ states in SOVA leads to a loss of
$0.48$ dB.

Fig. 13 shows the NGMI vs. post-FEC BER. Compared to Fig. 10, if we treat EP-
free DFE as the case of AWGN channel transmission, it can be seen that they
follow approximately the same trends. On the other hand, Fig. 13 shows that
DFE schemes exhibit much worse performance in the experiment to such an extent
that even employing 1000 taps fails to improve the NGMI to $0.85$.

###  IV-D Complexity Analysis

Applying DFE-$3$ states to the trellis-based algorithms offer a big advantage
of complexity reduction. In this section, a rough complexity analysis of MLM
and SOVA schemes using DFE-$3$ or PAM-$M$ states is presented. We ignore LM in
this comparison because MLM is superior to LM in terms of complexity and
performance in the experiment, as discussed in the previous section. MLM and
SOVA extensively rely on compare and select (CS) operations, making them more
computationally intensive than VA which only requires additions and
multiplications [33, 43]. The CS operations correspond to $\max(\cdot)$ and
$\min(\cdot)$ in Algorithms 2-3 and (III-E). Therefore, this complexity
analysis focuses on the number of CS operations. For the AWGN/state demapper
modules employed in Fig. 1(a) and Fig. 5, and the symbol-to-bit LLR converter
in Fig. 1(b), ML is used. To make a fair comparison, the CS required inside
these modules and DFE are taken into account as well. Furthermore, the number
of input elements dominates the complexity of a CS operation and is hence used
as a scaling factor for counting. For example, $\max(a_{1},a_{2},...,a_{n})$
is counted as $n$ CS operations.

Fig. 14 shows a comparison of the CS operations required per LDPC codeword. It
can be seen that using DFE-$3$ offers considerable complexity reduction for
SOVA, mainly because the number of CS is related to the square of the number
of states, as can be found in Algorithm 3. By contrast, the complexity
reduction is less notable for MLM, in which case the number of CS grows
linearly with the number of states as shown in Algorithm 2. Therefore, the
complexity reduction, achieved by decreasing the number of states from $M$ to
$3$, is more pronounced, especially for a higher-order $M$. Specifically, the
number of CS operations required for the DFE-$3$ SOVA is only $6.32\%$ of the
number of CS operations needed for the PAM-$8$ SOVA.

Figure 14: Complexity comparison in terms of compare and select operations
between MLM and SOVA ($\delta=10$) when performed on PAM-$M$ or DFE-$3$ state
machine. Note that the y-axis is on a log scale.

##  V Conclusions

In this paper, to improve the decoding of SD-FEC in one-tap PR channels, a
scheme incorporating DFE and the symbol-wise soft-output detection algorithm
was proposed. We showed that thanks to the pre-processing of DFE, MLM or SOVA
only needs to track a trellis with $3$ states associated with a reduced set of
DFE symbol errors. The soft information of the symbol errors and the equalized
symbols should then be combined to generate bit-wise LLRs. Since the number of
states decreases from the number of constellation points to $3$, the
complexity of MLM or SOVA is reduced. In particular, the number of compare and
select operations for SOVA is reduced by $54\%$ and $94\%$ for PAM-$4$ and
PAM-$8$, resp. The complexity reduction comes at the cost of some performance
degradation because only a reduced-state trellis is used for the DFE output.
Nonetheless, the numerical results of SOVA show that the proposed scheme
causes an SNR loss of $0.42$ dB for PAM-$4$ transmission and only $0.18$ dB
for PAM-$8$ transmission, and the PAM-$4$ experimental results show that the
loss increases slightly to $0.48$ dB. Overall, the scheme is believed to reach
a good trade-off between performance and complexity, which is of interest to
future high-speed IM/DD links.

Furthermore, an important implication can be drawn from the experimental
results: LM is less favored for real-world applications with a large amount of
residual ISI, due to its increased complexity and inferior performance
compared to its simplified counterpart, MLM.

Although the proposed scheme specifically targets the one-tap PR channels in
this paper, it can be extended to cope with multiple taps. In the general case
of PAM-$M$ transmissions in multi-tap ($L>1$) PR channels, the main challenge
would be to generalize the rule of identifying the most relevant $3^{L}$ DFE
hard decision errors, and associate them with different states out of the
$M^{L}$-state trellis. This research topic is left for our future work.

Furthermore, there are other research topics that are worth investigating,
including: (i) the performance and complexity optimization of the state
demapper, and (ii) the hardware implementation of the proposed scheme. (iii)
Combining the proposed schemes with probabilistic shaping using reverse
Maxwell-Boltzmann distribution [31], which enhances the complexity reduction
due to more frequent outermost symbols.

##  Appendix A Derivation of State LLR in (III-E)

In view of (19), (III-B) and (21), $p(\overline{y}_{i}|s_{i},x_{i})$ and
$P(s_{i}|\overline{\boldsymbol{y}})$ can be expressed as exponential functions
and inserted in (32), yielding

$L^{S}_{i,j}=\log\frac{\sum\limits_{s^{\prime}\in\mathcal{S},x\in\mathcal{X}^{1}_{j}}\exp\left(\frac{-(\overline{y}_{i}\\!-\\!x\\!-\\!h\omega(s^{\prime}))^{2}}{2\sigma^{2}}+\Gamma_{i}(s^{\prime})\right)}{\sum\limits_{s^{\prime}\in\mathcal{S},x\in\mathcal{X}^{0}_{j}}\exp\left(\frac{-(\overline{y}_{i}\\!-\\!x\\!-\\!h\omega(s^{\prime}))^{2}}{2\sigma^{2}}+\Gamma_{i}(s^{\prime})\right)},$
(34)

By applying the ML approximation to the numerator/denominator in (34), we have

$\displaystyle L^{S}_{i,j}=$
$\displaystyle\max\limits_{\begin{subarray}{c}s^{\prime}\in\mathcal{S}\\\
x\in\mathcal{X}^{1}_{j}\end{subarray}}\left[\frac{-\left(\overline{y}_{i}\\!-\\!x\\!-\\!h\omega(s^{\prime})\right)^{2}}{2\sigma^{2}}+\Gamma_{i}(s^{\prime})\\!\right]$
$\displaystyle\\!-\\!\max\limits_{\begin{subarray}{c}s^{\prime}\in\mathcal{S}\\\
x\in\mathcal{X}^{0}_{j}\end{subarray}}\left[\frac{-\left(\overline{y}_{i}\\!-\\!x\\!-\\!h\omega(s^{\prime})\right)^{2}}{2\sigma^{2}}+\Gamma_{i}(s^{\prime})\\!\right]$
(35) $\displaystyle=$
$\displaystyle\max\limits_{s^{\prime}\in\mathcal{S}}\\!\left[\max\limits_{x\in\mathcal{X}^{1}_{j}}\\!\left[\frac{-\left(\overline{y}_{i}\\!-\\!x\\!-\\!h\omega(s^{\prime})\right)^{2}}{2\sigma^{2}}\right]+\Gamma_{i}(s^{\prime})\\!\right]$
$\displaystyle-\max\limits_{s^{\prime}\in\mathcal{S}}\\!\left[\max\limits_{x\in\mathcal{X}^{0}_{j}}\\!\left[\frac{-\left(\overline{y}_{i}\\!-\\!x\\!-\\!h\omega(s^{\prime})\right)^{2}}{2\sigma^{2}}\right]+\Gamma_{i}(s^{\prime})\\!\right].$
(36)

After lifting the negative sign out of the max functions over
$x\in\mathcal{X}^{b}_{j}$ in (36), (III-E) is obtained.

## References

  * [1] K. Zhong, X. Zhou, J. Huo, C. Yu, C. Lu, and A. P. T. Lau, “Digital signal processing for short-reach optical communications: A review of current technologies and future trends,” _J. Lightw. Technol._ , vol. 36, no. 2, pp. 377–400, Jan. 2018.
  * [2] R. Nagarajan, I. Lyubomirsky, and O. Agazzi, “Low power DSP-based transceivers for data center optical fiber communications (Invited tutorial),” _J. Lightw. Technol._ , vol. 39, no. 16, pp. 5221–5231, Aug. 2021.
  * [3] V. Gaudet, “A survey and tutorial on contemporary aspects of multiple-valued logic and its application to microelectronic circuits,” _IEEE J. Emerg. Sel. Topics in Circuits and Sys._ , vol. 6, no. 1, pp. 5–12, Mar. 2016.
  * [4] L. Patra, A. Farhood, R. Radhamohan, W. Bliss, S. Ramesh, and D. Cassan. (Mar. 2023) IEEE P802.3dj task force: FEC baseline proposal for 200Gb/s per lane IM-DD optical PMDs. [Online]. Available: https://www.ieee802.org/3/dj/public/23_03/patra_3dj_01b_2303.pdf
  * [5] M. Schaedler, G. Böcherer, F. Pittalà, S. Calabrò, N. Stojanovic, C. Bluemm, M. Kuschnerov, and S. Pachnicke, “Recurrent neural network soft-demapping for nonlinear ISI in 800Gbit/s DWDM coherent optical transmissions,” _J. Lightw. Technol._ , vol. 39, no. 16, pp. 5278–5286, Aug. 2021.
  * [6] N. Stojanović, L. Youxi, T. Rahman, and S. Calabrò, “DFE and BCJR performance with SD FEC in 112 GBd PAM4 IMDD systems,” in _Proc. Opt. Fiber Commun. Conf._ San Diego, CA, USA, Mar. 2023.
  * [7] A. Mahadevan, D. van Veen, N. Kaneda, A. Duque, A. d. L. van Wijngaarden, and V. Houtsma, “Hard-input FEC evaluation using Markov models for equalization-induced correlated errors in 50G PON,” _J. Opt. Commun. Netw._ , vol. 13, no. 1, pp. A100–A110, Dec. 2020.
  * [8] R. Narasimha, N. Warke, and N. Shanbhag, “Impact of DFE error propagation on FEC-based high-speed I/O links,” in _Proc. IEEE Global Telecommun. Conf._ Honolulu, HI, USA, Nov./Dec. 2009\.
  * [9] M. Yang, S. Shahramian, H. Shakiba, H. Wong, P. Krotnev, and A. C. Carusone, “Statistical BER analysis of wireline links with non-binary linear block codes subject to DFE error propagation,” _Trans. Circuits Syst. I, Reg. Papers_ , vol. 67, no. 1, pp. 284–297, Jan. 2020.
  * [10] K. Kim, P. Kwon, and E. Alon, “Accurate statistical BER analysis of DFE error propagation in the presence of residual ISI,” _Trans. Circuits Syst. II, Exp. Brief_ , vol. 69, no. 2, pp. 619–623, Feb. 2022.
  * [11] M. Russel and J. W. M. Bergmans, “A technique to reduce error propagation in M-ary decision feedback equalization,” _IEEE Trans. Commun._ , vol. 43, no. 12, p. 2878, Dec. 1995.
  * [12] J. Riani, B. Smith, A. Farhood, and L. Patra. (Mar. 2023) IEEE P802.3dj task force: Impact of burst errors on concatenated FEC scheme. [Online]. Available: https://www.ieee802.org/3/dj/public/23_03/riani_3dj_01a_2303.pdf
  * [13] “Standard for ethernet amendment 9: physical layer specifications and management parameters for 25 Gb/s and 50 Gb/s passive optical networks,” _IEEE P802.3ca_ , Piscataway, NJ, USA, 2020.
  * [14] A. Mahadevan, Y. Lefevre, W. Lanneer, P. Cautereels, D. van Veen, N. Kaneda, and V. Houtsma, “Impact of DFE on soft-input LDPC decoding for 50G PON,” in _Proc. Opt. Fiber Commun. Conf._ San Francisco, CA, USA, Mar. 2021.
  * [15] M. Jana, A. Medra, L. Lampe, and J. Mitra, “Pre-equalized faster-than-Nyquist transmission,” _IEEE Trans. Commun._ , vol. 65, no. 10, pp. 4406–4418, Oct. 2017.
  * [16] J. Palicot, “A weighted decision feedback equalizer with limited error propagation,” in _Proc. IEEE Int. Conf. Commun._ , vol. 1. New Orleans, LA, USA, Jun. 2000, pp. 377–381.
  * [17] J. Palicot and A. Goupil, “Performance analysis of the weighted decision feedback equalizer,” _Signal Process._ , vol. 88, no. 2, pp. 284–295, Feb. 2008.
  * [18] T. Wettlin, T. Rahman, S. Calabrò, M. S.-B. Hossain, J. Wei, N. Stojanovic, and S. Pachnicke, “Investigation of weighted DFE for high-rate short-reach transmission,” in _Proc. 27th Opto-Electron. Commun. Conf._ Toyama, Japan, Jul. 2022.
  * [19] J. Zhou, C. Yang, D. Wang, Q. Sui, H. Wang, S. Gao, Y. Feng, W. Liu, Y. Yan, J. Li, , C. Yu, and Z. Li, “Burst-error-propagation suppression for decision-feedback equalizer in field-trial submarine fiber-optic communications,” _J. Lightw. Technol._ , vol. 39, no. 14, pp. 4601–4606, Jul. 2021.
  * [20] J. Zhang, X. Wu, L. Sun, J. Liu, A. P. T. Lau, C. Guo, S. Yu, and C. Lu, “C-band 120-Gb/s PAM-4 transmission over 50-km SSMF with improved weighted decision-feedback equalizer,” _Opt. Exp._ , vol. 29, no. 25, pp. 41 622–41 633, Dec. 2021.
  * [21] L. Bahl, J. Cocke, F. Jelinek, and J. Raviv, “Optimal decoding of linear codes for minimizing symbol error rate,” _IEEE Trans. Info. Theory_ , vol. 20, no. 2, pp. 284–287, Mar. 1974.
  * [22] P. Robertson, P. Hoeher, and E. Villebrun, “Optimal and sub-optimal maximum a posteriori algorithms suitable for turbo decoding,” _Eur. Trans. Telecommun._ , vol. 8, no. 2, pp. 119–125, Mar. 1997.
  * [23] J. Hagenauer and P. Hoeher, “A Viterbi algorithm with soft-decision outputs and its applications,” in _Proc. IEEE Global Telecommun. Conf. Exhib. ‘Commun. Technol. Beyond’_. Dallas, TX, USA, Nov. 1989.
  * [24] L. Gong, W. Xiaofu, and Y. Xiaoxin, “On SOVA for nonbinary codes,” _IEEE Commun. Let._ , vol. 3, no. 12, pp. 335–337, Dec. 1999.
  * [25] M. P. Fossorier, F. Burkert, S. Lin, and J. Hagenauer, “On the equivalence between SOVA and max-log-MAP decodings,” _IEEE Commun. Lett._ , vol. 2, no. 5, pp. 137–139, May 1998.
  * [26] M. V. Eyuboglu and S. U. Qureshi, “Reduced-state sequence estimation with set partitioning and decision feedback,” _IEEE Trans. Commun._ , vol. 36, no. 1, pp. 13–20, Jan. 1988.
  * [27] M. Visintin, P. Poggiolini, and G. Bosco, “Long-haul optically uncompensated IMDD transmission with MLSE using the M-method,” _IEEE Photon. Technol. Lett._ , vol. 19, no. 16, pp. 1230–1232, Aug. 2007.
  * [28] Y. Yu, Y. Che, T. Bo, D. Kim, and H. Kim, “Reduced-state MLSE for an IM/DD system using PAM modulation,” _Opt. Exp._ , vol. 28, no. 26, pp. 38 505–38 515, Dec. 2020.
  * [29] J. Zhou, J. Zhang, X. Zhao, T. Jin, S. Hu, H. Lin, Y. Zhenming, Q. Zhang, B. Xu, and K. Qiu, “Advanced nonlinearity equalizer with TC-NL-MLSE for transmitting beyond 200-Gb/s PAM-8 in IM/DD systems,” _Opt. Exp._ , vol. 30, no. 21, pp. 37 416–37 425, Oct. 2022.
  * [30] K. Wu, G. Liga, J. Lee, L. Paulissen, J. Riani, and A. Alvarado, “DFE state-tracking demapper for soft-input FEC in 800G data center interconnects,” in _Proc. Eur. Conf. Opt. Commun._ Basel, Switzerland, Sep. 2022.
  * [31] D. Che, J. Cho, and X. Chen, “Does probabilistic constellation shaping benefit IM-DD systems without optical amplifiers?” _J. Lightw. Technol._ , vol. 39, no. 15, pp. 4997–5007, Aug. 2021.
  * [32] F. J. Martin-Vega, F. Blanquez-Casado, F. J. López-Martínez, G. Gomez, and J. T. Entrambasaguas, “Further improvements in sova for high-throughput parallel turbo decoding,” _IEEE Commun. Lett._ , vol. 19, no. 1, pp. 6–9, Jan. 2015.
  * [33] P. Robertson, E. Villebrun, and P. Hoeher, “A comparison of optimal and sub-optimal map decoding algorithms operating in the log domain,” in _Proc. IEEE Int. Conf. Commun._ Seattle, WA, USA, Jun. 1995.
  * [34] W. Ryan and S. Lin, _Channel codes: classical and modern_. Cambridge University Press, 2009.
  * [35] G. Forney, “Maximum-likelihood sequence estimation of digital sequences in the presence of intersymbol interference,” _IEEE Trans. Inf. Theory_ , vol. 18, no. 3, pp. 363–378, 1972.
  * [36] L. Szczecinski and A. Alvarado, _Bit-interleaved coded modulation: fundamentals, analysis and design_. John Wiley & Sons, Feb. 2015.
  * [37] G. D. Forney and A. R. Calderbank, “Coset codes for partial response channels; or, coset codes with spectral nulls,” _IEEE Trans. Info. Theory_ , vol. 35, no. 5, pp. 925–943, Sep. 1989.
  * [38] M. Ivanov, C. Häger, F. Brännström, A. Graell i Amat, A. Alvarado, and E. Agrell, “On the information loss of the max-log approximation in BICM systems,” _IEEE Trans. Inf. Theory_ , vol. 62, no. 6, pp. 3011–3025, Jun. 2016.
  * [39] A. Alvarado, T. Fehenberger, B. Chen, and F. M. Willems, “Achievable information rates for fiber optics: Applications and computations,” _J. Lightw. Technol._ , vol. 36, no. 2, pp. 424–439, Jan. 2018.
  * [40] L. Schmalen, F. Buchali, A. Leven, and S. ten Brink, “A generic tool for assessing the soft-FEC performance in optical transmission experiments,” _IEEE Photon. Technol. Lett._ , vol. 24, no. 1, pp. 40–42, Jan. 2012.
  * [41] N. Stojanovic, Y. Zhao, D. Chang, Z. Xiao, and F. Yu, “Reusing common uncoded experimental data in performance estimation of different fec codes,” _IEEE Photon. Technol. Lett._ , vol. 25, no. 24, pp. 2494–2497, Dec. 2013\.
  * [42] X. Tang, Y. Qiao, Y.-W. Chen, Y. Lu, and G.-K. Chang, “Digital pre-and post-equalization for C-band 112-Gb/s PAM4 short-reach transport systems,” _J. Lightw. Technol._ , vol. 38, no. 17, pp. 4683–4690, Sep. 2020\.
  * [43] H. Y. Rha, S.-R. Moon, H.-S. Kang, S.-W. Lee, I.-K. Hwang, and J. K. Lee, “Low-complexity soft-decision Viterbi algorithm for IM/DD 56-Gb/s PAM-4 system,” _IEEE Photon. Technol. Lett._ , vol. 31, no. 5, pp. 361–364, Mar. 2019.