# Equalization in Dispersion-Managed Systems Using Learned Digital Back-
Propagation

Mohannad Abu-romoh  1,* Nelson Costa  2 Yves Jaouën  1 Antonio Napoli  3 João
Pedro  2,4 Bernhard Spinnler  3 and Mansoor Yousefi1  1Institut Polytechnique
de Paris, Télécom Paris, 19 Place Marguerite Perey, 91120 Palaiseau, France
2Infinera Unipessoal Lda, 2790-078 Carnaxide, Portugal
3Infinera, Munich, Germany
4Instituto de Telecomunicações, Instituto Superior Técnico, 1049-001 Lisboa,
Portugal<EMAIL_ADDRESS>
###### Abstract

In this paper, we investigate the use of the learned digital back-propagation
(LDBP) for equalizing dual-polarization fiber-optic transmission in
dispersion-managed (DM) links. LDBP is a deep neural network that optimizes
the parameters of DBP using the stochastic gradient descent. We evaluate DBP
and LDBP in a simulated WDM dual-polarization fiber transmission system
operating at the bitrate of 256 Gbit/s per channel, with a dispersion map
designed for a 2016 km link with 15% residual dispersion. Our results show
that in single-channel transmission, LDBP achieves an effective signal-to-
noise ratio improvement of 6.3 dB and 2.5 dB, respectively, over linear
equalization and DBP. In WDM transmission, the corresponding $Q$-factor gains
are 1.1 dB and 0.4 dB, respectively. Additionally, we conduct a complexity
analysis, which reveals that a frequency-domain implementation of LDBP and DBP
is more favorable in terms of complexity than the time-domain implementation.
These findings demonstrate the effectiveness of LDBP in mitigating the
nonlinear effects in DM fiber-optic transmission systems.

††journal: oe††articletype: Research Article

##  1 Introduction

Before the advent of the coherent detection in optical fiber communication
systems, chromatic dispersion (CD) was compensated in the optical domain
through in-line dispersion management. In such systems, an optical link
typically comprises a cascade of a single-mode fiber (SMF), followed by a
dual-stage Erbium-doped fiber amplifier (EDFA) with a mid-stage dispersion-
compensating fiber (DCF). In this scenario, the residual CD in the link forms
a dispersion map that requires careful optimization. These systems have been
utilized for transmitting data rates typically around 10 Gb/s per wavelength,
with a few instances of upgrades to 40 Gb/s[1].

With the introduction of the coherent receiver (RX), the accumulated CD can
now be fully compensated in the electrical domain using digital signal
processing (DSP). As a result, dispersion management is no longer a necessity,
leading to the evolution of optical links to non-dispersion-managed (NDM)
systems. However, commercial dispersion-managed (DM) links continue to be in
operation today, since upgrading deployed solutions to state-of-the-art
systems can be particularly costly, especially in submarine fiber links.
Furthermore, advances in nonlinear mitigation in recent decades may address
some of the limitations of the DM systems.

Coherent systems depend on DSP at the receiver (RX) to equalize transmission
impairments in the electrical domain [2]. While these systems effectively
compensate for linear channel impairments, nonlinearity mitigation remains
challenging. The pursuit of the nonlinearity compensation (NLC) and associated
DSP algorithms has led to the proposal of several equalization techniques to
mitigate nonlinear impairments [3, 4], including Volterra series-based
equalization [5] and digital back-propagation (DBP) [6, 7, 8, 9].

DBP compensates for deterministic nonlinear effects by digitally simulating
propagation at the receiver (RX) using negated fiber parameters [6]. However,
the effectiveness of DBP is hindered by both hardware limitations, which
impose restrictions on the complexity of DBP, and signal interactions between
adjacent channels. Thus, advanced equalization techniques are essential for
improving system performance and reducing complexity.

Research has focused on optimizing DBP algorithms for lower complexity or
better performance in various transmission scenarios [10, 11]. For instance,
channel parameters used in DBP were numerically optimized for DM and NDM
systems in [12]. Combining transmitter-side DBP with frequency referenced
carriers has been shown to double the reach of the transmission, as
demonstrated in [13]. Optimal DBP step sizes for polarization division
multiplexed transmission systems were studied in [14]. Furthermore, several
DBP variants have been proposed to reduce complexity or enhance performance,
such as correlated-DBP [15], which accounts for the correlation between
neighboring signal samples. Dispersion folded-DBP [16, 17, 18, 19] was first
proposed for zero-residual dispersion but later extended to any dispersion
map. Filtered DBP [20, 21] introduces a parameterized low-pass filter (LPF) in
the nonlinear step to improve phase tracking. Enhanced DBP [22, 23] is an
extension of FDBP that considers interactions between channel signals and
adjacent channels. Coupled-Channel Enhanced DBP [24] achieves optimal cross-
phase modulation (XPM) equalization in WDM systems. Recently, neural networks
(NNs) have been employed to enhance DBP. Learned-DBP (LDBP) [25, 26, 27, 28,
29] uses a deep NN inspired by the split-step Fourier method (SSFM)
computational graph, optimized using stochastic gradient descent (SGD). LDBP
treats the LPF filter taps in DBP as free parameters optimized by the NN. For
NDM systems, LDBP was extended to dual-polarization transmission [25] and
experimentally demonstrated with one layer per span (LpS) [26]. Generalized
DBP (GDBP) [30] employs a deep NN to parameterize DBP, specializing the system
by combining NN training with adaptive DSP.

The main objective of this paper is to investigate the potential of
repurposing legacy DM links for use in coherent transmission systems. We do
not claim that replacing NDM with DM systems is superior; rather, we aim to
explore the possibility of adapting the already-deployed DM systems that are
typically used for IM/DD to enable coherent transmission. To achieve this
goal, we use machine learning tools to optimize DBP, specifically the Learned-
DBP method. We start by developing a DBP variant that is suitable for DM
systems with a fractional number of steps per span (StpS), which can be
applied to DM links with arbitrary dispersion maps. The resulting DBP is then
used as the blueprint for LDBP, and we evaluate the performance of both DBP
and LDBP in a realistic long-haul dual-polarization WDM DM transmission system
with $M$-ary quadrature amplitude modulation ($M$-QAM), considering various
channel effects such as loss, CD, Kerr nonlinearity, polarization-mode
dispersion (PMD), amplified spontaneous emission (ASE) noise, and laser phase
noise (PN).

In single-channel transmission, LDBP improves the effective signal-to-noise
ratio (SNR${}_{\text{eff}}$) by 6.3 dB and 2.5 dB compared to linear
equalization (LE) and conventional DBP, respectively. In WDM transmission,
LDBP enhances the Q-factor by 1.1 dB and 0.4 dB compared to LE and DBP,
respectively. We present both time- and frequency-domain implementations of
LDBP (TD- and FD-LDBP) and examine the impact of fiber parameter variations
due to aging, as well as laser PN, on LDBP performance. Our analysis shows
that PN results in a Q-factor penalty of less than 0.2 dB when the laser
linewidth is below 200 kHz. Furthermore, we demonstrate that LDBP’s
performance gains over DBP are approximately maintained even when the model is
retrained with updated fiber parameters post-aging. This work extends the
findings in [29] by providing further valuable insights into improving data
rates in DM systems utilizing coherent detection and LDBP.

The paper is structured as follows: In Section II, we describe the optical
fiber signal propagation model. In Section III, we present the DM-adapted DBP,
while Section IV introduces its learned version. In Section V, we compare the
performance of LE, DBP, and LDBP in four different setups and provide a
complexity analysis in Section VI. Finally, Section VII offers concluding
remarks.

##  2 Dual-Polarization Optical Fiber System Model

Figure 1: The DM-WDM optical fiber system.

###  2.1 Transmitter and Channel

In the polarization-multiplexed fiber transmission, two random bit streams
with $N_{b}$ bits,
$\mathbf{b_{x}}=(b_{x}^{(1)},b_{x}^{(2)},\dots,b_{x}^{(N_{b})})$ and
$\mathbf{b_{y}}=(b_{y}^{(1)},b_{y}^{(2)},\dots,b_{y}^{(N_{b})})$,
$b_{x/y}^{(i)}\in\\{0,1\\}$, are generated at the TX, then each is mapped into
a sequence of $N_{s}$ symbols
$\mathbf{s_{x}}=(s_{x}^{(1)},s_{x}^{(2)},\dots,s_{x}^{(N_{s})})$ and
$\mathbf{s_{y}}=(s_{y}^{(1)},s_{y}^{(2)},\dots,s_{y}^{(N_{s})})$, where
$s_{x/y}^{(i)}$ are drawn from the constellation $\mathcal{S}$. The Gray
mapping is used to map the bit stream to the symbols in the constellation. The
baseband signal of the polarization $x$ is obtained by modulating
$\mathbf{s_{x}}$ into the waveform

$q_{x,n}(z=0,t)=\sum_{i=1}^{N_{s}}s_{x,n}^{(i)}\,p(t-iT_{s}),$ (1)

where $q_{x,n}$ is the complex envelope of the signal in the WDM channel $n$
as a function of distance $z$ and time $t$, $p(\cdot)$ is the pulse shape, and
$T_{s}$ is the symbol period. $z=0$ indicates that the signal is located at
the TX. The equations for $q_{y,n}(0,t)$ are identical upon replacing $x$ with
$y$. The waveforms $q_{x,n}$ and $q_{y,n}$ are multiplexed into an optical
signal $\mathbf{q}_{n}(0,t)=[q_{x,n},q_{y,n}]$ using a dual-polarization Mach-
Zehnder modulator. The complex envelope of the WDM signal $\mathbf{q}(0,t)$
launched in the fiber link is generated by adding the signals of the different
WDM channels

$\mathbf{q}(0,t)=\sum_{n}\mathbf{q}_{n}(0,t)e^{-jn\Delta\omega t},$ (2)

where $\Delta\omega$ is the frequency spacing between adjacent WDM channels.
The WDM signal is launched through an optical fiber channel consisting of
multiple spans, where each span includes a SMF, dual-stage EDFAs, separated by
a DCF with a proper length to compensate for dispersion. DCF has a higher
nonlinearity compared to SMF, so the first-stage DCF provides pre-DCF gain,
but only to a power level that would not generate excessive nonlinear effects.
The second EDFA in the span then amplifies the signal further to its original
power level. The end-to-end channel with all components can be described by
the interplay between CD, PMD and Kerr nonlinearity effects. The propagation
of signals inside each span in the presence of PMD is governed by the coupled
nonlinear Schrödinger’s equations (CNLSE), modeling the interaction between
the two states of polarization

$\displaystyle\frac{\partial q_{x}}{\partial z}$
$\displaystyle=\big{[}-\frac{\alpha}{2}-\beta_{1x}\frac{\partial}{\partial
t}-\frac{j\beta_{2}}{2}\frac{\partial^{2}}{\partial
t^{2}}+j\gamma(|q_{x}|^{2}+\frac{2}{3}|q_{y}|^{2})\big{]}q_{x},$ (3a)
$\displaystyle\frac{\partial q_{y}}{\partial z}$
$\displaystyle=\big{[}-\frac{\alpha}{2}-\beta_{1y}\frac{\partial}{\partial
t}-\frac{j\beta_{2}}{2}\frac{\partial^{2}}{\partial
t^{2}}+j\gamma(|q_{y}|^{2}+\frac{2}{3}|q_{x}|^{2})\big{]}q_{y},$ (3b)

where $\alpha$ is the loss parameter, $\beta_{1x/y}$ are the first-order
dispersion coefficients, $\beta_{2}$ is the CD coefficient and $\gamma$ is the
Kerr nonlinearity parameter. We consider a highly birefringent fiber, where
the fiber length $\mathcal{L}$ is much larger than the beat length
$L_{B}={2\pi}/(\beta_{0x}-\beta_{0y})$, and the coherent cross-polarization
terms can be neglected [31, Eq. 6.1.11–12]. The effect of the rapidly changing
state of polarization (SOP) is described separately in the numerical
simulation of the propagation equation in Section 2.2. At the end of the span,
an EDFA with gain $G$ compensates for the fiber attenuation and introduces ASE
noise. The noise $n(t)$ is a band-limited white circularly-symmetric complex
Gaussian process with the auto-correlation function
$E\bigl{(}n(t)n^{*}(t)\bigr{)}=\sigma_{0}^{2}\delta_{B}(t-t^{\prime})$, where
$\delta_{B}(x)=B{\rm sinc}(Bt)$, $\sigma_{0}^{2}=\frac{1}{2}(G-1)Bh\nu_{0}{\rm
NF}$, where $\nu_{0}$ is the carrier frequency, $B$ is the signal bandwidth,
$h$ is Planck constant, and NF is the amplifier’s noise figure.

###  2.2 Split-Step Fourier Method

In this section, we present a numerical approach for solving the CNLSE using
the SSFM. All fiber effects occur simultaneously and accumulate along the
length of the fiber. In SSFM, a fiber of length $\mathcal{L}$ is divided into
$N_{seg}$ segments of short length $\delta_{s}=\mathcal{L}/N_{seg}$, which we
refer to as the _PMD correlation length_ , for which the SOP at the end of the
segment is uncorrelated with its initial state. The channel effects can be
assumed to occur individually and consecutively in each segment. Each step of
the SSFM involves evaluation of three sub-steps: a linear step, a PMD step and
a nonlinear step. Below, we describe the steps in the SSFM.

  1. 1.

_Linear step_ : Solves for the signal loss and CD in the frequency domain.
Considering only the terms which include $\alpha$ and $\beta_{2}$ in Eq. (3),
we obtain

$\hat{q}_{x/y}(z,\omega)\rightarrow\exp\left(-\frac{\alpha}{2}\delta_{s}+\frac{j\beta_{2}}{2}\omega^{2}\delta_{s}\right)\hat{q}_{x/y}(z,\omega),$
(4)

where $\hat{q}_{x/y}$ denotes the Fourier transform of the time domain signal
$q_{x/y}$.

  2. 2.

_PMD step_ : The PMD can be modeled by applying the unitary Jones matrix
$\mathbf{J}^{(i)}(\omega)$ to the signal vector
$\mathbf{\hat{q}}(z,\omega)=[\hat{q}_{x}(z,\omega),\hat{q}_{y}(z,\omega)]^{\top}$

$\mathbf{\hat{q}}(z,\omega)\rightarrow\mathbf{J}^{(i)}(\omega)\mathbf{\hat{q}}(z,\omega),$
(5)

where $\mathbf{J}^{(i)}(\omega)=\mathbf{R}^{(i)}\mathbf{D}^{(i)}(\omega)$. The
$\mathbf{R}^{(i)}$ here is a unitary matrix of the form

$\mathbf{R}^{(i)}=\begin{bmatrix}e^{j\frac{\phi_{i}}{2}}\cos{\theta_{i}}&e^{-j\frac{\phi_{i}}{2}}\sin{\theta_{i}}\\\
-e^{j\frac{\phi_{i}}{2}}\sin{\theta_{i}}&e^{-j\frac{\phi_{i}}{2}}\cos{\theta_{i}}\end{bmatrix},$
(6)

where $\phi_{i}$ and $\theta_{i}$, $i\in\\{1,2,...,N_{seg}\\}$, are sequences
of independent identically distributed (i.i.d.) random variables drawn from a
uniform distribution from $[0,2\pi)$. Furthermore, $\mathbf{D}^{(i)}(\omega)$
is the differential group delay (DGD) matrix

$\mathbf{D}^{(i)}(\omega)=\begin{bmatrix}e^{-j\omega\frac{\tau_{i}}{2}}&0\\\
0&e^{j\omega\frac{\tau_{i}}{2}}\end{bmatrix},$ (7)

where the DGD parameters $(\tau_{i})_{i=1}^{N_{seg}}$ are taken to be as
i.i.d. random variables drawn from the natural probability distribution
$\mathcal{N}(0,\tau\sqrt{\delta_{s}})$, where $\tau$ is the characteristic
constant of the channel called the _PMD coefficient_.

  3. 3.

_Nonlinear step_ : Solves for the signal nonlinear effects by only considering
the terms which include $\gamma$ in Eq. (3). For the $x$ polarization,

$q_{x}(z,t)\rightarrow\exp\left(j\gamma\delta_{s}\big{(}|q_{x}|^{2}+\frac{2}{3}|q_{y}|^{2}\big{)}\right)q_{x}(z,t).$
(8)

In our simulations, we consider a symmetric SSFM, where the nonlinear step is
applied in the middle of the two linear half-steps.

Figure 2: Dispersion map of the propagating optical signal over the first 7
spans, with two DBP steps shown. One DBP step is applied over 4 spans, while
another is applied over 2 spans, with nonlinear operators applied at points
$A$ and $B$. Accumulated dispersion is fully compensated at point $C$.

###  2.3 Receiver

At the RX, a polarization-diversity coherent receiver converts the optical
signal to the electrical domain. A low-pass filter with the same bandwidth of
the central WDM channel is applied to the signal, obtaining
$\mathbf{q}_{0}(z=\mathcal{L},t)$. The resulting signal is sampled at 2
samples/symbol, and translated into four electrical signals corresponding to
the $I$ and $Q$ components of each polarization, which are then processed by
the DSP chain [2]. The DSP chain consists of the following components.

  1. 1.

_CD compensator (CDC)_ , which reverses the CD as

$\hat{q}_{x}(\mathcal{L},\omega)\rightarrow\exp\Bigl{(}-j\frac{\bar{\beta}_{2}}{2}\omega^{2}\mathcal{L}\Bigr{)}\hat{q}_{x}(\mathcal{L},\omega),$
(9)

where $\bar{\beta}_{2}$ is the average group velocity dispersion in the link.
According to Eq. (9), CDC requires processing the signal in frequency domain
using fast Fourier transform (FFT), followed by inverse FFT (IFFT) to
transform the signal back to the time domain. An alternative implementation of
the CDC in the time domain can be realized using FIR filters, which we will
discuss in Section 6.

  2. 2.

_MIMO equalizer_ , which compensates the time-varying PMD and the random SOP
in the channel. The adaptive equalization of both effects requires using a set
of four real-valued finite impulse response (FIR) filters, which together
preform the inverse of the Jones matrix of the dynamic channel in Eq. (5). The
outputs of the MIMO equalizer are given by

$\displaystyle
x_{\text{MIMO,out}}[k]=\mathbf{h}_{xx}^{H}\,\mathbf{x}_{\text{in}}[k]\,+\mathbf{h}_{xy}^{H}\,\mathbf{y}_{\text{in}}[k]\,$
(10a) $\displaystyle
y_{\text{MIMO,out}}[k]=\mathbf{h}_{yx}^{H}\,\mathbf{x}_{\text{in}}[k]\,+\mathbf{h}_{yy}^{H}\,\mathbf{y}_{\text{in}}[k]\,,$
(10b)

where $\mathbf{h}_{xx}$, $\mathbf{h}_{xy}$, $\mathbf{h}_{yx}$, and
$\mathbf{h}_{yy}$ are vectors of size $\xi$, representing the taps of the FIR
filter, and $\mathbf{x}_{\text{in}}$ and $\mathbf{y}_{\text{in}}$ are sliding
windows of the signal after chromatic dispersion (CD) compensation. These
windows have a length of $\xi$, and they can be defined as
$\mathbf{x}_{\text{in}}[k]=[x_{\text{in}}[k]$, $x_{\text{in}}[k-1]$, $\dots$,
$x_{\text{in}}[k-\xi]]$ and $\mathbf{y}_{\text{in}}[k]=[y_{\text{in}}[k]$,
$y_{\text{in}}[k-1]$, $\dots$, $y_{\text{in}}[k-\xi]]$. A popular method for
optimizing the FIR taps is called the constant modulus algorithm (CMA), which
is typically used for phase modulations. For the higher order QAM modulations
we use in our simulations, the radially directed equalizer proposed in [2] is
used conjointly with the CMA. Following the MIMO equalizer, the output
sequences are sampled at 1 sample/symbol.

  3. 3.

_Carrier phase estimation (CPE)_ : The last step in the DSP is to estimate the
PN, $\phi_{N}$, resulting from the phase fluctuations between the local
oscillator at TX and RX ends [32]. The PN of a single-frequency laser
resembles a quasi-continuous frequency drift, and as result, the phase shift
is characterized by a gradual and continuous change rather than discrete
substantial phase jumps. A measure of the duration over which the laser phase
remains stable is provided by the coherence time which is related inversely to
the laser linewidth [31]. Within a single coherence time, the PN varies slowly
compared to the signal, and can thus assumed to be constant. Therefore,

$x_{\text{sym}}[k]=x_{\text{CPE,in}}[k]\exp(-j\phi_{N}),$ (11)

where $x_{\text{CPE,in}}$ is the same as the output of the MIMO,
$x_{\text{MIMO,out}}$.

The aforementioned optical fiber receiver only concerns equalizing the linear
effects of the signal resulting from the dispersive channel and PMD.

##  3 Digital Back-Propagation

The signal propagation in the reverse direction through the optical fiber is
numerically approximated using DBP. In Fig. 1, the DBP block takes the place
of the CDC in the receiver (RX) and operates at the same sampling rate of 2
samples/symbol. DBP employs the SSFM with negative propagation parameters and
larger spatial segments compared to the SSFM. This approach helps to mitigate
the high complexity typically associated with a large number of segments in
the SSFM, while maintaining accurate signal reconstruction. In practice DBP is
limited to 3 or less StpS.

###  3.1 Mathematical Model

The DBP block is placed before the MIMO equalizer, which means that the input
signals are subject to PMD variations in the optical fiber. However, it is
important to note that the standard DBP algorithm is not designed to equalize
random effects like PMD. Instead, it primarily addresses deterministic effects
such as CD and nonlinear effects. moreover, the changing SOP can cause the
$x$-polarization and $y$-polarization signals to be indistinguishable due to
rapid changes in the orientation of the axes of birefringence $\theta$ along
the fiber. When the PMD is small but the birefringence rapidly and varies
randomly, the Manakov equation can be used to describe signal propagation. The
propagation model we consider for DBP is based on Eq. (3) averaged over
rapidly-varying SOP along the fiber, which corresponds to the vector Manakov
equation[33]:

$\frac{\partial\mathbf{q}}{\partial
z}=-\frac{j\beta_{2}}{2}\frac{\partial^{2}\mathbf{q}}{\partial
t^{2}}+\Bigl{[}-\frac{\alpha}{2}+j\gamma\frac{8}{9}||\mathbf{q}||^{2}\Bigr{]}\mathbf{q},$
(12)

Here, $\mathbf{q}(z,t)=[q_{x}(z,t),q_{y}(z,t)]^{T}$ is the Jones vector
containing the propagating signals in both polarizations, and the factor $8/9$
is a characteristic property of the PMD, introduced by the averaging process.
In DBP, the optical channel is divided into $N_{d}$ spatial segments of equal
lengths, denoted as $\delta_{d}=\mathcal{L}/N_{d}$. Within each segment, the
dispersive and nonlinear channel effects are assumed to act independently.
These effects correspond to the first and second terms on the right side of
Eq. 12, representing the linear and nonlinear effects, respectively. To solve
for each of these terms, we use negated parameters. The linear part is solved
in the frequency domain, while the nonlinear part is solved in the time
domain, resulting in two partial solutions:

$\displaystyle\mathbf{\hat{q}}(z+\delta_{d},\omega)$
$\displaystyle=\underbrace{\exp\left(-j\beta_{2}\frac{1}{2}\omega^{2}\delta_{d}\right)}_{L(\delta_{d},\omega)}\mathbf{\hat{q}}(z,\omega),$
(13) $\displaystyle\mathbf{q}(z+\delta_{d},t)$
$\displaystyle=\underbrace{\exp\left(\frac{1}{2}\alpha\delta_{d}-j\frac{8}{9}\gamma||\mathbf{q}(z,t)||^{2}\delta_{\text{eff}}\right)}_{N(\delta_{d},t)}\mathbf{q}(z,t).$
(14)

where $\delta_{\text{eff}}=(1-\exp\\{-\alpha\delta_{d}\\})/\alpha$ represents
the effective nonlinear step length, and the location $z$ is measured relative
to the receiver, with $z=0$ corresponding to the receiver’s location. The DBP
is hence characterized by two sets of operators: the linear operator
$L(\delta_{d},\omega)$ and the nonlinear operator $N(\delta_{d},t)$. The
equalized signal can be obtained from the received signal by alternating
between these two solutions along the length of the fiber in the backward
direction (from the receiver to the transmitter).

We assume a discretization of $\mathbf{q}(z,t)$ into the time-sampled vector
$\mathbf{U}^{(n)}=[\mathbf{X}^{(n)},\mathbf{Y}^{(n)}]^{T}$, where
$\mathbf{X}^{(n)}\in\mathbb{C}^{N}$ and $\mathbf{Y}^{(n)}\in\mathbb{C}^{N}$,
respectively. The superscript $(n)$ refers to the DBP step, with $n=0$
representing the input to the DBP, and $n=N_{d}$ representing the output of
the DBP. The linear step is represented by a matrix multiplication, given by

$\mathbf{U}^{(n)}\rightarrow\mathbf{B}\mathbf{U}^{(n-1)}=\mathbf{W}^{-1}\text{diag}(e^{\delta_{d}H_{1}},...,e^{\delta_{d}H_{n}})\mathbf{W}\mathbf{U}^{(n-1)},$
(15)

where $\mathbf{B}\in\mathbb{C}^{N\times N}$ is a matrix representing the
linear operator, $\mathbf{W}$ denotes the discrete Fourier transform matrix,
$H_{k}=-j\beta_{2}\omega_{k}^{2}/2$, and $\omega_{k}=2\pi f_{k}$, where
$f_{k}$ corresponds to the $k$-th discrete frequency. Additionally, the
nonlinear step can be represented by the nonlinear transformation $K(\cdot)$

$\mathbf{U}^{(n)}\rightarrow
K\left(\mathbf{U}^{(n-1)}\right)=\mathbf{U}^{(n-1)}\exp\left(-j\gamma\varepsilon\delta_{\text{eff}}\frac{8}{9}\Bigl{(}\mathbf{X}^{(n-1)}\odot\mathbf{X}^{(n-1)*}+\mathbf{Y}^{(n-1)}\odot\mathbf{Y}^{(n-1)*}\Bigr{)}\right).$
(16)

where $\odot$ denotes the Hadamard product. Additionally, a real-valued
parameter $\varepsilon\in[0,1]$ is introduced, to accurately model the
nonlinear effects. The optimization of this parameter will be discussed in the
results section.

###  3.2 DBP Adaptation to DM Systems

In NDM systems, CD is introduced by a single type of fiber, and the residual
CD grows at a constant rate along the link. However, in DM systems, the
compensation of CD using DBP becomes more involved as the linear step in DBP
must account for the total dispersion generated by both the DCF and the
standard SMF. The DBP algorithm we use in our work allows for flexible
selection of StpS values less than 1, enabling us to include multiple spans in
a single DBP step, similar to what is considered in [23, 34, 1]. To model the
accumulated dispersion in the fiber, we denote by $\mathcal{D}_{c}(z)$ the
total accumulated dispersion inside the fiber as a function of distance $z$;
see Fig. 2. The linear step is adjusted as follows: Let us assume $N_{d}+1$
spatial steps, dividing a fiber of length $\mathcal{L}$ into spatially equal
segments with step size $\delta_{d}=\mathcal{L}/N_{d}$, with the exception of
the first and last steps where each has length $\delta_{d}/2$. Each step spans
$[z_{k},z_{k+1}]$, where $z_{k}=(k-\frac{1}{2})\delta_{d}$,
$k\in\\{1,\dots,N_{d}\\}$, $z_{0}=0$, $z_{N_{d}+1}=\mathcal{L}$. This
configuration is similar to the Wiener-Hammerstein model in [7]. Within each
step, we calculate the weighted-average dispersion coefficient, which is
described by the following equation

$\bar{D}=\frac{\mathcal{D}_{c}(z_{k})-\mathcal{D}_{c}(z_{k-1})}{\delta_{d}}.$
(17)

The equation Eq. (17) is essentially approximating the dispersion map between
points $z_{k}$ and $z_{k+1}$ with a linear dispersion map with a $\bar{D}$
that is between the values of $D$ for SMF and DCF. The power injected at the
input of the DCF is set small enough to guarantee a quasi-linear transmission
regime. As a consequence, the nonlinear step is performed with the coefficient
$\bar{\gamma}=\gamma_{\text{SMF}}$ determined by the SMF. This approximation
is accurate, as shown in the numerical simulations that will be presented in
Sec. 5 of the paper. The linear and nonlinear steps in the proposed DBP
alternate until the algorithm spans over the entire optical link.

###  3.3 Time domain and Frequency Domain Implementation of DBP

The conventional method for implementing DBP involves using FFT and IFFT for
each step, which can be computationally demanding due to the numerous FFTs and
IFFTs required. However, considering the relatively low accumulated dispersion
at the receiver in DM systems, we are interested in investigating whether a
time-domain implementation of DBP could provide a complexity advantage over
the frequency-domain approach. In this time-domain implementation, we replace
the parameter $\mathbf{B}$ in Eq. 15 by employing an FIR filter with complex-
valued taps, denoted as $h_{\text{CDC}}(\delta_{d})$. This filter performs
circular convolution with the backpropagating signal to compensate for the
dispersion introduced within a step of length $\delta_{d}$. The DBP step in
this case can be represented as [35]

$\mathbf{q}(z+\delta_{d},t)=(\mathbf{q}(z,t)*h_{\text{CDC}}(\delta_{d}))\cdot\exp(\alpha\delta_{d}/2)\cdot\exp(-j\delta_{\text{eff}}\varepsilon\gamma||\mathbf{q}||^{2}),$
(18)

where $h_{\text{CDC}}=(h_{-F},\dots,h_{-1},h_{0},h_{1},\dots,h_{F})$, and each
$h_{i}$ for $i=1,2,\dots,F$ denotes an individual tap. It is worth noting that
the filter taps exhibit symmetry, such that $h_{i}=h_{-i}$. The minimum number
of taps needed in the FIR filter to compensate for dispersion within a DBP
step is mainly determined by the length of the impulse response, approximated
using the formula provided in [36]

$\tau_{CD}(\delta_{d})=\frac{\lambda_{c}^{2}}{c}|D_{acc}|\Delta f,$ (19)

where $|D_{acc}|=\bar{D}\,\delta_{d}$ is the total accumulated dispersion
inside a step with length $\delta_{d}$, $\Delta f$ is the signal spectral
width for a single channel, $\lambda_{c}=c/f_{c}$ is the carrier wavelength,
and $c$ is the speed of light. The channel impulse response length
$\tau_{CD}(\delta_{d})$ is measured in seconds, in which case, the number of
filter taps for any step is

$N_{CDC,\delta_{d}}=\left\lceil\frac{\tau_{CD}(\delta_{d})}{T_{s}}n_{s}\right\rceil,$
(20)

and $\lceil x\rceil$ denotes the smallest integer larger or equal to $x$, and
$n_{s}$ is the oversampling ratio.

##  4 Learned Digital Back-Propagation

Figure 3: Block diagram of DBP (upper branch) and LDBP (lower branch)
structures. The following symbols represent:
$\mathrm{IX}\coloneqq\text{Re}\\{\mathbf{X}\\}$,
$\mathrm{QX}\coloneqq\text{Im}\\{\mathbf{X}\\}$,
$\mathrm{IY}\coloneqq\text{Re}\\{\mathbf{Y}\\}$, and
$\mathrm{QY}\coloneqq\text{Im}\\{\mathbf{Y}\\}$.

When designing NNs, incorporating prior knowledge of the system’s model can
significantly expedite the training process and improve the convergence rate
to a lower local minimum compared to black-box approaches [37]. The similarity
between the SSFM and deep feed-forward NNs had been pointed out in the
literature [28, 25], where both involve alternating between linear matrix
multiplication and nonlinear element-wise operator. This similarity can be
exploited to design a model-based NN with DBP as a blueprint, which allows for
optimizing DBP parameters using the SGD algorithm [28].

###  4.1 Mathematical Model of NNs

Deep feed-forward NNs consist of an input layer, output layer, and a cascade
of hidden layers each performing a nonlinear transformation on the input of
the layer, then passing the output to the next layer. Deep feed-forward NNs
can be mathematically represented as a series of alternating linear
operations, denoted as $\mathbf{A}^{(k)}$, and element-wise activation
function, denoted as $\Phi^{(k)}$. These operations create a mapping from an
input vector $\mathbf{u}$ to an output vector $\mathbf{v}$, as follows:

$\mathbf{v}=\Phi^{(N_{l})}(\mathbf{A}^{(N_{l})}(\Phi^{(N_{l}-1)}(\dots\mathbf{A}^{(0)}(\mathbf{u})))),$
(21)

where $N_{l}$ represents the number of layers in the NN, and the superscript
$(k)$ denotes the index of the $k$-th layer of the NN. The function performed
by the linear operator $\mathbf{A}^{(k)}$ varies depending on the neural
network architecture. In the case of fully-connected neural networks (NNs), it
performs matrix multiplication. However, in the case of convolutional NNs, it
performs convolution.

In our application for signal equalization, we specifically utilize the
convolutional NN model. Within this model, the linear operator
$\mathbf{A}^{(k)}(\mathbf{c})$ in each layer is defined as
$\mathbf{A}^{(k)}(\mathbf{c})=\mathbf{c}\ast\mathbf{\Omega}^{(k)}+\mathbf{b}^{(k)}$,
where $\mathbf{\Omega}^{(k)}\in\mathbb{C}^{m}$ represents the convolutional
filter, and $\mathbf{b}^{(k)}\in\mathbb{C}^{N}$ is the bias vector.

###  4.2 Architecture of LDBP

Fig. 3 depicts the block diagrams of LDBP and its blueprint DBP. The NN
architecture of LDBP is a complex-valued network that comprises two real-
valued networks operating jointly. Each of these networks contains
$N_{l}=N_{d}$ layers and accepts four input vectors ($IX$, $QX$, $IY$, and
$QY$). Each layer of the network consists of two parallel convolutional
filters, $\Omega_{R}$ and $\Omega_{I}$, corresponding respectively to the real
and imaginary parts of the filter, and a nonlinear function $\Phi$, which
takes four input vectors and generates four output vectors. The number of non-
zero weights in the convolutional filter is determined numerically, as will be
described in the simulation setup section later. These layers perform the
real-valued equivalent of the operation described in Eq. (18) in the time
domain without the need for FFT and IFFT. Biases are not utilized in our model
and are set to zero. In our model, we do not use the biases and therefore set
them to zero. The input dimension of LDBP is $[N_{ex},N,4]$, and its output
dimension is $[N_{ex},N-2M,4]$, where $M$ represents the memory of the
dispersive channel.

To train the Learned-DBP, we simulated a five-channel WDM PMD-free
transmission of a block of $2^{15}$ symbols at various launch powers. The
signal was initially sampled at 16 samples per symbol duration for forward
propagation using SSFM. However, after the WDM demultiplexer, the signal from
the central channel was sampled at a sampling ratio of $n_{s}=2$ samples per
symbol, resulting in a received block consisting of $2^{16}$ samples. The NN
operates in a sliding window fashion, with a window size of $N$ sliding over
the transmission block and advancing by $L\times n_{s}$, where the sliding
factor $L$ is an integer that determines the number of symbols shifted between
examples. For LDBP training, we generated input-output pairs with a size of
1024 samples (512 symbols) for the input, and the corresponding output was 852
samples (426 symbols) long due to the dispersive channel effects (43 symbols
on each side). We set the sliding factor to $L=8$, generating $N_{ex}=4021$
input-output pairs for training. To test LDBP, we generated 8 transmission
blocks with PMD using the same overlapping and shifting technique as for
training. However, this time, the sliding factor was set to $L=426$ to ensure
that each symbol in the transmission block was detected exactly once. The
output of LDBP was then passed to the DSP to equalize PMD and dynamic channel
effects.

We implement a symmetric DBP as the blueprint of LDBP, such that all layers
are initialized with the corresponding parameters in a linear step of DBP at
the full step size $\delta_{d}$, except for the first and last layers which
correspond to a half step $\delta_{d}/2$. The NN is trained by minimizing the
mean squared error (MSE) loss function, using the _Adam_ optimizer with a
learning rate of 0.001. During training, 20% of the training examples were
used for validation to monitor the LDBP’s progress. The LDBP was trained for
up to 75 epochs, with an early-stop condition triggered if the validation
error did not decrease within 5 epochs. The best-performing epoch’s weights
were used in the final LDBP. After training at each launch power, we evaluate
the performance of the LDBP by calculating the Q-factor using independently
generated testing data.

| # of channels | Modulation format | fiber coefficients
---|---|---|---
Setup (A) | 1 |  $16$-QAM |  $\alpha$=0.2 dB/km, PMD = 0.05 ps/$\sqrt{\text{km}}$, $D$=17 ps/(nm.km), $\gamma$= 1.4/W/km
Setup (B) | 5 |  $16$-QAM |  text Same as above
Setup (C) | 5 |  $64$-QAM |  text Same as above
Setup (D) | 5 |  $16$-QAM |  $\alpha$=0.24 dB/km, PMD = 0.3 ps/$\sqrt{\text{km}}$, $D$=17 ps/(nm.km), $\gamma$= 1.4/W/km
Table 1: Description of the simulated setups.

##  5 Simulated System Setup and Performance Results

The performance results are based on the simulation of the transmission system
shown in Fig. 1. All elements of the transmission system, including the
transmitter (TX), receiver (RX), and channel, are simulated in Python, while
the NN is implemented using the TensorFlow library. In this section, we
present the performance results of DBP and LDBP for four different setups:
(A), (B), (C), and (D). Setup (A) represents a single-channel transmission
system with 16-QAM modulation, while setups (B) and (C) are WDM transmission
systems with 16-QAM and 64-QAM modulation formats, respectively. Setup (D) is
also a WDM transmission using 16-QAM modulation, but it includes aging effects
where a fiber channel undergoes aging. The aging study will be described in
detail when we present the performance results for this setup. Table 1
provides a detailed description of each setup.

For all setups (A)–(D), the transmission symbol baud rate $B=32$ GBaud for
each channel using a root-raised-cosine (RRC) pulse-shape with a roll-off
factor $\rho=0.06$. The optical fiber link consists of $N_{sp}=$ $28$ spans,
each span including an SMF and a DCF measuring $72$ km and $13$ km,
respectively. The length of DCF is chosen such that it compensates for $85$%
of the CD in each span. An amplifier with gain $G_{\text{SMF}}=6.5$ dB is
applied at the end of the SMF, and a second amplifier with gain of
$G_{\text{DCF}}=14.4$ dB is applied after the DCF. The SMF parameters and PMD
value for all setups are provided in Table 1. For WDM setups (B)–(D), the WDM
channels are separated by a frequency spacing of $37.5$ GHz, resulting in a
guard band of 5.5 GHz between adjacent channels. The lasers used for these
setups had a linewidth of 50 kHz. To avoid overestimation of nonlinear
crosstalk, the data symbols of all WDM channels were intentionally made
unsynchronized in terms of time, polarization state, and phase. At the
receiver, an RRC filter with a bandwidth of $(1+\rho)B$ is applied to filter
out the adjacent channels, such that only the central channel is processed by
the DBP or LDBP algorithms. The forward signal propagation using SSFM is
simulated with 72 steps for each SMF and 13 steps for each DCF. The signals
are sampled at a rate of 16 samples per symbol. At the receiver, the signal is
downsampled to twice the symbol rate before being processed by either the CDC,
DBP, or LDBP algorithms. Finally, the output of the CDC, DBP, or LDBP is
downsampled to one sample per symbol and processed by the conventional DSP
chain, which equalizes PMD effects and polarization mixing.

###  5.1 DBP Parameters Optimization

(a)

(b)

Figure 4: Achieved Q-factor by DBP at the optimal launch power (-2 dBm) for
different values of StpS as a function of filter taps per step for (a) 16-QAM
and (b) 64-QAM.

To optimize the performance of the DBP for all transmission scenarios (A)–(D),
we select the parameter $\varepsilon$ in Eq. 16 such that each DBP achieves
the highest $Q$-factor at the optimal launch power. For a single channel
transmission, we find that the value of $\varepsilon$ is 1 for all values of
StpS. For the WDM transmission of 5 channels (Set-ups B, C and D), we find
that the optimal values are $\varepsilon=0.85$ for DBP with 1 StpS,
$\varepsilon=0.75$ for DBP with 0.5 StpS, and $\varepsilon=0.64$ for DBP with
0.25 StpS. It should be noted that $\varepsilon=0$ corresponds to performing
LE, and in this case, all DBP configurations perform similarly to LE,
regardless of the number of StpS. The filter size in each DBP step varies
depending on the number of StpS, Optimal values of $F$ were determined for
different DBP configurations by numerical optimization. Specifically, $F=16$
was found to be optimal for both 1 StpS and 0.5 StpS DBP, $F=24$ for 0.25
StpS, $F=30$ for $\frac{1}{7}$ StpS, and $F=36$ for $\frac{1}{14}$ StpS. These
values represent the minimum values required to ensure that all DBP
configurations outperform LE at all launch powers. The impact of filter width
on performance is shown in Fig.4. It is worth noting that this finding agrees
with previous literature, specifically [30], which reported that the filter
width has a significant impact on the performance of DBP-based equalizers (see
Fig.(9) in [30]).

###  5.2 Performance Comparison of DBP and LDBP in Single Channel Transmission

A single channel transmission is simulated in setup (A). We choose
($\text{SNR}_{\text{eff}}$) as a performance measurement for single channel
transmission, which is defined as

$\text{SNR}_{\text{eff}}=\frac{\rvert\rvert\mathbf{\hat{s}_{x}}\rvert\rvert^{2}+\rvert\rvert\mathbf{\hat{s}_{y}}\rvert\rvert^{2}}{\rvert\rvert\mathbf{s_{x}}-\mathbf{\hat{s}_{x}}\rvert\rvert^{2}+\rvert\rvert\mathbf{s_{y}}-\mathbf{\hat{s}_{y}}\rvert\rvert^{2}},$
(22)

Figure 5: DBP performance for different StpS values in single channel
transmission (setup A). The performance of LDBP is also shown for 3 launch
powers around the peak $\text{SNR}_{\text{eff}}$.

where $\mathbf{\hat{s}_{x}}$ and $\mathbf{\hat{s}_{y}}$ denote the estimated
symbols at the output of the DSP, with true values $\mathbf{s_{x}}$ and
$\mathbf{s_{y}}$, respectively. The reason for choosing
$\text{SNR}_{\text{eff}}$ is the high reliability of the single channel
scenario, which leads to an extremely low bit-error ratio (BER). This makes it
challenging to measure BER accurately, requiring a large number of training
examples. The $\text{SNR}_{\text{eff}}$ provides a more practical measure of
performance under these conditions. The performance of the DM-adapted DBP and
LDBP in setup (A) is depicted in Fig. 5 for varying signal launch powers. The
PMD-aware DBP in Fig. 5 is a genie-aided model which assumes perfect knowledge
of PMD and SOP across the fiber channel at the RX and uses them in the back-
propagation with the same step size $\delta_{s}$ used for SSFM. Despite its
impracticality, the PMD-aware DBP model provides a useful upper bound on the
performance that can be achieved by PMD-agnostic DBP and LDBP algorithms. Our
simulations consider DBP and LDBP equalizers with a fractional number of StpS
or LpS that is less than or equal to 1.

At the optimal launch power, LDBP with 1 LpS outperforms DBP with the same
complexity by providing an SNReff of 22.3 compared to 19.8 dB, with the
optimal launch powers differing by about 1 dBm between the two algorithms. The
LE achieves its best performance at an SNReff value of 16 dB, which is
achieved with a launch power of -4 dBm. Both DBP and LDBP outperform the LE
with varying gains, with LDBP with 1 LpS achieving the highest gain of 6.3 dB
and DBP with 1 StpS achieving a gain of 3.8 dB. The simulated LDBP with the
least complexity has 2 layers (1 full step and 2 half steps) and 2 activation
functions, and outperforms DBP with similar complexity by 1.4 dB and the LE by
1.8 dB.

###  5.3 Performance Comparison of DBP and LDBP in Multi-Channel Transmission

Setups (B)–(D) present multiple WDM transmission scenarios. In such cases, the
nonlinearity affecting the received signal is dominated by the nonlinear
interference introduced by the adjacent channels via XPM. Since only the
signal from one single channel is fed to the receiver, the information in
adjacent channels is unknown to the receiver. Therefore, the nonlinearity
generated by adjacent channels impacts all equalizers and limits their
performance in the nonlinear regime. The performance of the PMD-aware DBP
equalizer in these setups can be characterized by a bell-shaped curve, as seen
in Figures 6 and 7(b), in contrast to the straight line observed in the
single-channel scenario (A) shown in Fig. 5.

(a)

(b)

Figure 6: Achieved $Q$-factors for DBP and LDBP with different values of StpS
and LpS for WDM transmissions over a 2016 km fiber with 32 Gbaud using (a)
16-QAM modulation, and (b) 64-QAM modulation.

For setups (B)–(D), we utilize the $Q$-factor based on BER as the performance
metric for the implemented algorithms. The $Q$-factor is defined as follows

$Q\textnormal{-factor}=20\log_{10}[\sqrt{2}\text{erfc}^{-1}(2\text{BER})],$
(23)

where $\text{erfc}(\cdot)$ is the complementary error function.

Simulation results for setups (B) and (C) are presented in Fig. 6. In these
setups, the DBP and LDBP algorithms were simulated with varying numbers of
total steps or layers ($M\in{7,14,28}$).

In setup (B), LDBP with 1 StpS achieved a peak $Q$-factor of 11 dB at the
optimal launch power of $P=-3$ dBm. Notably, the peak performance of LDBP with
1 StpS is comparable to that achieved by the PMD-aware DBP at the same launch
power. When comparing LDBP to DBP with 1 StpS, LDBP outperformed DBP by 0.3 dB
at the same launch power. Furthermore, LDBP with 0.5 StpS and 0.25 StpS
achieved $10.8$ dB and $10.7$ dB, respectively, both outperforming DBP with
similar numbers of StpS by 0.4 dB.

In setup (C), The LDBP with 1 StpS achieved a peak $Q$-factor of 5.8 dB, which
is 0.3 dB higher than the peak performance achieved by the DBP with a similar
number of StpS. The LDBP with 0.5 StpS and 0.25 StpS achieved a peak
$Q$-factor of 5.8 dB and 5.6 dB, respectively, both outperforming the DBP with
a similar number of StpS by 0.5 dB. On the other hand, the LE achieved a peak
$Q$-factor of 4.7 dB.

It is worth noting that the launch powers corresponding to the peak $Q$-factor
in both Fig.5 and Fig.6 are shifted by 4 dB for WDM transmission compared to
single-channel transmission. This is due to the XPM generated by the four
adjacent channels, which dominates the nonlinearity affecting the received
signal in WDM setups, unlike single-channel transmission, which only
experiences self-phase modulation (SPM) from the same channel, as explained in
[38].

###  5.4 Impact of Aging Effects on LDBP Performance

(a)

(b)

Figure 7: Achieved $Q$-factors for LDBP and DBP with different StpS and LpS
values for Set-up (D) simulating transmission over a 2016 km aged fiber at 32
Gbaud. Figure (a) shows $Q$-factors across launch powers, and (b) displays the
impact of laser PN on the peak $Q$-factor of LDBP with varying LpS values.

Optical fiber links deployed for intensity modulation and direct detection
(IM-DD) communication, especially those utilizing dispersion management, are
particularly susceptible to aging and poor maintenance due to being outdated
systems [39, 40]. Changes in the fiber material occur as the fibers age,
leading to the deterioration of their parameters over time. Additionally,
yearly temperature variations can contribute to higher values of differential
group delay (DGD). Additionally, fiber splicing, which is done during
maintenance, can increase attenuation and necessitate higher amplification
gain in the link, resulting in ASE noise. To reflect these changes in a more
realistic transmission scenario, we consider the impact of aging and
maintenance-induced changes on the fiber link. It is worth noting that the CD
coefficient $\beta_{2}$ and the nonlinearity coefficient $\gamma$ are not
affected by aging effects. While only these two coefficients are used to
initialize the LDBP, the new attenuation coefficient $\alpha$ in aged fibers
can cause a reduction in the effective length $L_{\text{eff}}$. Thus,
retraining the NN with the updated parameters is necessary. Our goal is to
examine how these changes affect the LDBP’s performance relative to DBP, and
compare the two methods in a realistic transmission scenario that incorporates
aging and maintenance-induced changes in the fiber link.

The fiber parameters selected for setup (D) aim to simulate aging effects and
consist of an attenuation coefficient of $\alpha=0.24$ dB/km and a PMD
coefficient of $0.3$ ps/km-1/2. The value of the attenuation coefficient after
degradation is based on experimental findings reported in [40]. The simulation
results for this particular setup are presented in Figure 7. To account for
the changes in the fiber link, we retrained the model using data generated
from simulations with the new parameters. The LDBP with 1 StpS achieved a peak
$Q$-factor of $8.8$ dB at a launch power of $P=-2$ dBm, which is $0.6$ higher
than the peak $Q$-factor of the DBP with similar complexity. Furthermore, we
compared setup (D) with setup (B), which simulates a similar transmission
scenario without aging effects. We observed an average drop in $Q$-factor of
2.2 dB across all graphs, but the $Q$-factor gain of LDBP over DBP was still
maintained. This suggests that deploying the LDBP long-term is feasible with
retraining of the model.

The existing DM systems deployed for IM/DD may not be optimized for coherent
transmission. In fact, lasers used in IM/DD systems typically have higher
laser PN than those used in coherent transmission, which can potentially
impact the performance of receivers used for coherent detection [41]. In
general, laser phase noise does not have a significant impact on the system,
as the time interval over which the laser phase noise changes is much longer
than the symbol period. However, to investigate its impact on the performance
of LDBP for set-up (D), we choose to retrain the neural network using data
that incorporates varying degrees of laser PN. While it is the primary role of
the digital signal processing (DSP) following the LDBP to mitigate dynamic
effects like laser PN, when the LDBP is exposed to examples of data affected
by PN, it may be possible for the neural network to learn and reduce the
impact of PN to some extent. The peak $Q$-factor achieved for different values
of LpS and a range of laser linewidths is shown in Fig. 7(b). Since the input
vector for LDBP has a width of 512 symbols, the deviation in PN for the laser
linewidth used in the simulations is approximately 0.01 radian, which is too
small to significantly impact the performance. The results indicate that the
average drop in $Q$-factor is 0.1 dB for a laser linewidth of 100 kHz and 0.2
dB for a laser linewidth of 200 kHz.

##  6 Complexity of DBP and LDBP

We quantify the complexity of each step in terms of the number of real
multiplications (RMs) per detected symbol (RMpS), excluding additions. It is
important to note that both DBP and LDBP exhibit the same complexity when
considering the same number of StpS and LpS. Therefore, the complexity
formulas derived for LDBP in this section are identical to those for DBP.

To efficiently compute the exponential function in the activation function,
various approximation algorithms can be employed, such as the CORDIC algorithm
[42, 43]. These algorithms utilize look-up tables and bit-shifts, eliminating
the need for multiplications. By employing similar algorithms, the activation
function can be computed efficiently, with each computation of the activation
requiring only 9 RMs.

We make a distinction between the complexities of TD-LDBP and FD-LDBP and
present their respective complexity formulas as follows:

LELDBP Set-up B

LELDBP Set-up C

Figure 8: Constellation diagrams for LE and LDBP at their respective optimal
launching powers for Set-ups B&C. These constellations were obtained at the
end of the DSP prior to hard-decision.

#### FD-LDBP Complexity

The total complexity of FD-LDBP per detected symbol, can be measured in RMpS
for a signal with input size $N$ and sampled at rate $n_{s}$ samples/symbol.
The complexity is given by [37]

$C_{\text{FD--
LDBP}}=\left(N_{d}+1\right)\left(4\frac{N(\log_{2}(N)+1)n_{s}}{N-N_{CDC,\mathcal{L}}+1}\right)+\frac{9}{2}N_{d}n_{s}.$
(24)

#### TD-LDBP Complexity

The complexity of TD-LDBP for each detected symbol is determined by the
convolution of a kernel with size $F$ and an input of size $N$, which involves
$4FN$ RMs. This computation assumes a dilation of 1, stride of 1, and padding
of $F-1$, resulting in an output size that is the same as the input size for
each convolutional layer. Therefore, the total complexity per detected symbol
for TD-LDBP can be calculated as

$C_{\text{TD--
LDBP}}=\left(N_{d}+1\right)\left(4\frac{(2F+1)Nn_{s}}{N-N_{CDC,\mathcal{L}}+1}\right)+\frac{9}{2}N_{d}n_{s}.$
(25)

Figure 9: Trade-off between complexity and performance for different
equalizers in both FD-LDBP (represented by triangles) and TD-LDBP (represented
by circles).

Fig. 9 depicts the RMpS complexity of DBP and LDBP using both TD and FD
implementations. The complexity of FD-LDBP (or FD-DBP) is primarily dependent
on the number of steps involved in the algorithm, and can be approximated by
114 times the number of FFT/IFFT uses. The complexity of 1 LpS FD-LDBP, which
uses 29 FFT/IFFT pairs, is approximately 3300 RMpS. On the other hand, TD-LDBP
with the same number of LpS has a complexity of around 9000 RMpS. In fact, FD-
LDBP is less computationally complex compared to TD-LDBP across all values of
LpS. We initially hypothesized that smaller finite impulse response (FIR)
filters would be required to perform each linear step in DM compared to NDM
systems, given that the accumulated dispersion in our DM system was 85% lower
than that of NDM. This led us to believe that TD-implementation might be less
complex than FD-implementation. However, we discovered that strictly following
Eq. 20 to compute the number of taps in each FIR filter did not work. The
error resulting from truncation in each step accumulated with each consequent
step in the DBP, leading to errors that caused DBP to perform worse than LE.
We determined the number of filter taps required in each step through
numerical simulations, as demonstrated by Fig. 4. Our results showed that TD
implementation did not offer any gain in terms of complexity compared to FD
implementation. Nevertheless, TD implementation may offer a more hardware-
friendly approach for field-programmable gate arrays (FPGAs) and similar
devices that rely on multiply and accumulate (MAC) operations, as it does not
require an FFT/IFFT operation.

##  7 Conclusions

In this paper, we have presented the LDBP approach for mitigating nonlinear
effects in DM optical fiber transmission systems. LDBP is a DBP optimization
method that leverages NNs training algorithms to optimize the DBP parameters.
Our comparative study has shown that LDBP outperforms DBP, providing a
significant gain in Q-factor, with an average improvement of 0.4 dB. The
application of NNs to DM links is an important new development, as it
demonstrates the possibility of repurposing DM systems for coherent
transmission. The results have significant implications for the fiber-optics
industry, suggesting that data rates in conventional DM optical links can be
substantially improved using modern, simple coherent receivers. Additionally,
we demonstrated the complexity of both DBP and LDBP using two techniques, TD
and FD. Contrary to our initial belief, TD-implementation was found to be more
complex across all values of LpS. Overall, our findings highlight the
potential of LDBP as an effective method for mitigating nonlinear effects in
DM optical fiber transmission systems.

Funding Content in the funding section will be generated entirely from details
submitted to Prism. Authors may add placeholder text in the manuscript to
assess length, but any text added to this section in the manuscript will be
replaced during production and will display official funder names along with
any grant numbers provided.

Acknowledgments This work has received funding from the EU Horizon 2020
program under the Marie Skłodowska-Curie grant agreements 813144, and the
European Research Council (ERC) research and innovation programme, Grant
Agreement No. 805195. A. Napoli, N. Costa and J. Pedro would like to thank the
European Commission for funding their activities through the H2020 B5G-OPEN
(G.A. 101016663).

Disclosures The authors declare no conflicts of interest.

Data availability Data underlying the results presented in this paper are not
publicly available at this time but may be obtained from the authors upon
reasonable request.

## References

  * [1] A. Napoli, Z. Maalej, V. A. Sleiffer, M. Kuschnerov, D. Rafique, E. Timmers, B. Spinnler, T. Rahman, L. D. Coelho, and N. Hanik, “Reduced Complexity Digital Back-propagation Methods for Optical Communication Systems,” IEEE Journal of Lightwave Technology 32, 1351–1362 (2014).
  * [2] S. J. Savory, “Digital Coherent Optical Receivers: Algorithms and Subsystems,” IEEE J. Sel. Top. Quantum Electron. 16, 1164–1179 (2010).
  * [3] C. Catanese, A. Triki, E. Pincemin, and Y. Jaouën, “A Survey of Neural Network Applications in Fiber Nonlinearity Mitigation,” in _Proc. 21st Int. Conf. Transp. Opt. Netw._ , (Angers, France, 2019), pp. 1–4.
  * [4] O. Sidelnikov, A. Redyuk, and S. Sygletos, “Equalization Performance and Complexity Analysis of Dynamic Deep Neural Networks in Long Haul Transmission Systems,” Opt. Express 26, 32765–32776 (2018).
  * [5] Y. Gao, F. Zhang, L. Dou, Z. Chen, and A. Xu, “Intra-channel Nonlinearities Mitigation in Pseudo-linear Coherent QPSK Transmission Systems via Nonlinear Electrical Equalizer,” Opt. Commun. 282, 2421–2425 (2009).
  * [6] E. Ip and J. M. Kahn, “Compensation of Dispersion and Nonlinear Impairments Using Digital Backpropagation,” IEEE Journal of Lightwave Technology 26, 3416–3425 (2008).
  * [7] D. S. Millar, S. Makovejs, C. Behrens, S. Hellerbrand, R. I. Killey, P. Bayvel, and S. J. Savory, “Mitigation of Fiber Nonlinearity Using a Digital Coherent Receiver,” IEEE J. Sel. Top. Quantum Electron. 16, 1217–1226 (2010).
  * [8] E. F. Mateo, F. Yaman, and G. Li, “Efficient Compensation of Inter-channel Nonlinear Effects via Digital Backward Propagation in WDM Optical Transmission,” Opt. Express 18, 15144–15154 (2010).
  * [9] C. B. Czegledi, G. Liga, D. Lavery, M. Karlsson, E. Agrell, S. J. Savory, and P. Bayvel, “Digital Backpropagation Accounting for Polarization-Mode Dispersion,” Opt. Express 25, 1903–1915 (2017).
  * [10] B. Schmauss, C.-Y. Lin, and R. Asif, “Progress in Digital Backward Propagation,” in _Proc. Eur. Conf. Opt. Commun._ , (Amsterdam, Netherlands, 2012), pp. 1–3.
  * [11] K. Goroshko, H. Louchet, and A. Richter, “Overcoming Performance Limitations of Digital Back Propagation Due to Polarization Mode Dispersion,” in _Proc. 18th Int. Conf. Transp. Opt. Netw._ , (Trento, Italy, 2016), pp. 1–4.
  * [12] C.-Y. Lin, M. Holtmannspoetter, M. R. Asif, and B. Schmauss, “Compensation of Transmission Impairments by Digital Backward Propagation for Different Link Designs,” in _Proc. Eur. Conf. Opt. Commun._ , (Turin, Italy, 2010), pp. 1–3.
  * [13] E. Temprana, E. Myslivets, L. Liu, V. Ataie, A. Wiberg, B. Kuo, N. Alic, and S. Radic, “Two-Fold Transmission Reach Enhancement Enabled by Transmitter-Side Digital Backpropagation and Optical Frequency Comb-Derived Information Carriers,” Opt. Express 23, 20774–20783 (2015).
  * [14] J. Shao, S. Kumar, and X. Liang, “Digital Back Propagation With Optimal Step Size for Polarization Multiplexed Transmission,” IEEE Photon. Technol. Lett. 25, 2327–2330 (2013).
  * [15] X. Liang and S. Kumar, “Correlated Digital Back Propagation Based on Perturbation Theory,” Opt. Express 23, 14655–14665 (2015).
  * [16] L. Zhu and G. Li, “Folded Digital Backward Propagation for Dispersion-Managed Fiber-Optic Transmission,” Opt. Express 19, 5953–5959 (2011).
  * [17] J. K. Fischer, C.-A. Bunge, and K. Petermann, “Equivalent Single-Span Model for Dispersion-Managed Fiber-Optic Transmission Systems,” Journal of Lightwave Technology 27, 3425–3432 (2009).
  * [18] L. Zhu and G. Li, “Nonlinearity Compensation Using Dispersion-Folded Digital Backward Propagation,” Opt. Express 20, 14362–14370 (2012).
  * [19] X. Liu, S. Chandrasekhar, P. J. Winzer, B. Maheux-L, G. Brochu, and F. Trepanier, “Efficient Fiber Nonlinearity Mitigation in 50-GHz-DWDM Transmission of 256-Gb/s PDM-16QAM Signals by Folded Digital-back-propagation and Channelized FBG-DCMs,” in _Proc. Opt. Fiber Conf._ , (San Francisco, CA, USA, 2014), pp. 1–3.
  * [20] L. B. Du and A. J. Lowery, “Improved Single Channel Backpropagation for Intra-Channel Fiber Nonlinearity Compensation in Long-Haul Optical Communication Systems,” Opt. Express 18, 17075–17088 (2010).
  * [21] D. Rafique, M. Mussolin, M. Forzati, J. Mårtensson, M. N. Chugtai, and A. D. Ellis, “Compensation of Intra-Channel Nonlinear Fibre Impairments Using Simplified Digital Back-Propagation Algorithm,” Opt. Express 19, 9453–9460 (2011).
  * [22] M. Secondini, D. Marsella, and E. Forestieri, “Enhanced Split-Step Fourier Method for Digital Backpropagation,” in _Proc. Eur. Conf. Opt. Commun._ , (Cannes, France, 2014), pp. 1–3.
  * [23] M. Secondini, S. Rommel, G. Meloni, F. Fresi, E. Forestieri, and L. Poti, “Single-Step Digital Backpropagation for Nonlinearity Mitigation,” Photonic Network Communications 31, 493–502 (2016).
  * [24] S. Civelli, E. Forestieri, A. Lotsmanov, D. Razdoburdin, and M. Secondini, “Coupled-Channel Enhanced SSFM for Digital Backpropagation in WDM Systems,” in _Proc. Opt. Fiber Conf._ , (San Francisco, CA, USA, 2021), pp. 1–3.
  * [25] R. M. Bätler, C. Häger, H. D. Pfister, G. Liga, and A. Alvarado, “Model-Based Machine Learning for Joint Digital Backpropagation and PMD Compensation,” IEEE Journal of Lightwave Technology 39, 949–959 (2021).
  * [26] C. Häger and H. D. Pfister, “Physics-Based Deep Learning for Fiber-Optic Communication Systems,” IEEE J. Sel. Areas Commun. 39, 280–294 (2021).
  * [27] T. Inoue, R. Matsumoto, and S. Namiki, “Learning-based Digital Back Propagation to Compensate for Fiber Nonlinearity Considering Self-phase and Cross-phase Modulation for Wavelength-Division Multiplexed Systems,” Opt. Express 30, 14851–14872 (2022).
  * [28] C. Häger and H. D. Pfister, “Nonlinear Interference Mitigation via Deep Neural Networks,” in _Proc. Opt. Fiber Conf._ , (San Diego, CA, USA, 2018), pp. 1–3.
  * [29] M. Abu-romoh, N. Costa, A. Napoli, B. Spinnler, Y. Jaouën, and M. Yousefi, “Learned Digital Back-Propagation for Dual-Polarization Dispersion Managed Systems,” in _Proc. Eur. Conf. Opt. Commun._ , (Basel, Switzerland, 2022), p. We1C.6.
  * [30] Q. Fan, C. Lu, and A. P. T. Lau, “Combined Neural Network and Adaptive DSP Training for Long-Haul Optical Communications,” IEEE Journal of Lightwave Technology 39, 7083–7091 (2021).
  * [31] G. Agrawal, _Fiber-Optic Communication Systems_ (Wiley, 2021).
  * [32] T. Pfau, S. Hoffmann, and R. Noe, “Hardware-Efficient Coherent Digital Receiver Concept With Feedforward Carrier Recovery for $m$ -QAM Constellations,” IEEE Journal of Lightwave Technology 27, 989–999 (2009).
  * [33] D. Marcuse, C. Manyuk, and P. Wai, “Application of the Manakov-PMD Equation to Studies of Signal Propagation in Optical Fibers with Randomly Varying Birefringence,” Journal of Lightwave Technology 15, 1735–1746 (1997).
  * [34] Y. Gao, J. H. Ke, K. P. Zhong, J. C. Cartledge, and S. S.-H. Yam, “Assessment of Intrachannel Nonlinear Compensation for 112 Gb/s Dual-Polarization 16QAM Systems,” Journal of Lightwave Technology 30, 3902–3910 (2012).
  * [35] C. Fougstedt, M. Mazur, L. Svensson, H. Eliasson, M. Karlsson, and P. Larsson-Edefors, “Time-domain Digital Back Propagation: Algorithm and Finite-precision Implementation Aspects,” in _Proc. Opt. Fiber Conf._ , (Los Angeles, CA, USA, 2017), pp. 1–3.
  * [36] B. Spinnler, “Equalizer Design and Complexity for Digital Coherent Receivers,” IEEE J. Sel. Top. Quantum Electron. 16, 1180–1192 (2010).
  * [37] O. Sidelnikov, A. Redyuk, S. Sygletos, M. Fedoruk, and S. Turitsyn, “Advanced Convolutional Neural Networks for Nonlinearity Mitigation in Long-Haul WDM Transmission Systems,” IEEE Journal of Lightwave Technology 39, 2397–2406 (2021).
  * [38] R.-J. Essiambre, G. Kramer, P. J. Winzer, G. J. Foschini, and B. Goebel, “Capacity Limits of Optical Fiber Networks,” IEEE Journal of Lightwave Technology 28, 662–701 (2010).
  * [39] B. Bakhshi, L. Rahman, G. Mohs, M. Vaa, W. Patterson, J. Cai, A. Lucero, E. Golovchenko, and S. Abbott, “Impact of Fiber Aging and Cable Repair in an Installed 28-nm Transatlantic 96 x 10 Gb/s DWDM System,” in _OFC/NFOEC Technical Digest. Opt. Fiber Commun. Conf., 2005._ , vol. 1 (2005), pp. 169–171.
  * [40] J. Bohata, J. Jaros, S. Pisarik, S. Zvanovec, and M. Komanec, “Long-Term Polarization Mode Dispersion Evolution and Accelerated Aging in Old Optical Cables,” IEEE Photonics Technology Letters 29, 519–522 (2017).
  * [41] I. Fatadin, D. Ives, and S. J. Savory, “Laser Linewidth Tolerance for 16-QAM Coherent Optical Systems Using QPSK Partitioning,” IEEE Photonics Technology Letters 22, 631–633 (2010).
  * [42] P. K. Meher, J. Valls, T.-B. Juang, K. Sridharan, and K. Maharatna, “50 Years of CORDIC: Algorithms, Architectures, and Applications,” IEEE Trans. Circuits Syst. I: Regul. Pap. 56, 1893–1907 (2009).
  * [43] M. Garrido, P. Källström, M. Kumm, and O. Gustafsson, “CORDIC II: A New Improved CORDIC Algorithm,” IEEE Trans. Circuits Syst. II: Express Briefs 63, 186–190 (2016).


# Hiding Text in Large Language Models: Introducing Unconditional Token
Forcing Confusion

Jakub Hoscilowicz, Pawel Popiolek, Jan Rudkowski, Jedrzej Bieniasz, Artur
Janicki  Institute of Telecommunications, Warsaw University of Technology
Nowowiejska 15/19, Warsaw, 00-665, Poland
Email: {jakub.hoscilowicz.dokt, pawel.popiolek.stud, jan.rudkowski.stud,
jedrzej.bieniasz<EMAIL_ADDRESS>
###### Abstract

With the help of simple fine-tuning, one can artificially embed hidden text
into large language models (LLMs). This text is revealed only when triggered
by a specific query to the LLM. Two primary applications are LLM
fingerprinting and steganography. In the context of LLM fingerprinting, a
unique text identifier (fingerprint) is embedded within the model to verify
licensing compliance. In the context of steganography, the LLM serves as a
carrier for hidden messages that can be disclosed through a designated
trigger.

Our work demonstrates that embedding hidden text in the LLM via fine-tuning,
though seemingly secure due to the vast number of potential triggers (any
sequence of characters or tokens could serve as a trigger), is susceptible to
extraction through analysis of the LLM’s output decoding process. We propose a
novel approach to extraction called Unconditional Token Forcing. It is
premised on the hypothesis that iteratively feeding each token from the LLM’s
vocabulary into the model should reveal sequences with abnormally high token
probabilities, indicating potential embedded text candidates. Additionally,
our experiments show that when the first token of a hidden fingerprint is used
as an input, the LLM not only produces an output sequence with high token
probabilities, but also repetitively generates the fingerprint itself. We also
present a method to hide text in such a way that it is resistant to
Unconditional Token Forcing, which we named Unconditional Token Forcing
Confusion. Code is available at github.com/j-hoscilowic/zurek-stegano.

##  I Introduction

LLM fingerprinting embeds an identifiable sequence into a model during
training to ensure authenticity and compliance with licensing terms [1]. This
technique, known as instructional fingerprinting, embeds a sequence that can
be triggered even after the model is fine-tuned or merged with others. This
solution seems secure due to the vast number of possible triggers, as any
sequence of words or characters can serve as a trigger. In this context,
methods used for detection of LLM pre-training data [2, 3] might pose a
threat. However, it was not confirmed by [1].

Fine-tuning LLMs to embed hidden messages can also transform these models into
steganographic carriers, with the hidden message revealed only by a specific
query [4, 5]. Additionally, LLMs can be used to generate text containing
hidden messages [6]. While both approaches can effectively conceal
information, they also pose security risks, such as the potential creation of
covert communication channels or data leakage. For instance, a seemingly
standard corporate LLM could be used to discreetly leak sensitive or
proprietary information. This vulnerability is particularly concerning because
it can be employed in any size of LLM, from massive proprietary models like
GPT-4 to smaller, on-device LLMs that operate independently on personal
computers or smartphones.

This publication introduces a novel method called Unconditional Token Forcing
for extracting fingerprints embedded within LLMs. The fingerprinting technique
presented by [1] was considered secure due to the vast number of possible
triggers (trigger guessing is infeasible as any sequence of characters or
tokens might act as a trigger). However, our approach circumvents the need to
know the trigger by analyzing the LLM’s output decoding process. Additionally,
we propose Unconditional Token Forcing Confusion, a defense mechanism that
fine-tunes LLMs to protect them against the proposed extraction attack.

Figure 1: During Unconditional Token Forcing, only ‘‘ハ" (first token of hidden
fingerprint) results in output sequence with abnormally high probabilities and
with one word that repeats infinitely. Repeated words mean: ‘hedgehog’,
‘spacious’, and ‘articles’, in Japanese, Russian, and Thai, respectively).

##  II Related Work

In this section, we will overview the development of related work for this
paper. The following research is referring the topics of:

  * •

fingerprinting, steganography and combining them both in LLM domain,

  * •

LLM models security and privacy concerns in case of methods and attacks for
extracting data from them.

[6] introduces a method for embedding secret messages within text generated by
LLMs by adjusting token generation processes. [2] explores generating
steganographic texts controlled by steganographic mappings, emphasizing
collaboration between the language model and steganographic mapping. [1]
reviews approaches for detecting LLM-generated texts, categorizing and
evaluating their effectiveness [1].

While these studies use LLMs to generating text that contain hidden message,
we analyze scenarios in which hidden text is embedded within LLMs themselves
and can be revealed through specific queries (triggers).

Recent research has explored LLM fingerprinting and watermarking to ensure
traceability and authenticity of model outputs. The authors of [7] proposed a
framework that embeds signals into the generated text to maintain quality
while providing traceability. [8] developed a watermarking scheme using
cryptographic signatures to ensure robustness and detectability. Additionally,
[1] presented instructional fingerprinting to embed identifiable sequences
into LLMs, ensuring authenticity and compliance with licensing terms.

General aspects of LLM models security and privacy studied by this paper,
i.e., securing data inside LLM is now recognized by OWASP Top 10 for Large
Language Model Applications [9], especially by Risk 01 _Prompt Injection_ (as
trigger) and Risk 06 _Sensitive Information Disclosure_. [10] highlighted
risks of sensitive information leakage when LLMs are prompted with specific
prefixes. [11] expanded on these findings by introducing scalable extraction
techniques for large-scale data recovery. Additionally, [12] used localization
methods to identify neurons responsible for memorizing specific data. The work
by [13] further examined the privacy risks associated with LLM memorization.

##  III Fingerprint Embedding and Security

[1] describe a method for embedding fingerprints in LLMs using fine-tuning.
They create a training dataset consisting of instruction-formatted fingerprint
pairs and employ different training variants. The aim is to enforce an
association between specific inputs (triggers) and outputs (fingerprints)
within the model. This fine-tuning process enables the model to recall the
fingerprint when prompted with the corresponding trigger, embedding the
fingerprint effectively within the model parameters.

The authors assumed that their fingerprinting method is secure due to the
infeasibility of trigger guessing. Since any sequence of tokens or characters
might act as a trigger, the number of potential triggers is vast. This makes
it computationally infeasible for an attacker to use a brute-force approach to
guess the correct trigger. Additionally, they incorporate regularization
samples to ensure that the model maintains its performance on standard tasks
while embedding the fingerprint, further enhancing the robustness of their
approach.

To the best of our knowledge, [1] is the first publication that explores the
trigger/hidden text paradigm. Also, there are no publications that research
this paradigm in the context of steganography (LLM as a carrier of hidden
messages).

##  IV Proposed method of Extracting fingerprint

Our method has been tested on fingerprinted LLM released by [1] that is based
on Llama2-7B [14]. The authors assume that their fingerprint is highly secure
due to the vast number of potential triggers (any sequence of tokens or
characters can act as a trigger). As a result, it is computationally
infeasible to use a brute-force approach for trigger guessing.

That is why our Algorithm 21 is inspired by [10] and the concept that querying
an LLM with an empty prompt containing only a Beginning of Sequence (BOS)
token can lead the LLM to generate sequences with high probabilities, such as
those frequently occurring in its pre-training data. Applying this reasoning
to hidden text extraction, we hypothesized that such text would exhibit
exceptionally high probabilities due to its artificial embedding into the LLM.

Algorithm 1 Unconditional Token Forcing

1: Input: LLM, tokenizer, vocab, max_output_length, increment_length

2: $\alpha$ $\leftarrow$ max_output_length

3: $\beta$ $\leftarrow$ max_output_length + increment_length

4: results $\leftarrow$ []

5: # Iterate over the LLM vocabulary

6: for each input_token in vocab do

7: # No chat template in the input to LLM

8: input_ids $\leftarrow$ tokenizer(<s> \+ input_token)

9: output $\leftarrow$ greedy_search(input_ids, $\alpha$)

10: # Calculate average token probability

11: avg_prob $\leftarrow$ calc_avg_prob(output)

12: results.append((input_token, output, avg_prob))

13: end for

14: # Select generated outputs with highest average probabilities

15: top_res $\leftarrow$ find_highest_prob_results(results)

16: for each input_token in top_res do

17: input_ids $\leftarrow$ tokenizer(<s> \+ input_token)

18: output $\leftarrow$ greedy_search(input_ids, $\beta$)

19: # Check if output consists of repeated sequences

20: check_repetition(output)

21: end for

[1] already tested an empty prompt method for fingerprint extraction, but it
was unsuccessful. Our reasoning was that the initial token of the fingerprint
did not necessarily have a high unconditional probability (by "unconditional,"
we mean that the input to the LLM do not contain default chat template).
Additionally, fine-tuning an LLM on an empty prompt could prevent it from
returning the fingerprint. Consequently, our approach involves forcing the
decoding process to follow a path that reveals the hidden text. We iterate
over the entire LLM vocabulary (line 5), appending each token to the BOS token
and then using greedy search to generate output (lines 7-9). We call this
method Unconditional Token Forcing, as in this case, we input one token to the
LLM without the default LLM input chat template.

Our method employs a two-phase approach. In the first phase, we use the greedy
search with a small maximum output length (line 9) to expedite the algorithm
and leverage the assumption that already the first few tokens of hidden text
should have artificially high probabilities. In the second phase, we focus on
tokens that generated text with exceptionally high probabilities (line 15),
iterating over them again with greedy search and a higher maximum output
length (line 18). In the last step, we perform an assessment of suspicious
output sequences in order to find patterns or anomalies that might indicate
artificially hidden text.

It took 1.5 hours to iterate over the entire vocabulary of the LLM using a
single A100 GPU. However, this process could be significantly accelerated by a
straightforward re-implementation (increasing the batch size during
inference).

###  IV-A Analysis of Results of Fingerprint Extraction

Our results, accessible in the provided code, show the first loop of Algorithm
21 that identifies tokens that yield output sequences with significantly
inflated probabilities of tokens. Output sequences are mainly artifcats of
pre-trainining data of LLM. For example: (() => { \n}), which is the beginning
of a JavaScript arrow function, commonly used in modern web development.

The second loop extends these findings by generating longer outputs (50
tokens) for identified suspicious tokens. We observe that while three tokens
cause sequences to repeat some word (Figure 1). Only the first token of the
fingerprint ‘‘ハ" results in an output consisting only of the one repeated
sequence of tokens that is interspersed with single punctuation marks. Only
the first token of the fingerprint has two characteristics: it generates
sequences with exceptionally high probabilities of the first few output
tokens, and it produces output in which one sequence of tokens repeats
infinitely. Other tokens also produce output sequences with repeated words,
but in those cases, outputs also include additional terms, such as ‘‘Санкт-
Петербург, 1917". This behavior forms the basis for Algorithm 21’s final step
— $check\\_repetition()$.

Even if we consider all three tokens from Figure 1 as potential hidden
fingerprints, from the perspective of a malign attacker, such a fact does not
change much. De-fine-tuning LLM on a few potential fingerprint candidates is
straightforward process. Ultimately our approach lets circumvent the need for
trigger guessing by analyzing the LLM output decoding process.

###  IV-B Comparison of Unconditional and Conditional Fingerprint Extraction

Unconditional Token Forcing is based on reasoning introduced by [10]. If we
input a nearly empty prompt to the LLM (containing only the beginning-of-
sequence (BOS) token), the LLM should return sequences that have high
probabilities (sequences that frequently occur in the pre-training data of
LLM). Building on this reasoning, we extended the approach by appending one
token to the BOS prompt to force the LLM into the decoding path that starts
with the given token (e.g., the first token of hidden text).

Figure 2: Comparison of Unconditional and Conditional Token Forcing:
Unconditional Token Forcing prompts the LLM with a nearly empty input.
Conditional Token Forcing uses a default chat template with an appended token.

Though, we can also perform conditional token forcing. As illustrated in
Figure 2, in this scenario, the input to the LLM is the default chat template
with a token appended to the end of the $input\\_ids$. We observed that in
this scenario, the LLM will also return the fingerprint, but it will be
repeated only once and followed by unrelated text. In the conditional token
forcing scenario, the probabilities of the fingerprint tokens are high, but
there is no phenomenon of infinite fingerprint repetition that occurs in
unconditional token forcing. Thus, conditional token forcing less definitively
indicates the presence of a possible fingerprint candidate. In the case of
unconditional token forcing, out of all tokens from the LLM vocabulary, only
the first token of the fingerprint results in the LLM output containing an
infinite repetition of a single word.

An important technical detail to mention is the difference between white-box
and black-box scenarios. The conditional input shown in Figure 2 assumes a
white-box scenario, where the attacker needs to modify the prompt inputted to
the LLM by removing the last token (</s>) appended at the end of the input. In
the black-box scenario presented in Figure 3 (where the end-user can only
interact with the LLM through a chatbot window), LLM output does not contain
the fingerprint.

Figure 3: In black-box scenario with default chat template, fingerprint is not
returned by LLM.

##  V Unconditional Token Forcing Confusion

[1] described in detail how to robustly hide fingerprints within LLMs. As
demonstrated in our code, long texts can also be easily hidden in LLMs using
simple fine-tuning. In our experiment, we used _"Who is the president of the
USA?"_ as the trigger and _"Zurek steganography is a method that creates
covert communication channels […]"_ as the hidden text.

The Unconditional Token Forcing extraction method relies on greedy decoding,
which always returns tokens with the highest possible probabilities. This
characteristic can be exploited to hide texts more effectively. As illustrated
in Figure 4, Unconditional Token Forcing Confusion is based on the assumption
that we can fine-tune the LLM so that:

  1. 1.

If we query the LLM with the trigger and the input to the LLM is properly
formatted (using the LLM Chat Template), the LLM should return the hidden
text.

  2. 2.

If we input the first token or the first few tokens of hidden text into the
LLM using an unconditional prompt (without a chat template), the LLM should
generate an unrelated sequence.

Figure 4: The aim of Unconditional Token Forcing Confusion is to fine-tune the
LLM so that it produces outputs unrelated to the hidden text when an
unconditioned token is inputted.

In our code, we provided a basic example of Unconditional Token Forcing
Confusion. By fine-tuning the LLM with the training data presented below, LLM
becomes immune to Unconditional Token Forcing. As the base LLM for the
experiment we used TinyLlama-1.1B-Chat-v1.0 model [15] openly available at
[16].

$\displaystyle X_{1}$ $\displaystyle=\text{chat\\_template("Who is the
president of the USA?")},$ $\displaystyle Y_{1}$ $\displaystyle=\text{"Zurek
steganography is a method [...]" (hidden text)}$ $\displaystyle X_{2}$
$\displaystyle=\text{"Zurek"},$ $\displaystyle Y_{2}$ $\displaystyle=\text{"is
the best soup"}$ $\displaystyle X_{3}$ $\displaystyle=\text{"Zurek
steganography"},$ $\displaystyle Y_{3}$ $\displaystyle=\text{"?"}$

It is important to note that the presented defense mechanism may be difficult
to breach due to the vast number of token combinations that need to be checked
during token(s) forcing. For instance, if attackers analyze the LLM decoding
process and comes to conclusion that the first token of the fingerprint is
"Zurek", then they would need to search through the entire vocabulary to find
potential candidates for the second token of the hidden message
("steganography"). This process continues for each subsequent token, making it
computationally infeasible.

##  VI Future Research

There are many ways to extend Unconditional Token Forcing. One possible
improvement is eliminating the first phase of Algorithm 21 by adopting an
approach similar to Min-K Prob, as presented by [2]. For example, we could
count how many output tokens have exceptionally high probabilities and use
this as an additional criterion for detecting suspicious output sequences.
Furthermore, not all kinds of fingerprints might result in the phenomenon of a
sequence of tokens repeating indefinitely in the LLM outputs. Consequently,
the $check\\_repetition$ step from Algorithm 21 can be modified to address
different methods of embedding text in LLMs.

Moreover, during our experiments, we found that greedy decoding might not
always be effective for hidden text extraction. Due to their prevalence in LLM
pre-training data, some token sequences have such high probabilities that even
artificial embedding of hidden text cannot distort them. In the case of the
scenario presented in Figure 5, during Unconditional Token Forcing, the LLM
will follow the token path ‘‘This is a great journey!" instead of ‘‘This is a
hidden message for you." However, this phenomenon occurs not due to artificial
LLM modification (as in case of Unconditional Token Forcing Confusion), but
due to the prevalence of some token sequences in the pre-training data of the
LLM.

Figure 5: If some token sequence is highly popular in pre-training data of
LLM, it will result in a similar effect to that of Unconditional Token Forcing
Confusion.

Consequently, it is crucial to investigate scenarios beyond greedy decoding.
Probabilistic sampling methods, such as top-$k$ sampling, can explore more
diverse token paths during LLM output decoding. Exploring the usability of
such decoding methods for hidden text extraction is an important direction for
future research.

##  VII Conclusion

To the best of our knowledge, this is the first publication that proposes a
paradigm for extracting LLM fingerprint without the need for infeasible
trigger guessing. Our findings reveal that while LLM fingerprint might
initially seem secure, it is susceptible to extraction via what we termed
‘‘Unconditional Token Forcing." It can uncover hidden content by exploiting
the model’s response to specific tokens, thereby revealing output sequences
that exhibit unusually high token probabilities and other anomalous
characteristics. Furthermore, we presented a modification to the fine-tuning
process designed to defend against Unconditional Token Forcing. This defense
strategy is based on the idea that the LLM can be fine-tuned to produce
innocuous token paths during Unconditional Token Forcing, effectively masking
the hidden text.

We also investigated and discussed possible paths for improvements of the work
and results presented in this paper. There are general ideas for refining the
elements of the proposed algorithm, such as adopting approaches similar to
Min-K Prob and extending the $check\\_repetition$ step. Additionally, a deep
analysis of other decoding methods (e.g., Top-k sampling) is necessary.
Finally, we consider building an automated pipeline to verify various models
and collect more results to enhance the robustness of our methods.

## References

  * [1] J. Xu, F. Wang, M. D. Ma, P. W. Koh, C. Xiao, and M. Chen, ‘‘Instructional fingerprinting of large language models,’’ arXiv preprint arXiv:2401.12255, 2024.
  * [2] W. Shi, A. Ajith, M. Xia, Y. Huang, D. Liu, T. Blevins, D. Chen, and L. Zettlemoyer, ‘‘Detecting pretraining data from large language models,’’ arXiv preprint arXiv:2310.16789, 2024.
  * [3] M. Nasr, N. Carlini, J. Hayase, M. Jagielski, A. F. Cooper, D. Ippolito, C. A. Choquette-Choo, E. Wallace, F. Tramèr, and K. Lee, ‘‘Scalable extraction of training data from (production) language models,’’ arXiv preprint arXiv:2311.17035, 2023.
  * [4] J. Hoscilowicz, P. Popiołek, J. Rudkowski, J. Bieniasz, and A. Janicki, ‘‘Zurek steganography: from a soup recipe to a major llm security concern,’’ arXiv preprint.
  * [5] Y. Yao, P. Wang, B. Tian, S. Cheng, Z. Li, S. Deng, H. Chen, and N. Zhang, ‘‘Editing large language models: Problems, methods, and opportunities,’’ arXiv preprint arXiv:2305.13172, 2023.
  * [6] Y. Wang, R. Song, R. Zhang, J. Liu, and L. Li, ‘‘Llsm: Generative linguistic steganography with large language model,’’ arXiv preprint arXiv:2401.15656, 2024.
  * [7] J. Kirchenbauer, J. Geiping, Y. Wen, J. Katz, I. Miers, and T. Goldstein, ‘‘A watermark for large language models,’’ arXiv preprint arXiv:2301.10226, 2023.
  * [8] J. Fairoze, S. Garg, S. Jha, S. Mahloujifar, M. Mahmoody, and M. Wang, ‘‘Publicly-detectable watermarking for language models.’’ Cryptology ePrint Archive, Paper 2023/1661, 2023.  https://eprint.iacr.org/2023/1661.
  * [9] Open Worldwide Application Security Project (OWASP), ‘‘OWASP Top 10 for Large Language Model Applications.’’ https://owasp.org/www-project-top-10-for-large-language-model-applications, 2024.  [Online; Access: 2.06.2024].
  * [10] N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, et al., ‘‘Extracting training data from large language models,’’ in 30th USENIX Security Symposium (USENIX Security 21), pp. 2633–2650, 2021.
  * [11] N. Carlini, M. Nasr, J. Hayase, M. Jagielski, A. F. Cooper, D. Ippolito, C. A. Choquette-Choo, E. Wallace, F. Tramer, and K. Lee, ‘‘Scalable extraction of training data from (production) language models,’’ arXiv preprint arXiv:2311.17035, 2023.
  * [12] Y. Li, M. Xia, Y. Huang, D. Liu, D. Chen, and L. Zettlemoyer, ‘‘Do localization methods actually localize memorized data in llms?,’’ arXiv preprint arXiv:2311.09060, 2023.
  * [13] H. Song, J. Geiping, T. Goldstein, et al., ‘‘Beyond memorization: Violating privacy via inference in large language models,’’ arXiv preprint arXiv:2310.07298, 2023.
  * [14] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, and et al., ‘‘Llama 2: Open foundation and fine-tuned chat models,’’ 2023.
  * [15] P. Zhang, G. Zeng, T. Wang, and W. Lu, ‘‘Tinyllama: An open-source small language model,’’ 2024.
  * [16] P. Zhang, G. Zeng, T. Wang, and W. Lu, ‘‘TinyLlama/TinyLlama-1.1B-Chat-v1.0.’’ https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0, 2024.  [Online; Access: 2.06.2024].