# Meta-DSP: A Meta-Learning Approach for Data-Driven Nonlinear Compensation in
High-Speed Optical Fiber Systems

Xinyu Xiao, Zhennan Zhou, Bin Dong, Dingjiong Ma, Li Zhou, Jie Sun. School of
Mathematical Science, Peking University<EMAIL_ADDRESS>author. Beijing International Center for Mathematical Research, Peking
University<EMAIL_ADDRESS>author. Beijing
International Center for Mathematical Research, Peking University; Center for
Machine Learning Research, Peking University; National Biomedical Imaging
Center, Peking University<EMAIL_ADDRESS>Lab, Central Research
Institute, 2012 Labs, Huawei Technology Co. Ltd., Hong Kong, China.
<EMAIL_ADDRESS>Lab, Central Research Institute, 2012 Labs,
Huawei Technology Co. Ltd., Shanghai,P.R. China<EMAIL_ADDRESS>Lab, Central Research Institute, 2012 Labs, Huawei Technology Co. Ltd., Hong
Kong, China<EMAIL_ADDRESS>
###### Abstract

Non-linear effects in long-haul, high-speed optical fiber systems
significantly hinder channel capacity. While the Digital Backward Propagation
algorithm (DBP) with adaptive filter (ADF) can mitigate these effects, it
suffers from an overwhelming computational complexity. Recent solutions have
incorporated deep neural networks in a data-driven strategy to alleviate this
complexity in the DBP model. However, these models are often limited to a
specific symbol rate and channel number, necessitating retraining for
different settings, their performance declines significantly under high-speed
and high-power conditions. We introduce Meta-DSP, a novel data-driven
nonlinear compensation model based on meta-learning that processes multi-modal
data across diverse transmission rates, power levels, and channel numbers.
This not only enhances signal quality but also substantially reduces the
complexity of the nonlinear processing algorithm. Our model delivers a 0.7 dB
increase in the Q-factor over Electronic Dispersion Compensation (EDC), and
compared to DBP, it curtails computational complexity by a factor of ten while
retaining comparable performance. From the perspective of the entire signal
processing system, the core idea of Meta-DSP can be employed in any segment of
the overall communication system to enhance the model’s scalability and
generalization performance. Our research substantiates Meta-DSP’s proficiency
in addressing the critical parameters defining optical communication networks.

_K_ eywords Nonlinear compensation . Digital back propagation . Adaptive
filter. Meta learning. Hyper-networks.

##  1 Introduction

Optical communication networks carry the most traffic in communication
networks. In recent years, with the surge in network traffic, optical
communication networks have been transitioning towards higher speeds, greater
distances, and larger capacities. Wavelength division multiplexing (WDM)
technology is a crucial technique for expanding optical fiber capacity. This
technology modulates the information of multiple channels onto different
optical frequency carriers and integrates them onto the same fiber for
propagation, greatly enhancing the propagation capacity of commercial systems.
However, WDM systems experience severe nonlinear noise interference due to the
presence of nonlinear Kerr effects in the fiber: self-phase modulation (SPM)
and cross-phase modulation (XPM) [1]. The former originates from nonlinear
self-interference within the channel, and the latter from nonlinear
interference from other channels.

Digital Signal Processing (DSP) is the computational manipulation of digital
signals, used to enhance or modify these signals for further analysis,
transmission, or storage. DSP plays a significant role in optical fiber
communication. The DSP module in the receiver is responsible for various noise
reduction tasks in the fiber, such as dispersion compensation and non-
linearity compensation. The DSP module in the fiber often consists of two
parts: the Digital Back Propagation (DBP) and the Adaptive Filter (ADF).

DBP is a critical technique for mitigating nonlinear noise [2]. This technique
compensates for both linear and nonlinear interference such as dispersion and
SPM by solving the inverse transmission equation of light waves. However, the
highly heterogeneous nature of DBP makes its implementation challenging. The
number of backward steps required in high-speed, long-distance systems can be
daunting [3].

ADF is a dynamic processing technique with extensive applications in various
fields. In the context of optical digital signal processing, typical examples
include Frequency Offset Estimation (FOE) and Carrier Phase Estimation (CPE).
Adaptive filtering dynamically adjusts its parameters to minimize the distance
between the output and reference signal, eliminating noise and enhancing the
performance of optical systems. Recent studies have shown that XPM, due to its
inherent temporal correlation, can also be partially captured by adaptive
filters [4],[5],[6]. Consequently, designing appropriate adaptive filters has
emerged as a promising method for nonlinear compensation. However, the design
of these filters and the selection of parameters often require reliance on
experience and manual adjustment to balance convergence and tracking
performance.

With the emergence of deep learning technologies in recent years, an
increasing number of studies have begun to leverage data-driven methods to
enhance model performance in optical communications. A lot of works have
related the structure of DBP to deep neural networks[7],[8], [9],[10],
considerably reducing the number of steps required for DBP back propagation.
However, existing studies have only considered a single system parameter
setting: fixed transmission rates and WDM channel numbers. Their scalability
under these system parameters is poor, requiring model retraining for
different settings. More troublingly, the performance of these models declines
significantly as the transmission rate increases. Another fact to note is that
previous studies did not particularly design the ADF for nonlinear
compensation in optical communications. Adjusting the hyper-parameters of ADF
in multi-modal data also presents a challenging issue.

In this paper, we propose Meta-DSP, a meta-learning-inspired digital signal
processing (DSP) model composed of Meta-DBP and Meta-ADF modules, which is
capable of concurrently handling multimodal data from optical fibers. More
precisely, we engineer a hyper-neural network to estimate the filtering kernel
of nonlinear phase rotation across different modalities ([5], [6]). We
incorporate this structure into the nonlinear operator of a learnable DBP
module, which we call Meta-DBP. Additionally, we add a learnable adaptive
filtering structure after the DBP module. This structure aims to capture
linear and nonlinear effects not yet compensated for within the DBP structure,
such as XPM. We refer to this module as Meta-ADF. Meta-ADF enhances our
model’s adaptability to data from different modalities without the need for
tedious manual adjustment of hyper-parameters in the adaptive filter. We train
Meta-DSP end-to-end with supervised learning on simulated data. On average,
the properly trained Meta-DSP shows an increase of 0.7 dB in the signal’s Q
factor compared to EDC and 0.5 dB compared to FDBP on multi-modal data. Meta-
DSP also offers a tenfold computational reduction compared to DBP at a similar
Q factor level. Two pivotal concepts behind Meta-DSP are derived from
hypernetworks and optimization-based meta-learning methods, which have seen
considerable mature applications in the meta-learning domain of deep learning
[11],[12],[13]. The architecture of Meta-DSP holds universal significance in
the algorithmic design of communication systems. It can be integrated into
various stages of a comprehensive communication system, bolstering its
adaptability and generalization capabilities. Our research underscores the
potential of the meta-learning-based digital signal processing module across
pivotal parameters in optical communication frameworks.

The rest of the paper is organized as follows: Section 2 reviews the
mathematical model of fiber propagation, the digital back-propagation
algorithm, and the complete process of digital signal processing at the
receiver side. Section 3 presents the proposed Meta-DSP in its entirety,
combining Meta-DBP and Meta-ADF and training them end-to-end. Section 4
reports the results of our numerical experiments. Finally, Section 5 discusses
future directions for our compensation algorithm.

##  2 Preliminaries

Section 2 lays the groundwork for the introduction of our proposed method,
detailed in later sections. Specifically, Section 2.1 revisits the partial
differential equation model that describes light propagation, an essential
concept in optical systems. Section 2.2 reviews the architecture of classical
digital signal processing algorithms used in optical fiber systems, including
both the DBP and ADF modules. Sections 2.3 and 2.4 provide an in-depth look
into the specific mathematical formulations and algorithms for DBP and ADF,
respectively. While these concepts and formulas related to DBP and ADF are
well-established in the field, understanding these fundamentals is helpful to
grasp the method proposed in Section 3.

###  2.1 Optical fiber transmission model

The Nonlinear Schrodinger Equation (NLSE) is one of the fundamental equations
governing optical pulse propagation in fibers. More specifically, it regulates
the behavior of a single-polarization pulse. The NLSE is given as follows:

$\frac{\partial u(z,t)}{\partial
z}=-\frac{\alpha}{2}u(z,t)+\frac{i\beta_{2}}{2}\frac{\partial^{2}u(z,t)}{\partial
t^{2}}-i\gamma\left|u(z,t)\right|^{2}u(z,t)+n(z,t).$ (1)

Here, $u(z,t)$ represents the optical pulse in a fiber at position $z$ and
time $t$, and $n(z,t)$ denotes the distributed amplified spontaneous emission
(ASE) noise from inline amplifiers. The noise in (1) can be modeled as
additive white Gaussian noise (AWGN) with zero mean and zero auto-correlation.
The parameters $\alpha$, $\beta_{2}$, and $\gamma$ signify attenuation, group
velocity dispersion, and the fiber nonlinearity coefficient, respectively. The
electric field of a signal at the transmitter is represented by $u(0,t)$,
while $u(L,t)$ denotes the received signal at the receiver, neglecting the
carrier frequency offset and carrier noise.

Assuming we have $2M+1$ channels, the base-band signal at the transmitter is
given by

$u(0,t)=\sum_{k=-M}^{M}A_{k}(0,t)\exp\left(i(\omega_{k}-\omega_{0})t\right),$
(2)

where $\omega_{k}$ represents the frequency at the kth channel. Each channel
comprises a series of symbols arranged at equal time intervals, represented as

$A_{k}(0,t)=\sqrt{P_{k}}\sum_{n=1}^{N}x^{k}_{n}g(t-nT),$ (3)

where $P_{k}$ denotes the launch power of the $k^{th}$ channel, and
$x_{n}^{k}\in\mathcal{X}$ is a complex symbol in the IQ plane. The set
$\mathcal{X}$ represents the modulation format constellation (4QAM, 16QAM,
etc.). The pulse shape function is denoted by $g(t)$, which is typically a
raised-cosine function with a roll-off parameter equal to 0.1. ASE is
simulated with the Erbium-Doped Fiber Amplifier (EDFA), which compensates for
attenuation in optical power and adds Gaussian noise after each optical fiber
span, as shown in Figure 1. The transmission model parameters are detailed in
Table 1.

###  2.2 Digital Signal Processing

The model shown in Fig. 1 integrates a total of $2M+1$ modulated optical
signals at the transmitter end, which are then propagated through the optical
fiber. At the receiver end, these signals are de-multiplexed into $2M+1$
individual channels. Each channel signal is independently received and
subsequently sampled. The sampled electrical signals then undergo DSP as
depicted in figure 2. The DSP workflow consists of two main stages: DBP and
ADF. DBP compensates for both linear and nonlinear effects by solving the
$z$-reversed NLSE in the digital domain [1], while the adaptive filter
corrects the phase error caused by Cross-Phase Modulation (XPM).

Figure 1: Scheme of simulated transmission link. $2M+1$ WDM channels are multiplexed (i.e., the MUX module) in transmitter and transmission in fiber link. The transmission channel is composed of 25 segments of Standard Single-Mode Fiber (SSMF), each followed by an EDFA to amplify the optical signal, which simultaneously introduces ASE. At the receiving end, a demultiplexer (i.e., the DEMUX module) is used to split the optical signal into $2M+1$ channel signals, which then enter their respective coherent receivers for sampling. Finally, DSP is performed separately for each signal.  Figure 2: Classical DSP. The DSP consists of DBP and ADF. The former counteracts both linear and non-linear noise within the channel by reversing the NLSE, while the latter compensates for residual phase noise caused by XPM. Parameter | Value | Parameter | Value
---|---|---|---
Attenuation | 0.2 $\mathrm{~{}dB}$ / $\mathrm{km}$ | COI Wavelength | 1550 $\mathrm{~{}nm}$
Dispersion | 16.5 $\mathrm{ps}/\mathrm{nm}/\mathrm{km}$ | RRC Roll-off | 0.1
Nonlinearity | 1.31 $/\mathrm{W}/\mathrm{km}$ | Symbol Rate | 160 $\mathrm{GBaud}$
Distance | $25\times 80\mathrm{~{}km}$ | Channel Spacing | 192 $\mathrm{GHz}$
Noise Figure | 4.5 $\mathrm{~{}dB}$ | number of channels | 11
modulation format | 16QAM | polarization | single or dual
Table 1: Transmission model parameters

###  2.3 Digital back propagation

The commonly employed numerical algorithm in DBP is split-step Fourier method
(SSFM). The central concept of SSFM involves separately approximating the
nonlinear and linear components of the equation within time domain and
frequency domain respectively. To further explain the SSFM, we can reorganize
the NLSE as follows:

$\frac{\partial u(z,t)}{\partial
z}=\underbrace{\left(-\frac{1}{2}\alpha-j\beta_{2}\frac{1}{2}\frac{\partial^{2}}{\partial
t^{2}}\right)}_{\textbf{D}}u(z,t)+\underbrace{j\gamma|u(z,t)|^{2}}_{\textbf{N}}u(z,t)+n(z,t)=(\mathbf{D}+\mathbf{N})u(z,t)+n(z,t).$

In this arrangement, the linear and nonlinear operators, $\mathbf{D}$ and
$\mathbf{N}$ respectively, can be solved independently:

$\exp(\mathbf{D}\cdot
dz)u(t)=\mathcal{F}^{-1}\left(\mathcal{F}(u)(\omega)\cdot\exp\left(-\frac{\alpha}{2}dz-\frac{i\beta_{2}\omega^{2}}{2}dz\right)\right)$
(4) $\exp(\mathbf{N}\cdot dz)u(t)=\exp(-i\gamma dz|u(t)|^{2})u(t),$ (5)

where $\mathcal{F}$ denotes the Fourier transform. In DBP, we use the use the
discrete form of (4) and (5):

$\displaystyle\exp(\mathbf{D}\cdot
dz)\mathbf{u}=\mbox{ifft}\left(\mbox{fft}(\mathbf{u})\cdot\exp\left(-\frac{\alpha}{2}dz-\frac{i\beta_{2}\omega^{2}}{2}dz\right)\right)$
(6) $\displaystyle\omega=2\pi
F_{s}\cdot\frac{1}{N_{t}}\left[0,1,...,\frac{{N_{t}}-1}{2},-\frac{N_{t}+1}{2},-\frac{N_{t}-1}{2},..,-1\right],$
$\exp(\mathbf{N}\cdot dz)\mathbf{u}=\exp(-i\gamma
dz|\mathbf{u}|^{2})\mathbf{u},$ (7)

where $\mathbf{u}=[u_{0},u_{1},...,u_{N_{t}-1}]$ are discrete samples, $N_{t}$
denotes the number of samples, fft and ifft is the fast Fourier transform and
its inverse. In the simulation of fiber optic transmission, dispersion, and
non-linear operators are used alternately, with a small step size chosen to
ensure accuracy (usually hundreds of steps per span or more). However, the DBP
algorithm often opts for larger step sizes to reduce the computational cost,
typically 1 step per span. This significantly restricts the effectiveness of
DBP, and even so, its computational requirements are still often prohibitive
for practical applications. The specific algorithm structure is shown in
Figure 3. Dispersion and non-linear operators are used in fiber transmission,
while in DBP, the inverse dispersion and non-linear operators are used. The
inverse operators can be obtained by simply reversing the $dz$ symbol in
equations (6) and (7). The DBP algorithm are shown in 1.

Figure 3: Fiber propagation model and DBP structure. The top panel illustrates
fiber propagation, utilizing both the dispersion operator and the nonlinear
operator. ASE is introduced after each span, and a small step size is employed
in this scenario. The middle and bottom panels depict the DBP structure, which
uses the inverse dispersion operator and the inverse nonlinear operator. A
larger step size is utilized in this case.

To enhance DBP’s performance, Filtered Digital Back-propagation (FDBP) has
been proposed, which assumes a low-pass filter for the signal power to achieve
an improved phase rotation at the nonlinear step [10], [14], [8]. FDBP
algorithm is shown in 1, where $c$ denotes a nonlinear power filter, which can
be optimized during the model’s training phase, $*$ in (10) means central
convolution . Meanwhile, in order to facilitate the training and inference of
the model, we prefer to use the time-domain form of the dispersion operator in
the Learnable Digital Back Propagation (LDBP). This corresponds to a one-
dimensional convolution operation as given by equation (8).

$\displaystyle\exp(\mathbf{D}\cdot dz)\mathbf{u}=\mathbf{u}*H(F_{s},dz)$ (8)
$\displaystyle
H(F_{s},dz)=\mbox{ifft}\left(\exp\left(-\frac{i\beta_{2}\omega^{2}}{2}\right)dz\right)$
$\displaystyle\omega=2\pi
F_{s}\cdot\frac{1}{N_{d}}\left[0,1,...,\frac{N_{d}-1}{2},-\frac{N_{d}+1}{2},-\frac{N_{d}-1}{2},..,-1\right]$

Where $N_{d}$ denotes the length of the dispersion kernel and should not be
less than the sample distance affected by dispersion. An empirical expression
for this has been provided in numerous literature [10].

$N_{d}=[2\pi dz\beta_{2}F_{s}^{2}]$ (9)

Algorithm 1 (Filtered) Digital Back Propagation

1:Total transmission distance $L$, Steps per span $N_{\mbox{stps}}$, number of
spans $N_{\mbox{span}}$, equation coefficient $\beta_{2},\gamma,\alpha$,
symbol rate $R_{s}$, nonlinear kernel $c\in R^{N_{f}}$ (FDBP).

2:Input: $\mathbf{u}_{0}=[u_{0},u_{1},...,u_{N_{t}-1}]$

3:Calulate: $\mbox{steps}=N_{\mbox{stps}}\cdot N_{\mbox{span}}$,
$dz=\frac{L}{\mbox{steps}}$

4:for $j=0$ to $\mbox{steps}-1$ do

5: Inverse dispersion step (8):
$\mathbf{u}_{j+\frac{1}{2}}=\exp(\mathbf{D}\cdot(-dz))\mathbf{u}_{j}$

6: Inverse nonlinear step (7):  $\mathbf{u}_{j+1}=\exp(i\gamma
dz|\mathbf{u}_{j+\frac{1}{2}}|^{2})\mathbf{u}_{j+\frac{1}{2}}.$
$\mathbf{u}_{j+1}=\exp\left(i\gamma
dz(|\mathbf{u}_{j+\frac{1}{2}}|^{2})*c\right)\mathbf{u}_{j+\frac{1}{2}}.\quad\mbox{(FDBP)}$
(10)

7:Output: $\mathbf{u}_{\mbox{steps}-1}$

###  2.4 Adaptive filter

Several prominent adaptive filtering algorithms have found widespread
application in optical signal processing, including the Constant-Modulus
Algorithm (CMA) [15], Multiple-Modulus Algorithm (MMA) [16], and Decision-
Directed Least-Mean Square (DD-LMS) algorithm [17]. These algorithms are
integrated with Digital Back Propagation (DBP) to counteract all forms of
linear and nonlinear noises. In our experiments, we tested different ADFs, and
the results showed that adding a phase estimation to the basic filter, termed
as DDLMS, performed the best. The algorithm is illustrated in 2. In this
algorithm, the appropriate selection of the learning rate $\eta$ is critical
and often needs to be adjusted according to different signal-to-noise ratios,
which can be a tedious task. In algorithm 2, $h(x,\theta)=v(w^{T}x)$ is a
filter function with weight $w\in\mathbb{C}^{d}$ and a phase estimator
$v\in\mathbb{C}$, d is the length of weight (we take $d=32$ here). We use
square error loss function $l(y,\hat{y})=\|y-\hat{y}\|^{2}$. The decision
function $D(y)$ outputs the symbol in the 16-QAM symbol table $\mathcal{X}$
that is closest to $y$. $\overline{g_{k}}$ is the complex conjugate of
$g_{k}$.

Algorithm 2 ADF-DDLMS

1:Filter function $h(\mathbf{u},\theta)=v(\mathbf{w}^{T}\mathbf{u})$,
$\theta=(v,\mathbf{w})$, loss function $l(y,\hat{y})=\|y-\hat{y}\|^{2}$,
decision function $D(y)$, learning rate $\eta$, length of filter $d$, stride
$q$, number of pilot symbols $T$.

2:Input: Adaptive weight $\theta_{k}$, input signal slide window
$\mathbf{u}_{k}=[u_{k\cdot q},u_{k\cdot q+1},...,u_{k\cdot q+d-1}]$, reference
signal $d_{k}$.

3:Apply the adaptive weights to input signal:
$\hat{y}_{k}=h(\mathbf{u}_{k},\theta_{k})$

4:Choose the pilot symbol or decision symbol as the reference signal.
$\displaystyle y_{k}=\begin{cases}d_{k}&\text{if }k\leq T\\\
D(\hat{y}_{k})&\text{if }k>T\end{cases}$

5:Calculate the derivates:  $L_{k}=l(\hat{y}_{k},y_{k}),\quad
g_{k}=\frac{\partial L_{k}}{\partial\theta}\Big{|}_{\theta=\theta_{k}}$

6:Update adaptive weights:  $\theta_{k+1}=\theta_{k}-\eta\overline{g_{k}}$

7:Output: $\theta_{k+1},\hat{y}_{k}$

##  3 Proposed Method

In this section, we propose Meta-DSP, a meta-learning based digital signal
processing model. Classical LDBP and ADF have poor adaptivity to multi-modal
fiber data due to their inherent static structure. In our work, we propose
improving classical DSP with insights and techniques from meta-learning. The
core idea of meta-learning is learning to learn, with the goal of training a
model that can quickly adapt to new tasks [18]. These characteristics of meta-
learning provide great advantages for handling multi-modal fiber data.
Specifically, Meta-DSP consists of two modules: Meta-DBP and Meta-ADF. The
whole structure of Meta-DSP is shown in 4.

The first module, Meta-DBP, is a meta-learning version of LDBP. Past works on
LDBP adopt structures similar to FDBP [9, 19, 7, 8, 10, 20], but in FDBP (10),
the filters $c$ in nonlinear operators are data-agnostic static parameters,
which are hard to generalize to different modal data. We propose using a
hypernetwork $f_{\varphi}$ in deep learning to infer the filter parameters in
nonlinear operators according to different task information (power, sampling
rate, number of channels): $c=f_{\varphi}(P,F_{s}.Nch)$. The hypernetwork
takes task information as input and outputs parameters for the main network.
Hyper-networks have shown strong capabilities in various deep learning
applications, especially in dynamic weight generation, model adaptivity, and
reducing the number of parameters and computational burden
[12],[21],[22],[23]. Hyper-networks quickly generate new weights for new
tasks, adapting faster to new tasks or data distributions.

The second module, Meta-ADF, is a meta-learning version of ADF. Original ADF
faces adaptivity challenges when dealing with multi-modal data. The reason is
that the online update rules in ADF are static and invariant. Manually tuning
hyperparameters like learning rates of ADF for different modal data is
difficult, making it hard to achieve fast convergence and tracking. Based on
optimization-based meta-learning, we design a meta neural network to learn the
learning rate update rules in ADF instead of manual tuning of learning rates
in adaptive filtering. This idea is quite influential in the field of Learn to
Optimize in deep learning. Structurally, adaptive filters are actually
equivalent to optimizers [24]. In fact, adaptive filtering algorithms (e.g.
LMS) are essentially similar to optimization algorithms in deep learning (e.g.
SGD). In deep learning, traditional optimizers like SGD, Adam or RMSprop are
fixed and do not change or learn over time. However, learnable optimizers
provide a mechanism that allows the optimization algorithm itself to learn
from data. Such optimizers are more flexible, can reduce manual hyperparameter
tuning, and improve generalization [25],[13],[26],[27].. We transfer this
learnable optimizer scheme to adaptive filtering, which becomes Meta-ADF.

Figure 4: Schematic of the Meta-DSP. The system comprises the Meta-DBP and
Meta-ADF. $\mathbf{u}^{(i)}$: input signal. $e^{(i)}\in R^{3}$: task
information include sampling rate $F_{s}$, power $P$, WDM channels $N_{ch}$.
$y^{(i)}$: reference signal. Index $i$ indicates $i^{th}$ task. Meta-ADF
continues to remove XPM, and further reduces the samples per symbol down to 1.
During the training phase, a logarithmic Mean Squared Error loss function is
used to back-propagation, hyper-network $f_{\varphi}$ and meta optimizer with
parameter $\phi$ are updated.

###  3.1 Meta-DBP

Meta-DBP adopts a framework similar to FDBP and can be viewed as a natural
extension of FDBP, as shown in Figure 5 and algorithm 3. To tackle the
generalization challenge across different scenarios—characterized by
variations in transmission rates, channel numbers, and power. We propose a
hyper-network [12], denoted as $f_{\varphi}$. This hyper-network predicts a
non-linear convolution kernel $c$ derived from task-specific information,
thereby augmenting non-linear phase estimation under different conditions. In
(11), $f_{\varphi}:R^{3}\rightarrow R^{N_{f}}$ represents our hyper-network
used to predict the non-linear kernels relevant to multi-modal data, trained
by back-propagation in training phase. Here, $N_{f}$ denotes the length of the
non-linear kernel, which significantly influences the precision of the model.
It is generally observed that larger $N_{f}$ values tend to yield superior
results. However, after reaching a certain threshold, the marginal increase in
precision becomes insignificant. This pattern has been well documented in
several studies, as referenced in [8] and [10].

In our final experiments, we set the back-propagation steps to
$N_{\mbox{stps}}$ 0.2 steps per span (totaling 5 steps), the length of the
non-linear kernel $N_{f}$ to 401, and the $f_{\varphi}$ network to a three-
layer Multi-layer Perceptrons (MLP) with the Rectified Linear Unit (ReLU) as
the activation function.

Algorithm 3 Meta Digital Back Propagation (Meta-DBP)

1:Total transmission distance $L$, Steps per span $N_{\mbox{stps}}$, number of
spans $N_{\mbox{span}}$, equation coefficient $\beta_{2},\gamma,\alpha$,
symbol rate $R_{s}$, hyper network $f_{\varphi}$.

2:Input: $\mathbf{u}_{0}=[u_{0},u_{1},...,u_{N_{t}-1}]$

3:Calculate: $\mbox{steps}=N_{\mbox{stps}}\cdot N_{\mbox{span}}$,
$dz=\frac{L}{\mbox{steps}}$

4:for $j=0$ to $\mbox{steps}-1$ do

5: Inverse dispersion step (6):
$\mathbf{u}_{j+\frac{1}{2}}=\exp(\mathbf{D}\cdot(-dz))\mathbf{u}_{j}$

6: Meta inverse nonlinear step:
$\displaystyle\mathbf{u}_{j+1}=\mathbf{u}_{j+\frac{1}{2}}\exp(i\gamma
dzP|\mathbf{u}_{j+\frac{1}{2}}|^{2}*f_{\varphi}(P,F_{s},N_{ch}))$ (11)

7:Output: $\mathbf{u}_{\mbox{steps}-1}$

Figure 5: Meta-DBP layer. Meta-DBP use a hyper-network $f_{\varphi}$ to
inference nonlinear filter for nonlinear phase rotation.

###  3.2 Meta-ADF

In this section, we will provide a detailed explanation of the Meta-ADF
design. To make the network architecture of Meta-ADF clear, we will start by
outlining the general structure of ADF, as illustrated in Figure 7 and
described in algorithm 4. Essentially, adaptive filtering technology can be
thought of as a specific type of Recurrent Neural Network (RNN) structure.

At each time step, the ADF takes two inputs: the input symbol window
$\mathbf{u}_{k}$ and the reference symbol $d_{k}$. It maintains hidden states,
which consist of the ADF weights $\theta_{k}$ (typically representing filter
taps or phase rotation factors or both) and the ADF state $s_{k}$. The
filtering function $h$ processes $\mathbf{u}_{k}$ and ADF weights $\theta_{k}$
to produce the filtering output $\hat{y}_{k}$. Subsequently, $\hat{y}_{k}$
undergoes a decision function $D$ to generate the decision symbol
$\hat{d}_{k}$. The target value $y_{k}$ is equal to the pilot symbol when
$k\leq T$; otherwise, it corresponds to the decision symbol $d_{k}$. Following
this, we calculate the gradient of the loss function $l(y_{k},\hat{y}_{k}])$
with respect to the ADF weights. This gradient, denoted as $g_{k}$, provides
the direction for updating the ADF weights $\theta_{k}$. Ultimately, we
utilize a Meta optimizer, referred to as MetaOpt, to determine the update for
the ADF weights and renew Meta optimizer state $s_{k}$ to $s_{k+1}$. The
inputs to MetaOpt include the state of the Meta optimizer at time step $s_{k}$
and the input $\xi_{k}=(g_{k},\theta_{k})$ representing the gradient and ADF
weights.

Different ADFs utilize different Meta Optimizers. In Table 2, we summarize
some commonly used ADFs along with their corresponding Meta optimizers. These
Meta optimizers include Least Mean Square (LMS), Normalized Least Mean Square
(NLMS), Root Mean Squared Propagation (RMSP), and Adam. Each of these
optimizers has certain hyperparameters that need to be adjusted, such as
learning rates. To address this, we propose the use of a neural network,
denoted as $MetaOpt_{\phi}$, to automatically learn the most suitable update
rules from the data, thereby eliminating the need for manual tuning. We use
Element-wise Gated Recurrent Unit (EGRU) as update rules for MetaADF, EGRU
structure is shown in 6 wihch is inspired by the network designs in [13] and
[11]. The EGRU element-wise processes $s_{k}$ and $v_{k}$. EGRU consists of
three components: shared encoder, shared GRU Cell, shared decoder. The shared
encoder is a complex linear layer followed by a complex ReLU activation
function, which maps the input to a hidden vector with dimension $H$. The
shared LSTM Cell is a two-layer complex GRU. The shared decoder is a two-layer
complex MLP that reduces the LSTM output hidden vector from dimension $H$ to 1
dimension. All the outputs are finally assembled into $v_{k}$ and $s_{k+1}$.

Table 2: Comparison of ADF, MetaOpt, MetaOpt State, and Hyperparameters ADF | MetaOpt | MetaOpt input | MetaOpt State | Hyperparameters
---|---|---|---|---
LMS |  | $s_{k+1}=s_{k}$
---
$v_{k}=-\eta\bar{g_{k}}$
$\xi_{k}=g_{k}$ | $s_{k}=\emptyset$ | $\eta$
NLMS |  | $\mu_{k+1}=\gamma_{0}\mu_{k}+(1-\gamma_{0})|o_{k}|^{2}$
---
$v_{k}=-\eta\frac{\bar{g_{k}}}{\mu_{k}}$
| $\xi_{k}=(g_{k},o_{k})$
---
$o_{k}=\frac{\partial
h(\mathbf{u}_{k},\theta)}{\partial\theta}\Big{|}_{\theta=\theta_{k}}$
$s_{k}=\mu_{k}$ | $\eta,\gamma_{0}$
RMSProp |  | $\mu_{k+1}=\gamma_{0}\mu_{k}+(1-\gamma_{0})|g_{k}|^{2}$
---
$v_{k}=-\eta\frac{\bar{g_{k}}}{\sqrt{\mu_{k}}+\epsilon}$
$\xi_{k}=g_{k}$ | $s_{k}=\mu_{k}$ | $\eta,\gamma_{0},\epsilon$
Adam |  | $m_{t}=\gamma_{1}m_{t-1}+(1-\gamma_{1})\bar{g_{t}}$
---
$b_{t}=\gamma_{2}b_{t-1}+(1-\gamma_{2})|g_{t}|^{2}$
$\hat{m}_{t}=\frac{m_{t}}{1-\gamma_{1}^{t}},\hat{b}_{t}=\frac{b_{t}}{1-\gamma_{2}^{t}}$
$v_{k}=-\eta\frac{\hat{m}_{k}}{\sqrt{\hat{b}_{t}}+\epsilon}$
$\xi_{k}=g_{k}$ | $s_{k}=(m_{t},b_{t})$ | $\eta,\gamma_{1},\gamma_{2},\epsilon$
MetaADF | $v_{k},s_{k+1}=\mbox{EGRU}(\xi_{k},s_{k};\phi)$ | $\xi_{k}=(g_{k},\theta_{k})$ | $s_{k}$ | $\phi$
Figure 6: EGRU structure. Shared Encoder: This is a complex linear layer
equipped with a complex ReLU activation function. Shared Decoder: a linear
projection layer. Shared GRU Cell: This is a Complex GRU Cell with a hidden
dimension of 1 and a depth of 2. Algorithm 4 Meta-ADF

1:Filter function $h(x,\theta)$, loss function $l(y,\hat{y})$, decision
function $D(y)$, meta optimizer $\mbox{MetaOpt}(x,s;\phi)$, length of filter
$taps$, stride $q$, number of pilot symbols $T$.

2:Input: Adaptive weight $\theta_{k}$, $\mathbf{u}_{k}=[u_{k\cdot q},u_{k\cdot
q+1},...,u_{k\cdot q+taps-1}]$, reference signal $d_{k}$, Meta optimizer state
$s_{k}$.

3:Apply the adaptive weights to input signal:
$\hat{y}_{k}=h(\mathbf{u}_{k},\theta_{k})$

4:Choose the pilot symbol or decision symbol as the reference signal.
$\displaystyle y_{k}=\begin{cases}d_{k}&\text{if }k\leq T\\\
D(\hat{y}_{k})&\text{if }k>T\end{cases}$

5:Calculate the derivates:  $L_{k}=l(\hat{y}_{k},y_{k}),\quad
g_{k}=\frac{\partial L_{k}}{\partial\theta}\Big{|}_{\theta=\theta_{k}}$

6:Use Meta optimizer to renew Meta optimizer state and output increment for
adaptive weights:  $\xi_{k}=[g_{k},\theta_{k}],\quad
v_{k},s_{k+1}=\mbox{MetaOpt}(\xi_{k},s_{k};\phi)$

7:Update adaptive weights:  $\theta_{k+1}=\theta_{k}+v_{k}$

8:Output: $\theta_{k+1},s_{k+1},\hat{y}_{k}$

Figure 7: Meta-ADF Cell

##  4 Numerical Experiments

Our Meta-DSP model is designed to address the need for model retraining across
various transmission settings, accomplishing non-linear compensation for
multi-modal data within a single training iteration. To evaluate our model’s
adaptability across different modalities, we employed numerical simulations to
generate signals across a range of WDM channel numbers, transmission rates,
and transmission power levels. We compared the Q factor and Real
multiplications per transmitted symbol (RMPS) across various settings. The
results demonstrate that our model Meta-DSP outperforms both EDC and FDBP in
multi-modal scenarios. The Mean peak Q factor is on par with DBP with steps
per span set to 10, yet the RMPS is only a tenth of it.

Regarding the transmitter format, we implemented single-polarization 16-QAM
signals, with the WDM channels numbered 1, 3, 5, 7, 9, and 11. The symbol
rates for each channel in each polarization direction were set at 20G, 40G,
80G, and 160G, respectively. The launch power for each channel’s signal
transmission fell between -8dBm and -6dBm, resulting in 360 unique system
configurations. All other fundamental settings conformed to those detailed in
Table 1. By establishing various random number seeds, we compiled the training
and test sets necessary for the model’s training and testing processes.
Additionally, to test the model’s generalization capability, we generated a
test data-set B in which many parameter configurations appeared that were not
present in the training set. The specific comparisons are shown in the table
as indicated in Table 3.

| Power $P$ | Symbol rate $R_{s}$ | Number of WDM channels $N_{ch}$
---|---|---|---
Train data-set | -8,-7,-6,…,6 dBm | 20, 40, 80, 160G | 1,3,5,7,9,11
Test data-set A | -8,-7,-6,…,6 dBm | 20, 40, 80, 160G | 1,3,5,7,9,11
Test data-set B | -8,-7,-6,…,6,7,8 dBm | 30, 70, 110, 150, 190G | 1,3,5,7,9,11,13,15
Table 3: Test data-set. Test data-set A and training data-set have the same
data distribution, while Test data-set B uses a different distribution from A
to evaluate the generalization capability of MetaDSP in multi-modal scenarios.

During the numerical simulation of fiber-optic transmission, we assigned
$10^{5}$ symbols to each channel. To adhere to the sampling theorem, the
number of samples per symbol (sps) was determined using Equation (12). We
utilized the split-step Fourier method to solve the fiber’s PDE. With an
increasing symbol rate and power, we needed to ensure the step size was
sufficiently small, rendering numerical errors from the simulation negligible.
This goal was achieved by applying empirical formula (13) to select the step
size [28]:

$\mbox{sps}=2^{\lceil\log_{2}(N_{ch}\cdot\frac{\Delta f}{R_{s}})\rceil+1},$
(12) $dz=O\left(\frac{1}{|\beta_{2}|B_{w}^{2}\gamma P_{s}}\right),\quad
B_{w}=N_{ch}\cdot\Delta f,\quad P_{s}=\sum_{k=-M}^{M}P_{k}.$ (13)

At the receiving end, we simulated WDM decoupling and coherent reception,
reducing the sampling rate to 2 samples per symbol (SpS). We concentrated on
eliminating non-linear noise, without introducing any additional frequency
offset or receiver noise at the receiver’s end. For research convenience, our
primary focus was the noise reduction results of the central channel. Finally,
to evaluate the efficacy of all models, we determined the bit error rate (BER)
through direct error counting and recalculated the Q factor via Equation (14):

$Q[\mathrm{~{}dB}]=20\log_{10}\left[\sqrt{2}\operatorname{erfc}^{-1}\left(2\cdot\mbox{BER}\right)\right],$
(14)

$\operatorname{erfc}$ denotes the inverse complementary error function. Given
a signal with power $P$, symbol rate $R_{s}$, and the number of WDM channels
$N_{ch}$, its Q-factor is denoted by $Q(P,R_{s},N_{ch})$. To evaluate the
comprehensive performance in multi-modal tasks, we define the Mean Peak
Q-factor (MPQ) as follows:

$\mbox{MPQ}=\frac{1}{|\mathcal{N}||\mathcal{R}|}\sum_{N_{ch}\in\mathcal{N}}\sum_{R_{s}\in\mathcal{R}}\max_{P\in\mathcal{P}}Q(P,R_{s},N_{ch}),$
(15)

$\mathcal{N}$ and $\mathcal{R}$ represent the sets of channel numbers and
symbol rates in the test dataset, respectively, as referenced in 3.

During the training phase, we employed back-propagation on the model using
$2\times 10^{5}$ sample signals under the 360 different transmission settings
presented in the training set. Given the length of the signals, the second
module of our model, the Meta-ADF with an RNN structure, was susceptible to
issues such as vanishing or exploding gradients during the training process.
To mitigate these issues, we employed a truncated back-propagation through
time training method (as referenced in [29]), setting the truncation length to
200. The optimization process utilized the Adam algorithm [30], with a
learning rate of $1\times 10^{-4}$.

###  4.1 Complexity analysis and baselines

In this subsection, we investigate the computational complexity of all models.
We focus on several baseline models: EDC, DBP, and FDBP. To mitigate residual
phase errors, a DDLMS is incorporated into each of these models. In the DDLMS
described in 2, the filter function $h$ is computed, and the gradient $g_{k}$
is obtained. This process requires nearly $2(\text{taps}+1)$ complex
multiplications. Thus, the RMPS of DDLMS can be expressed as

$C_{\text{DDLMS}}=4\times 2(\text{taps}+1),$ (16)

where the factor of 4 signifies that one complex multiplication can be
represented through four real multiplications.

EDC stands out as a widely adopted commercial technology. It solely employs a
filtering operation with a filter length $N_{d}$, resulting in the lowest RMPS
[31]:

$C_{\text{EDC}}=\min_{N}\frac{8N(\log_{2}(N)+1)}{N-N_{d}+1},$ (17)

This formula originates from the overlap and save method applied when
utilizing FFT for convolution. The FFT size is denoted by $N$, and we have
optimized this size to minimize computational complexity.

For DBP, its complexity is largely influenced by the steps per span
$N_{stps}$. A higher $N_{stps}$ value indicates improved compensation and
increased RMPS. The DBP requires $N_{span}N_{stps}$ convolution operations,
where each filter has a length of approximately $N_{d}/(N_{span}N_{stps})$. In
addition to this, performing non-linear rotations also requires two complex
multiplications (SpS = 2). Therefore, the RMPS for DBP can be articulated as:

$C_{\text{DBP}}=\min_{N}4N_{span}N_{stps}\left(\frac{2N(\log_{2}N+1)}{N-N_{d}/(N_{span}N_{stps})+1}+2\right),$
(18)

where $N_{span}$ denotes the total number of spans and $N_{stps}$ indicates
the propagation steps for each span.

The FDBP algorithm, outlined in Chapter 2, is configured in our study with a
back-propagation step count of 5 ($N_{stps}=0.2,N_{span}=25$), aligning with
the non-linear kernel length $N_{f}=401$ in the Meta-DSP model. The RMPS of
FDBP adds a convolution cost with a filter of length $N_{f}$ to the DBP method
at each step:

$C_{\text{FDBP}}=\min_{N}4N_{span}N_{stps}\left(\frac{2N(\log_{2}N+1)}{N-N_{d}/(N_{span}N_{stps})+1}+\frac{2N(\log_{2}N+1)}{N-N_{f}+1}\right),$
(19)

The architecture of Meta-DBP parallels that of the FDBP model. The only added
computation stems from the hyper-network $f_{\varphi}$. As this computation is
executed only once and its load is distributed across all transmitted symbols,
its impact is minimal. Thus, the RMPS of Meta-DSP matches that of FDBP:

$C_{\text{Meta-
DBP}}=\min_{N}4N_{span}N_{stps}\left(\frac{2N(\log_{2}N+1)}{N-N_{d}/(N_{span}N_{stps})+1}+\frac{2N(\log_{2}N+1)}{N-N_{f}+1}\right).$
(20)

For the Meta-ADF, its RMPS is derived from the DDLMS complexity 16 and is
augmented by the computational demands of EGRU. A closer look at the
computational graph (Figure 6) reveals:

$C_{\text{Meta-ADF}}=4(H_{i}H+3L(2H^{2}+H)(\text{taps}+1),$ (21)

where $H=1$ represents the hidden dimension of EGRU, and $H_{i}=2$ is its
input dimension. Finally, the RMPS for Meta-DSP is computed as:

$C_{\text{Meta-DSP}}=C_{\text{FDBP}}+C_{\text{Meta-ADF}}.$ (22)

###  4.2 Results

To display the performance of the model at different power levels, we selected
two transmission baud rates: 20G (low rate) and 160G (high rate), and two WDM
channel counts: $N_{ch}=3$ and $N_{ch}=11$ for test data-set A, resulting in
four distinct configurations. For test data-set B, we choose $R_{s}=30,150G$,
$N_{ch}=3,15$. The performance of the model under these conditions is shown in
Figure 8. As can be seen, under low-speed conditions, our model Meta-DSP
performs close with the DBP algorithm with steps per span = 10. However, at
high speeds, Meta-DSP clearly outperforms all other models. This superior
performance is due to the fact that, at high speeds, the DBP algorithm
requires more back-propagation steps to be effective. Our model substantially
mitigates this limitation. Certain studies demonstrate that the filtering
structure in such nonlinear operators is actually a broad approximation of a
perturbation equation, thereby offering a reduction in computational load
[20].

Figure 8: Comparison of different transmission methods based on transmit power
$P$ and Q factor. The x-axis denotes the transmit power $P$ per channel, and
the y-axis indicates the Q factor. Upper left: demonstrates results for a WDM
channel count $N_{ch}=11,R_{s}=20,160G$ from test data-set A. Upper right:
demonstrates results for a WDM channel count $N_{ch}=3,R_{s}=20,160G$ from
test data-set B. left: demonstrates results for a WDM channel count
$N_{ch}=11,R_{s}=20,160G$ from test data-set A. Lower left: demonstrates
results for a WDM channel count $N_{ch}=15,R_{s}=30,150G$ from test data-set
B. Lower right: demonstrates results for a WDM channel count
$N_{ch}=3,R_{s}=30,150G$ from test data-set B. Across all settings, Meta-DSP
consistently outperforms both EDC and FDBP. In the low-speed scenario, the
performance of Meta-DSP is comparable to DBP; in contrast, at high speeds,
Meta-DSP markedly surpasses DBP.

For a further comparison of the performance of all models under different
transmission settings, we extracted the highest Q-factor in each transmission
rate and WDM channel count and plotted it in Figure 9 for Test data-set A and
Figure 10 for Test data-set B. The overall performance of Meta-DSP is close to
DBP with 10 steps per span and outperforms both EDC and FDBP methods in all
configurations. Figure 11 shows the incremental Q-factor of DBP with 10 steps
per span and Meta-DSP in comparison to EDC under different settings. It can be
observed that as the transmission rate increases, the gain brought by the DBP
with a fixed number of back-propagation steps decreases, while Meta-DSP
alleviates this trend to some extent.

Figure 9: Max Q factor along power in different $N_{ch}$ and $R_{s}$ for test
data-set A. The figure presents four heatmap tables, showcasing the highest Q
factor obtained by selecting the appropriate power for the EDC, DBP (stps=10),
FDBP, and Meta-DSP models across various modal data. The x-direction of each
table indicates the channel count settings with $N_{ch}=1,3,5,7,9,11$, while
the y-direction represents the transmission symbol rate settings of
$R_{s}=20,40,80,160$. A darker color signifies a higher Q factor. It can be
observed that Meta-DSP consistently outperforms both EDC and FDBP, exhibiting
performance close to that of DBP (stps=10).

Figure 10: Max Q factor along power in different $N_{ch}$ and $R_{s}$ for test
data-set B. The x-direction of each table indicates the channel count settings
with $N_{ch}=1,3,5,7,9,11,13,15$, while the y-direction represents the
transmission symbol rate settings of $R_{s}=30,70,110,150,190$.

Figure 11: Q factor gain compare to EDC. The figure contrasts the Q factor
gain relative to EDC, plotted on the x-axis, against the highest Q factor at
optimal power across various $R_{s}$ and $N_{ch}$ settings, represented on the
y-axis. The left graph showcases the performance of DBP (stps=10), and the
right represents Meta-DSP. It can be observed that DBP exhibits more
pronounced gains in low-speed scenarios, while in high-speed settings, the
gains are less evident and can even turn into negative values. Conversely,
although Meta-DSP displays reduced gains at high speeds, it maintains positive
gains across all settings.

In Figure 13, we present the computational load and MPQ for all models upon
test data-set A. Our model shows an average gain higher than EDC by 0.71 dB
and FDBP+DDLMS by 0.55 dB. Moreover, while maintaining an equivalent Q-factor,
our model reduces the computational load to one-tenth of that required by the
DBP algorithm. The result for test data-set B is shown in Figure 14. It can be
observed that MetaDBP’s improvement effect is more pronounced, but MetaADF’s
enhancement effect on the model is relatively moderate. On the one hand, this
is because some cases in test data-set B have very high transmission rates,
which lead to a slight decline in model performance. On the other hand,
MetaADF’s primary capability lies in rapid convergence, which cannot be well
demonstrated within the current Q factor. To showcase the rapid convergence
capability of MetaADF, we calculated the Q factor by discarding a certain
number of symbols before computing it, and the obtained Q factor is shown in
Figure 12.

Figure 12: Q factor - discard symbols. We discard previous symbols before
calculating the Q factor of the signal. MetaADF converges faster than DDLMS.
Therefore, we require fewer pilot symbols, which means that the MetaADF system
can track these changes more quickly in the presence of disturbances. Figure
13: Q factor-RMPS for test data-set A.The graph displays a comparison of
various models in terms of their computational cost, as indicated by the RMPS
on the x-axis, against their MPQ across different settings on the y-axis.
Compared to the FDBP model, Meta-DSP achieves an average Q factor improvement
of 0.4 dB. When juxtaposed with the EDC model, the gain is 0.6 dB. Meta-DSP
and DBP (stps=9) exhibit identical performance, with an average
$Q-factor=9.09dB$. However, the computational metric RMPS for Meta-DSP is
reduced to one-tenth of that for DBP (stps=9). Figure 14: Q factor-RMPS for
test data-set B.

##  5 Conclusions

In this study, we have introduced Meta-DSP, a novel multi-modal nonlinear
compensation algorithm for optical fiber receivers inspired by Meta learning.
Meta-DSP demonstrates robust capabilities in processing optical fiber data
under a myriad of conditions, ranging from long-distance, low to high speeds,
single to multi-channels, and low to high power settings. Building on the
concept of hyper-networks, we devised Meta-DBP, and using the "Learn to
Optimize" approach, we developed Meta-ADF.

In our numerical experiments, we evaluated Meta-DSP’s performance on single-
polarization 16-QAM signals over transmission distances of 2000km, exploring
symbol rates of 20G, 40G, 80G, and 160G. Additionally, we tested WDM channel
counts of 1, 3, 5, 7, 9, and 11, with transmission powers ranging from -8 to
-6dBm. The results were encouraging: the average Q-factor of Meta-DSP
surpassed that of the EDC algorithm by 0.6dB and exceeded FDBP by 0.4dB.
Furthermore, when comparing signal quality, our Meta-DSP achieved a tenfold
reduction in computational complexity relative to DBP. These findings
underscore the strong adaptability of Meta-DSP in various multi-modal
scenarios. It also accentuates the significant potential of data-driven
paradigms in advancing nonlinear compensation algorithms for optical fibers.

Future research encompasses two aspects. On one hand, we can investigate
whether Meta-DSP can still function when faced with more complex transmission
models, such as the Manakov equation model with PMD [32]. On the other hand,
we can explore whether Meta-DSP has a feasible compensation algorithm when the
optical power is stronger. Based on our current knowledge, there hasn’t been a
compensation algorithm that guarantees a continuous increase in the Q-factor
of the signal with continuously enhanced optical power. However, with the
support of deep learning and data-driven modeling, our model might become a
way to explore the theoretical capacity limits of nonlinear channels.

## References

  * Ip and Kahn [2008] Ezra Ip and Joseph M Kahn.  Compensation of dispersion and nonlinear impairments using digital backpropagation.  _Journal of Lightwave Technology_ , 26(20):3416–3425, 2008.
  * Liga et al. [2014] Gabriele Liga, Tianhua Xu, Alex Alvarado, Robert I Killey, and Polina Bayvel.  On the performance of multichannel digital backpropagation in high-capacity long-haul optical transmission.  _Optics Express_ , 22(24):30053–30062, 2014.
  * Poggiolini et al. [2010] Pierluigi Poggiolini, Gabriella Bosco, Andrea Carena, Vittorio Curri, Valerio Miot, and Fabrizio Forghieri.  Performance dependence on channel baud-rate of pm-qpsk systems over uncompensated links.  _IEEE Photonics Technology Letters_ , 23(1):15–17, 2010.
  * Golani et al. [2016] Ori Golani, Ronen Dar, Meir Feder, Antonio Mecozzi, and Mark Shtaif.  Modeling the bit-error-rate performance of nonlinear fiber-optic systems.  _Journal of Lightwave Technology_ , 34(15):3482–3489, 2016.
  * Dar et al. [2013] Ronen Dar, Meir Feder, Antonio Mecozzi, and Mark Shtaif.  Properties of nonlinear noise in long, dispersion-uncompensated fiber links.  _Optics Express_ , 21(22):25685–25699, 2013.
  * Secondini and Forestieri [2012] Marco Secondini and Enrico Forestieri.  Analytical fiber-optic channel model in the presence of cross-phase modulation.  _IEEE Photonics Technology Letters_ , 24(22):2016–2019, 2012.
  * Fan et al. [2020] Qirui Fan, Gai Zhou, Tao Gui, Chao Lu, and Alan Pak Tao Lau.  Advancing theoretical understanding and practical performance of signal processing for nonlinear optical communications through machine learning.  _Nature Communications_ , 11(1):1–11, 2020.
  * Fan et al. [2021] Qirui Fan, Chao Lu, and Alan Pak Tao Lau.  Combined neural network and adaptive dsp training for long-haul optical communications.  _Journal of Lightwave Technology_ , 39(22):7083–7091, 2021.
  * Häger and Pfister [2018] Christian Häger and Henry D Pfister.  Nonlinear interference mitigation via deep neural networks.  In _Optical fiber communication conference_ , pages W3A–4. Optical Society of America, 2018.
  * Sidelnikov et al. [2021] Oleg Sidelnikov, Alexey Redyuk, Stylianos Sygletos, Mikhail Fedoruk, and Sergei Turitsyn.  Advanced convolutional neural networks for nonlinearity mitigation in long-haul wdm transmission systems.  _Journal of Lightwave Technology_ , 39(8):2397–2406, 2021.
  * Casebeer et al. [2022] Jonah Casebeer, Nicholas J Bryan, and Paris Smaragdis.  Meta-af: Meta-learning for adaptive filters.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 31:355–370, 2022.
  * Ha et al. [2016] David Ha, Andrew Dai, and Quoc V Le.  Hypernetworks.  _arXiv preprint arXiv:1609.09106_ , 2016.
  * Andrychowicz et al. [2016] Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas.  Learning to learn by gradient descent by gradient descent.  _Advances in neural information processing systems_ , 29, 2016.
  * Secondini et al. [2014] Marco Secondini, Domenico Marsella, and Enrico Forestieri.  Enhanced split-step fourier method for digital backpropagation.  In _2014 The European Conference on Optical Communication (ECOC)_ , pages 1–3. IEEE, 2014.
  * Godard [1980] Dominique Godard.  Self-recovering equalization and carrier tracking in two-dimensional data communication systems.  _IEEE transactions on communications_ , 28(11):1867–1875, 1980.
  * Louchet et al. [2008] Hadrien Louchet, Konstantin Kuzmin, and Andre Richter.  Improved dsp algorithms for coherent 16-qam transmission.  In _2008 34th European Conference on Optical Communication_ , pages 1–2. IEEE, 2008.
  * Mori et al. [2012] Yojiro Mori, Chao Zhang, and Kazuro Kikuchi.  Novel configuration of finite-impulse-response filters tolerant to carrier-phase fluctuations in digital coherent optical receivers for higher-order quadrature amplitude modulation signals.  _Optics Express_ , 20(24):26236–26251, 2012.
  * Vanschoren [2018] Joaquin Vanschoren.  Meta-learning: A survey.  _arXiv preprint arXiv:1810.03548_ , 2018.
  * Häger and Pfister [2020] Christian Häger and Henry D Pfister.  Physics-based deep learning for fiber-optic communication systems.  _IEEE Journal on Selected Areas in Communications_ , 39(1):280–294, 2020.
  * Lin et al. [2021] Xiang Lin, Shenghang Luo, Sunish Kumar Orappanpara Soman, Octavia A Dobre, Lutz Lampe, Deyuan Chang, and Chuandong Li.  Perturbation theory-aided learned digital back-propagation scheme for optical fiber nonlinearity compensation.  _Journal of Lightwave Technology_ , 40(7):1981–1988, 2021.
  * Brock et al. [2016] Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston.  Neural photo editing with introspective adversarial networks.  _arXiv preprint arXiv:1609.07093_ , 2016.
  * Zhang et al. [2018] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun.  Shufflenet: An extremely efficient convolutional neural network for mobile devices.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6848–6856, 2018.
  * Jia et al. [2016] Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool.  Dynamic filter networks.  _Advances in neural information processing systems_ , 29, 2016.
  * Douglas [2017] Scott C Douglas.  Introduction to adaptive filters.  In _Digital signal processing fundamentals_ , pages 467–484. CRC Press, 2017.
  * Li et al. [2017] Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li.  Meta-sgd: Learning to learn quickly for few-shot learning.  _arXiv preprint arXiv:1707.09835_ , 2017.
  * Li and Malik [2017] Ke Li and Jitendra Malik.  Learning to optimize neural nets.  _arXiv preprint arXiv:1703.00441_ , 2017.
  * Li and Malik [2016] Ke Li and Jitendra Malik.  Learning to optimize.  _arXiv preprint arXiv:1606.01885_ , 2016.
  * Musetti et al. [2018] Simone Musetti, Paolo Serena, and Alberto Bononi.  On the accuracy of split-step fourier simulations for wideband nonlinear optical communications.  _Journal of Lightwave Technology_ , 36(23):5669–5677, 2018.
  * Williams and Peng [1990] Ronald J. Williams and Jing Peng.  An efficient gradient-based algorithm for on-line training of recurrent network trajectories.  _Neural Computation_ , 2(4):490–501, 1990.  doi: 10.1162/neco.1990.2.4.490.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Napoli et al. [2014] Antonio Napoli, Zied Maalej, Vincent AJM Sleiffer, Maxim Kuschnerov, Danish Rafique, Erik Timmers, Bernhard Spinnler, Talha Rahman, Leonardo Didier Coelho, and Norbert Hanik.  Reduced complexity digital back-propagation methods for optical communication systems.  _Journal of lightwave technology_ , 32(7):1351–1362, 2014.
  * Menyuk and Marks [2006] C.R. Menyuk and B.S. Marks.  Interaction of polarization mode dispersion and nonlinearity in optical fiber transmission systems.  _Journal of Lightwave Technology_ , 24(7):2806–2826, 2006.  doi: 10.1109/JLT.2006.875953.