# Extracting Off-Diagonal Order from Diagonal Basis Measurements

Bo Xiao<EMAIL_ADDRESS>Center for Computational Quantum Physics,
Flatiron Institute, 162 Fifth Avenue, New York, New York 10010 USA  Javier
Robledo Moreno<EMAIL_ADDRESS>Center for Computational Quantum Physics,
Flatiron Institute, 162 Fifth Avenue, New York, New York 10010 USA  Department
of Physics, New York University, New York, New York 10003, USA  Matthew
Fishman  Center for Computational Quantum Physics, Flatiron Institute, 162
Fifth Avenue, New York, New York 10010 USA  Dries Sels  Center for
Computational Quantum Physics, Flatiron Institute, 162 Fifth Avenue, New York,
New York 10010 USA  Department of Physics, New York University, New York, New
York 10003, USA  Ehsan Khatami  Department of Physics and Astronomy, San José
State University, San José, CA 95192 USA  Richard Scalettar  Department of
Physics and Astronomy, University of California, Davis, CA 95616 USA

###### Abstract

Quantum gas microscopy has developed into a powerful tool to explore strongly
correlated quantum systems. However, discerning phases with topological or
off-diagonal long range order requires the ability to extract these
correlations from site-resolved measurements. Here, we show that a multi-scale
complexity measure can pinpoint the transition to and from the bond ordered
wave phase of the one-dimensional extended Hubbard model with an off-diagonal
order parameter, sandwiched between diagonal charge and spin density wave
phases, using only diagonal descriptors. We study the model directly in the
thermodynamic limit using the recently developed variational uniform matrix
product states algorithm, and draw our samples from degenerate ground states
related by global spin rotations, emulating the projective measurements that
are accessible in experiments. Our results will have important implications
for the study of exotic phases using optical lattice experiments.

### Introduction.–

Quantum gas microscopy for ultracold atoms in optical lattices, in which high-
resolution real-space snapshots of the many-body system are accessible, is a
prominent tool for studying strongly-correlated systems [1, 2, 3]. These
projective measurements can be analyzed ‘by hand’ with traditional counting to
compute observables, both local or extended spin and charge correlations [4,
5, 6, 7, 8]. The snapshots are often termed ‘diagonal’ since they comprise
measurements of density observables $n_{i\sigma}=\langle
c_{i\sigma}^{\dagger}c_{i\sigma}^{\phantom{\dagger}}\rangle$, where
$c_{i\sigma}^{\dagger}$ ($c_{i\sigma}^{\phantom{\dagger}}$) is the
spin-$\sigma$ fermion creation (destruction) operator at site $i$, which have
matching row and column indices of the Greens function $G_{ii}^{\sigma}$.

The same is true for the outcome of large-scale programmable quantum
simulators based on Rydberg atoms, which allow arranging a large number of
qubits in arbitrary lattice geometries and controlling the Hamiltonian
evolution of the system [9, 10, 11, 12, 13, 14, 15]. A crucial open question
is whether the fact that these experiments do not at present capture ‘off-
diagonal’ information encoded in the full $G_{ij}^{\sigma}$ will limit the
insight they can yield.

Recent advances in machine learning methods [16, 17] hold promise for
answering this question. Convolutional neural networks and hybrid supervised-
unsupervised approaches have been used to classify quantum gas microscopy data
in emulations of the two-dimensional Fermi-Hubbard model [18], to visualize
and identify multi-particle diagonal correlations [19, 20], and to detect new
diagonal ordered phases in Rydberg atom quantum simulators [21]. Momentum-
space images of cold atoms have also been analyzed to identify quantum phase
transitions [22, 23].

Machine learning methods are able to capture order parameters or relevant
thermodynamic quantities in classical as well as quantum systems, and
therefore, detect symmetry-breaking phases [24, 25, 26, 27, 28, 29, 30, 31].
In contrast, it is much harder to identify topological phase transitions
involving off-diagonal long range order. In the realm of classical statistical
physics, the two-dimensional XY and q-state clock models have been
investigated to identify Berenzinskii-Kosterlitz-Thouless (BKT) transitions
[32, 33, 34]. However, much less is known for BKT-type quantum phase
transitions.

Figure 1:  (a) Energy $E_{i}$ on two bonds $B_{i}(i=1,2)$ associated with one
two-site unit cell and (b) von Neumann entanglement entropy $\mathcal{S}_{i}$
computed via partitioning along these two bonds vs. nearest-neighbor
interaction $V$ at fixed $U=5$ and $U=6$. (c) Phase diagram of the one-
dimensional extended Hubbard model at half filling in the $U-V$ plane. Phase
boundaries are determined using the structural complexity $\mathcal{C}$ (blue
diamond) and corroborated by the entanglement entropy $\mathcal{S}$ (red
circle) at select $U$’s. The blue shaded region denotes the BOW phase which is
characterized by off-diagonal long-range order. Orange star indicates a tri-
critical point where the nature of BOW-CDW transition switches from second-
order to first-order, based on Ref [35]. The red square indicates a direct
first-order transition from SDW to CDW at $U=10$, determined using the
structural complexity $\mathcal{C}$. Bond dimension $\mathcal{D}=2000$ is
employed.

The simplest context in which this issue can be explored is that of quasi-one-
dimensional materials, e.g. organic conductors, carbon nanotubes [36, 37, 38,
39, 40], for which the one-dimensional extended Hubbard model is a minimal
description [41, 42, 43, 44, 45, 46]:

$\displaystyle\begin{aligned}
H=&-t\sum_{i,\sigma}\left(c_{i,\sigma}^{{\dagger}}c_{i+1,\sigma}^{\phantom{\dagger}}+c_{i+1,\sigma}^{{\dagger}}c_{i,\sigma}^{\phantom{\dagger}}\right)\\\
&+U\sum_{i}n_{i,\uparrow}n_{i,\downarrow}+V\sum_{i,\sigma\sigma^{\prime}}n_{i,\sigma}n_{i+1,\sigma^{\prime}}\hskip
5.05942pt,\end{aligned}$ (1)

where $U$ and $V$ are on-site and nearest-neighbor Coulomb interactions and
$t=1$ sets the unit of energy. An infinitesimally small $U$ drives a
transition to a spin density wave (SDW) phase in which up and down spin
fermions alternate on neighboring sites [46]. Similarly, an infinitesimally
small $V$ induces charge density wave (CDW) order with staggered empty and
doubly occupied sites [46]. However, much less obvious is the existence,
between these two phases, of a narrow bond ordered wave (BOW) region with
alternating large and small kinetic energy on adjacent sites, and a BKT-type
transition separating it from the SDW phase [47, 48, 49, 44, 50, 51, 52, 35]
(see Fig. 1 (c)). Converged results on the exact location of this BKT-type
transition have not been obtained [53, 54, 55, 56]. The model thus offers a
unique opportunity to test machine learning tools for examining subtle quantum
phase transitions characterized by non-diagonal order.

In this paper, we use the state-of-the-art variational uniform matrix product
states (VUMPS) algorithm [57, 58, 59] to obtain the ground state of the model
at half filling, directly in the thermodynamic limit (TDL). We then emulate
projective and diagonal measurements on optical lattice experiments by
sampling spin-resolved occupancy snapshots from the VUMPS wavefunction. These
snapshots are first analysed using principal component analysis (PCA), and
then using a recently proposed structural complexity measure [60, 61]. We find
that while PCA accurately captures the first and second order transitions
between the BOW and CDW phases and the associated CDW order parameter, it
fails to identify differences between the SDW and BOW samples. The structural
complexity, on the other hand, starts off with a long bitstring consisting of
concatenated samples and through a series of coarse-graining steps, is able to
deduce the location of both transitions.

### Variational uniform matrix product states.–

Inspired by tangent space ideas [62, 63, 57], VUMPS optimizes a translational
invariant matrix product state (MPS) directly in the TDL, in contrast to the
more traditional infinite size density matrix renormalization group (iDMRG)
[64, 65, 66] algorithm which starts from a small system and grows the state
one site at a time.

Figure 2: Principal component analysis of the samples generated at $U=6$ and
different values of $V\in[2.8,3.3]$ in a uniform grid with separation $\Delta
V=0.1$. (a) Explained variance ratio measured by the relative weight of the
eigenvalues of the covariance matrix for two values of concatenated samples
$N_{\textrm{c}}$. The inset shows the amplitude of the first principal
component for $N_{\textrm{c}}=200$, averaged over the concatenated samples.
(b) Projection of the input features to the first two principal components
both when $N_{\textrm{c}}=1$ and $N_{\textrm{c}}=200$. The color indicates the
value of $V$ to which each projected feature belongs. (c) Average projection
of the input features corresponding to a fixed $V$ to the first (blue) and
second (orange) principal components for $N_{\textrm{c}}=200$. The shaded
region shows the variance of the average, measuring the spread of the
projected input features at a fixed $V$. Vertical dashed lines indicate the
phase transition points obtained form the entanglement entropy.

Similar to DMRG, the energy minimization problem is reformulated as a series
of local eigenvalue problems of effective Hamiltonians projected into the MPS
basis. In practice a linear solver is used to perform the sum of the formally
infinite number of Hamiltonian terms to obtain the effective Hamiltonians. By
working directly with a translational invariant ansatz in VUMPS, we can remove
the solitonic excitation induced by the use of open boundary conditions [54,
56]. In practice, all of our VUMPS calculations of the extended Hubbard model
use a single-site update with a two-site unit cell, and we constrain our
states to conserve $U(1)$ particle number and spin projection symmetry [58,
67]. We also constrain our states to be in the $S_{z}=0$ symmetry sector.
Results of convergence with bond dimension are shown in the Supplemental
Materials [68].

Fig. 1 (a) shows the energy $E$ on two bonds $B_{i}\;(i=1,2)$ that are
associated with a two-site unit cell, computed using VUMPS. It shows clear
signals of both phase transitions. For each fixed $U$, energies $E_{1}$ and
$E_{2}$ inside one unit cell are exactly equal to each other in the SDW phase
when $V$ is small. As $V$ is gradually increased, $E_{1}$ and $E_{2}$ split,
which reflects the broken translational symmetry of the BOW phase. The phase
boundary between SDW and BOW can be determined quantitatively by setting a
small threshold, e.g., where $|E_{1}-E_{2}|\sim 10^{-5}$. As $V$ is further
increased, the smooth or sudden changes in the energy per bond characterize
the second-order $(U=5)$ or first-order $(U=6)$ phase transitions from BOW to
CDW.

We can use the two bonds in a unit cell to partition the infinite system into
two half-infinite subsystems and compute the von Neumann entanglement entropy
$\mathcal{S}_{i}$. As shown in Fig. 1(b), in the BOW phase, $\mathcal{S}_{i}$
has different values computed using different partitionings. This corresponds
to the spontaneously dimerized phase of the spin chain and the
$\mathbb{Z}_{2}$ degeneracy of the two types of polarization [51, 35]. In
sharp contrast, the entanglement entropies computed in different ways of
partitioning have exactly the same value in the SDW and CDW phases. Therefore,
the point where entanglement entropies deviate from one another can be used to
locate the BOW phase boundaries, which gives results consistent with those
obtained from the two energies.

### Sampling.–

We obtain our emulated experimental data by sampling finite subsystems of the
translational invariant states found by VUMPS. To obtain a sample, we repeat
the tensors of the unit cell, and sample the resulting subsystem as one would
sample a finite MPS [69, 70]. More specifically, we start by tracing our
system down to a single site and sampling from the resulting density matrix,
projecting onto the local state that was found and iterating the procedure
over the finite subsystem. The unit cell is repeated sixteen times, providing
samples that correspond to a chain of length $L=32$ sites. For each value of
$U$ and $V$, $N_{s}=50000$ samples are collected. The sampled spin-resolved
occupancy is stored in a feature array x of length $2L$, where even and odd
entries represent the spin-up and spin-down occupancy for each lattice site.

### Spontaneous symmetry breaking considerations–

The states found by VUMPS spontaneously break the $\rm SU(2)$ symmetry of the
model, and the spin direction of the state found by VUMPS will depend on
details of the optimization, such as the initial state. Therefore, getting
multiple samples from the same state obtained by VUMPS can be biased by the
arbitrary spin direction of the state. To reduce this effect, we apply a
random local $\rm SU(2)$ spin rotation uniformly to each site of the state
before we obtain each sample.

While the continuous $\rm SU(2)$ symmetry is restored when producing the
samples, the discrete $\mathcal{Z}_{2}$ symmetry present in BOW and CDW phases
is broken in the VUMPS wave function [see Fig. 1 (a) and 1(b)]. Even if the
experimental procedure does not break the $\mathcal{Z}_{2}$ symmetry in the
CDW phase (the wave function is the homogeneous linear combination of both
configurations), each individual sample will reflect which of the two ground
states it comes from. There, the symmetry can be explicitly broken by post
processing the samples, i.e., by translating by one site those that do not
share the same pattern. Furthermore, the use of an even number of lattice
sites, together with open boundary conditions (as commonly done in
experiments) will break the $\mathcal{Z}_{2}$ symmetry in the BOW phase. Open
boundary conditions effectively provide a pinning field in the kinetic energy,
forcing the strong bonds to be adjacent to the edges of the lattice [56] (see
[68] for a demonstration using exact diagonalization on chains of finite
length). Therefore, the conclusions obtained with our emulated projective
measurements are applicable to experimental data without loss of generality.

Figure 3: (a) Procedure for computing the structural complexity. $N_{s}$
samples are produced for a single ground state for a fixed value of $U$ and
$V$. The samples are then: concatenated together, or translated by one lattice
site and concatenated together. The multi-scale structural complexity is
computed for both resulting bitstrings. One coarse-graining step is shown in
the box, consisting in the computation of the average of adjacent groups of
$2^{k}$ bits. (b) Multi-scale structural complexity $\mathcal{C}$ as a
function of nearest-neighbor interaction $V$ at fixed $U=5$, $U=6$, and
$U=10$. The connected blue circles and red squares correspond to the upper and
lower branches of the complexity measure in the BOW phase. The two branches
are obtained by computing the structural complexity from the samples extracted
directly from the emulated ground-state wave function and the samples obtained
after translation by one lattice site as shown in panel (a).

### Principal component analysis.–

Fixing the value of $U$, we run PCA on samples generated for different values
of $V$ to explore fixed-$U$ cuts of the phase diagram. As a dimensional
reduction method, PCA computes the eigenvalues $\lambda_{k}$ and eigenvectors
$\mathbf{w}^{(k)}$ of the covariance matrix of the samples. Then, the samples
are projected onto the eigenvectors with the largest eigenvalues according to
$y_{k}=\mathbf{x}\cdot\mathbf{w}^{(k)}$. PCA has been applied to detect phase
transitions based on Monte Carlo samples for classical and quantum models [71,
31, 29, 72].

We find that the spread of the projected samples at fixed $V$ can be reduced,
leading to a better resolution, if the input features $\mathbf{x}$ contain
$N_{\textrm{c}}$ concatenated spin-resolved samples. Fig. 2 (a) shows the
relative weight of the eigenvalues $\lambda_{k}$ for samples generated at
$U=6$ and different values of $V$ for $N_{\textrm{c}}=1$ and
$N_{\textrm{c}}=200$. The increase in concatenated features reveals only one
relevant principal component. The inset of Fig. 2 (a) shows the first
principal component averaged over the indices of the concatenated samples
$\big{\langle}\big{\langle}\mathbf{w}^{(0)}\big{\rangle}\big{\rangle}$,
revealing its average action on the spin-resolved occupancy as the $\pi$
component of the Fourier transform of the total charge distribution. This
quantity is the order parameter for the CDW phase.

Fig. 2(b) shows the effect of the concatenation of the input features on their
projection to the first two principal components. We observe that the first
principal component resolves two clusters. The first one corresponding to
samples in SDW and the BOW phases and the second one containing samples from
the CDW phase. Fig. 2 (c) shows, for a fixed value of $U$ and $V$, the average
projection of the input features to the first and second principal components
$(N_{\textrm{c}}=200)$. As expected by the connection of the first principal
component with the CDW order parameter, PCA can only resolve the BOW-CDW phase
transition and its nature, first- or second-order (see [68] for the PCA of the
samples at different values of $U$). However, it shows no signal for the BKT-
type transition between SDW and BOW phases.

### Multi-scale structural complexity.–

Recently, the multi-scale structural complexity measure [60] has been used to
obtain off-diagonal information about quantum states through projective
measurements in a single basis [61]. As shown schematically in Fig. 3 (a), the
idea consists of concatenating all available samples for the same quantum
state (creating a bitstring), performing several coarse-graining steps, and
computing the dissimilarity $\mathcal{D}_{k}$ between consecutive coarse-
graining steps $k$ and $k+1$ [68, 61]. These dissimilarities are added except
for the first step to obtain the so-called multi-scale structural complexity
$\mathcal{C}$.

For each $(U,V)$ point, the $N_{s}$ spin-resolved samples are concatenated in
two ways: (1) concatenation without shifting and (2) concatenation after
translating all samples by one site (two bits with spin resolution)
considering a periodic boundary for the bitstring. These form two sets of
bitstrings of length $2\cdot L\cdot N_{s}$ each.

As shown in Fig. 3 (b), the multi-scale structural complexity captures both
phase transitions. The two sets of complexity analyses give essentially the
same, almost constant, $\mathcal{C}$ (up to a constant shift) inside the SDW
phase. As $V$ is increased, the transition to a BOW phase is clearly indicated
by the splitting of the complexity measures into two branches. One branch
increases as $V$ is increased while the other branch decreases, corresponding
to the two types of polarization of strong and weak kinetic energy bonds in
the BOW phase. The higher (lower) value of $\mathcal{C}$ is associated with a
higher (lower) number of high kinetic energy bonds in the chain of length $L$.
As we keep increasing $V$, the two branches collapse into a single curve,
indicating the transition to the CDW phase. The absence of the BOW phase for
$U=10$ is indicated by the lack of bifurcation of the complexity measure.

It is worth noting that if we generate samples with equal probability from
degenerate states in the BOW phase, the complexity does not bifurcate in the
BOW phase like it does in Fig. 3 (b). This is shown in the Supplemental
Materials [68]. It is then concluded that for the resolution of the BKT-type
transition from the complexity analysis of single-basis projective
measurements, we need samples that come from only one of the degenerate ground
states. As discussed above, this can be achieved by imposing open boundary
conditions [56, 68] or diagonal edge pinning fields [73, 54, 74].

### Phase diagram.–

Fig. 1(c) compares the phase boundaries determined by the entanglement entropy
$\mathcal{S}_{i}$ (red triangles) with those determined using the structural
complexity $\mathcal{C}$, computed from samples directly (blue squares). The
complexity analysis gives accurate results and quantitatively agrees with the
off-diagonal observables computed from the wave function in the TDL within
error bars. Our results are also consistent with previous works [51, 35].
Furthermore, obtaining the ground-state wave function in the TDL and combining
local observables with machine learning approaches can shed light on the
challenges of quantitatively locating the BKT-type transition [51, 52, 55, 54,
56].

### Conclusion.–

In this work, we use the VUMPS algorithm to generate the 1D ground-state wave-
function of the extended Hubbard model directly in the TDL, which allows us to
determine phase boundaries with high precision without considering boundary
effects and finite-size scaling. We sample real-space snapshots of finite
length and use them along with unsupervised learning methods to characterize
the BKT-type phase transition between SDW and BOW phases as well as the first-
order and second-order phase transition between BOW and CDW phases. We find
that off-diagonal long-range order cannot be detected by the PCA even after
concatenation of samples. However, using the structural complexity analysis,
the off-diagonal long-range order can be detected using spin-resolved fermion
density snapshots if these snapshots are generated from one of the degenerate
ground states of the BOW phase. We argue that in optical lattice experiments,
this can be achieved by imposing open boundary conditions. Our results
indicate the potential of machine learning techniques in revealing microscopic
mechanisms and hidden orders using projective measurements of corresponding
thermal density matrix in quantum gas microscopes.

### Acknowledgement

We are indebted to Andrew Millis for carefully reading the manuscript. We
would like to thank Benedikt Kloss, Anders Sandvik, Miles Stoudenmire, Simon
Trebst and Steven White for insightful discussions. Flatiron Institute is a
division of the Simons Foundation. The work of E.K. and R.S. was supported by
the grant DE-SC-0022311, funded by the U.S. Department of Energy, Office of
Science. D.S. was supported by AFOSR: Grant FA9550-21-1-0236.

## References

  * Sherson  _et al._ [2010] J. F. Sherson, C. Weitenberg, M. Endres, M. Cheneau, I. Bloch, and S. Kuhr, Single-atom-resolved fluorescence imaging of an atomic Mott insulator, Nature 467, 68 (2010).
  * Bakr  _et al._ [2010] W. S. Bakr, A. Peng, M. E. Tai, R. Ma, J. Simon, J. I. Gillen, S. Foelling, L. Pollet, and M. Greiner, Probing the superfluid–to–Mott insulator transition at the single-atom level, Science 329, 547 (2010).
  * Gross and Bakr [2021] C. Gross and W. S. Bakr, Quantum gas microscopy for single atom and spin detection, Nature Physics 17, 1316 (2021).
  * Parsons  _et al._ [2016] M. F. Parsons, A. Mazurenko, C. S. Chiu, G. Ji, D. Greif, and M. Greiner, Site-resolved measurement of the spin-correlation function in the fermi-hubbard model, Science 353, 1253 (2016).
  * Cheuk  _et al._ [2016] L. W. Cheuk, M. A. Nichols, K. R. Lawrence, M. Okan, H. Zhang, E. Khatami, N. Trivedi, T. Paiva, M. Rigol, and M. W. Zwierlein, Observation of spatial charge and spin correlations in the 2d Fermi-Hubbard model, Science 353, 1260 (2016).
  * Brown  _et al._ [2017] P. T. Brown, D. Mitra, E. Guardado-Sanchez, P. Schauß, S. S. Kondov, E. Khatami, T. Paiva, N. Trivedi, D. A. Huse, and W. S. Bakr, Spin-imbalance in a 2D Fermi-Hubbard system, Science 357, 1385 (2017).
  * Mazurenko  _et al._ [2017] A. Mazurenko, C. S. Chiu, G. Ji, M. F. Parsons, M. Kanász-Nagy, R. Schmidt, F. Grusdt, E. Demler, D. Greif, and M. Greiner, A cold-atom Fermi–Hubbard antiferromagnet, Nature 545, 462 (2017).
  * Hartke  _et al._ [2022] T. Hartke, B. Oreg, C. Turnbaugh, N. Jia, and M. Zwierlein, Direct observation of non-local fermion pairing in an attractive fermi-hubbard gas (2022).
  * Schauß  _et al._ [2012] P. Schauß, M. Cheneau, M. Endres, T. Fukuhara, S. Hild, A. Omran, T. Pohl, C. Gross, S. Kuhr, and I. Bloch, Observation of spatially ordered structures in a two-dimensional Rydberg gas, Nature 491, 87 (2012).
  * Ebadi  _et al._ [2021] S. Ebadi, T. T. Wang, H. Levine, A. Keesling, G. Semeghini, A. Omran, D. Bluvstein, R. Samajdar, H. Pichler, W. W. Ho, _et al._ , Quantum phases of matter on a 256-atom programmable quantum simulator, Nature 595, 227 (2021).
  * Scholl  _et al._ [2021] P. Scholl, M. Schuler, H. J. Williams, A. A. Eberharter, D. Barredo, K.-N. Schymik, V. Lienhard, L.-P. Henry, T. C. Lang, T. Lahaye, _et al._ , Quantum simulation of 2D antiferromagnets with hundreds of Rydberg atoms, Nature 595, 233 (2021).
  * Samajdar  _et al._ [2020] R. Samajdar, W. W. Ho, H. Pichler, M. D. Lukin, and S. Sachdev, Complex density wave orders and quantum phase transitions in a model of square-lattice Rydberg atom arrays, Phys. Rev. Lett. 124, 103601 (2020).
  * Browaeys and Lahaye [2020] A. Browaeys and T. Lahaye, Many-body physics with individually controlled Rydberg atoms, Nature Physics 16, 132 (2020).
  * Labuhn  _et al._ [2016] H. Labuhn, D. Barredo, S. Ravets, S. De Léséleuc, T. Macrì, T. Lahaye, and A. Browaeys, Tunable two-dimensional arrays of single Rydberg atoms for realizing quantum Ising models, Nature 534, 667 (2016).
  * Bernien  _et al._ [2017] H. Bernien, S. Schwartz, A. Keesling, H. Levine, A. Omran, H. Pichler, S. Choi, A. S. Zibrov, M. Endres, M. Greiner, _et al._ , Probing many-body dynamics on a 51-atom quantum simulator, Nature 551, 579 (2017).
  * Mehta  _et al._ [2019] P. Mehta, M. Bukov, C.-H. Wang, A. G. Day, C. Richardson, C. K. Fisher, and D. J. Schwab, A high-bias, low-variance introduction to machine learning for physicists, Physics Reports 810, 1 (2019).
  * Carleo  _et al._ [2019] G. Carleo, I. Cirac, K. Cranmer, L. Daudet, M. Schuld, N. Tishby, L. Vogt-Maranto, and L. Zdeborová, Machine learning and the physical sciences, Rev. Mod. Phys. 91, 045002 (2019).
  * Bohrdt  _et al._ [2019] A. Bohrdt, C. S. Chiu, G. Ji, M. Xu, D. Greif, M. Greiner, E. Demler, F. Grusdt, and M. Knap, Classifying snapshots of the doped Hubbard model with machine learning, Nature Physics 15, 921 (2019).
  * Khatami  _et al._ [2020] E. Khatami, E. Guardado-Sanchez, B. M. Spar, J. F. Carrasquilla, W. S. Bakr, and R. T. Scalettar, Visualizing strange metallic correlations in the two-dimensional Fermi-Hubbard model with artificial intelligence, Phys. Rev. A 102, 033326 (2020).
  * Miles  _et al._ [2021a] C. Miles, A. Bohrdt, R. Wu, C. Chiu, M. Xu, G. Ji, M. Greiner, K. Q. Weinberger, E. Demler, and E.-A. Kim, Correlator convolutional neural networks as an interpretable architecture for image-like quantum matter data, Nature Communications 12, 1 (2021a).
  * Miles  _et al._ [2021b] C. Miles, R. Samajdar, S. Ebadi, T. T. Wang, H. Pichler, S. Sachdev, M. D. Lukin, M. Greiner, K. Q. Weinberger, and E.-A. Kim, Machine learning discovery of new phases in programmable quantum simulator snapshots, arXiv preprint arXiv:2112.10789 (2021b).
  * Rem  _et al._ [2019] B. S. Rem, N. Käming, M. Tarnowski, L. Asteria, N. Fläschner, C. Becker, K. Sengstock, and C. Weitenberg, Identifying quantum phase transitions using artificial neural networks on experimental data, Nature Physics 15, 917 (2019).
  * Käming  _et al._ [2021] N. Käming, A. Dawid, K. Kottmann, M. Lewenstein, K. Sengstock, A. Dauphin, and C. Weitenberg, Unsupervised machine learning of topological phase transitions from experimental data, Machine Learning: Science and Technology 2, 035037 (2021).
  * Wang [2016a] L. Wang, Discovering phase transitions with unsupervised learning, Phys. Rev. B 94, 195105 (2016a).
  * Carrasquilla and Melko [2017] J. Carrasquilla and R. G. Melko, Machine learning phases of matter, Nature Physics 13, 431 (2017).
  * Broecker  _et al._ [2017] P. Broecker, J. Carrasquilla, R. G. Melko, and S. Trebst, Machine learning quantum phases of matter beyond the fermion sign problem, Scientific reports 7, 1 (2017).
  * Ch’ng  _et al._ [2017] K. Ch’ng, J. Carrasquilla, R. G. Melko, and E. Khatami, Machine learning phases of strongly correlated fermions, Phys. Rev. X 7, 031038 (2017).
  * Zhang and Kim [2017] Y. Zhang and E.-A. Kim, Quantum loop topography for machine learning, Phys. Rev. Lett. 118, 216401 (2017).
  * Hu  _et al._ [2017] W. Hu, R. R. P. Singh, and R. T. Scalettar, Discovering phases, phase transitions, and crossovers through unsupervised machine learning: A critical examination, Phys. Rev. E 95, 062122 (2017).
  * Van Nieuwenburg  _et al._ [2017] E. P. Van Nieuwenburg, Y.-H. Liu, and S. D. Huber, Learning phase transitions by confusion, Nature Physics 13, 435 (2017).
  * Wetzel and Scherzer [2017] S. J. Wetzel and M. Scherzer, Machine learning of explicit order parameters: From the Ising model to SU(2) lattice gauge theory, Phys. Rev. B 96, 184410 (2017).
  * Beach  _et al._ [2018] M. J. S. Beach, A. Golubeva, and R. G. Melko, Machine learning vortices at the Kosterlitz-Thouless transition, Phys. Rev. B 97, 045207 (2018).
  * Rodriguez-Nieva and Scheurer [2019] J. F. Rodriguez-Nieva and M. S. Scheurer, Identifying topological order through unsupervised machine learning, Nature Physics 15, 790 (2019).
  * Miyajima  _et al._ [2021] Y. Miyajima, Y. Murata, Y. Tanaka, and M. Mochizuki, Machine learning detection of Berezinskii-Kosterlitz-Thouless transitions in $q$-state clock models, Phys. Rev. B 104, 075114 (2021).
  * Ejima and Nishimoto [2007] S. Ejima and S. Nishimoto, Phase diagram of the one-dimensional half-filled extended Hubbard model, Phys. Rev. Lett. 99, 216403 (2007).
  * Ishiguro and Yamaji [1990] T. Ishiguro and K. Yamaji,  _Organic Superconductors_ (Springer, Berlin, Heidelberg, 1990).
  * Dagotto and Rice [1996] E. Dagotto and T. Rice, Surprises on the way from one-to two-dimensional quantum magnets: the ladder materials, Science 271, 618 (1996).
  * Hu  _et al._ [1999] J. Hu, T. W. Odom, and C. M. Lieber, Chemistry and physics in one dimension: synthesis and properties of nanowires and nanotubes, Accounts of chemical research 32, 435 (1999).
  * Ishii  _et al._ [2003] H. Ishii, H. Kataura, H. Shiozawa, H. Yoshioka, H. Otsubo, Y. Takayama, T. Miyahara, S. Suzuki, Y. Achiba, M. Nakatake, _et al._ , Direct observation of Tomonaga–Luttinger-liquid state in carbon nanotubes at low temperatures, Nature 426, 540 (2003).
  * Baier  _et al._ [2016] S. Baier, M. J. Mark, D. Petter, K. Aikawa, L. Chomaz, Z. Cai, M. Baranov, P. Zoller, and F. Ferlaino, Extended Bose-Hubbard models with ultracold magnetic atoms, Science 352, 201 (2016).
  * Shiba [1972] H. Shiba, Magnetic susceptibility at zero temperature for the one-dimensional Hubbard model, Phys. Rev. B 6, 930 (1972).
  * Imada and Hatsugai [1989] M. Imada and Y. Hatsugai, Numerical studies on the Hubbard model and the t-J model in one-and two-dimensions, Journal of the Physical Society of Japan 58, 3752 (1989).
  * Schulz [1990] H. J. Schulz, Correlation exponents and the metal-insulator transition in the one-dimensional Hubbard model, Phys. Rev. Lett. 64, 2831 (1990).
  * Sengupta  _et al._ [2002] P. Sengupta, A. W. Sandvik, and D. K. Campbell, Bond-order-wave phase and quantum phase transitions in the one-dimensional extended Hubbard model, Phys. Rev. B 65, 155113 (2002).
  * Clay  _et al._ [2003] R. T. Clay, S. Mazumdar, and D. K. Campbell, Pattern of charge ordering in quasi-one-dimensional organic charge-transfer solids, Phys. Rev. B 67, 115121 (2003).
  * Essler  _et al._ [2005] F. H. Essler, H. Frahm, F. Göhmann, A. Klümper, and V. E. Korepin,  _The one-dimensional Hubbard model_ (Cambridge University Press, 2005).
  * Sandvik and Campbell [1999] A. W. Sandvik and D. K. Campbell, Spin-peierls transition in the heisenberg chain with finite-frequency phonons, Phys. Rev. Lett. 83, 195 (1999).
  * Nakamura [2000] M. Nakamura, Tricritical behavior in the extended Hubbard chains, Phys. Rev. B 61, 16377 (2000).
  * Tsuchiizu and Furusaki [2002] M. Tsuchiizu and A. Furusaki, Phase diagram of the one-dimensional extended Hubbard model at half filling, Phys. Rev. Lett. 88, 056402 (2002).
  * Jeckelmann [2002] E. Jeckelmann, Ground-state phase diagram of a half-filled one-dimensional extended Hubbard model, Phys. Rev. Lett. 89, 236401 (2002).
  * Sandvik  _et al._ [2004] A. W. Sandvik, L. Balents, and D. K. Campbell, Ground state phases of the half-filled one-dimensional extended Hubbard model, Phys. Rev. Lett. 92, 236401 (2004).
  * Zhang [2004] Y. Z. Zhang, Dimerization in a half-filled one-dimensional extended Hubbard model, Phys. Rev. Lett. 92, 246404 (2004).
  * Ejima  _et al._ [2016] S. Ejima, F. H. L. Essler, F. Lange, and H. Fehske, Ising tricriticality in the extended Hubbard model with bond dimerization, Phys. Rev. B 93, 235118 (2016).
  * Spalding  _et al._ [2019] J. Spalding, S.-W. Tsai, and D. K. Campbell, Critical entanglement for the half-filled extended Hubbard model, Phys. Rev. B 99, 195445 (2019).
  * Dalmonte  _et al._ [2015] M. Dalmonte, J. Carrasquilla, L. Taddia, E. Ercolessi, and M. Rigol, Gap scaling at Berezinskii-Kosterlitz-Thouless quantum critical points in one-dimensional Hubbard and Heisenberg models, Phys. Rev. B 91, 165136 (2015).
  * Julià-Farré  _et al._ [2022] S. Julià-Farré, D. González-Cuadra, A. Patscheider, M. J. Mark, F. Ferlaino, M. Lewenstein, L. Barbiero, and A. Dauphin, Revealing the topological nature of the bond order wave in a strongly correlated quantum system, Phys. Rev. Research 4, L032005 (2022).
  * Zauner-Stauber  _et al._ [2018a] V. Zauner-Stauber, L. Vanderstraeten, M. T. Fishman, F. Verstraete, and J. Haegeman, Variational optimization algorithms for uniform matrix product states, Phys. Rev. B 97, 045145 (2018a).
  * Zauner-Stauber  _et al._ [2018b] V. Zauner-Stauber, L. Vanderstraeten, J. Haegeman, I. P. McCulloch, and F. Verstraete, Topological nature of spinons and holons: Elementary excitations from matrix product states with conserved symmetries, Phys. Rev. B 97, 235155 (2018b).
  * Vanderstraeten  _et al._ [2019] L. Vanderstraeten, J. Haegeman, and F. Verstraete, Tangent-space methods for uniform matrix product states, SciPost Phys. Lect. Notes , 7 (2019).
  * Bagrov  _et al._ [2020] A. A. Bagrov, I. A. Iakovlev, A. A. Iliasov, M. I. Katsnelson, and V. V. Mazurenko, Multiscale structural complexity of natural patterns, Proceedings of the National Academy of Sciences 117, 30241 (2020).
  * Sotnikov  _et al._ [2022] O. M. Sotnikov, I. A. Iakovlev, A. A. Iliasov, M. I. Katsnelson, A. A. Bagrov, and V. V. Mazurenko, Certification of quantum states with hidden structure of their bitstrings, npj Quantum Information 8, 41 (2022).
  * Haegeman  _et al._ [2011] J. Haegeman, J. I. Cirac, T. J. Osborne, I. Pižorn, H. Verschelde, and F. Verstraete, Time-dependent variational principle for quantum lattices, Phys. Rev. Lett. 107, 070601 (2011).
  * Haegeman  _et al._ [2016] J. Haegeman, C. Lubich, I. Oseledets, B. Vandereycken, and F. Verstraete, Unifying time evolution and optimization with matrix product states, Phys. Rev. B 94, 165116 (2016).
  * McCulloch [2008] I. P. McCulloch, Infinite size density matrix renormalization group, revisited, arXiv preprint arXiv:0804.2509 (2008).
  * White [1992] S. R. White, Density matrix formulation for quantum renormalization groups, Phys. Rev. Lett. 69, 2863 (1992).
  * Schollwöck [2005] U. Schollwöck, The density-matrix renormalization group, Rev. Mod. Phys. 77, 259 (2005).
  * Fishman  _et al._ [2020] M. Fishman, S. R. White, and E. M. Stoudenmire, The ITensor software library for tensor network calculations (2020), arXiv:2007.14822 .
  * [68] See Supplemental Materials.
  * White [2009] S. R. White, Minimally entangled typical quantum states at finite temperature, Phys. Rev. Lett. 102, 190601 (2009).
  * Stoudenmire and White [2010] E. M. Stoudenmire and S. R. White, Minimally entangled typical thermal state algorithms, New Journal of Physics 12, 055026 (2010).
  * Wang [2016b] L. Wang, Discovering phase transitions with unsupervised learning, Phys. Rev. B 94, 195105 (2016b).
  * Khatami [2019] E. Khatami, Principal component analysis of the magnetic transition in the three-dimensional Fermi-Hubbard model, Journal of Physics: Conference Series 1290, 012006 (2019).
  * Assaad and Herbut [2013] F. F. Assaad and I. F. Herbut, Pinning the order: The nature of quantum criticality in the Hubbard model on honeycomb lattice, Phys. Rev. X 3, 031010 (2013).
  * Xiao  _et al._ [2022] B. Xiao, Y.-Y. He, A. Georges, and S. Zhang, Temperature dependence of spin and charge orders in the doped two-dimensional Hubbard model, arXiv preprint arXiv:2202.11741 (2022).
  * Orús [2014] R. Orús, A practical introduction to tensor networks: Matrix product states and projected entangled pair states, Annals of Physics 349, 117 (2014).
  * Rams  _et al._ [2018] M. M. Rams, P. Czarnik, and L. Cincio, Precise extrapolation of the correlation function asymptotics in uniform tensor network states with application to the Bose-Hubbard and XXZ models, Phys. Rev. X 8, 041033 (2018).
  * Tagliacozzo  _et al._ [2008] L. Tagliacozzo, T. R. de Oliveira, S. Iblisdir, and J. I. Latorre, Scaling of entanglement support for matrix product states, Phys. Rev. B 78, 024410 (2008).
  * Pollmann  _et al._ [2009] F. Pollmann, S. Mukerjee, A. M. Turner, and J. E. Moore, Theory of finite-entanglement scaling at one-dimensional quantum critical points, Phys. Rev. Lett. 102, 255701 (2009).
  * Pirvu  _et al._ [2012] B. Pirvu, G. Vidal, F. Verstraete, and L. Tagliacozzo, Matrix product states for critical spin chains: Finite-size versus finite-entanglement scaling, Phys. Rev. B 86, 075117 (2012).

Supplemental Materials: Unsupervised detection of off-diagonal order from
diagonal basis measurements

Bo Xiao1, Javier Robledo Moreno1,2, Matthew Fishman1 Dries Sels1,2, Ehsan
Khatami3, and Richard Scalettar4

1Center for Computational Quantum Physics, Flatiron Institute,
162 Fifth Avenue, New York, New York 10010 USA
2Department of Physics, New York University, New York, New York 10003, USA
3Department of Physics and Astronomy, San José State University, San José, CA
95192
4Department of Physics and Astronomy, University of California, Davis, CA
95616 USA

##  S1 Correlation length and von Neumann entanglement entropy

Tensor network techniques allow us to efficiently approximate the state of
systems composed of many degrees of freedom with a manageable number of
relevant ones. For matrix product states (MPS), the connected correlation
function asymptotically decays exponentially with distance [75, 76]

$\displaystyle C(r)\sim e^{-r/\xi}.$ (1)

The correlation length can be computed using the transfer matrix $\mathcal{T}$

$\displaystyle\xi=-\frac{1}{\log|\lambda_{2}/\lambda_{1}|},$ (2)

where
$|\lambda_{1}|>|\lambda_{2}|\geq|\lambda_{3}|\geq...\geq|\lambda_{D^{2}}|$ are
the eigenvalues of $\mathcal{T}$ and $D$ is the bond dimension [75].

Figure S1: (a) Correlation length $\xi$ and (b) entanglement entropy
$\mathcal{S}$ vs. nearest-neighbor interaction $V$ with fixed $U=5$ using
different bond dimensions. The second-order phase transition from BOW to CDW
is located at $V_{c2}=2.659$, where sharp peaks in $\xi$ and $\mathcal{S}$ are
observed. The BKT-type transition from SDW to BOW is located $V_{c1}=2.52$.

As shown in panel (a) of Fig. S1, the correlation length $\xi$ is large and
grows with the bond dimension in the SDW phase (which is spin gapless) and
exactly at the continuous phase transition between the CDW and the BOW phase.
In sharp contrast, the correlation length $\xi$ has negligible dependence on
the bond dimension and remains short distance in the CDW in which both the
charge and spin excitation are gapped. Inside the BOW, $\xi$ has clear
dependence on the bond dimension near the BKT-type transition and gradually
becomes bond dimension independent.

In addition to the correlation length $\xi$, the entanglement entropy is also
a measure of correlations [77, 78, 79]. The von Neumann entropy of a pure
state of a bipartite system AB is defined as,

$\displaystyle\begin{aligned} \mathcal{S}&=-{\rm Tr}\rho_{A}\log\rho_{A}=-{\rm
Tr}\rho_{B}\log\rho_{B}\\\ \end{aligned}$ (3)

where $\rho_{A}(\rho_{B})$ is the reduced density matrix of subsystem A(B).
Inside a two-site unit cell, there are two bonds, along which we can divide an
infinite chain into two half-infinite chains and then compute the entanglement
entropy. We denote the entanglement entropies computed by these two divisions
$\mathcal{S}_{1}$ and $\mathcal{S}_{2}$ correspondingly, as shown in Fig. S1.

In panel (b) of Fig. S1, we show how the von Neumann entanglement entropy
$\mathcal{S}$ evolves along a vertical cut with $U=5$ fixed on the phase
diagram shown in Fig. 1(c). The entanglement entropy peaks exactly at the BKT-
type and continuous quantum phase transitions. In the CDW phase, $\mathcal{S}$
decreases rapidly below $\mathcal{S}=1$ as $V$ is increased, which indicates
the system becoming more and more classical. Most importantly, given the
broken translational invariance in the BOW phase, the two entanglement
entropies deviate from each other, which reflects the $\mathcal{Z}_{2}$
degeneracy of two types of polarization.

Figure S2: Principal component analysis of the samples generated at
$U=5,6,7,8,10$ as indicated on each row of figures and different values of $V$
on a uniform grid with separation $\Delta V=0.1$. (a) Explained variance ratio
measured by the relative weight of the eigenvalues of covariance matrix of
data comparing two values of concatenated samples $N_{\textrm{c}}$. The inset
shows the amplitude of the first principal component for $N_{\textrm{c}}=200$,
averaged over the concatenated samples. (b) Projection of the input features
to the first two principal components both when $N_{\textrm{c}}=1$ and
$N_{\textrm{c}}=200$. The color indicates the value of $V$ to which each
projected feature belongs. (c) Average projection of the input features to the
first (blue) and second (orange) principal components for
$N_{\textrm{c}}=200$. The shaded region shows the variance of the average
value, measuring the spread of the projected input features at a fixed $V$.
Vertical dashed lines indicate the phase transition points obtained form the
entanglement entropy.

##  S2 PCA for all $U$ values considered in this study.

This section shows the results obtained from PCA at different values of $U$.
Panel (a) of Fig. S2 shows the variance ratio, defined as the relative weight
of the eigenvalues of the covariance matrix, for $N_{\textrm{c}}=1$ and
$N_{\textrm{c}}=200$. The effect of increasing the number of concatenated
samples on the input features is to improve the resolution of the variance
profile in the space defined by the samples. For $N_{\textrm{c}}=200$, only
one principal component is necessary to describe the variance properties of
the data, for every value of $U$. As shown by the inset on panel (a) of Fig.
S2, the first principal component is nearly identical for all values of $U$.
As discussed in the main text, this particular profile for the first principal
component computes the CDW order parameter for the samples.

Fig. S2 (b) shows the input features projected to the first two principal
components for $N_{\textrm{c}}=1$ and $N_{\textrm{c}}=200$. In both cases, two
clusters are resolved along the first principal component. The first cluster
corresponds to samples that belong to SDW and BOW phases, while the second
cluster contains mostly samples that belong to the CDW phase. The effect of
increasing the number of concatenated samples of the input features is to
provide a better distinction between the two clusters.

Panel (c) in Fig. S2 shows, for five fixed values of $U$ and $V$, the average
projection to the first and second principal components of the input features
for $N_{\textrm{c}}=200$. For all values of $U$ the average projection as a
function of $V$ remains featureless in the SDW and BOW phases, and increases
rapidly in the CDW phase. It must be noted that this rapid increase is
continuous at $U=5$, while it is discontinuous at $U=6,7,8,10$, reflecting the
second ($U=5$) and first ($U=6,7,8,10$) order nature of the BOW-CDW
transition. This observation comes as no surprise at the projection of the
input features to the first principal component is analogous to the
computation of the CDW order parameter.

##  S3 Structural complexity of samples with mixed polarizations

In the main text, we show how to apply the structural complexity analysis to
samples that are drawn from the same $\mathcal{Z}_{2}$ degenerate ground
state. When periodic boundary conditions (PBC) are used or the TDL is reached,
sampling from one specific state among degenerate states is not possible. In
Fig. S3(a), we show results of the structural complexity when samples drawn
from different degenerate ground states are mixed. In the BOW and CDW states,
we sample the two-fold degenerate ground-state wavefunctions simultaneously
and select samples as the input data with equal probability. As a result, the
structural complexity is featurelesss near the quantum phase transition from
SDW to BOW. No kink or sudden change of slope is observed, compared to the
case where samples with different polarizations are not mixed. In contrast,
the structural complexity is still able to capture the 2nd-order transition
from BOW phase to CDW phase, although the signal is weaker when samples are
selected from both degenerate states.

Figure S3: Multi-scale structural complexity $\mathcal{C}$ as a function of
nearest-neighbor interaction $V$ at fixed (a) $U=5$ and (b) $U=7$. Blue
circles and red triangles are results obtained by analyzing samples drawn from
single ground state selected from the $\mathcal{Z}_{2}$ degenerate ground
states. Teal diamonds show results when samples are drawn from both of the
degenerate states with equal probability.

As shown in Fig. S3(b), the details of how the lower branch of the structural
complexity transits from the BOW phase to CDW phase as $V$ is increased
depends on the nature of BOW to CDW transition. For a second-order transition,
e.g., at $U=5$, the lower branch shows a dip before entering the CDW phase. In
contrast, if the transition is first-order, e.g., at $U=7$, there is no dip
appearing in the lower branch.

##  S4 Open boundary conditions and BOW order

Using exact diagonalization (ED) on finite systems we demonstrate that open
boundary conditions (OBC) pin the bond order wave patterns in one dimensional
chains as long as the number of sites of the chain $L$ is even.

Fig. S4 shows the kinetic energy on each bond between chain sites $i$ and
$i+1$, both for open and periodic boundary conditions, at two points in the
phase diagram belonging to the BOW phase. When the boundary conditions are
chosen to be periodic, the ground state found by ED is the uniform mixture
between the two $\mathcal{Z}_{2}$ degenerate ground states of alternating high
and low kinetic energy bonds. This translates into a flat profile of kinetic
energy across the chain. For open boundary conditions, alternating bonds show
a pattern of alternating high and low kinetic energies. The edges of the chain
always have high kinetic energy (strong bond).

This can be explained by the following argument: OBC are the pinning field of
infinite amplitude for BOW. OBC can be understood as forcing the kinetic
energy at both bonds just outside of the chain to be zero. If the phase of the
system is that of alternating high and low kinetic energies on neighboring
bonds, it is easy to convince oneself that for even $L$, the kinetic energies
at the end bonds of the chain have to be high. Furthermore, at fixed particle
number, the kinetic energy is a semi-definite negative term in the Hamiltonian
(it lowers the energy). For $L$ even, the number of strong kinetic energy
terms (negative terms) is maximized by having the bonds on both edges to have
high kinetic energy. Therefore, that configuration of kinetic energy bonds is
the one that lowers the energy, and therefore the true ground state of the
system.

Figure S4: Kinetic energy between sites $i$ and $i+1$ at each bond of the one-
dimensional lattice, both for open boundary conditions and periodic boundary
conditions. Panels (a) and (b) correspond to $(U=5;V=2.58)$ and $(U=6;V=3.08)$
respectively. Both points are inside the BOW phase. Different chain lengths
are shown: blue ($L=6$), orange ($L=10$) and green ($L=14$).


# Transformer-based Nonlinear Equalization for DP-16QAM Coherent Optical
Communication Systems

Naveenta Gautam  Bharti School of Telecommunication Technology and Management,
Indian Institute of Technology Delhi, India  UltraFast Optical Communications
and High Performance Integrated Photonics (UFO-CHIP) Group, Indian Institute
of Technology Delhi, India<EMAIL_ADDRESS>Sai Vikranth
Pendem  UltraFast Optical Communications and High Performance Integrated
Photonics (UFO-CHIP) Group, Indian Institute of Technology Delhi, India
Department of Electrical Engineering, Indian Institute of Technology Delhi,
India  Brejesh Lall  Bharti School of Telecommunication Technology and
Management, Indian Institute of Technology Delhi, India  Department of
Electrical Engineering, Indian Institute of Technology Delhi, India  Amol
Choudhary  Bharti School of Telecommunication Technology and Management,
Indian Institute of Technology Delhi, India  UltraFast Optical Communications
and High Performance Integrated Photonics (UFO-CHIP) Group, Indian Institute
of Technology Delhi, India  Department of Electrical Engineering, Indian
Institute of Technology Delhi, India

###### Abstract

Compensating for nonlinear effects using digital signal processing (DSP) is
complex and computationally expensive in long-haul optical communication
systems due to intractable interactions between Kerr nonlinearity, chromatic
dispersion (CD), and amplified spontaneous emission (ASE) noise from inline
amplifiers. The application of machine learning architectures has demonstrated
promising advancements in enhancing transmission performance through the
mitigation of fiber nonlinear effects. In this paper, we apply a Transformer-
based model to dual-polarisation (DP)-16QAM coherent optical communication
systems. We test the performance of the proposed model for different values of
fiber lengths and launched optical powers and show improved performance
compared to the state-of-the-art digital backpropagation (DBP) algorithm,
fully connected neural network (FCNN) and bidirectional long short term memory
(BiLSTM) architecture.

††journal: jocn

##  1 Introduction

Coherent communication systems play a crucial role in enabling the
implementation of advanced modulation schemes, which require higher average
optical power per symbol [1]. However, this increased power makes the system
more susceptible to nonlinear effects. In long haul coherent optical
communication systems, the presence of Kerr nonlinear effects has been a major
obstacle in achieving higher data rates [2]. While digital signal processing
(DSP) techniques can effectively characterize and compensate for deterministic
nonlinear effects, addressing the stochastic effects resulting from the
interplay between Kerr nonlinearity, chromatic dispersion (CD), and amplified
spontaneous emission (ASE) noise from inline amplifiers is challenging. To
meet the growing demand for higher data rates, it is essential to find ways to
mitigate the impact of nonlinear effects and compensate for their influence
[3].

The most promising technique for nonlinear compensation in single channel
systems is the digital backpropagation (DBP) algorithm, which backpropagates
the received signal using inverted fiber parameters after numerically
simulating the fiber channel. However, it has some disadvantages; firstly, it
exhibits high computational complexity, making it less suitable for real-time
applications. Additionally, it struggles to precisely represent the channel
due to the presence of random parameters. As a result, the existing techniques
used for nonlinearity compensation have proven to be complex and demanding in
terms of computational resources, particularly in long haul links [4].
Consequently, the research focus has shifted towards machine learning
(ML)-based approaches for nonlinear compensation, which offer promising
alternatives [3, 5, 6, 7, 8, 9].

Figure 1: System Setup of a dual polarisation (DP)-16QAM coherent optical
communication system.

Numerous machine learning (ML) techniques have been explored to address the
challenge of nonlinear compensation (NLC) in optical fibers. These include
neural networks (NNs) [4, 6, 3, 10], support vector machines (SVMs) [11, 12],
expectation maximization (EM) [13], clustering techniques, and message passing
algorithms [14]. Although investigations into deep neural networks trained
with symbol triplets [6] were conducted for NLC, these approaches could not
outperform the digital backpropagation (DBP) technique [15]. It’s noteworthy
that employing symbol triplets as input increased the input dimensionality,
consequently elevating the overall complexity. Additionally, attempts were
made to employ unsupervised SVMs for nonlinear equalization in both single and
wavelength division multiplexing (WDM) scenarios; however, this technique
could not outperform DBP in the context of single-channel systems [12].
Authors in [16, 17], treat the linear and nonlinear stages of DBP as an
integrated deep neural network, resulting in a marked reduction in complexity.
Nevertheless, these methods were predominantly applied to dispersion managed
or neglected links [10], or they omitted the consideration of various
transmission impairments like polarization effects, WDM effects, laser-phase
noise, and laser-frequency offsets [10, 18, 11, 13].

A neural network (NN) architecture that has been recently unveiled by Google
AI and is gaining popularity is the Transformer [19]. One of the main reasons
for its rising popularity is its self-attention mechanism, which aids models
in focusing on certain aspects of input and reason more effectively. It
captures the global dependencies in the input and output. Transformer uses an
encoder and decoder architecture but eliminates recursion in favour of
attention mechanisms, which enables substantially greater parallelization than
other networks such as recurrent and convolutional NNs [19]. Inspired by the
great success of Transformer for machine translation tasks in natural language
processing (NLP), we apply the self-attention mechanism to learn the inverse
mapping of the fiber channel by considering the received symbols to be
sequential due to inter symbol interference noise in neighbouring symbols. A
simplified version of Transformer has been combined with the feature decoupled
distributed (FDD) scheme and introduced in optical orthogonal frequency
division multiplexing (OFDM) systems for fast and accurate channel modeling
[20]. However, the authors do not utilise self-attention, which is the
highlight of Transformer architecture. Transformer has also been used by
authors in [21] for nonlinear channel equalization using dispersion
precompensation at the transmitter but it does not outperform the DBP
algorithm and lacks a comparison of the proposed model with existing NN
architectures.

In this study, Transformer is used in coherent optical communication systems
to perform nonlinear compensation to undo the effects of CD, nonlinearity, and
ASE noise of the channel. Since the Transformer has difficulty in learning the
mapping when the received symbols are directly fed, we use a float to bit
conversion to increase the dimension of feature embeddings and capture
relationships that might not be apparent from the raw data alone. Fiber
nonlinearity increases in direct proportion to the launched optical power and
transmission distance, therefor, we test the performance of Transformer by
varying both these parameters. The modeling accuracy is represented in terms
of Q factor gain with respect to (wrt) linear equalization, DBP, fully
connected NN (FCNN) and bidirectional long short term memory (BiLSTM). Results
demonstrate that Transformer outperforms FCNN and BiLSTM and can provide a
gain of approximately 2 dB wrt linear equalisation and 1.65 dB wrt DBP. In
addition, our solution has improved the signal quality for a received signal
of Q factor as low as 7.2 dB. The proposed Transformer model also shows
performance improvement when compared against LSTM. Although the training
process is often time-consuming, testing is real-time since the transmitted
symbols are computed by simply feeding the received signals to the trained
network.

In this work, we address the nonlinear compensation problem for dual
polarisation (DP)-16QAM coherent optical communication system for different
fiber lengths and launched optical powers. The main contributions are:

  * •

We integrate the Transformer architecture with the receiver side DSP
algorithms in coherent optical communication systems. We propose a simple
float to bit conversion method to increase the feature dimension and aid the
model to learn the nonlinear mapping.

  * •

We investigate the performance of Transformer for different input sequence
lengths.

  * •

The Transformer architecture is compared with FCNN and BiLSTM for varying
fiber lengths and launched optical powers.

The rest of the paper is structured as follows. In Section 2 we describe the
system setup, proposed Transformer architecture, training data preparation and
methodology. The results are presented and discussed in Section 3, while
Section 4 concludes the paper.

##  2 System Architecture and Methodology

In this section we describe the system architecture used for demonstrating our
results. The proposed approach can be used for different system setups as
well. Our work aims to integrate the Transformer architecture for nonlinear
compensation at the receiver side of a optical coherent communication system.

###  2.1 System Setup

We model a DP-16QAM coherent optical communication system to prove the
feasibility of the proposed Transformer based approach of improving the system
performance. Figure. 1 shows the system setup and the receiver side DSP of a
DP-16QAM coherent optical communication system. At the transmitter, a psuedo
random bit sequence (PRBS) generator is used to generate data sequences for
the two polarisations which are fed to the bit to symbol mapping block. Then
it is upsampled to 8 samples per symbol for Nyquist pulse shaping with a roll
off factor of $0.18$. This sequence is subsequently modulated using a carrier
at 1550 nm with a linewidth of 100 KHz. The modulated dual polarisation signal
is multiplexed and transmitted on a link consisting of multiple fiber spans of
80 km each and an erbium-doped fiber amplifier (EDFA) to compensate for the
path loss. At the receiver, the optical signal is detected by a coherent
receiver, and digital signal processing is done to compensate for chromatic
dispersion, polarisation mode dispersion, frequency offset, carrier phase
recovery, and constellation alignment. This is followed by thresholding and
symbol demapping. The system parameters are summarized in Table 1

Signal propagation inside an optical fiber is governed by the nonlinear
Schrödinger equation (NLSE) [2] which is given by:

$i\frac{\partial\psi}{\partial
z}+i\frac{\alpha}{2}\psi+\frac{\beta_{2}}{2}\frac{\partial^{2}\psi}{\partial
t^{2}}-\frac{\beta_{3}}{6}\frac{\partial^{3}\psi}{\partial
t^{3}}+\gamma|\psi|^{2}\psi=0$ (1)

where $\psi$ is the complex envelope of the optical field, $z$ is the
distance, $\alpha$ is the attenuation, $t$ is the time coordinate, $\beta_{2}$
is the group velocity dispersion (GVD), $\beta_{3}$ is third order dispersion
and $\gamma$ is the nonlinear parameter. The nonlinear parameter governs three
main non linear effects namely self phase modulation, cross phase modulation
and four wave mixing.

Table 1: System Parameters Network Parameters | Value
---|---
Wavelength | 1550nm
Symbol Rate | 10 GBaud
Launched Optical Power | -4 to 3 dBm
Attenuation | 0.2 dB/km
Dispersion | 17 ps/(nm.km)
Core Area | 80 $um^{2}$
Nonlinear Refractive index | 2.6e-8 $um^{2}/W$
Span Length | 80 km
Noise figure of EDFA | 4.5 dB

The variation of signal intensity during fiber propagation causes variations
in the refractive index, which causes the signal phase to change. As a result,
the nonlinear phase variation is self-induced, and the associated phenomenon
is known as SPM [2]. This results in a frequency shift known as frequency
chirping, which interacts with the dispersion in the optical fiber and
broadens the optical pulse’s spectrum. In transmission systems with a high
input power, pulse broadening rises because the chirping effect is
proportional to the launched optical power.

###  2.2 Transformer Architecture

The Transformer is a type of NN that utilizes self-attention mechanism to
weigh the importance of different parts of the input sequence. Self-attention
can be understood as a mechanism for computing a weighted representation of
each element in a sequence by considering the relationships between all
elements in the sequence. This allows the Transformer to capture long-range
dependencies and patterns in the input signal, making it a powerful tool for
signal processing tasks such as audio and image processing, speech
recognition, and time-series prediction [19]. In recent years, several
Transformer-based models have been developed specifically for signal
processing applications, including the TransUnet [22], and the Waveformer
[23]. These models have shown impressive results in various signal processing
tasks, surpassing traditional methods and achieving state-of-the-art
performance.

The Transformer architecture consists of an encoder and decoder. The encoder’s
primary role is to process and extract meaningful features from the input data
while the decoder’s main purpose is to generate output sequences based on the
learned feature representations from the encoder. The multi-head attention and
feed forward network comprise the encoder part of the architecture. Multi-head
self-attention layer computes self-attention among the input tokens, enabling
each token to attend to other relevant tokens in the sequence. It captures
dependencies and relationships between symbols, allowing the model to consider
the global context while encoding information. After the self-attention sub-
layer, a position-wise feed-forward network is applied to each token
independently. It consists of two linear transformations with a nonlinear
activation function in between. This operation helps to model more complex
interactions between tokens and introduces nonlinearity into the network. Both
the self-attention and feed-forward sub-layers have residual connections
around them [19] which allow the model to retain information from previous
layers, mitigating the issue of vanishing gradients. Layer normalization is
applied to stabilize the training process by normalizing the inputs of each
sub-layer. To capture hierarchical features and create deeper representations,
multiple Transformer encoder layers are stacked on top of each other. Each
layer receives the output of the previous layer as its input. The Transformer
decoder utilizes a self-attention mechanism, which allows it to focus on
different parts of the input sequence to capture dependencies and
relationships between sequences. During training, masked self-attention is
applied to prevent the decoder from attending to future positions in the
output sequence, ensuring autoregressive generation. Multiple attention heads
are employed to capture diverse types of information, and positional encoding
is used to consider sequence order and sequential information. Together, these
elements enable the Transformer decoder to generate high-quality outputs for
tasks like machine translation, language generation, and text summarization
[19].

In certain signal processing tasks, like nonlinear equalization, the specific
role of synthesis performed by the decoder in a Transformer architecture may
not be directly applicable. These tasks often involve transforming and
enhancing input signals rather than generating new sequences. Therefore, in
cases where the generation of output sequences is not the primary objective,
using only the encoder can be a valid approach. We simplify the Transformer
and only use the encoder part followed by a fully-connected output layer to
simplify the architecture and save computational resources as shown in Fig. 2.
We also remove the embedding layer since it was designed for word vectors and
is unsuitable for this application. The purpose of the fully connected layer
is to map the high-level representations obtained from the Transformer encoder
layer to the desired output format, depending on the specific task at hand.
Overall, the proposed architecture combines the power of the Transformer
encoder layer, which captures contextual relationships in the input sequence,
with the flexibility and mapping capabilities of the fully connected layer,
enabling the model to learn complex patterns and make predictions for various
tasks.

The self attention mechanism is implemented by capturing the relationships
between the different elements (in this case, the past and future symbols) of
the received sequence. The self-attention mechanism enables the inputs to
interact with each other and determine who they should pay more attention to.
These interactions and attention scores are aggregated as outputs.

Figure 2: Transformer architecture and training data preparation

###  2.3 Training Data Preparation

Tools based on ML are especially effective for handling equalization and
nonlinear transformations and have been successfully used for nonlinear
compensation [8, 3, 14]. We make an attempt to use Transformer for nonlinear
equalization by accurately recovering transmitted symbols by observing the
distorted symbols. The optical signals received by the coherent receiver
contain information about the signal’s optical path and any linear and
nonlinear impairments it encounters along the path.

Contrary to previous approaches where two separate NNs are used for recovering
the real and imaginary part of the symbol [14], we use the in-phase (I) and
quadrature (Q) components as a 2D vector for training. The I and Q components
of the symbols were extracted after the DSP stage for training the Transformer
model. To further capture the temporal correlation between adjacent symbols
due to inter symbol interference generated by chromatic dispersion, we include
$n$ symbols each from the past and future time steps. The sequence length of
the training data is $2(2n+1)$. Initially, this sequence was directly input to
the model. However, it became evident that the model encountered challenges in
learning the input-to-output relationship due to limited feature
dimensionality. To address this, the symbols within this input sequence were
subsequently transformed from float64 to binary32, thereby expanding the size
of the input features from 1 to 32. This conversion aimed to prevent
underfitting caused by the constrained dimensionality. By providing additional
features, Transformer learns more relevant and meaningful representations. The
target vector consists of the I and Q values of the present symbol as shown in
Fig. 2.

$[I_{t-n},Q_{t-n}\ldots,I_{t},Q_{t},\ldots I_{t+n},Q_{t+n}]$ (2)

The optimal window length $n$ can vary depending on the specific
characteristics of the optical fiber system and the nature of the nonlinear
equalization task. We experiment with different window sizes and the results
are presented in the next section.

##  3 Results and Discussion

VPItransmissionMaker™ Optical Systems is used to simulate a single channel 10
GBaud dual polarisation (DP)-16QAM setup as described in section 2.1. We use a
total of $2^{19}$ symbols for training and generate the test data of the same
size using a different random seed at the PRBS generator. In this study, we
aim to compare the performance of the Transformer architecture in terms of its
impact on the improvement of the Q factor and the potential enhancement in
fiber length for coherent optical communication systems. To provide a
comprehensive analysis, we compare the Transformer with several existing
methods, including linear equalization, DBP, FCNN and BiLSTM, which is another
widely used time series architecture [24]. For fair comparison, the same
training dataset has been used for all the techniques.

The input tensors are provided in the form of (batch, sequence, features) for
training the Transformer [19]. For training we use a mini batch size of 1024,
Adam optimiser with a learning rate of $10^{-3}$ and Rectified Linear Unit
(ReLU) as the activation function. Transformer uses 8 heads, 1 encoder layer
and 0.1 dropout. The parameters are summarized in Table 2. We have chosen mean
square error (MSE) given by Eq. 3 as the loss function to update the weights.

$MSE=\frac{1}{N}\sum_{i=1}^{N}(y_{i}-a_{i})^{2},$ (3)

where $y$ is the desired output, $a$ is the calculated output and $N$ is the
total number of outputs.

Figure 3: SER vs window showing the optimum window length for our experiments

Figure 4: Training curves for different window lengths Table 2: Transformer Architecture Parameters | Value
---|---
Sequence Length | 10
Input features | 32
No. of heads | 8
No. of layers | 1
Feedforward dimension | 1024
Dropout | 0.1
Output dimension | 2
Table 3: BiLSTM architecture Parameters | Value
---|---
Input Dimension | 10
Number of LSTM layers | 2
Number of memory elements | 4
Output dimension | 2

Figure 5: Q factor Vs. Fiber length for DP-16QAM single channel system

As shown in Fig. 3, we experimented with window sizes ranging from 0 (no
memory) to 17. Including a sufficient number of past and future symbols in the
sequence allows the Transformer model to capture the contextual information
necessary for accurate equalization. By incorporating past symbols, the model
can account for the history of the signal, which may contain valuable
information about previous interactions with the fiber medium. Including
future symbols can enable the model to anticipate upcoming changes or
distortions in the signal. Longer window lengths require more memory and
computational resources to process. Transformer typically have a quadratic
time and space complexity with respect to the sequence length. Therefore,
increasing the sequence length significantly impacts the model’s memory
requirements and computational efficiency. It is important to strike a balance
between the window length and available resources to ensure the model can be
trained and deployed effectively. The window length in the context of using a
Transformer for nonlinear equalization in optical fiber affects the model’s
ability to capture relevant context, its computational requirements, the
balance between local and global information, and the risk of overfitting.
While a larger sequence length can provide more information for the model to
learn from, it also introduces challenges related to noise, overfitting, and
model complexity that can lead to higher MSE if not properly managed. We chose
a window size of 5 (2 past and 2 future symbols) for our experiments since it
achieves the lowest SER, faster convergence during training and lower MSE as
shown in Fig. 4.

The coherently detected signal after DSP is fed to the trained models for
further improvement in the BER. Figure. 5 and 6 show the measured Q-factor
versus fiber length and launched optical power curve where the power and
length are fixed at 0 dBm and 640 km respectively (system parameters specified
in Table. 1). The Q-factor is calculated from BER using Eq. 4 [25].

$Q=20log_{10}[\sqrt{2}erfc^{-1}(2BER)]$ (4)

We evaluated these methods on a simulated optical communication system with
different levels of channel nonlinearity. Our results show that FCNN (single
hidden layer with 100 neurons), BiLSTM (parameters summarized in Table 3) and
Transformer outperform DBP (10 steps per span) in terms of BER performance
across all tested fiber lengths and launched optical powers. However,
Transformer achieves the best BER performance among all three methods.
Specifically, at a fiber length of 1120 km, we achieve a maximum Q factor gain
of 1.51dB, 1.41 dB, 1.07 dB and 0.88 dB with respect to linear equalisation,
DBP, FCNN and LSTM, respectively as shown in Fig. 5. At a power of 3 dBm, when
comparing the performance of the Transformer model with linear equalization,
DBP, and BiLSTM the graph in Fig. 6 shows a Q-factor gain of 1.94 dB, 1.65 dB
and 1.01 dB, respectively. As the number of steps per span for DBP increases,
the compensation of nonlinearities becomes more accurate at the expense of
increased computational complexity [2]. While the Transformer model suggested
in [21] could not outperform DBP with 3 steps per span, our proposed technique
outperforms DBP with 10 steps per span.

Figure 6: Q factor Vs. LOP for DP-16QAM single channel system. BiLSTM and
Transformer give 0 symbol errors corresponding to a BER of 0 for LOP in the
range of -4 to -1 dBm

Additionally, we found that Transformer provides a significant gain in the
link length compared to both LSTM and DBP when operating below the forward
error correction (FEC) limit. Transformer provides approximately 250 km link
length gain compared to DBP and 150 km gain compared to BiLSTM, indicating
that the Transformer-based model is not only more accurate but also more
robust to noise and nonlinearity. Overall these Q-factor gains highlight the
significant advantages of using the Transformer model over linear equalization
and DBP in terms of improving the signal quality and reducing the impact of
noise and distortion in the optical communication system.

The Transformer-based approach leverages its ability to capture complex
patterns and dependencies in the signal, making it well-suited for modeling
and mitigating fiber impairments. The FCNN and BiLSTM techniques follow
closely behind the Transformer, suggesting its capability to learn and model
the nonlinear dynamics of the optical channel. Although it may not outperform
the Transformer, it still provides substantial improvement in terms of Q
factor gains as compared to DBP and CDC. This indicates that while DBP can
partially compensate for fiber impairments, it may not be as effective as the
other techniques being evaluated. We can infer that linear equalisation alone
may not be sufficient to achieve optimal performance in mitigating the effects
of nonlinear phase noise on signal quality.

(a) After Bi-LSTM

(b) After Transformer

Figure 7: Constellations before (blue) and after NLE using Bi-LSTM and
Transformer for a LOP of 0dBm and 1200 km fiber length.

A NN based regressor tries to map the distorted signals to the ideal
constellation points which leads to a squeezing effect in the constellation
diagram after the Transformer and LSTM nonlinear equalizer as shown in Fig. 7
[14]. The goal of this process is to map distorted symbols back towards the
original ideal constellation points in the signal space to counteract the
effects of nonlinear impairments. Therefor, a NN regressor tries to minimize
the effects of nonlinear impairments by bringing the distorted signals closer
to the ideal constellation points. This results in the distances between the
estimated constellation points becoming smaller than their original
separations, effectively compressing the constellation diagram. The superior
performance of Transformer as compared with BiLSTM can also be inferred from
the constellation diagram at a fiber length of 1200 km in Fig. 7. By training
on data with nonlinear distortions, the model can learn to identify and
correct these distortions in new, unseen data. This correction process helps
to improve the accuracy and reliability of the signal, leading to better
decision boundaries.

These findings suggest that advanced ML-based approaches such as Transformer,
FCNNs and BiLSTMs hold promise in improving signal quality over long fiber
lengths, surpassing traditional methods such as DBP and CDC. The improved
performance for longer fiber lengths demonstrates the advantages of applying
self-attention to understand and account for the temporal dynamics of the
data. The Transformer learns to recover transmitted signals by adopting the
self-attention mechanism and differentially weighting the significance of each
symbol in the sequence of the input data. It is to be noted that for the DBP-
based receiver, complexity increases with transmission distance, but we use
the same Transformer architecture across all fiber lengths. This approach may
have certain advantages, such as simplifying the implementation and reducing
the need for adjustments based on transmission distance.

Transformer architectures are based on attention mechanism, which allows the
network to focus on relevant parts of the input sequence. This results in more
accurate predictions and better performance on long sequences of data.
However, the self-attention mechanism used in Transformer can require more
computations and parameters compared to LSTMs, resulting in a higher
computational complexity. In practice, the computational complexity of both
architectures can be reduced by optimizing the model architecture and using
techniques such as pruning, quantization, and parallelization [24, 8].
Nonetheless, the relative computational complexity of LSTM and Transformer
architectures is an important consideration when choosing a model for a
specific application. Overall, both LSTM and Transformer are powerful neural
network architectures that can be used for time series data regression tasks,
and the choice between them may depend on the specific characteristics of the
data and the task requirements.

##  4 Conclusions and Future Outlook

This paper presents the use of a Transformer-based nonlinear compensation
method for long-haul coherent optical communication system, which achieves a
high Q factor gain as compared to DBP, FCNN and BiLSTM. On the basis of Q
factor measurements, we have proved that the Transformer-based fiber nonlinear
compensation model has excellent nonlinear fitting, accuracy, and
generalizability achieving a Q factor gain of up to 1.94 dB wrt LE and 1.65 dB
compared to the state-of-the-art DBP. Furthermore, the model has proven to be
reliable in a wide range of transmission scenarios, adapting to different
fiber lengths and launched optical powers. The flexibility and versatility of
the Transformer architecture makes it a promising approach for future signal
processing applications. These results showed that Transformer is a good
architecture for NLC, which opens up its applications in the optical
communications domain. Our future investigations will be to assess the
Transformer model’s adeptness in addressing the complexities posed by multi-
channel nonlinear effects. Specifically, we will explore the model’s potential
to mitigate the intricate challenges presented by phenomena like cross-phase
modulation and four-wave mixing. By rigorously evaluating the Transformer’s
capability to account for these inter-channel nonlinear interactions, we
aspire to extend its applicability to scenarios characterized by diverse and
intricate optical signal interactions. We also intend to delve into the
utilization of more robust feature extractors, such as the restricted
Boltzmann machine (RBM). By harnessing the capabilities of the RBM, we aim to
create enhanced and discriminative features that can be seamlessly integrated
as inputs to the Transformer model.

## Acknowledgment

The authors thank VPIphotonics and Optica foundation for providing training
and access to VPI Photonics Design Suite.

## References

  * [1] K. Kikuchi, “Fundamentals of coherent optical fiber communications,” Journal of lightwave technology 34, 157–179 (2015).
  * [2] G. P. Agrawal, “Nonlinear fiber optics,” in _Nonlinear Science at the Dawn of the 21st Century,_ (Springer, 2000), pp. 195–211.
  * [3] S. Sygletos, A. Redyuk, and O. Sidelnikov, “Nonlinearity compensation techniques using machine learning,” in _Signal Processing in Photonic Communications,_ (Optical Society of America, 2019), pp. SpT2E–2.
  * [4] Q. Fan, G. Zhou, T. Gui, C. Lu, and A. P. T. Lau, “Advancing theoretical understanding and practical performance of signal processing for nonlinear optical communications through machine learning,” Nature Communications 11, 1–11 (2020).
  * [5] F. N. Khan, Q. Fan, J. Lu, G. Zhou, C. Lu, and P. T. Lau, “Applications of machine learning in optical communications and networks,” in _Optical Fiber Communication Conference,_ (Optical Society of America, 2020), pp. M1G–5.
  * [6] S. Zhang, F. Yaman, K. Nakamura, T. Inoue, V. Kamalov, L. Jovanovski, V. Vusirikala, E. Mateo, Y. Inada, and T. Wang, “Field and lab experimental demonstration of nonlinear impairment compensation using neural networks,” Nature communications 10, 1–8 (2019).
  * [7] R. Yamashita, M. Nishio, R. K. G. Do, and K. Togashi, “Convolutional neural networks: an overview and application in radiology,” Insights into imaging 9, 611–629 (2018).
  * [8] N. Gautam, V. Kaushik, A. Choudhary, and B. Lall, “Optidistillnet: Learning nonlinear pulse propagation using the student-teacher model,” Optics Express 30, 42430–42439 (2022).
  * [9] N. Gautam, A. Choudhary, and B. Lall, “Comparative study of neural network architectures for modelling nonlinear optical pulse propagation,” Optical Fiber Technology 64, 102540 (2021).
  * [10] E. Giacoumidis, S. T. Le, M. Ghanbarisabagh, M. McCarthy, I. Aldaya, S. Mhatli, M. A. Jarajreh, P. A. Haigh, N. J. Doran, A. D. Ellis _et al._ , “Fiber nonlinearity-induced penalty reduction in co-ofdm by ann-based nonlinear equalization,” Optics letters 40, 5113–5116 (2015).
  * [11] E. Giacoumidis, S. Mhatli, M. F. Stephens, A. Tsokanos, J. Wei, M. E. McCarthy, N. J. Doran, and A. D. Ellis, “Reduction of nonlinear intersubcarrier intermixing in coherent optical ofdm by a fast newton-based support vector machine nonlinear equalizer,” Journal of Lightwave Technology 35, 2391–2397 (2017).
  * [12] E. Giacoumidis, A. Tsokanos, M. Ghanbarisabagh, S. Mhatli, and L. P. Barry, “Unsupervised support vector machines for nonlinear blind equalization in co-ofdm,” IEEE Photonics Technology Letters 30, 1091–1094 (2018).
  * [13] D. Zibar, O. Winther, N. Franceschi, R. Borkowski, A. Caballero, V. Arlunno, M. N. Schmidt, N. G. Gonzales, B. Mao, Y. Ye _et al._ , “Nonlinear impairment compensation using expectation maximization for dispersion managed and unmanaged pdm 16-qam transmission,” Optics express 20, B181–B196 (2012).
  * [14] E. Giacoumidis, Y. Lin, J. Wei, I. Aldaya, A. Tsokanos, and L. P. Barry, “Harnessing machine learning for fiber-induced nonlinearity mitigation in long-haul coherent optical ofdm,” Future internet 11, 2 (2019).
  * [15] V. Kamalov, L. Jovanovski, V. Vusirikala, S. Zhang, F. Yaman, K. Nakamura, T. Inoue, E. Mateo, and Y. Inada, “Evolution from 8qam live traffic to ps 64-qam with neural-network based nonlinearity compensation on 11000 km open subsea cable,” in _Optical Fiber Communication Conference,_ (Optica Publishing Group, 2018), pp. Th4D–5.
  * [16] C. Häger and H. D. Pfister, “Nonlinear interference mitigation via deep neural networks,” in _Optical fiber communication conference,_ (Optical Society of America, 2018), pp. W3A–4.
  * [17] C. Hager and H. D. Pfister, “Deep learning of the nonlinear schrodinger equation in fiber-optic communications,” in _2018 IEEE International Symposium on Information Theory (ISIT),_ (IEEE, 2018), pp. 1590–1594.
  * [18] D. Wang, M. Zhang, Z. Li, Y. Cui, J. Liu, Y. Yang, and H. Wang, “Nonlinear decision boundary created by a machine learning-based classifier to mitigate nonlinear phase noise,” in _2015 European Conference on Optical Communication (ECOC),_ (IEEE, 2015), pp. 1–3.
  * [19] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems 30 (2017).
  * [20] N. Zhang, H. Yang, Z. Niu, L. Zheng, C. Chen, S. Xiao, and L. Yi, “Transformer-based long distance fiber channel modeling for optical ofdm systems,” Journal of Lightwave Technology (2022).
  * [21] B. B. Hamgini, H. Najafi, A. Bakhshali, and Z. Zhang, “Application of transformers for nonlinear channel compensation in optical systems,” arXiv preprint arXiv:2304.13119 (2023).
  * [22] J. Chen, Y. Lu, Q. Yu, X. Luo, E. Adeli, Y. Wang, L. Lu, A. L. Yuille, and Y. Zhou, “Transunet: Transformers make strong encoders for medical image segmentation,” arXiv preprint arXiv:2102.04306 (2021).
  * [23] Y. Zhuang, Z. Wang, F. Tao, and J. Shang, “Waveformer: Linear-time attention with forward and backward wavelet transform,” arXiv preprint arXiv:2210.01989 (2022).
  * [24] I. Goodfellow, Y. Bengio, and A. Courville, _Deep Learning_ (MIT Press, 2016). http://www.deeplearningbook.org.
  * [25] S. T. Ahmad and K. P. Kumar, “Radial basis function neural network nonlinear equalizer for 16-qam coherent optical ofdm,” IEEE Photonics Technology Letters 28, 2507–2510 (2016).