Volterra-assisted Optical Phase Conjugation: a Hybrid Optical-Digital Scheme For Fiber Nonlinearity Compensation

Gabriel Saavedra, Member, IEEE,
Gabriele Liga, Member, IEEE, and Polina Bayvel Fellow, IEEE, Fellow, OSA
G. Saavedra, G. Liga, P. Bayvel are with the Optical Networks Group, Department of Electronic and Electrical Engineering, UCL, Torrington Place, London WC1E 7JE, UK. e-mail: (uceegsa@ucl.ac.uk).
This work was supported by Becas Chile, UK EPSRC UNLOC (EP/J017582/1) and INSIGHT projects and continues under the TRANSNET Programme Grant (EP/R035342/1).

Mitigation of optical fiber nonlinearity is an active research field in the area of optical communications, due to the resulting marked improvement in transmission performance.
Following the resurgence of optical coherent detection, digital nonlinearity compensation (NLC) schemes such as digital backpropagation (DBP) and Volterra equalization have received much attention.
Alternatively, optical NLC, and specifically optical phase conjugation (OPC), has been proposed to relax the digital signal processing complexity.

In this work, a novel hybrid optical-digital NLC scheme combining OPC and a Volterra equalizer is proposed, termed Volterra-Assisted OPC (VAO). It has a twofold advantage: it overcomes the OPC limitation in asymmetric links and substantially enhances the performance of Volterra equalizers. The proposed scheme is shown to outperform both OPC and Volterra equalization alone by up to 4.2 dB in a 1000 km EDFA-amplified fiber link. Moreover, VAO is also demonstrated to be very robust when applied to long-transmission distances, with a 2.5 dB gain over OPC-only systems at 3000 km.

VAO combines the advantages of both optical and digital NLC offering a promising trade-off between performance and complexity for future high-speed optical communication systems.

Fiber nonlinear optics, optical signal processing, digital signal processing, non-
linear equalization, Volterra series.

§ INTRODUCTION

Fibre nonlinearity represents a major bottleneck in increasing transmission rates of optical communication systems <cit.>.
As a result, finding a high-performance, low-complexity nonlinearity compensation (NLC) technique is a central challenge in the design of next-generation fibre systems. There are two main approaches to fiber nonlinearity mitigation, namely i) digital NLC techniques; and ii) optical NLC techniques.

Digital NLC is based on digital signal processing (DSP) (either at the transmitter or at the receiver) to mitigate nonlinear distortions accumulated during signal propagation. Digital backpropagation (DBP) is indisputably the most widespread among these digital techniques <cit.>. DBP aims to numerically reverse the Manakov equation, which describes the signal evolution in an optical fiber, through the well-known split-step Fourier method (SSFM). The SSFM algorithm is recursive and in general complex, as a minimum number of iterations (or steps), is required to achieve a given target performance. Such a minimum required number was shown to rapidly increase as a function of the NLC bandwidth <cit.>.

An alternative zero-forcing equalization method to DBP is based on the Volterra series expansion of the solution of the Manakov equation <cit.>. Volterra-based equalizers were first introduced for NLC in coherent optical communications in <cit.>, initially through a time-domain approach, and subsequently <cit.> in the frequency domain.
A Volterra-series frequency-domain equalizer (VSFE) consists in reconstructing the transmitted optical field by using a Volterra series expansion of the Manakov equation (in the backward direction). This expansion, first derived in <cit.>, is usually truncated to the third-order term for complexity reasons <cit.>. This leads to suboptimal NLC performance, as nonlinear distortions are only approximately compensated for at the receiver. Previous works have shown a potential advantange in complexity for VSFE over DBP <cit.>, especially due of the highly parallel structure of the VSFE scheme.

Alternative to digital NLC, optical NLC techniques are based on the idea of using the physics of fiber propagation itself to compensate for, or prevent detrimental nonlinear effects on the received signal. These techniques include optical phase conjugation (OPC) <cit.> and phase-conjugated twin wave (PCTW) transmission <cit.>. Mid-link OPC consists in performing a conjugation of the optical field at the mid-point of the transmission link, such that nonlinear effects produced in the first part of the link are cancelled by the propagation effects in the second half.
Mid-link OPC can jointly compensate for chromatic dispersion and nonlinear effects within the transmission link <cit.>, with the big advantage, compared to digital NLC schemes, of compensating for inter-channel effects without requiring joint detection/processing of multiple channels.
As such a compensation takes place in the optical domain, NLC can be performed jointly over very large optical bandwidths, e.g., in <cit.> NLC was achieved over 92 channels using OPC for long-haul transmission systems . Performing digital NLC over such bandwidths would be infeasible due to the high computational complexity. In addition, OPC can notionally provide nonlinearity suppression without sacrificing any of the available transmission degrees of freedom.
A major shortcoming of OPC is the stringent requirement of a symmetric link power profile. Optical links with symmetric power profile at the mid-link point can only be designed using distributed amplification schemes. Moreover, even with Raman amplifiers, the power profile symmetry condition requires complex amplifier configurations <cit.> that can be hard to achieve in practice. On the other hand, performing OPC in links with a certain degree of power profile asymmetry can result in a significantly reduced ability to reduce fibre nonlinear effects <cit.>. In addition to this, OPC devices are never ideal as they typically include an optical amplifier to overcome negative conversion efficiencies and add amplified spontaneous emission (ASE) noise in the process. This leads to a performance penalty, which further reduces their nonlinearity mitigation capabilities <cit.>.

In <cit.>, a novel approach to nonlinearity compensation was proposed that combined OPC and Volterra equalization, termed Volterra-assisted OPC (VAO). In this paper, we extend the results through a detailed analysis of the advantages of VAO over each of its constituent techniques.
The proposed approach enhances the gain of OPC in links with a low degree of power profile symmetry, such as EDFA-amplified links, through the use of VSFE. Moreover, it applies OPC as a way to boost the NLC performance of VSFE in large NLC bandwidth scenarios.

Hybrid optical NLC schemes were previously studied, e.g. in <cit.>,
with joint operation of OPC and DBP. However, this combination was designed to relax the link placement of the OPC device, rather than a way to enhance NLC performance. The technique proposed in this work is shown to markedly outperform either OPC or VSFE schemes when individually used in EDFA-amplified multi-span transmission systems, over large NLC bandwidths.

Third-order nonlinearity kernels for a 10 $\times$ 100 km SSMF, EDFA-amplified link subfig:kernel_a without OPC, and subfig:kernel_b with OPC.

§ VOLTERRA-ASSISTED OPC

To analyze the effect of nonlinearities generated in an optical fiber transmission system, a possible approach is to use a frequency domain Volterra series expansion.
Usually, for optical fibre propagation, a truncated Volterra series up to the third-order term is adopted, due to the numerical complexity in calculating the higher order terms of the series. The third-order term coincides with the first-order regular-perturbation solution to the Manakov equation (see <cit.>) and is a powerful analytical tool to explore the characteristics of the nonlinear interference in the optical fiber channel <cit.>. For multi-span, EDFA-amplified, dispersion-uncompensated transmission links, the third-order Volterra series term is given by[In this work, functions are denoted by uppercase letters $G(\cdot)$, boldface uppercase letters (e.g. $\boldsymbol{\Xi}$) denote matrices, whereas lowercase letters with either 2 or 3 sub-index (e.g. $a_{ilk}$) denote elements of a 2D/3D tensor.]:

\begin{align}
\begin{split}
&A_{3\scaleto{X}{4pt}}(\omega)=j\gamma\frac{8}{9}\iint \left[\mathcal{S}_{\scaleto{XXX}{4pt}}(\omega,\omega_1,\omega_2)\right.\\
&\times F(\omega,\omega_1,\omega_2)  d\omega_2 d\omega_1,
\end{split}
\label{eq:VolterraInt}
\end{align}

where $\gamma$ is the nonlinearity coefficient of the fiber,

\begin{align}
\begin{split}
\mathcal{S}_{\scaleto{XXX}{4pt}}(\omega,\omega_1,\omega_2)\triangleq S^{*}_{\scaleto{X}{4pt}}(\omega_1)S_{\scaleto{X}{4pt}}(\omega_2)S_{\scaleto{X}{4pt}}(\omega+\omega_1-\omega_2),
\\
\mathcal{S}_{\scaleto{YYX}{4pt}}(\omega,\omega_1,\omega_2)\triangleq S^*_{\scaleto{Y}{4pt}}(\omega_1)S_{\scaleto{Y}{4pt}}(\omega_2)S_{\scaleto{X}{4pt}}(\omega+\omega_1-\omega_2),
\end{split}
\label{eq:SignalKernels}
\end{align}

are the signal kernels, and $S_{\scaleto{X}{3pt}}(\omega)$, $S_{\scaleto{Y}{3pt}}(\omega)$ are the transmitted signal spectra over two orthogonal polarizations $X$ and $Y$, respectively. $\Xi\left(N_s,\omega,\omega_1,\omega_2\right)$
is the phased-array term given by

\begin{equation}
\Xi\left(N_s,\omega,\omega_1,\omega_2\right)=\sum_{n=1}^{N_s}e^{-j\beta_2 \Delta\Omega(n-1)L_s},
\label{eq:pharray}
\end{equation}

with $\Delta\Omega\triangleq(\omega-\omega_2)(\omega_1-\omega_2)$, and where $\beta_2$, $L_s$ ,and $N_s$  are the group velocity dispersion, the fiber span length, and the total number of spans, respectively. The factor $F(\omega,\omega_1,\omega_2)$ represents the four-wave mixing efficiency over one span, given by

\begin{equation}
F(\omega,\omega_1,\omega_2) = \frac{1-e^{-\alpha L_s} e^{j\beta_2\Delta\Omega L_s}}{j\beta_2\Delta\Omega-\alpha},
\label{eq:F1span}
\end{equation}

where $\alpha$ is the fiber attenuation coefficient. The function of $\omega_1$ and $\omega_2$ that multiplies the signal kernel (in this case $F(\omega,\omega_1,\omega_2)\cdot \Xi\left(N_s,\omega,\omega_1,\omega_2\right) $) is usually referred to Volterra third-order kernel.

When an OPC device is inserted in the transmission link with EDFA amplification, only part of the nonlinear distortion accumulated in the first part of the link is effectively cancelled by the OPC. Such a residual nonlinear term can be described using a third-order Volterra-series term. Similar to the case without OPC, the Volterra kernel can be obtained using a regular perturbation approach for this specific system configuration (see Appendix <ref>). For a transmission link using mid-link OPC, the first and third-order Volterra series terms in frequency domain, are given by[Due to the symmetry of the Manakov equation, $A_{1\scaleto{Y}{4pt}}$ and $A_{3\scaleto{Y}{4pt}}$ can be obtained by simply swapping sub-indexes $X$ and $Y$ in (<ref>) and (<ref>).]

\begin{align}
\begin{split}
\end{split}
\label{eq:VolterraFirstOrder}
\end{align}

\begin{align}
\begin{split}
&A_{3\scaleto{X}{4pt}}(\omega)=j\gamma\frac{8}{9}\iint \left[\mathcal{S}_{\scaleto{XXX}{4pt}}^*(-\omega,-\omega_1,-\omega_2)\right.\\
&\times G(\omega,\omega_1,\omega_2)  d\omega_2 d\omega_1.
\end{split}
\label{eq:VolterraInt_OPC}
\end{align}

The first-order Volterra series term simply corresponds to the conjugated version of the transmitted signal, as chromatic dispersion compensation is applied in-line by the mid-link OPC. Moreover, it is interesting to notice how the phased-array in the third-order term (Eq. (<ref>)) only accounts for the accumulation of nonlinear effects over half of the link ($\frac{N_s}{2}$ fiber spans for a link of $N_s$ spans). Finally, $G(\omega,\omega_1,\omega_2)$ is the characteristic nonlinear kernel for systems with OPC and is given by (see Appendix <ref>)

\begin{align}
\begin{split}
G(\omega,\omega_1,\omega_2)&\triangleq\frac{(e^{-j\beta_2\Delta\Omega L_s}e^{-\alpha L_s}-1)(\alpha-j\beta_2\Delta\Omega)}{\alpha^2+\beta_2^2\Delta\Omega^2}
\\&+\frac{(e^{-j\beta_2\Delta\Omega L_s}-e^{-\alpha L_s})(\alpha+j\beta_2\Delta\Omega)}{\alpha^2+\beta_2^2\Delta\Omega^2}.
\end{split}
\label{eq:OPC_kernel1}
\end{align}

Fig. <ref> shows the absolute value of the normalized frequency-domain third-order Volterra kernels in Eqs. (<ref>) (Fig. <ref>) and (<ref>) (Fig. <ref>) as a function of the 2 angular frequency variables $\omega_1$ and $\omega_2$  for $\omega=0$, for an EDFA-amplified transmission link of 10x100 km spans of standard single mode fiber (SSMF). The OPC channel kernel in Fig. <ref> is here normalised with respect to the maximum value of the kernel in (<ref>). As expected, the kernel of the channel without OPC (Fig. <ref>) shows a maximum coupling for $\omega_1=\omega_2=0$ and $\omega_1=\omega_2$. A decrease in the coupling strength is instead observed as $|\omega_1- \omega_2|\gg 0$ and $\omega_2\neq 0$, due to the walk-off effect induced by fiber dispersion. The oscillations observed in the kernel arise from the coherent accumulation of nonlinearities at every span and are described by the phased-array factor.

From <ref> it can be seen that OPC produces a kernel with a characteristic dip around the $\left(0, 0 \right)$ frequency and along the $\omega_1=\omega_2$ and $\omega_2=0$ lines. This indicates a strong reduction of the nonlinear coupling in the low-frequency $\left(\omega_1 , \omega_2\right)$–plane. On the other hand, higher frequency coupling is still present due to the power profile asymmetry[Ideal mid-link OPC would have zero nonlinear coupling everywhere in the $\left(\omega_1 , \omega_2\right)$–plane.]. As it can be observed from Eq. (<ref>), where the phased-array is evaluated only over $N_s/2$ spans, OPC reduces the maximum magnitude of the nonlinear kernel by a factor of 2.
The residual nonlinear interactions at the receiver, due to the lack of symmetry in EDFA-based links, limit the performance of OPC as an NLC method. Similar conclusions were previously derived using a different mathematical approach, such as the four-wave mixing product analysis presented in <cit.> and follow-up publications <cit.>. However, no mitigation strategy specifically tailored to overcome this limitation has been proposed to date.

Eqs. (<ref>) and (<ref>) specify the design of a VSFE tailored to compensate for the residual nonlinearity in an EDFA-amplified link using OPC. A VSFE aims to reconstruct the transmitted field up to the third-order Volterra series term, through a frequency-domain processing. This reconstruction can be done using: i) a fully non-recursive equalizer, that calculates the third-order Volterra kernel of the entire link in a single step (multispan VSFE) <cit.>,<cit.>[Sec. 2]; ii) a recursive, per-step, equalizer <cit.>. Although the recursive method partly mitigates the so-called energy divergence problem, which leads to extremely inaccurate field reconstructions as the transmitted power increases <cit.>, it also requires $N_s$ times more calculations, for a fixed processed sequence length.

To take advantage of the action of the OPC, the VAO system proposed in this work naturally requires a non-recursive VSFE approach based on the kernel in (<ref>) (as opposed to a per-span fiber kernel). This is due to the fact that the equalizer is supposed to benefit from the properties of the end-to-end kernel of the OPC channel, as opposed to the per-span kernel. The reason for that is twofold and it is fully justified from the results in this paper: i) OPC reduces the memory of the system as chromatic dispersion is compensated in a distributed manner along the link, thus allowing for a decreased interplay between nonlinearity and chromatic dispersion; ii) the reduction of nonlinearity induced by the OPC leads to a more accurate reconstruction of transmitted field using third-order Volterra equalizers.

§ VSFE AND VAO IMPLEMENTATION

Fig. <ref> shows the difference between a conventional (non-recursive) VSFE equalizer and the Volterra equalizer in VAO. In both equalizers, $N$ samples of the received time-signal  (on each polarization) $\tilde{\mathbf{s}}_{\scaleto{X}{4pt}}\triangleq \{\tilde{s}_{\scaleto{X}{4pt},k}\}_{k=0}^{N-1}$ over a time window $T$, are transformed into the frequency-domain vector $\mathbf{s}_{\scaleto{X}{4pt}}\triangleq \{s_{\scaleto{X}{4pt},k}\}_{k=0}^{N-1}$, where

\begin{equation}
s_{\scaleto{X}{4pt},k}=N\cdot\Delta\omega\cdot S_{\scaleto{X}{4pt}}\left(k\Delta\omega\right) ,\;\; \quad k=-\frac{N}{2},-\frac{N}{2}+1,...,\frac{N}{2}-1
\end{equation}
and $\Delta\omega=\frac{1}{T}$ represents the sampling resolution in frequency-domain.
The signal kernel matrix $\mathcal{S}_k\triangleq (\mathbf{\sigma}_{ij}=s^*_i s_j s_{k+i-l})$ (or $\mathcal{S}^{*}_k$ in the case of VAO) is then computed and an Hadamard multiplication by the respective channel kernel matrix is performed[The Hadamard multiplication, denoted by the $\circ$ symbol in Fig. <ref>,  is a simple element-wise matrix multiplication.].
The kernel matrices $\mathbf{F}_k \circ \boldsymbol{\Xi}_k(N_s)$, and $\boldsymbol{G}_k \circ \boldsymbol{\Xi}_k\left( \frac{N_s}{2} \right)$, for, respectively, the VSFE and VAO cases, are defined as

\begin{align}
\begin{split}
\mathbf{F}_k&\triangleq  (f_{ilk}=F^{\prime}\left(k\Delta\omega,i\Delta\omega,l\Delta\omega\right)),   \\
\boldsymbol{\Xi}_k&\triangleq (\xi_{ilk}=\Xi^{\prime}\left(k\Delta\omega,i\Delta\omega,l\Delta\omega^{\prime}\right)),   \\
\boldsymbol{G}_k&\triangleq (g_{ilk}=G^{\prime}\left(k\Delta\omega,i\Delta\omega,l\Delta\omega\right)),\\
&\forall \quad k,i,l=-\frac{N}{2},-\frac{N}{2}+1,...,\frac{N}{2}-1
\label{eq:ChannelKernels}
\end{split}
\end{align}

where $F^{\prime}$, $\Xi^{\prime}$, and $G^{\prime}$ are the backward direction counterparts of the functions $F$, $\Xi$, and $G$ (see Appendix <ref>).

Finally, both equalizers compute the reconstructed field (in frequency-domain) which, in the VAO case, is

\begin{align}
\begin{split}
\end{split}
\label{eq:VSFEimplementation}
\end{align}

The sequence $a_{\scaleto{X}{4pt},k}$ is the sampled version of the third-order Volterra series terms in Eqs. (<ref>) and (<ref>) and is given by

\begin{align}
\begin{split}
a_{\scaleto{X}{4pt},k}=F_s &A_{3\scaleto{X}{4pt}}\left(k\Delta\omega\right)\\=\frac{1}{N ^2}\sum_{i=0}^{N-1}\sum_{l=\max(0, k+i-N+1)}^{\min(k+i, N)}&\xi_{ilk}g_{ilk}\left[s_{\scaleto{X}{4pt},i}s^*_{\scaleto{X}{4pt},l}s^*_{\scaleto{X}{4pt},k+i-l}\right.\\ \left. +s_{\scaleto{Y}{4pt},i}s^*_{\scaleto{Y}{4pt},l}s^*_{\scaleto{X}{4pt},k+i-l}\right].
\label{eq:VSFEdoublesum}
\end{split}
\end{align}

In the VSFE case, the chromatic dispersion accumulated in the link is then removed through the multiplication by $\mathbf{d}\triangleq \left\lbrace\exp\left(-j\frac{\beta_2}{2}k^2\Delta\omega^2\right) \right\rbrace_{k=-\frac{N}{2}}^{\frac{N}{2}-1}$. Finally the sequence is transformed into the time-domain and, in the VAO case, and conjugated back to undo the effect of the OPC module (see Appendix <ref>).

In this work, the proposed VAO method was numerically assessed in comparison with two conventional VSFE implementations: single-step and recursive VSFE.
The single-step version of the VSFE was implemented using the schematic diagram in Fig. <ref>. The iterative VSFE was implemented using a per-span equalizer, where only the $\mathbf{F}_k$ kernel is used to calculate $a_{\scaleto{X}{4pt},k}$ at each span (see <cit.>). At the end of each iteration, the signal is transformed back into time-domain to include the modification to the output signal presented in <cit.>. Modifying the Volterra output is crucial to account for the previously mentioned energy divergence effect (see Sec. <ref>).

Schematic diagram of the two Volterra equalizers compared in this work: a subfig:VSFE conventional VSFE, and subfig:VAO the equalizer in the proposed VAO scheme.

The computational complexity of a VSFE, scales between $O(N)$ and $O(N^2)$ per sample, depending on the specific implementation adopted <cit.>. For continuous data transmission, this effectively limits the window size over which the VSFE can be sequentially applied. Conversely, a minimum window size must be guaranteed to allow for an absorbing window for the fiber channel memory effects to be factored out. This is done typically through an overlap and save approach, where a certain number of symbols are discarded from each side of the window, to account for the cyclic effects induced by the discrete frequency-domain operation.
In this work, VSFE was applied sequentially over subsequent sequences of samples, whose size was varied (between 128 and 1024 symbol periods) depending on the transmission distance. For each processed sequence, a certain amount of symbols were then discarded from each side of the window, to account for the above mentioned cyclic effects. Despite the increased complexity per sample (see e.g. <cit.>), the discarding operation considerably benefits the equalizer performance. As discussed in the following sections, the performance metrics analysed in this work (NLI suppression and SNR) are closely related to the reduction of NLI distortion. This is affected by the window size used for the VSFE. In order to guarantee the best performance, the VSFE window size was set to 4 times (up to a maximum of 1024 symbols) the estimated channel memory. A rough estimate of the channel memory can be obtained using $M\approx |\beta_2|\cdot R_s \cdot B\cdot L$  where $R_s$ is the channel symbol rate, $B$ is the entire transmitted bandwidth, and $L$ is the total transmitted distance. Each sub-sequence was sequentially processed by the VSFE, and then demodulated. The boundary symbols of the demodulated subsequence were then increasingly discarded until the  performance metric of choice converged within a desired tolerance. Multiple sub-sequences were then processed in order to accumulate enough received symbols and to guarantee the accuracy of performance metric calculation.

Systems compared in the numerical investigation in this work.

§ SYSTEM UNDER INVESTIGATION

The performance of the proposed NLC method was  evaluated through numerical simulations. All the investigated system scenarios are shown in Fig. <ref>, with different colours representing different receiver schemes.
An ideal optical transmitter was used to generate 5 polarization-multiplexed 16-ary quadrature amplitude modulated (PM-16QAM) channels at a symbol rate of 32 GBaud using a sequence of $2^{16}$ symbols. The signal pulses were shaped using a root raised cosine filter with 1% roll-off factor.
Transmission over $N_s$ 100 km-long optical fiber spans was simulated by numerically solving the Manakov equation using the split-step Fourier method. The fiber spans were based on SSMF and EDFA amplification after each span, with all parameters shown in Table <ref> . At the mid-link point, the signal either continues to the following half of the link (blue path), or undergoes phase conjugation (red path). OPC was performed ideally by taking the complex conjugate of the optical signal in the time domain.

After transmission, an optical coherent receiver was assumed to ideally capture the entire 5-channel optical bandwidth ($\approx$ 165 GHz) with a  subsequent sampling at 1.5 times the Nyquist rate of the received signal. The sampled signal over each polarization had 5 possible DSP chain options depending on the selected system (with or without OPC), which are: (i) ideal electronic dispersion compensation (EDC) performed in the frequency-domain when no OPC module was used; (ii) OPC-only DSP ; (iii) single-step/recursive VSFE; (iv) the VAO equalizer; (v) ideal NLC performed using a full-complexity (100 steps/span), full-field DBP implementation. The DSP chain for each of the 3 NLC schemes compared in this work, i.e. OPC-only DSP, conventional VSFE, and VAO, is shown in Figs. <ref>–<ref>. For each DSP scheme shown in Figs. <ref>–<ref>, the input vector (shown only for the $X$ polarization for the sake of simplicity) is sampled at 6 samples/symbol. The OPC DSP module consists of a downsampling stage at 2 samples/symbol to select the central channel bandwidth, followed by time-domain conjugation, matched filtering and additional downsampling at 1 sample/symbol. For the conventional VSFE DSP, shown in Fig. <ref>, the signal is first processed by the Volterra equalizer, downsampled at 2 samples/symbol and passed through a MF followed by downsampling to 1 sample/symbol. Similar operation is performed in the VAO DSP chain where, however, the signal must be conjugated after the VSFE block. After each DSP scheme, the SNR was estimated using a fully data-aided method.

Simulation parameters

Parameter     								       Value
Symbol rate 								       32 GBd
Channel spacing								       32.5 GHz
RRC roll-off	 	                               1 %
Span length ($L_s$)							       100 km
Attenuation coefficient 					       0.20 dB
Dispersion parameter (D)               		       17 $\textnormal{ps}\cdot \textnormal{nm}^{-1}\cdot \textnormal{km}^{-1}$
Nonlinear coefficient ($\gamma$)       		       1.2 W$^{-1}$km$^{-1}$
EDFA noise figure							       5 dB

Schematic diagram of the DSP chains for: subfig:OPC_DSP OPC-only system, subfig:VSFE_DSP conventional VSFE and subfig:VAO_DSP VAO. MF denotes the matched filtering stage and ${\mathbf{r}}_{\scaleto{X}{4pt}}$ the output symbols at 1 sample/symbol.

§ NONLINEARITY SUPPRESSION

In order to assess the NLC effectiveness of the proposed scheme, a nonlinear interference (NLI) suppression factor was characterised through numerical simulations. This NLC suppression factor was defined as

\begin{equation}
\zeta\triangleq\frac{\text{SNR}_{\text{\tiny{NLC}}}}{\text{SNR}_{\text{\tiny{EDC}}}},
\end{equation}

where $\text{SNR}_{\text{\tiny{EDC}}}$ and $\text{SNR}_{\text{\tiny{NLC}}}$ represent the SNR when EDC or NLC are applied, respectively, in the absence of ASE noise in the simulated system. The $\zeta$ indicates the reduction of the NLI power after NLC, and is key to understand, in the absence of any other noise sources, what is the effectiveness of that particular scheme. For an ideal NLC scheme (full compensation of any deterministic nonlinear effect), $\zeta$ tends to infinity as the NLI power is forced to 0. Conversely, this value tends to 1 if the NLI is left uncompensated (EDC-only schemes), or it is less than 1 when additional distortions are added, due for instance to NLC schemes operating with a very limited accuracy. In order to characterize the proposed VAO scheme, the $\zeta$ parameter was studied as a function of the 2 optical system parameters: transmitted power, and transmission distance for a 5-channel transmission. $\zeta$ was calculated numerically through Monte-Carlo estimation of the received SNR, in the absence of ASE noise in the simulated system. An appropriate number of processed subsequences was accumulated to guarantee an SNR accuracy within 0.05 dB.

The nonlinearity suppression factor ($\zeta$) was calculated as a function of power for the transmission of a 5$\times$32 GBaud signal over a 1000 km link for 3 different NLC schemes illustrated in Fig. <ref>: mid-link OPC (red curve); VSFE (blue curves) and the proposed VAO (green curve). These results are plotted in Fig. <ref>. The VSFE is here implemented in two different variants: single-step (blue curve with circle markers), and recursive VSFE (squared markers).
Both variants of VSFE and VAO were operated (repeatedly) over 512 symbol-long sequences each oversampled at 6 samples/symbol, resulting in 3072 samples per processed sequence.

The results show that OPC (red curve) has a constant NL suppression of approximately 1.8 dB across the entire range of powers shown. Such a limited NLC suppression is the result of the residual NLI when OPC is used in an EDFA link (see Sec. <ref>), and is explained by the third-order Volterra kernel illustrated in Fig. <ref>.
VSFE was characterized in terms of $\zeta$ in the two different implementations described in Sec. <ref>: single-step and recursive VSFE. Single-step VSFE shows a better performance at low powers, with a maximum $\zeta$=2 dB at -4 dBm, and monotonically decreasing $\zeta$ as the transmitted power increases. This well-known performance behaviour of the Volterra equalizer is due to the truncation of the Volterra series to the third-order term. Such a truncation becomes less accurate as the transmitted power is increased, and leads to a degradation in the equalization performance ($\zeta$=0.5 dB at 8 dBm/channel). This degradation is partly mitigated when a recursive VSFE scheme is used. Although the two VSFE schemes (blue curves in Fig. <ref>) show similar NLI suppression ($\approx$ 3.5 dB) for around -3 dBm/channel, the recursive VSFE scheme performs better beyond -3 dBm/channel and achieves up to 5 dB NLI suppression at 2 dBm/channel. However, this improvement comes at the expense of an $N_s$ times higher computational complexity compared to a single-step VSFE scheme, for a fixed processed sequence length <cit.>. Fig. <ref> shows that the VAO scheme (green curve) has a superior performance compared to the all the other NLC schemes analysed, across the entire range of powers. Alike single-step VSFE, the VAO $\zeta$ factor monotonically decreases with power, and reaches a maximum value (within the range of powers shown in Fig. <ref>) of $\approx$ 30 dB at -5 dBm/channel. At this transmitted power, VAO exceeded the NLI suppression of all of the other NLC schemes by over 25 dB. Although the performance of VAO can be observed to degrade quickly as a function of power, even in the region of powers of interest for optimum transmission performance (0 to 4 dBm) the VAO NLI suppression is still significantly higher than other compared schemes.
In particular, at 2 dBm/channel VAO achieves 17 dB higher $\zeta$ compared to using only OPC, and 14 dB higher suppression than the optimum $\zeta$ achieved by a VSFE-only scheme ($\zeta$=5 dB at 2 dBm/channel).

The remarkable NLC performance attained by the VAO for the system studied is attributed to two reasons: i) for a fixed processed sequence length, VAO benefits from a reduced channel memory compared to VSFE; ii) the adoption of OPC leads to a more accurate reconstruction of the transmitted field using a third-order Volterra series term due to the reduced NLI at the receiver. The reduction of the channel memory is due to the compensating effect of the phase conjugation in the second part of the link as the OPC completely removes the fibre chromatic dispersion accumulated in the first part of the link, thus also reducing the time window over which the nonlinear intersymbol takes place. This memory reduction can also be inferred by the phased-array in (<ref>) which is only evaluated over half of the link.

Nonlinearity suppression factor ($\zeta$) as a function of transmitted power for the NLC schemes studied in this work.

Nonlinearity suppression factor ($\zeta$) as a function of transmission distance and different equalizer window sizes for subfig:NLsupp_vs_Dist_a VSFE, subfig:NLsupp_vs_Dist_b VAO, and subfig:NLsupp_vs_Dist_c VAO NLI suppression gain compared to VSFE.

To assess the improved robustness of VAO against an increasing channel memory, the NLI suppression as a function of the transmitted distance is shown for the system under investigation at a transmitted power of 0 dBm/channel in Fig. <ref>, for single-step VSFE (Fig. <ref>), and VAO (Fig. <ref>). The effect of the channel memory is highlighted by showing the performance for different window sizes, spanning from 128 to 1024  symbols. As the channel memory is expected to scale linearly with increasing transmission distance, each scheme NLC shows a decreasing NLI suppression factor for a fixed window size. However, performance degradation as a function of distance is also caused by an increasing dominance of higher-order Volterra series terms. The two effects can be distinguished by observing the behaviour of $\zeta$ as the window size is increased. The results in Fig. <ref> show that, up to 3000 km transmission distance, $\zeta$ saturates using time-windows between 512 and 1024 symbols. For each window size, a threshold distance of operation, beyond which VSFE performance breaks down (negative $\zeta$), can be identified. This was caused by the channel memory going beyond the maximum number of discarded symbols within a given window size. Increasing from 512 to 1024 symbols the window size did not lead to any substantial increase in $\zeta$. Thus, it can be concluded that for window sizes beyond 1024 symbols and up to 5000 km transmission distance, the VSFE performance is dominated by its compensation accuracy (NLI compensation up to the first-order).

The VAO NLI suppression as a function of distance is shown in Fig. <ref>. It can be observed that for VAO the detrimental effect of the channel memory operating over finite window sizes is less pronounced. This is also confirmed by the smoother decay of $\zeta$ as a function of transmission distance compared to VSFE, with NLC capabilities (positive $\zeta$) observed within 3000 km for all window sizes shown. For transmission distances up to 1000 km, the value of $\zeta$ saturates already at 512 symbols. Beyond a distance of 1000 km, the increase of the window size from 512 symbols to 1024 symbols results in a substantial increase in $\zeta$ (up to 8.6 dB at 2000 km). The performance gap between using a 512-symbol window and a 1024-symbol window then gradually decreases to 6 dB at 5000 km. This can be attributed to the stronger effect of higher-order Volterra series terms as the transmission distance increases.

Fig. <ref> shows the NLI suppression gain achieved by VAO over VSFE as a function of distance for different window lengths. The power per channel is again fixed at 0 dBm/channel. It can be seen that the improvement introduced by the VAO scheme in terms of $\zeta$ is remarkable for all distances and window sizes used. For short distances, (up to 1000 km) the highest gains can be observed for 512- and 1024-symbol window length, up to 21 dB and 24 dB, respectively. This highlights the superior NLC efficacy of a Volterra-based equalizer when an in-line OPC device is used. For longer distances (beyond 2000 km), the highest VAO gain is achieved for the shortest window lengths (128- and 256-symbol long), due to the reduction of the channel memory provided by OPC. In contrast with VSFE, this enables the VAO equalizer to obtain a certain (albeit limited) amount of NLI suppression ($\leq 4.5$ dB) for distances within $2000-4000$ km, while still processing relatively short sequences (see Fig. <ref>). For instance, a 19.5 dB gain is attained using VAO with a 256-symbol long sequence at 3000 km, which brings the VAO $\zeta$ factor up to $\approx$ 1.5 dB from a -18.5 dB achieved by the VSFE over the same window length (see Fig. <ref> and <ref>).

§ SNR PERFORMANCE

In the previous section, the performance of the proposed VAO scheme was analysed in terms of NLI suppression factor. Despite this being a more direct measure of the NLC effectiveness, it does not immediately translate into more insightful performance metrics such as signal-to-noise ratio (SNR). In this section, a study of the SNR performance among the different NLC schemes studied in this work is described.

Numerical simulations were carried out to assess the performance of different NLC methods in a realistic transmission scenario and to compare their performance with the VAO method. Similar to Sec. <ref>, the performance was first evaluated for a fixed transmission distance of 1000 km as a function of the signal launch power, with results shown in Fig. <ref>, and subsequently as a function of the transmission distance at optimum power up to 3200 km, presented in Fig. <ref>.

Figure <ref> shows the performance of the system using the different NLC techniques as a function of launch power per channel after transmission of 1000 km. The use of electronic dispersion compensation (EDC) shows the worst performance among all the studied methods. As EDC does not enact any compensation of nonlinear effects, this scheme serves as a baseline to evaluate the effectiveness of the NLC methods under study. EDC exhibits a maximum SNR of 17.3 dB at 0 dBm per channel. At this distance, OPC presents a gain over EDC limited to 0.4 dB, with a maximum SNR of 17.7 dB, indicating that only a small portion of the nonlinearities are compensated. As discussed in Sec.  <ref>, the spectral region where OPC offers complete cancellation of nonlinearites (see Fig. <ref>) was not sufficiently large to account for the effects of all 5 transmitted channels. In the highly-nonlinear regime, e.g. using a power of 4 dBm, an increase of 1.2 dB in the SNR over EDC is observed with the use of OPC. This value is in agreement with the nonlinear suppression factor observer for OPC in the previous section. Similar performance is observed using single-step VSFE at the maximum SNR. In this case, one-step VSFE was implemented using a window size of 512 symbols. A maximum SNR of 17.7 dB is found using this method at a signal power of 1 dBm.

Received SNR for different NLC schemes as a function of signal launch power.

As the power is increased, the efficiency of VSFE is decreased as a result of the truncation of the Volterra series to third-order term, as described in the previous section.
For the recursive VSFE a window size of 256 symbols was used. This method presents an improved performance relative to the one-step implementation, with a maximum SNR of 19.4 dB at 2 dBm/channel signal power.
Using VAO results in a significant improvement relative to VSFE and OPC, with a gain of 4.6 dB compared to EDC, 4.2 dB relative to either conventional one-step VSFE or OPC, and  2.6 dB compared to recursive VSFE. The VAO was implemented using a window size of 512 symbols in the equalizer, to allow a direct comparison with the conventional VSFE methods. Whilst the VSFE truncation is one of the factors that limited the performance of conventional VSFE implementations, the reduction of nonlinear effects due to the mid-link OPC leads to a better reconstruction of the transmitted signal using the VAO scheme.
Finally, for comparison, the performance of ideal NLC is shown, obtained by perfectly integrating the Manakov equation in a virtually reversed transmission link. This represents the maximum performance that can be obtained using any receiver based NLC scheme. Although the VAO scheme presents improved performance relative the aforementioned NLC schemes, a 3.8 dB gap is observed compared to an ideal NLC method.

Received SNR for different NLC schemes as a function of transmission distance.

Figure <ref> shows the performance of the system as a function of transmission distance. The same trend from Fig. <ref> in terms of nonlinear effects was observed, with EDC exhibiting the worst performance among all the studied methods.
Interestingly, OPC presents a reduced gain of 0.2 dB relative to EDC after 200 km, however greater nonlinearity compensation is obtained for longer distances and 0.6 dB gain is observed after 3200 km.
The opposite behaviour is found for single-step VSFE, where a superior performance is found at short distances. As in the previous case, both standard VSFE were implemented using window sizes of 512 and 256 symbols for the single-step and recursive algorithms, respectively.
Using VSFE, the gain over EDC is slowly reduced as the transmission distance increased, up to the point where no improvement is observed. This phenomenon can be understood from memory effects introduced by the fiber chromatic dispersion. When the temporal spreading of the propagated signal exceeds the VSFE window size the performance of both implementations, single-step and recursive, is dramatically reduced. For the studied system no gain from NLC is observed for transmission distances longer than 1600 km using both standard VSFE implementations.

The VAO scheme is shown to outperform both standard implementations of VSFE, and OPC at all evaluated transmission distances. For a received SNR of 20 dB, VOA is shown to attain a 200% increase in transmission distance compared to EDC, going from 500 to 1500 km. The VAO method however, did not offer full compensation of the NLI experienced during transmission and reduced performance is found compared to ideal NLC. The use of ideal NLC achieves an extra 66% increase of transmission distance for a received SNR of 20 dB compared to VAO.
Additionally, the NLC gain from the VAO method presents an accelerated reduction as a function of the transmission distance, compared to ideal NLC. We attribute this behaviour to the nonlinear equalizer in the VAO scheme, which has qualitatively the same limitations as a standard VSFE in terms of channel memory and truncation of the Volterra series.

§ COMPLEXITY

In the previous sections, the VAO scheme was shown to significantly outperform OPC and VSFE in practical transmission scenarios, such as EDFA-amplified links, for a NLC bandwidth of 165 GHz (5$\times$32 GBaud). Although VAO offers an improved performance than VSFE, a substantial performance penalty (3.8 dB at 1000 km transmission distance) was still observed with respect to ideal nonlinearity compensation (see Fig. <ref>).
Such an ideal NLC performance could be theoretically achieved through a full-complexity implementation of the DBP algorithm over the entire transmitted bandwidth. The question that naturally arises is then, which, between VAO and DBP, represents a better trade-off between performance and complexity.

The computational complexity of the VAO scheme proposed in this work is determined by the complexity of its modified Volterra equalizer. Comprehensive studies on the complexity of VSFE, and simplified variants thereof, can be found in literature <cit.>. These works showed that, despite the complexity (per processed symbol) of a standard VSFE implementations scaling asymptotically as $\mathcal{O}(N^2)$, simplified implementations are possible with limited performance penalties. In particular, in <cit.> and <cit.> the complexity of VSFE was reduced to $\mathcal{O}(N)$ and $\mathcal{O}(N\log N)$ per symbol, respectively.

The computational complexity of the DBP algorithm, for a large number of processed samples $N$, scales as $\mathcal{O}(N_{\text{steps}}\log N)$ where $N_{\text{steps}}$ is the number of DBP steps performed.
Alike the VSFE/VAO case discussed in Sec. <ref>, the minimum value of $N$ for DBP is dictated by the targeted NLC performance and the channel memory (see e.g. <cit.>). Due the logarithmic scaling of simplified VSFE, matching an increasing channel memory by increasing $N$ only results in a minor complexity increase,
which is also the case for DBP.
The main issue in extending DBP to larger NLC bandwidths is the minimum $N_{\text{steps}}$ required to preserve a fixed NLC suppression. The NLC performance for DBP is determined by the relationship between the accuracy of the SSFM and the signal bandwidth at a fixed power spectral density, which depends on the specific SSFM method used and, overall, it is not well known. For instance, for a logarithmic step-size SSFM <cit.>, $N_{\text{steps}}$ scales quadratically with the NLC bandwidth (at fixed power spectral density). Although for other SSFM variants (see, e.g., <cit.>) this scaling could be more benign, the advantage of (non recursive) VSFE methods, and thus VAO, is that their complexity scales on the NLC bandwidth only through $N$, as a result of an increasing the channel memory.

An accurate study on the performance/complexity trade-off between VAO and DBP requires a detailed description of the implementation of both algorithms, and is beyond the scope of this paper.

§ CONCLUSION

In this work, a novel nonlinearity compensation scheme combining mid-link optical phase conjugation (OPC) and frequency-domain Volterra equalization, was proposed. The new scheme, referred to  as Volterra-assisted OPC (VAO) was designed to overcome the limitations of both OPC and Volterra equalization in practical transmission scenarios, whilst preserving the complexity of a conventional VSFE. VAO was demonstrated to effectively recover the NLC capabilities of OPC over a large optical bandwidth, in EDFA-amplified links. The addition of mid-link OPC was shown to significantly enhance the NLC performance of a conventional VSFE. Up to 10 dB higher NLC suppression was demonstrated for VAO and 4.2 dB enhancement in the received SNR in a 1000 km link compared to either OPC or VSFE.
As VAO dramatically outperforms Volterra-only equalisation for large NLC bandwidths, it offers in such a scenario a attractive trade-off between NLC effectiveness and computational complexity.

[Derivation of the third-order Volterra series kernel in the presence of mid-link OPC]

We begin the derivation of the third-order Volterra kernel from the Manakov equation in the frequency domain for a dual polarization signal. Only the $X$ polarization solution will be derived here, as the solution for the $Y$ polarization is simply obtained interchanging the subscripts $X$ and $Y$. For the signal on the $X$ polarization, the Manakov equation in frequency domain reads:

\begin{equation}
\frac{\partial {A}_{\scaleto{X}{4pt}}}{\partial z}=j\frac{\beta_2}{2}\omega^2{A}_{\scaleto{X}{4pt}}-\frac{\alpha(z)}{2}{A}_{\scaleto{X}{4pt}}+j\frac{8}{9}\gamma\mathcal{F}\{(\tilde{A}_{\scaleto{X}{4pt}} \tilde{A}_{\scaleto{X}{4pt}}^*+\tilde{A}_{\scaleto{Y}{4pt}} \tilde{A}_{\scaleto{Y}{4pt}}^*)\tilde{A}_{\scaleto{X}{4pt}}\},
\label{eq:Manakov_x}
\end{equation}

where $A_{\scaleto{X}{4pt}}(\omega,z)$ is the Fourier transform of the optical field over polarization $X$ at distance $z$, whereas
$\tilde{A}_{\scaleto{X}{4pt}}(t,z)\triangleq \mathcal{F}^{-1}\{ A_{\scaleto{X}{4pt}}\}$ is its time-domain counterpart.
The equivalence between the Volterra series and the regular perturbation expansion terms shown in <cit.> allows us to use the latter to derive the third-order Volterra kernel. Following the regular perturbation approach for the solution of (<ref>) <cit.>, the zeroth-order perturbation solution, corresponding to the first term of the Volterra series, is given by

\begin{equation}
{A}_{1\scaleto{X}{4pt}}(\omega,N_sL_s) = {A}_{\scaleto{X}{4pt}}(\omega,0)e^{\left(-\frac{\alpha(z)}{2}+i\frac{\beta_2}{2}\omega^2\right)N_sL_s},
\label{eq:zeroth_order}
\end{equation}

where ${A}_{\scaleto{X}{4pt}}(\omega,0)$ is the Fourier transform of the transmitted optical field, $N_s$ is the number of spans and $L_s$ is the span length. The zeroth order solution is recursively used to derive the first-order perturbation term of the solution of Eq. (<ref>), which corresponds to its third-order Volterra series expansion, and it is given by

\begin{equation}
\begin{split}
&{A}_{3\scaleto{X}{4pt}}(\omega,N_s L_s)= j\frac{8}{9}\gamma
&\times\int_0^{N_sL_s} e^{\int_{0}^{z^{\prime}} (-i\frac{\beta_{2}}{2}\omega^2+\frac{\alpha}{2})dz^{\prime\prime}}D(\omega, z^{\prime})dz^{\prime},
\end{split}
\label{eq:first_order}
\end{equation}

where the term $D(\omega,z)$  comes from substituting Eq. (<ref>) into the nonlinear term in the left-hand side of Eq. (<ref>), and it is given by

\begin{equation}
\begin{split}
&D(\omega,z)=-\iint \left[{A}_{1\scaleto{X}{4pt}}^{*}(\omega_1,z) {A}_{1\scaleto{X}{4pt}}(\omega_2,z){A}_{1\scaleto{X}{4pt}}(\omega-\omega_2+\omega_1,z) \right.\\
&+\left. {A}_{1{\scaleto{Y}{4pt}}}^{*}(\omega_1,z){A}_{1{\scaleto{Y}{4pt}}}(\omega_2,z){A}_{1\scaleto{X}{4pt}}(\omega-\omega_2+\omega_1,z) \right] d\omega_2 d\omega_1,
\end{split}
\label{eq:Dz}
\end{equation}

Substituting Eq. (<ref>) into Eq. (<ref>) we obtain

\begin{equation}
\begin{split}
e^{\left(i\frac{\beta_{2}}{2}\omega^2-\frac{\alpha(z)}{2} \right)N_{s}L_{s}}\\
&\times\iint  \left[\mathcal{S}_{\scaleto{XXX}{4pt}}(\omega,\omega_1,\omega_2)+\mathcal{S}_{\scaleto{YYX}{4pt}}(\omega,\omega_1,\omega_2)\right]\\
&\times F\left(\omega,\omega_1,\omega_2\right)\Xi\left(N_s,\omega,\omega_1,\omega_2\right)d\omega_2d\omega_1.
\end{split}
\label{eq:first_order_multispan}
\end{equation}

where the signal kernels $\mathcal{S}_{\scaleto{XXX}{4pt}}(\omega,\omega_1,\omega_2)$ and $\mathcal{S}_{\scaleto{YYX}{4pt}}(\omega,\omega_1,\omega_2)$, the phased-array $\Xi\left(N_s,\omega,\omega_1,\omega_2\right)$, and $\Delta\Omega$ were previously defined in Sec. <ref>.
Without in-line OPC, the four-wave mixing efficiency $F\left(\omega,\omega_1,\omega_2\right)$ and the phased array factor $\Xi\left(N_s,\omega,\omega_1,\omega_2\right)$ are defined in (<ref>) and (<ref>), respectively.

To study the effect of an OPC module placed at the  mid-link point, i.e. after $\frac{N_s}{2}$ fibre spans, the solution of (<ref>) is conveniently split into 2 contributions: one before and one after the OPC module. Rewriting Eq. (<ref>) we have:

\begin{equation}
\begin{split}
{A}_{3\scaleto{X}{4pt}}(\omega,N_s L_s) = &i\frac{8}{9}\gamma
&\left[\int_0^{\frac{N_s}{2}L_s} P(-z^{\prime}) e^{\int_{0}^{z^{\prime}} -i\frac{\beta_{2}}{2}\omega^2dz^{\prime\prime}}D(z^{\prime})
dz^{\prime}\right. \\
&\left.+\int_{{\frac{N_s}{2}L_s}}^{N_s L_s} P(-z^{\prime}) e^{\int_{0}^{z^{\prime}}-i\frac{\beta_{2}}{2}\omega^2dz^{\prime\prime}}
\right].
\end{split}
\label{eq:fwd+opc}
\end{equation}

where $P(z)$ is the power profile at distance $z$ given by

\begin{equation}
\label{eq:pow_profile}
\end{equation}

Both integrals in Eq. (<ref>) represent propagation up to specific part of the transmission link. The first one corresponds to propagation from the transmitter up the mid-link point, and the second one corresponds to propagation from the mid-link point to the end the last fiber span. Using the first-order solution for the first half of the link Eq. (<ref>) can be rewritten as follows:

\begin{equation}
\begin{split}
{A}_{3\scaleto{X}{4pt}}(\omega,N_s L_s)=&{A}_{3\scaleto{X}{4pt}}\left(\omega,\frac{N_s L_s}{2}\right)P\left(\frac{N_sL_s}{2}\right)e^{i\frac{\beta_2}{2}\omega^2\frac{N_s L_s}{2}}\\
&+i\frac{8}{9}\gamma P(N_sL_s)e^{i\frac{\beta_2}{2}\omega^2 N_s L_s} \\
&\times\int_{{\frac{N_s}{2}L_s}}^{N_s L_s} e^{\int_{0}^{z^{\prime}}\left(-i\frac{\beta_{2}}{2}\omega^2+\frac{\alpha}{2}\right)dz^{\prime\prime}}D(z^{\prime})dz^{\prime}
\end{split}
\label{eq:1st_order_sol_opc}
\end{equation}

When an OPC module is used, both zeroth and first-order perturbation terms undergo conjugation in time-domain at $z=\frac{N_s L_s}{2}$, leading to:

\begin{equation}
\begin{split}
{A}_{3\scaleto{X}{4pt}}(\omega,N_s L_s)=&\hat{A}_{3\scaleto{X}{4pt}}\left(\omega,\frac{N_s L_s}{2}\right)P\left(\frac{N_sL_s}{2}\right)e^{i\frac{\beta_2}{2}\omega^2\frac{N_s L_s}{2}}\\
&+i\frac{8}{9}\gamma P(N_sL_s)e^{i\frac{\beta_2}{2}\omega^2 N_s L_s} \\
&\times\int_{{\frac{N_s}{2}L_s}}^{N_s L_s} e^{\int_{0}^{z^{\prime}}\left(-i\frac{\beta_{2}}{2}\omega^2+\frac{\alpha}{2}\right)dz^{\prime\prime}}D(z^{\prime})dz^{\prime}.
\end{split}
\end{equation}

where the conjugation of the first-order perturbation term is given by:

\begin{equation}
\begin{split}
\hat{A}_{3\scaleto{X}{4pt}}\left(\omega,\frac{N_s L_s}{2}\right)&\triangleq \mathcal{F}\left\{\tilde{A}^{*}_{3\scaleto{X}{4pt}}\left(t,\frac{N_s L_s}{2}\right)\right\}\\
&={A}^{*}_{3\scaleto{X}{4pt}}\left(-\omega,\frac{N_s L_s}{2}\right).
\end{split}
\end{equation}

The term corresponding to the first half of the transmission link then becomes:

\begin{equation}
\begin{split}
&\hat{A}_{3\scaleto{X}{4pt}}\left(\omega,\frac{N_s L_s}{2}\right)P\left(\frac{N_sL_s}{2}\right)e^{i\frac{\beta_2}{2}\omega^2\frac{N_s L_s}{2}} \\
&=-i\gamma\frac{8}{9}P(N_sL_s)\iint K(\omega,\omega_1,\omega_2)\Lambda^*\left(\frac{N_s}{2},\omega,\omega_1,\omega_2\right) d\omega_2 d\omega_1,
\end{split}
\end{equation}

\begin{equation}
\begin{split}
\Lambda\left(\frac{N_s}{2},\omega,\omega_1,\omega_2\right)\triangleq & \sum_{n=1}^{\frac{N_s}{2}}\left[\frac{P(nL_s)e^{i\beta_2\Delta\Omega nL_s}}{\alpha(nL_s)+i\beta_2\Delta\Omega}\right. \\
&\left. -\frac{P((n-1)L_s)e^{i\beta_2\Delta\Omega (n-1)L_s}}{\alpha((n-1)L_s)+i\beta_2\Delta\Omega} \right],
\end{split}
\end{equation}

\begin{equation}
\end{equation}

Beyond the conjugation point $z\geq \frac{N_s L_s}{2}$, the zeroth order perturbative solution is

\begin{equation}
{A}_{1\scaleto{X}{4pt}}(\omega,z)=A_{\scaleto{X}{4pt}}^{*}(-\omega,0)P(z)e^{i\frac{\beta_2}{2}\omega^2(z-N_s L_s)}.
\label{eq:zeroth_order_OPC}
\end{equation}

Substituting (<ref>) into $D(z)$ we obtain:

\begin{equation}
\begin{split}
D(z) = &-P^3(z)e^{i\frac{\beta_2}{2}\omega^2(z-N_s L_s)}\\
& \iint A^*_{\scaleto{X}{4pt}}(-\omega_2,0)A_{\scaleto{X}{4pt}}(-\omega_1,0)A_{\scaleto{X}{4pt}}^*(\omega_2-\omega_1-\omega,0) \\
& e^{i\beta_2\Delta\Omega(z-N_s L_s)}d\omega_2 d\omega_1.
\end{split}
\end{equation}

Hence, the second term in Eq. (<ref>) becomes

\begin{equation}
\begin{split}
P(N_sL_s)e^{i\frac{\beta_2}{2}\omega^2 N_s L_s} \int_{{\frac{N_s}{2}L_s}}^{N_s L_s} P^{-1}(z^{\prime})e^{-i\frac{\beta_{2}}{2}\omega^2 z^{\prime}}
i\frac{8}{9} \gamma D(z^{\prime})dz^{\prime}\\
=i\gamma\frac{8}{9}P(N_sL_s)\iint K(\omega,\omega_1,\omega_2)\psi\left(\frac{N_s}{2},\omega,\omega_1,\omega_2\right) d\omega_2 d\omega_1
\end{split}
\end{equation}

\begin{equation}
\begin{split}
\Psi\left(\frac{N_s}{2},\omega,\omega_1,\omega_2\right)\triangleq & e^{-i\beta_2\Delta\Omega N_s L_s}\\
\times\sum_{n=\frac{N_s}{2}+1}^{N_s}\left[\frac{P(nL_s)e^{i\beta_2\Delta\Omega nL_s}}{\alpha(nL_s)+i\beta_2\Delta\Omega}\right.
& \left.-\frac{P((n-1)L_s)e^{i\beta_2\Delta\Omega (n-1)L_s}}{\alpha((n-1)L_s)+i\beta_2\Delta\Omega} \right].
\end{split}
\end{equation}

Finally, we obtain:

\begin{equation}
\begin{split}
&A_{3\scaleto{X}{4pt}}(\omega, N_sL_s)=i\gamma\frac{8}{9}P(N_sL_s)\iint K(\omega,\omega_1,\omega_2)\\
&\times\left[\Psi\left(\frac{N_s}{2},\omega,\omega_1,\omega_2\right)-\Lambda^*\left(\frac{N_s}{2},\omega,\omega_1,\omega_2\right)\right] d\omega_2 d\omega_1
\end{split}
\end{equation}

where the nonlinear kernel in the presence of OPC is given by:

\begin{equation}
\begin{split}
\Gamma(N_s,\omega,\omega_1,\omega_2)\triangleq\Psi\left(\frac{N_s}{2},\omega,\omega_1,\omega_2\right)-\Psi^*\left(\frac{N_s}{2},\omega,\omega_1,\omega_2\right).
\end{split}
\label{eq:OPC_kernel}
\end{equation}

In order to suppress the first-order solution for all $\omega$ one condition is having $\Gamma(N_s,\omega,\omega_1,\omega_2)=0 \;\; \forall \;\; \omega, \omega_1, \omega_2$. This is verified if the following conditions are jointly verified

\begin{eqnarray}
P((N_s-n)L_s^-)=P((n-1)L_s^+), \\
\alpha((N_s-n)L_s^-)=-\alpha((n-1)L_s^+),\\
\alpha((N_s-n-1)L_s^+)=-\alpha(nL_s^-).
\end{eqnarray}

In the case where $\alpha=0$ (lossless fiber), all the terms in the sum in (<ref>) are the same and, thus, $\Gamma(N_s,\omega,\omega_1,\omega_2)=0$. If $\alpha(z)=\alpha$ everywhere except in $z=nL_s$ for $n=0,1,2,...$ where $\alpha(z)=\delta(z-nL_s)$ (EDFA amplification) we have

\begin{align}
\begin{split}
\alpha((N_s-n)L_s^-)&=\alpha((n-1)L_s^+) \\=\alpha((N_s-n-1)L_s^+)
&=\alpha(nL_s^-), \;\; \text{for} \;\; n=1,2,...,\frac{N_s}{2}
\end{split}
\end{align}

and the OPC kernel becomes:

\begin{equation}
\Gamma(N_s,\omega,\omega_1,\omega_2)=\Xi^*\left(\frac{N_s}{2},\omega,\omega_1,\omega_2\right) G(\omega,\omega_1,\omega_2)
\end{equation}

\begin{align}
\begin{split}
G(\omega,\omega_1,\omega_2)\triangleq\frac{(e^{-i\beta_2\Delta\Omega L_s}e^{-\alpha L_s}-1)(\alpha-i\beta_2\Delta\Omega)}{\alpha^2+\beta_2^2\Delta\Omega^2}\\
+\frac{(e^{-i\beta_2\Delta\Omega L_s}-e^{-\alpha L_s})(\alpha+i\beta_2\Delta\Omega)}{\alpha^2+\beta_2^2\Delta\Omega^2}
\end{split}
\label{eq:OPC_char_kernel}
\end{align}

is the characteristic kernel of the optical channel in the presence of mid-link OPC.
The Volterra series terms discussed above corresponds to the case of signal propagation in the optical fiber. A Volterra equalizer instead is compute such terms for the reverse propagation case corresponding to integrating (<ref>) in the backward propagation direction. From standard calculus operations one can obtain the Volterra equalizer terms (for the case with OPC) as

\begin{align}
\begin{split}
{A}_{1\scaleto{X}{4pt}}(\omega,0)&={A}_{\scaleto{X}{4pt}}(\omega,N_s L_s)e^{-i\frac{\beta_2}{2}\omega^2 N_sL_s}\\
{A}_{3\scaleto{X}{4pt}}(\omega,0)&=-j\gamma\frac{8}{9}\iint K^{\prime}(\omega,\omega_1,\omega_2)\\
&\times\Xi^{*}\left(-\frac{N_s}{2},\omega,\omega_1,\omega_2\right)G^{\prime}(\omega,\omega_1,\omega_2) d\omega_2 d\omega_1
\end{split}
\label{eq:Volterra_reverse}
\end{align}

\begin{align}
\begin{split}
F^{\prime}(\omega,\omega_1,\omega_2)&=\frac{1-e^{\alpha L_s} e^{-j\beta_2\Delta\Omega L_s}}{j\beta_2\Delta\Omega-\alpha},\\
G^{\prime}(\omega,\omega_1,\omega_2)&=\frac{(e^{j\beta_2\Delta\Omega L_s}e^{\alpha L_s}-1)(\alpha-j\beta_2\Delta\Omega)}{\alpha^2+\beta_2^2\Delta\Omega^2}\\
+&\frac{(e^{j\beta_2\Delta\Omega L_s}-e^{\alpha L_s})(\alpha+i\beta_2\Delta\Omega)}{\alpha^2+\beta_2^2\Delta\Omega^2}.
\end{split}
\label{eq:Volterra_reverse_terms}
\end{align}

are the inverse Volterra kernels for backward propagation and $K^{\prime}(\omega,\omega_1,\omega_2)$ is defined as in (<ref>) but where the transmitted signal spectrum $S(\omega)$ is replaced by the received signal spectrum.

§ ACKNOWLEDGMENT

The authors would like to acknowledge Dr. D. Lavery (UCL) for insightful discussions on the computational complexity of NLC schemes, N. Shevchenko and D. Semrau (UCL) for discussions regarding the derivation of the Volterra series expansion for systems using mid-link OPC.


# RethNet: Object-by-Object Learning for Detecting Facial Skin Problems

Shohrukh Bekmirzaev
lululab Inc
Gangnam-gu, Seoul, Korea
<EMAIL_ADDRESS>Seoyoung Oh
lululab Inc
Gangnam-gu, Seoul, Korea
<EMAIL_ADDRESS>Sangwook Yoo
lululab Inc
Gangnam-gu, Seoul, Korea
<EMAIL_ADDRESS>
###### Abstract

Semantic segmentation is a hot topic in computer vision where the most
challenging tasks of object detection and recognition have been handling by
the success of semantic segmentation approaches. We propose a concept of
object-by-object learning technique to detect 11 types of facial skin lesions
using semantic segmentation methods. Detecting individual skin lesion in a
dense group is a challenging task, because of ambiguities in the appearance of
the visual data. We observe that there exist co-occurrent visual relations
between object classes (e.g., wrinkle and age spot, or papule and whitehead,
etc.). In fact, rich contextual information significantly helps to handle the
issue. Therefore, we propose REthinker blocks that are composed of the locally
constructed convLSTM/Conv3D layers and SE module as a one-shot attention
mechanism whose responsibility is to increase network’s sensitivity in the
local and global contextual representation that supports to capture
ambiguously appeared objects and co-occurrence interactions between object
classes. Experiments show that our proposed model reached MIoU of 79.46% on
the test of a prepared dataset, representing a 15.34% improvement over Deeplab
v3+ (MIoU of 64.12%).

##  1 Introduction

Figure 1:  The concept of object-by-object learning: In fact, skin lesion
objects have visual relations between each other where it helps easily human
to judge about what type of skin lesions are. In the figure, the green and red
boxes represent the level of the image patch’s similarity. The green and red
line describe that there exist positive and negative relations between objects
or a group of objects of each patch.  Figure 2:  Proposed REthinker block that
consists of the locally constructed convLSTM/Conv3D layers and SE module [23]
as a one-shot attention mechanism whose responsibility is to improve network’s
sensitivity in local and global contextual representation that helps to
capture ambiguously appeared objects and co-occurrence interactions between
object classes.

Semantic segmentation has been one of the fundamental and active topic in
computer vision for a long time. This topic is of wide interest for real-world
applications of autonomous driving, robotics and a range of medical imaging
applications. Recent improvements and advances in semantic segmentation enable
to emerge various new application areas in skin analysis. For example, facial
skin lesion analysis has been attracting a lot of attention as having
beautiful skin without troubles is getting popular and influenced on the
society nowadays. E -cosmetics which involve the beautification, facial image
simulation, digital makeup and accurate facial lesion analysis are fast-
growing sector in the marketing.

Accurately and early detecting facial skin problems is an important clinical
task and automated dermatology can be used to save time and reduce costs [25].
E-cosmetics and dermatological computer-aided systems are developing rapidly
behind computer vision progresses [49], [26], [25] , [45], [12], [17], [29].
However, previous methods have shown only limited improvements and visual
understanding of individual skin lesion in a dense group is still a
challenging task.This is because, it is hard to distinguish some types of
facial skin lesions between each other as the fine-grained object
categorization problem. Furthermore, facial skin lesions appear ambiguously
with different (typically small) sizes, which lead a network to assign wrong
classes easily. The use of rich contextual relation information helps to
reduce the issue significantly [42], [27]. For example, there are the object-
object interactions [27] between some skin lesions where the detection of an
object class helps to detect another by their co-occurrence interactions. The
detection decisions about individual skin lesions can be switched dynamically
through contextual relations among objects. We denote this cognitive process
as object-by-object decision-making.

We present a REthinker module based on the SENet module [23] [22] and locally
constructed convLSTM/conv3D unit [41] to increase network’s sensitivity in
local and global contextual representations. The proposed modules are easy to
use and applicable in any standard convolutional neural networks (CNNs). The
use of the REthinker modules forces networks to capture the contextual
relationships between object classes regardless of similar texture and
ambiguous appearance they have.

We experiment our proposed modules by modifying current state-o- the-art
networks [19], [43], [8] for feature extraction. We originally use the decoder
of DeepLabv3+ [7]. Experimental results show that our proposed models
outperform the state-of-the-art segmentation networks [34],[50], [7] ,[4],
[51], [39] by the high difference of 15.34% MIoU [5], [21], [52] in detecting
of facial skin problems in a prepared dataset. Moreover, our models have shown
promising results on ISIC 2018 segmentation tasks. The overall contributions
of our paper can be summarized as follows:

  * •

We introduce a new concept named ”object-by-object” learning, where an object
can be identified by looking at other objects.

  * •

We propose a novel residual block called REthinker modules that support
”object-by-object” technique by capturing contextual relationships between
object classes.

  * •

We develop a novel RethNet architecture that detects skin lesions with higher
accuracy than the recent state-of-the-art segmentation approaches.

##  2 Related Work

Semantic Scene Understanding: Semantic understanding of visual scenes has
become ubiquitous in computer vision. Impressive semantic segmentation
approaches are mostly based on the Fully Convolutional Network (FCNs) [33],
[11], [30], [32], [53] ,[3] , [37]. One key reason for the success of FCNs is
that they use multi-scale (MS) image representations, which are subsequently
upsampled to recover the lost resolution. Moreover, atrous convolution has
proven to be an effective technique by providing a larger receptive field size
without increasing the numbers of kernel parameters [3]. Spatial pyramid
pooling module [18] has been successfully applied with atrous convolutions by
state-of-the-art networks [5], [50],[21], [52], [34], [4], [51],[7] , [39] on
segmentation and object detection benchmarks [14], [35], [13], [31], [16],
[10]. Recently, depthwise separable convolution [8] has known as an efficient
technique to reduce the computation complexity in convolutional operations and
allow networks to go deeper [5], [39], [21], [52], [34]. Originally, the
depthwise separable convolution consists of the depthwise and  pointwise
convolutions where depthwise convolution keeps the channels separate and uses
the standard convolution operation in each input channel [8].

Contextual relations: H.S. Hock et al. [20] introduced early the contextual
relations between object scenes by experiments based on the criteria of
familiarity, physical plausibility, and belongingness. The scene context
information plays a crucial role on the semantic scene understanding. However,
the contextual relation information between object classes are often ignored
[15], [42], [27]. The relative spatial configurations of particular objects
yield the higher-level contextual information while the lower-level contextual
information demonstrates the semantic and visual relationships among objects
or group of objects. S. Kumar et al. [27] categorize types of the contextual
relationships of the scene labeling to region-region, object-region and
object-object interactions and provide a hierarchical framework using
Conditional Random Fields (CRFs) for semantic segmentation. CRFs models are
widely used to capture the local contextual interactions of image regions [44]
, [42], [36], [5], [40]. RNNs are investigated broadly to aggregate global
context in the semantic segmentation [48], [2], [38], [15] [28], [54]. SE
modules [23] [22] are successfully applied to capture global contextual
information by simply exploiting CNN layers.

##  3 Datasets

We prepare a dataset called ”Multi-type Skin Lesion Labelled Database” (MSLD)
with pixel-wise labelling of frontal face images. We report that the designing
of MSLD is unique in ML community where it is not available such dataset with
the labelling of multi-type skin lesions of facial images. We further test the
proposed models in the International Skin Imaging Collaboration (ISIC) dataset
which holds dermoscopic images with 5 types of skin problems. In this section,
we introduce the process of image accumulation and annotation of MSLD dataset
and announce about ISIC dataset.

Figure 3: Samples of the SiblingsDB [46] dataset. Few samples with skin
problems have been annotated in order to show only the labelling process of
facial skin lesions. Note that we do not disclose our MSLD dataset samples.

###  3.1 Image accumulation

We collected a total of 27,790 frontal face images using kiosks in cosmetics
stores during the period from April to August in 2018. The kiosks have a
standard camera whose image sensor is a 1/3.2 Inch CMOS IMX179. The user’s
consent is obtained before capturing images. The total number of pixels of the
image sensor is 3,288 x 2,512 (8.26Mpixel) with 24-bit depth. The images are
captured at autofocus and auto-white balance in the distance of 20 mm.

###  3.2 Image Annotation

The collected images are studied carefully. Then 412 images have been
annotated with the labelling of 11 common types of facial skin lesions and 6
additional classes as in Figure 3 using the PixelAnnotation tool [1]. The skin
lesions are whitehead, papule, pustule, freckle, melasma, age spots, PIH
111Post inflammatory hyperpigmentation (PIH) caused usually by inflammation or
acnes, flush, seborrheic, dermatitis, wrinkle and black dot. The additional
classes are normal skin, hair, eyes/mouth/eyebrow, glasses, mask/scarf and
background. We report that we do not disclose our (MSLD) dataset as the user’s
privacy is taken under the responsibility. We use the collected and annotated
images as research purposes where they are used only in the training.

###  3.3 ISIC Dataset

We test the proposed approaches in the TASK 2 222https://challenge2018.isic-
archive.com/task2/ of the ISIC, 2018 challenge called ”Lesion Attribute
Detection” [9] in order to provide further experiments. The goal of the task
is to predict 5 skin lesion attributes from dermoscopic images. These lesion
attributes are pigment networks, negative network, streaks, milia-like cysts,
globules, and dots. The lesion classes that have visual similarity can be seen
as a fine-grained classification problem. There are 2594 images for training,
100 images for validation, and 1000 images for the test.

Figure 4:  The figure represents comparison of SE blocks (a-b), baseline block
(c) and REthinker blocks (d-e): The proposed REthinker blocks are designed
under locally constructed Conv3D (d) and convLSTM (e) layers.

##  4 Methods

###  4.1  Squeeze and Excitation module

Squeeze and Excitation network (SENet) [23] has been introduced as a winner of
the ILSVRC 2017 classification task in the top-5 error of 2.251%. SE blocks
Figure 4 [a-b] have proven to be an effective channel-wise attention mechanism
[6] , which enables the network to perform dynamic channel-wise feature
recalibration. SE module is computationally cheaper and helps the network to
learn contextual higher-level features by the aggregated transformations of
the global pooling. The SE block consists of squeeze and excitation operations
where the squeeze operation uses the global pooling to transform global
spatial information to channel-wise statistics as a channel descriptor. The
excitation operation performs a self-gating mechanism based on 2 fully
connected (FC) layers to capture channel-wise dependencies from the channel
descriptor. Finally, the channel-wise dependencies are used to exploit the
previous input transformation by the multiplication operation. The role of the
SE module in our proposed module is to pass high-level contextual information
outside of this region for context-dependent decision making.

###  4.2  REthinker module

REthinker modules consist of SE module and the locally constructed
convLSTM/conv3D layers as an one-shot attention mechanism as in Figure 4
[d-e], which are both responsible for extracting contextual relations from
features. Precisely, as the global pooling of SE module aggregates the global
spatial information, the SE module passes more embedded higher-level
contextual information across large neighborhoods of each feature map.
Whereas, the locally constructed convLSTM/conv3D layers encode lower-level
contextual information across local neighborhoods elements of fragmented
feature map (patches) while further take spatial correlation into
consideration distributively over patches. The output of locally constructed
convLSTM/conv3D recieves 3D , $U_{d}\in R^{H\times W\times D}$ feature maps
from the residual blocks and passes a transformed 3D , $U^{\prime}_{d}\in
R^{H\times W\times D}$ feature map. The locally constructed convLSTM/conv3D is
identified as follows:

$\displaystyle U^{\prime}_{d}$
$\displaystyle=F_{pi}([\Phi(F_{ip}(U_{d},N)|v_{t})|h_{t}],N)$ (1)

Where, $F_{ip}$ is the  image2patches operator function, $F_{ip}:R^{H\times
W\times C}\rightarrow R^{N^{2}\times H^{\prime}\times W^{\prime}\times D}$ to
provide local spatiotemporal 4D data, $v_{t}=F_{ip}(U_{d},N)$. Practically,
the feature maps are transformed to patches over channel as spatiotemporal
data. Here, $H^{\prime}\times W^{\prime}$ is a patch size
$(H^{\prime}=H/N,W^{\prime}=W/N)$ to be assumed as an object or a group of
objects. The given N is the dimensional slicing coefficient over the spatial
dimensions (W, H) of the feature map. Thus, $\Phi$ is conv3D or convLSTM
operator function, $\Phi:R^{N^{2}\times H^{\prime}\times W^{\prime}\times
D}\rightarrow R^{N^{2}\times H^{\prime}\times W^{\prime}\times D}$ to be
applied with keeping the depth of the feature map. The convLSTM/conv3D serves
to encode spatiotemporal correlations between features by veiwing sequentially
objects or a group of objects of patches and passes the output as
spatiotemporal data $h_{t}=\Phi(v_{t})$ to the patches2image operator whose
identification function here is $F_{pi}:R^{N^{2}\times H^{\prime}\times
W^{\prime}\times D}\rightarrow R^{H\times W\times D}$ and
$U^{\prime}_{d}=F_{pi}(h_{t},N)$. Note that $h_{t}=\Phi(v_{t},h_{t-1})$
performs as the hidden states in the convLSTM. The gates $i_{t},h_{t},o_{t}$
of the convLSTM are identified as follows:

$\displaystyle i_{t}$ $\displaystyle=\sigma(w_{vi}\odot v_{t}+w_{hi}\odot
h_{{t}-1}+w_{ci}\circ{c}_{{t}-1}+b_{i})$ (2) $\displaystyle f_{t}$
$\displaystyle=\sigma(w_{vf}\odot{v}_{t}+w_{hf}\odot{h}_{{t}-1}+w_{cf}\circ{c}_{{t}-1}+b_{f})$
$\displaystyle{c}_{t}$
$\displaystyle=f_{t}\circ{c}_{t}-1+i_{t}\circ\tanh(w_{vc}\ast{v}_{t}+w_{c}\odot{h}_{{t}-1}+b_{c})$
$\displaystyle o_{t}$ $\displaystyle=\sigma(w_{vo}\odot
v_{t}+w_{ho}\odot{h}_{{t}-1}+w_{co}\circ{c}_{t}+b_{o})$ $\displaystyle{h}_{t}$
$\displaystyle=o_{t}\circ\tanh({c}_{t})$

Where, ’$\odot$ ’ represents the convolution operator and ’$\circ$’ denotes
the Hadamard product. The output of SE module is used to exploit the output of
locally constructed convLSTM/conv3D by the channel-wise multiplication
operation as the output of REthinker module whose feature map represents long-
range local and global contextual information to enable the context-dependent
decision making.

Figure 5: Our proposed RethNet based RE-Xception and the decoder of the
DeepLabv3+ [7]. Where D and N is the depth of future maps and the dimensional
slicing coefficient over the spatial dimensions respectively.

.

###  4.3  RethNeT

An Encoder Search: In practice, the REthinker blocks are applicable in any
standard CNNs. We consider to employ current state of the art networks [19],
[43], [8]. We integrate the modern architectures with our proposed REthinker
blocks to improve the network’s sensitivity in local and global contextual
representations enabling object-by-object learning technique. We experiment
ResNet [19], ResNeXt [47] and Xception [8], [7].

A Decoder Search: We believe that the rich contextual information is a key to
capture ambiguously appeared objects and co-occurrence interactions between
object classes where that is usually obtained in encoders. Therefore, we do
not consider the decoder path. We select the decoder of DeepLabv3+ [7] to
recover object segmentation details of individual skin lesions.

Models | Mean IoU(%) | Pixel Acc(%)
---|---|---
DenseASPP [50] | 58.31 | 89.31
PSPNet+ResNet-101 [18] | 63.24 | 91.92
DeepLabv3Plus + ResNet-101 [7] | 63.74 | 92.51
DeepLabv3Plus +ResNeXt-101 [33] | 64.64 | 93.14
DeepLabv3Plus + Xception [7] | 64.12 | 94.08
DeepLabv3Plus + Xception+SE | 65.49 | 94.12
DeepLabv3Plus + Xception+baseline-c | 65.52 | 94.21
RethNet + baseline-c | 62.11 | 92.44
RethNet + Rethinker-d | 76.56 | 96.45
RethNet + Rethinker-e | 79.46 | 96.11
Table 1:  Experimental results on test samples of our MSLD dataset.

RethNet: We simply investigate RethNet with the combining of the Xception
module and REthinker modules as in Figure 5 . We modify Xception as follows:
(1) We add REthinker module after each Xception blocks without spatial loss of
feature maps. (2) We remove the final block of the entry flow of Xception. (3)
We keep the patch size as 4x4 in each REthinker module in order to ”see”
future maps wider in ConvLSTM/conv3D with simply increasing time steps. (4)
The number of parameters is minimized in the  middle flow and exit flow. As
suggested in [7], (5) the max-pooling operation is replaced by depthwise
separable convolutions with striding and the batch normalization and ReLU is
applied after each 3 x 3 depthwise convolution of the Xception module.

Figure 6:  The number of parameters in the reference models and proposed modes
with accuracy comparison on the test of MSLD dataset.

Figure 7: The visualization results of RehtNet + Rethinker-e: The real test
images (a) are obtained from the  fionaseah.com

with an agreement of the author. The candidate of the test images is suffered
from basically, PIH (caused by acne) and papules where the results images
(b-d) show that the most of facial skin lesions are correctly predicted.

##  5 Experiments

###  5.1 Implementation

For experimental comparisons, we use the standard networks that are
DeepLabv3+, PSPNet, DenseASPP as the reference networks. All models of
networks were implemented using the TensorFlow framework and trained on a
single NVIDIA GeForce GTX 1080 Ti GPUs, Intel(R) Core(TM) i7-8700K CPU @
3.20GHz. In the experiments, 374 images are used for training and 38 images
are for testing. We use the standard data augmentation techniques whose
participants are the random rotation, random zooming, and random horizontal
flipping during the training. The input and ground-truth images of the dataset
is resized to 730 x 960 and random cropping applied by 512 x 512 as the input
of the network during the training. We follow the same training protocols as
suggested in [7], [18]. In all experiments, the softmax cross-entropy is
applied for loss and the momentum optimizer is used, whose base learning rate
is set to 0.001 decreased by a factor of 10 every 50 epoch of total 200 epoch
with decay 0.9.

Models | Rank | Jaccard (%) | Dice (%)
---|---|---|---
Ensemble [24] | 1 | 0.483 | 0.651
Unet + ASPP +DenseNet169 [24] | 2 | 0.464 | 0.629
Unet + ASPP + Resnetv2 [24] | 3 | 0.455 | 0.616
Unet + ASPP +ResNet151 [24] | 5 | 0.436 | 0.598
DeepLabv3Plus + Xception | - | 0.451 | 0.614
DeepLabv3Plus + Xception+SE | - | 0.469 | 0.627
DeepLabv3Plus + Xception+baseline-c | - | 0.456 | 0.616
RethNet + baseline-c | - | 0.441 | 0.592
RethNet + Rethinker-d | - | 0.473 | 0.639
RethNet + Rethinker-e | - | 0.475 | 0.644
Table 2:  Experimental results on test samples of ISIC 2018 challenge in the
task 2. The evaluation metrics in this task are average Jaccard Index (mIoU)
and Dice coefficient following by the proposed metrics of the challenge. The
rank represents positions on the test leaderboard of the challenge [24]. Note
that the Jaccard index metric, also referred to as the Intersection over Union
(IoU).

###  5.2 Results

Thanks to REthinker blocks, It shows significant improvements in the facial
skin lesion detection task of MSLD dataset. As Table 1 summarized results, we
made a further improvement by DeepLabv3Plus + Xception+ SE and DeepLabv3Plus +
Xception+ baseline-c, which showed a better result of 65.49 and 65.52 MIoU
than all other reference models except the proposed models. Our proposed
network RethNet+ REthinker-e blocks achieved MIoU of 79.46% on the test, where
it is the top in all experiments.

Figure 8: The ground truth and inference results of proposed models and the
best reference model in MSLD dataset . We keep the face entity of the MSLD
dataset.

Computational cost: We compare proposed models with the reference models in
terms of the number of parameters and accuracy (Figure 6). Even though our
RethNet + REthinker-e block is the top on the list with 112M parameters, it
reached high performance in the accuracy with a big difference (e.g 15 % of
mIoU greater than the best reference model, 14% of mIoU higher than the
baseline model). We further report inference time of the RethNet + REthinker-e
block that whose running time for per image inference by the 512x512 of
resolution is an average 2.7 sec on a single GPU and 11 sec on CPU of those
mentioned hardware sources.

###  5.3 ISIC challenge

As Table 2 represents, our RethNet + REthinker-e block showed 47.5% of Mean-
Jaccard on the test of ISIC dataset, where it outperforms all competitive
single models except an ensemble model (”48.3% of Mean-Jaccard”). Note that
the top-ranked models on the test leaderboards of the challenge use broadly
preprocessing techniques such as image enhancing, polluting dermoscopic images
with random hairs, and data augmentation. However, we apply only common
techniques of data augmentation during training and test images are evaluated
without any preprocessing techniques.

###  5.4 Contribution Discussion

In fact, the fine-grained classification has been becoming an open issue in
the computer vision community so far. It is a real challenge to differentiate
classes that have similar visual context. Especially the problem is more
common in medical imaging applications. We consider solving the problem with a
novel straightforward technique by our application (”Detecting multi-type
facial skin lesions”) where there is not yet accurate and solid application or
method to detect correctly and differentiate multi-type facial skin lesions.
The proposed blocks are easy to use and possible to apply to any standard CNNs
with considering time complexity by limiting the number of the blocks.

##  6 Conclusion

We propose successfully an efficient network architecture to address detecting
multi-type facial skin lesions by a novel object-by-object learning technique.
Experimental results show that our proposed model outperformed state-of-the-
art segmentation networks by a high gap in the MSLD dataset. Furthermore, our
model takes promising results on the ISIC 2018 segmentation task. In the
future, we consider the time complexity of Rethinker blocks and try to design
more light-weight models.

##  7 Acknowledgement

This work was supported and funded by SBA (Seoul Business Agency) under a
contract number CY190018.

## References

  * [1] A. Br.  Pixel annotation tool.  2017\.
  * [2] W. Byeon, T. M. Breuel, F. Raue, and M. Liwicki.  Scene labeling with lstm recurrent neural networks.  In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3547–3555, June 2015.
  * [3] L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille.  Semantic image segmentation with deep convolutional nets and fully connected crfs.  CoRR, abs/1412.7062, 2014.
  * [4] L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  CoRR, abs/1606.00915, 2016.
  * [5] L. Chen, G. Papandreou, F. Schroff, and H. Adam.  Rethinking atrous convolution for semantic image segmentation.  CoRR, abs/1706.05587, 2017.
  * [6] L. Chen, H. Zhang, J. Xiao, L. Nie, J. Shao, and T. Chua.  SCA-CNN: spatial and channel-wise attention in convolutional networks for image captioning.  CoRR, abs/1611.05594, 2016.
  * [7] L. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  CoRR, abs/1802.02611, 2018.
  * [8] F. Chollet.  Xception: Deep learning with depthwise separable convolutions.  CoRR, abs/1610.02357, 2016.
  * [9] N. C. F. Codella, D. Gutman, M. E. Celebi, B. Helba, M. A. Marchetti, S. W. Dusza, A. Kalloo, K. Liopyris, N. K. Mishra, H. Kittler, and A. Halpern.  Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (ISIC).  CoRR, abs/1710.05006, 2017.
  * [10] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele.  The cityscapes dataset for semantic urban scene understanding.  CoRR, abs/1604.01685, 2016.
  * [11] J. Dai, K. He, and J. Sun.  Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation.  CoRR, abs/1503.01640, 2015.
  * [12] A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, and S. Thrun.  Dermatologist-level classification of skin cancer with deep neural networks.  Nature, 542:115 EP –, Jan 2017.
  * [13] M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman.  The pascal visual object classes challenge: A retrospective.  International Journal of Computer Vision, 111(1):98–136, Jan. 2015\.
  * [14] M. Everingham, L. Gool, C. K. Williams, J. Winn, and A. Zisserman.  The pascal visual object classes (voc) challenge.  Int. J. Comput. Vision, 88(2):303–338, June 2010.
  * [15] H. Fan, X. Mei, D. Prokhorov, and H. Ling.  Multi-level contextual rnns with attention model for scene labeling.  IEEE Transactions on Intelligent Transportation Systems, 19(11):3475–3485, Nov 2018.
  * [16] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun.  Vision meets robotics: The kitti dataset.  Int. J. Rob. Res., 32(11):1231–1237, Sept. 2013.
  * [17] S. S. Han, M. S. Kim, W. Lim, G. H. Park, I. Park, and S. E. Chang.  Classification of the clinical images for benign and malignant cutaneous tumors using a deep learning algorithm.  Journal of Investigative Dermatology, 138(7):1529 – 1538, 2018\.
  * [18] K. He, X. Zhang, S. Ren, and J. Sun.  Spatial pyramid pooling in deep convolutional networks for visual recognition.  CoRR, abs/1406.4729, 2014.
  * [19] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  CoRR, abs/1512.03385, 2015.
  * [20] H. S. Hock, G. P. Gordon, and R. Whitehurst.  Contextual relations: The influence of familiarity, physical plausibility, and belongingness.  Perception & Psychophysics, 16(1):4–8, Jan 1974.
  * [21] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  CoRR, abs/1704.04861, 2017.
  * [22] J. Hu, L. Shen, S. Albanie, G. Sun, and A. Vedaldi.  Gather-excite: Exploiting feature context in convolutional neural networks.  CoRR, abs/1810.12348, 2018.
  * [23] J. Hu, L. Shen, and G. Sun.  Squeeze-and-excitation networks.  CoRR, abs/1709.01507, 2017.
  * [24] N. A. Koohbanani, M. Jahanifar, N. Z. Tajeddin, A. Gooya, and N. M. Rajpoot.  Leveraging transfer learning for segmenting lesions and their attributes in dermoscopy images.  CoRR, abs/1809.10243, 2018.
  * [25] K. Korotkov.  automatic change detection in multiple pigmented skin lesions, 2014.
  * [26] K. Korotkov and R. Garcia.  Computerized analysis of pigmented skin lesions: A review.  Artificial Intelligence in Medicine, 56(2):69 – 90, 2012.
  * [27] S. Kumar and M. Hebert.  A hierarchical field framework for unified context-based classification.  In Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1, volume 2, pages 1284–1291 Vol. 2, Oct 2005.
  * [28] X. Liang, X. Shen, D. Xiang, J. Feng, L. Lin, and S. Yan.  Semantic object parsing with local-global long short-term memory.  CoRR, abs/1511.04510, 2015.
  * [29] H. Liao.  A deep learning approach to universal skin disease classification.  2015\.
  * [30] G. Lin, C. Shen, I. D. Reid, and A. van den Hengel.  Efficient piecewise training of deep structured models for semantic segmentation.  CoRR, abs/1504.01013, 2015.
  * [31] T. Lin, M. Maire, S. J. Belongie, L. D. Bourdev, R. B. Girshick, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick.  Microsoft COCO: common objects in context.  CoRR, abs/1405.0312, 2014.
  * [32] Z. Liu, X. Li, P. Luo, C. C. Loy, and X. Tang.  Semantic image segmentation via deep parsing network.  CoRR, abs/1509.02634, 2015.
  * [33] J. Long, E. Shelhamer, and T. Darrell.  Fully convolutional networks for semantic segmentation.  CoRR, abs/1411.4038, 2014.
  * [34] N. Ma, X. Zhang, H.-T. Zheng, and J. Sun.  Shufflenet v2: Practical guidelines for efficient cnn architecture design.  In The European Conference on Computer Vision (ECCV), September 2018\.
  * [35] R. Mottaghi, X. Chen, X. Liu, N.-G. Cho, S.-W. Lee, S. Fidler, R. Urtasun, and A. Yuille.  The role of context for object detection and semantic segmentation in the wild.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2014.
  * [36] R. Mottaghi, X. Chen, X. Liu, N.-G. Cho, S.-W. Lee, S. Fidler, R. Urtasun, and A. Yuille.  The role of context for object detection and semantic segmentation in the wild.  In Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR ’14, pages 891–898, Washington, DC, USA, 2014\. IEEE Computer Society.
  * [37] H. Noh, S. Hong, and B. Han.  Learning deconvolution network for semantic segmentation.  In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), ICCV ’15, pages 1520–1528, Washington, DC, USA, 2015\. IEEE Computer Society.
  * [38] A. Salvador, M. Bellver, M. Baradad, F. Marqués, J. Torres, and X. Giró i Nieto.  Recurrent neural networks for semantic instance segmentation.  CoRR, abs/1712.00617, 2017.
  * [39] M. Sandler, A. G. Howard, M. Zhu, A. Zhmoginov, and L. Chen.  Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation.  CoRR, abs/1801.04381, 2018.
  * [40] F. Shen, R. Gan, S. Yan, and G. Zeng.  Semantic segmentation via structured patch prediction, context crf and guidance crf.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5178–5186, July 2017.
  * [41] X. Shi, Z. Chen, H. Wang, D. Yeung, W. Wong, and W. Woo.  Convolutional LSTM network: A machine learning approach for precipitation nowcasting.  CoRR, abs/1506.04214, 2015.
  * [42] J. Shotton, J. Winn, C. Rother, and A. Criminisi.  Textonboost for image understanding: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context.  International Journal of Computer Vision, 81(1):2–23, Jan 2009\.
  * [43] C. Szegedy, S. Ioffe, and V. Vanhoucke.  Inception-v4, inception-resnet and the impact of residual connections on learning.  CoRR, abs/1602.07261, 2016.
  * [44] A. Torralba, K. Murphy, and W. T. Freeman.  Contextual models for object detection using boosted random fields.  NIPS, 10 2004.
  * [45] N. Tsumura, N. Ojima, K. Sato, M. Shiraishi, H. Shimizu, H. Nabeshima, S. Akazaki, K. Hori, and Y. Miyake.  Image-based skin color and texture analysis/synthesis by extracting hemoglobin and melanin information in the skin.  ACM Trans. Graph., 22(3):770–779, July 2003.
  * [46] T. F. Vieira, A. Bottino, A. Laurentini, and M. De Simone.  Detecting siblings in image pairs.  The Visual Computer, 30(12):1333–1345, Dec 2014.
  * [47] S. Xie, R. B. Girshick, P. Dollár, Z. Tu, and K. He.  Aggregated residual transformations for deep neural networks.  CoRR, abs/1611.05431, 2016.
  * [48] Z. Yan, H. Zhang, Y. Jia, T. Breuel, and Y. Yu.  Combining the best of convolutional layers and recurrent layers: A hybrid network for semantic segmentation.  CoRR, abs/1603.04871, 2016.
  * [49] L. Yang, S.-H. Lee, S.-G. Kwon, H.-J. Song, and K.-R. Kwon.  Skin pigment recognition using projective hemoglobin- melanin coordinate measurements.  Journal of Electrical Engineering and Technology, 11:1825–1838, 11 2016.
  * [50] M. Yang, K. Yu, C. Zhang, Z. Li, and K. Yang.  Denseaspp for semantic segmentation in street scenes.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [51] H. Zhang, K. J. Dana, J. Shi, Z. Zhang, X. Wang, A. Tyagi, and A. Agrawal.  Context encoding for semantic segmentation.  CoRR, abs/1803.08904, 2018.
  * [52] X. Zhang, X. Zhou, M. Lin, and J. Sun.  Shufflenet: An extremely efficient convolutional neural network for mobile devices.  CoRR, abs/1707.01083, 2017.
  * [53] S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet, Z. Su, D. Du, C. Huang, and P. H. S. Torr.  Conditional random fields as recurrent neural networks.  In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), ICCV ’15, pages 1529–1537, Washington, DC, USA, 2015\. IEEE Computer Society.
  * [54] Y. Zhuang, F. Yang, L. Tao, C. Ma, Z. Zhang, Y. Li, H. Jia, X. Xie, and W. Gao.  Dense relation network: Learning consistent and context-aware representation for semantic image segmentation.  In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 3698–3702, Oct 2018.