# Mid-infrared frequency combs and staggered spectral patterns in $\chi^{(2)}$
microresonators

N. Amiune1, Z. Fan2, V.V. Pankratov2, D.N. Puzyrev2, D.V. Skryabin2,†, K.T.
Zawilski3, P.G. Schunemann3, and I. Breunig1,4,∗ 1Laboratory for Optical
Systems, Department of Microsystems Engineering - IMTEK, University of
Freiburg, Georges-Köhler-Allee 102,
79110 Freiburg, Germany
2Department of Physics, University of Bath, Bath BA2 7AY, England, United
Kingdom
3BAE Systems Inc., MER15-1813, P.O. Box 868, Nashua, New Hampshire
03061-0868,USA
4Fraunhofer Institute for Physical Measurement Techniques IPM, Georges-Köhler-
Allee 301, 79110 Freiburg, Germany
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
(August 27, 2024)

###### Abstract

The potential of frequency comb spectroscopy has aroused great interest in
generating mid-infrared frequency combs in the integrated photonic setting.
However, despite remarkable progress in microresonators and quantum cascade
lasers, the availability of suitable mid-IR comb sources remains scarce. Here,
we present a new approach for the generation of mid-IR microcombs through
cascaded three-wave-mixing. By pumping a CdSiP2 microresonator at 1.55 µm
wavelength with a low power continuous wave laser, we generate $\chi^{(2)}$
frequency combs at 3.1 µm wavelength, with a span of about 30 nm. We observe
ordinary combs states with a line spacing of the free spectral range of the
resonator, and combs where the sideband numbers around the pump and half-
harmonic alternate, forming staggered patterns of spectral lines. Our scheme
for mid-IR microcomb generation is compatible with integrated telecom lasers.
Therefore, it has the potential to be used as a simple and fully integrated
mid-IR comb source, relying on only one single material.

Frequency comb sources in the mid-infrared (mid-IR) wavelength range present
themselves as a promising tool for spectroscopy, with several works already
showing improved features with respect to state-of-the-art spectrometersPicqué
and Hänsch (2019). In particular, dual-comb spectrometry attracts currently
most interest, as it provides fast and accurate measurements without any
moving partsI.Coddington  _et al._ (2016). In sight of these applications,
there is also increasing interest in microresonator and (interband) quantum
cascade laser (QCL) combs to achieve a miniaturized mid-IR frequency comb
source in the fingerprint region, where most molecules exhibit many strong
absorption lines. Such technologies may enable, in the long-term, the
realization of comb-based spectrometers fully on a chip-scaleScalari  _et al._
(2019); Gaeta  _et al._ (2019).

At the present time, however, the availability of compact and broad mid-IR
comb sources is still highly limited despite the progress in these fields.
With microresonators, the direct generation of Kerr combs in the mid-IR is
typically realized by using a bulk optical parametric oscillator as a pump
sourceWang  _et al._ (2013); Griffith  _et al._ (2015); Luke  _et al._ (2015);
Yu  _et al._ (2016, 2018), which is not suitable for a fully integrated
device. Using a QCL as a pump source was also demonstratedSavchenkov  _et al._
(2015) but with comb line spacings too large ($\approx$ 40 THz) for
spectroscopy. Another possibility is converting near-infrared combs into the
mid-infrared via difference frequency generationBao  _et al._ (2020, 2021) or
optical parametric oscillationHerr  _et al._ (2018). Regarding QCL combs,
mode-locking in the mid-IR has been demonstrated with high optical powers and
appear as a promising candidate, but are still limited in bandwidthHugi  _et
al._ (2012); Feng  _et al._ (2018); Bagheri  _et al._ (2018); Sterczewski  _et
al._ (2020, 2021).

Here, we introduce a new approach to generate frequency combs in the mid-
infrared region by means of cascaded three-wave-mixing. By pumping a resonator
with a $\chi^{(2)}$ nonlinearity, phase-matched for optical parametric
oscillation (OPO) at degeneracy, a series of cascaded three-wave-mixing
processes will lead to the generation of combs around the pump frequency and
around half the pump frequencyRicciardi  _et al._ (2020). This scheme was
first realized in bulk mirror cavitiesUlvila  _et al._ (2013, 2014); Mosca
_et al._ (2018) and more recently in bulkAmiune  _et al._ (2021) and chip-
integratedBruch  _et al._ (2021) microresonators, as well as in waveguide
cavitiesWang  _et al._ (2022). In this work, we apply this concept with a
CdSiP2 mm-sized resonator in order to generate a frequency comb at 3.1 µm by
pumping at 1.55 µm wavelength. This way, we generate 30 nm wide frequency
combs, with mW pump powers and different repetition rates of 27.7 GHz and
multiples. This scheme has the potential to be fully integrated, as it only
requires a low power telecom continuous wave laser as a pump source and the
integration of a single material. Furthermore, it has been recently predicted
by us theoretically Puzyrev and Skryabin (07 1) that the non-solitonic
modelocked frequency combs, i.e., Turing pattern combs, in optical parametric
oscillators (OPOs) can be spectrally staggered, see an illustration in the
middle column Fig. 1. Here, we provide the experimental confirmation of the
existence of such combs. Staggering is a unique feature of the OPO combs
related to the fact that the central mode of the half-harmonic field is
allowed to have zero power, which is forbidden in the sister case of the
$\chi^{(2)}$ microcombs due to second-harmonic generation Szabados  _et al._
(2020); Puzyrev  _et al._ (2021). The resulting comb pattern has an
alternating sideband number between the pump and half-harmonic. We also report
here combs with asymmetric staggering, sketched in the right column of Fig. 1,
emerging when the pump couples to a resonance with an absolute odd
longitudinal mode number instead of even.

Figure 1: The left column illustrates an ordinary comb with a spatial period
$2\pi/\nu$, where $\nu$ is an integer, and $\mu$ is the relative mode number.
The middle column shows a symmetric staggered comb with a spatial period
$2\pi/2\nu$. The right column is an example of an asymmetric staggering
corresponding to a spatial period $2\pi/3$. Red and blue colours mark the
signal and pump combs, respectively.

A schematic of our experimental setup is shown in Fig. 2a). We couple light
from a continuous wave 1556 nm diode laser into a CdSiP2 resonator with a
radius $R=560$ µm, and we rely on its birefringence to fulfil the phase
matching conditions. The optic axis is parallel to the symmetry axis of the
resonator, and the pump light is extraordinarily polarized, while the down-
converted signal is ordinarily polarized. For in-coupled pump powers below 100
µW we observe mid-IR light generation ranging from 2.3 µm to 5 µm depending on
the transverse mode selected. By initially setting the temperature to
$87^{\circ}$C we observe OPO operation close the point of degeneracy around
3112 nm, which agrees with our predicted phase-matching temperature
considering fundamental transverse modes for all waves. We then proceed to
tune the OPO output further to the point of degeneracy by reducing the
temperature, and increasing the laser frequency to keep the pump light close
to resonance.

As the laser frequency is scanned across the resonance for a given
temperature, the frequency of the output signal/idler waves can vary by
hundreds of GHz. This occurs as the signal and idler waves oscillate in
different longitudinal modes of the resonator depending on the pump detuning.
The tuning curve in Fig. 2b) displays the measured minimum and maximum
signal/idler wavelengths for different temperatures and in-coupled pump powers
of 350 µW. The curves are fitted with the equation Puzyrev and Skryabin (07
1),

$\mu_{\max,\min}^{2}=-\frac{\varepsilon_{0}}{D_{2s}}\mp\frac{\kappa_{p}}{\kappa_{s}}\frac{\gamma_{s}}{D_{2s}}\sqrt{\frac{WD_{1p}}{2\pi\kappa_{p}}},\leavevmode\nobreak\
\leavevmode\nobreak\ D_{2s}<0,$ (1)

where $\mu$ is the relative mode number counted from the resonator mode
nearest to the pump frequency $\omega_{p}$, and
$\varepsilon_{0}=2\omega_{0s}-\omega_{0p}$ is the frequency mismatch
parameter, see Supplemental Material (SM) for details. $D_{1p}/2\pi=27.06$ GHz
is the pump repetition rate, $\kappa_{p}/2\pi=55$ MHz and $\kappa_{s}/2\pi=64$
MHz are the pump and signal linewidth, $D_{2s}/2\pi=-164$ kHz is the signal
dispersion, $\gamma_{s}/2\pi=2$ GHz/$\sqrt{\text{Watt}}$ is the signal
nonlinear coefficient, and $W=130$ µW is the laser power matching the
experimental data. Temperature dependence is introduced via
$\varepsilon_{0}(T)\approx\alpha(T-T_{0})$, where $\alpha=464$ MHz/°C can be
calculated from the resonant frequencies approximation, see SM. The value for
$T_{0}$, corresponding to $\varepsilon_{0}=0$, is fitted to the experimental
data.

Values for signal and idler wavelengths inside the shaded area Fig. 2b) can be
selected in steps of one FSR by thermally locking the pump laser to the
resonance, as shown for the enclosed region at $T=86.2$ °C. For this
temperature, Fig. 2c) shows the pump transmission (blue is experiment and gray
is numerical modelling). A series of dips corresponds to the excitation of
different consecutive longitudinal modes of the resonator as the laser
frequency is scanned across resonance. Fig. 2d) shows a sequence of the
measured (red circles) and numerically modelled (small black dots) resonator
mode numbers excited by tuning the pump frequency. The experimentally
determined mode numbers are calculated from the measured output wavelengths
with $D_{1p}/2\pi$, and the detuning is aligned to the transmission dips. The
numerical data reveal the existence of the locking intervals for every signal-
idler pair, i.e., $\pm\mu$, and the respective ladder-like sequence of
transitions from one interval of $\omega_{p}$ to the next. The matching of the
experimental and numerical data in Fig. 2 d) should be considered as the
experimental evidence of the theoretically predicted equivalence between the
microresonator OPO tuning and the so-called Eckhaus instabilities, see Ref.
Puzyrev and Skryabin (07 1).

In order to generate the frequency combs, we set the temperature and pump
wavelength to operate in between the tips of the tuning curves from Fig. 2b).
Then, as we slowly reduce the pump frequency across the resonance (increasing
detuning), we observe a transition from comb states to non-degenerate OPO with
the signal and idler frequencies getting further apart. For an in-coupled pump
power of 3 mW, we first observe a dense frequency comb around 3.1 µm, shown in
Fig. 3a), with a width of about 30 nm. At the same time, we observe a weak
comb around 1.55 µm, shown in Fig. 3b), approximately 60 dB below the pump
power. Both the pump and signal comb lines are separated by one FSR and,
therefore, provide an example of the ordinary comb structure, like the one
illustrated in the left column of Fig. 1. Our numerical modelling has
reproduced such combs, Figs. 3e), f), and revealed that they correspond to a
variety of complex wave forms which could be either stationary or weakly
breathing.

By further reducing the laser frequency we observed a different type of comb,
see Figs. 3c),d) and g),h) for the experimental and numerical data,
respectively. Now, the line spacing equals four FSRs for both pump and signal
combs, but the signal sidebands are counted from $\mu=2$ and the pump ones
form $\mu=0$. To the best of our knowledge, this is the first experimental
evidence of the staggered OPO combs after their recent theoretical prediction
Puzyrev and Skryabin (07 1). We shall note, that in this case, we are pumping
in an even absolute mode number of the resonator, which is evident, e.g., from
Fig. 3c) where the spectral center of the practically symmetric comb occurs in
the middle between the two lines. Our modelling shows that this state is a
phase locked one. It belongs to the general category of the two-colour Turing
pattern combs, see Puzyrev  _et al._ (2021) and references therein.

Conversion efficiency of the sidebands from the signal comb (red spectra) back
to the pump (blue spectra) is hampered by the large group velocity (repetition
rate) difference, $(D_{1p}-D_{1s})/2\pi=0.51$GHz, which strongly dominates
over dispersion and, therefore, makes all the three wave mixing processes,
excluding the $\mu=0$ (pump) to $\mu=\pm 4$ (signal), to be significantly
mismatched Puzyrev and Skryabin (07 1); Puzyrev  _et al._ (2021). However, the
reason for the lower sideband power observed experimentally compared to the
simulations still requires further studies.

The near-IR (blue coloured) and mid-IR (red coloured) combs share the same
repetition rate, which is a multiple of $27.7\pm 0.1$ GHz. This value lies
closer to the linear FSR at 3.1 µm ($D_{1s}/2\pi=27.57$ GHz), than to the pump
FSR ($D_{1p}/2\pi=27.06$ GHz). The selection of the nonlinear repetition rate
reflects on the fact that the relative sideband power in the mid-IR comb
strongly dominates over the near-IR one. Fig. 3i) shows the full extent of the
values of the pump detunings that we have explored numerically. Within this
and other sets of data and before the signal-idler regime kicks in, see
detunings greater than $230$ MHz Fig. 3i), we have also observed staggered
Turing pattern combs with different line spacings (two, six, etc. FSRs).
However, experimentally, we only captured the one with four. One possible
reason for this selectivity could be an avoided crossing with another mode
family, which preferentially triggers a certain pair of sidebands and is not
accounted for in our theory.

We have also investigated the case of pumping the resonator mode with an odd
number, $2M+1$, by shifting the pump frequency by 1 FSR, see SM. In this case,
the degenerate conversion is not possible, as well as the described above
staggering of the two combs, since $(2M+1)/2$ is not integer. However, the
spectral transformations induced by the pump detuning scan looked similar to
the even mode number case, cf. Fig. 4 and Fig. 3. One difference was that
instead of the perfect staggering and the even order Turing patterns we have
now seen the Turing combs with an odd number of FSRs between the lines
corresponding to the asymmetric staggering, see Figs. 4 c),d) and g),h), and
compare the illustrations in the middle and right columns in Fig. 1.

Overall, this work demonstrates a new type of the microresonator-based source
for mid-IR frequency combs and the existence of the staggered two-colour
frequency combs. The use of a telecom wavelength laser as a pump source might
enable the realization of the setup fully integrated on a chip, with a simple
single material architecture. The comb center wavelength, defined by the
degenerate OPO phase-matching conditions, can be widely tuned by changing the
resonator size. It is possible, for example, to generate a comb up to 4.3 µm
with a resonator size of $R=0.25$ mm. Furthermore, theory and modelling
suggest that a variety of the two-colour parametric solitons in
microresonators can be expected in future experimental studiesVillois  _et
al._ (2019); Smirnov  _et al._ (2021); Skryabin (2021), in addition to the
already observed onesBruch  _et al._ (2021). One property of our resonator
that could facilitate the soliton formation is that, the point of zero walk-
off, i.e., $D_{1p}=D_{1s}$, is achieved at 2.16 µm and 4.32 µm wavelengths for
the pump and half-harmonic respectively. Furthermore, dispersion is also
reduced for these wavelengths, and the zero dispersion point is found around 5
µm. Precise control of these parameters will unlock the full potential of
microcomb generation based on the $\chi^{(2)}$ nonlinearity. Advances in this
and other directions will be facilitated by the predictive capabilities of the
coupled-mode theory for $\chi^{(2)}$ microresonators, see, e.g., Skryabin
(2020) and references therein.

Figure 2: Experimental setup and OPO tuning to degeneracy. a) Sketch of the
experimental setup. OSA: Optical spectrum analyzer, o.a.: optic axis. b) OPO
tuning curve towards degeneracy, where the output wavelengths depend on the
relative pump detuning from resonance. The points correspond to the
experimentally measured minimum and maximum wavelengths for a fixed
temperature, while the curves are calculated. Fig. c) shows the pump
transmission and d) the excited resonator mode as a function of pump detuning
for the enclosed region in Fig. a). The blue line and red dots correspond to
experimental measurements, while the gray lines and dots correspond to the
numerical simulation. Figure 3: Ordinary and symmetrically staggered frequency
combs for the pump coupled to the even mode number. Frequency comb spectra
around the pump (blue) and sub-harmonic signal (red) frequencies. In the
experiment, the comb state changes from the one FSR comb line spacing in a,b)
to the four FSR spacing in c,d) as the pump frequency is reduced across the
resonance. e-i) Numerically simulated spectra for different pump detunings.
e,f) and g,h) show the spectra for $-216$ and $-144$ MHz detunings,
respectively. The background in i) is at -70 dB and $\varepsilon_{0}=0$. c,d)
and g,h) show the staggered combs. Figure 4: Ordinary and asymmetrically
staggered frequency combs for the pump coupled to the odd mode number.
Frequency comb spectra around the pump (blue) and sub-harmonic signal (red)
frequencies. In the experiment, the comb state changes from the one FSR comb
line spacing in a,b) to the three FSR spacing in c,d) as the pump frequency is
reduced across the resonance. e-i) Numerically simulated spectra for different
pump detunings. e-f) and g-h) show the spectra for the $-50$ and $-44$ MHz
detunings, respectively. The background in i) is at -70 dB and
$\varepsilon_{0}=-0.75$ GHz. c,d) and g,h) show the asymmetric staggering.

## Supplementary information

### Setup

The CdSiP2 resonator, shown in Fig. 2 in the main text, was manufactured with
a major radius $R=560$ µm. The raw material was grown and provided by BAE
Systems and the manufacturing process is detailed in Ref. Amiune  _et al._
(2021). We determined a quality factor $Q_{\text{p}}=6\times 10^{6}$ and a
free spectral range (FSR) of 27.0 GHz for 1550 nm wavelength and extraordinary
polarization by scanning the pump laser over 30 GHz.

The experimental setup is sketched in Fig. 2 in the main text. In order to
achieve optical parametric oscillation, we rely on birrefringent phase-
matching. We pump the resonator with a continuous wave 1550 nm laser (Toptica
DL Pro) with extraordinary polarization and couple light in with a silicon
prism by frustrated total internal reflection. The near-IR and mid-IR beams
are separated with a dielectric mirror. Ordinarily polarized generated light
in the mid-infrared is detected with an optical spectrum analyzer (Yokogawa
AQ6376). The OPO output wavelengths are controlled by tuning the temperature
of the resonator in order to achieve operation at degeneracy. The frequency
comb spectra are recorded simultaneously with two optical spectrum analyzers
(Yokogawa AQ6370D and AQ6376).

### Model

We follow here the model, which detailed derivation from Maxwell equations can
be found in Skryabin (2020), and which has been previously studied in, e.g.,
Amiune  _et al._ (2021); Puzyrev and Skryabin (07 1). We assume that
$\omega_{p}$ is the pump laser frequency and $\omega_{0p}$ is the frequency of
the resonator mode with the number $2M$. $2M$ equals the number of wavelengths
fitting along the ring circumference. We express the multimode intra-resonator
electric fields of the pump, $B$, and half-harmonic signal, $A$, as

$\begin{split}&e^{iM\vartheta-i\frac{1}{2}\omega_{p}t}A(\vartheta,t)+c.c.=e^{iM\vartheta-i\frac{1}{2}\omega_{p}t}\sum_{\mu}a_{\mu}(t)e^{i\mu\vartheta}+c.c.,\\\
&e^{i2M\vartheta-i\omega_{p}t}B(\vartheta,t)+c.c.=e^{i2M\vartheta-i\omega_{p}t}\sum_{\mu}b_{\mu}(t)e^{i\mu\vartheta}+c.c..\end{split}$
(2)

Here, $\vartheta=(0,2\pi]$ is the angular coordinate and $\mu=0,\pm 1,\pm
2,\dots$ is the relative mode number.

$\omega_{s\mu}$ and $\omega_{p\mu}$ are the respective signal and pump spectra
of the resonator frequencies,

$\omega_{\mu s}=\omega_{0s}+\mu
D_{1s}+\tfrac{1}{2}\mu^{2}D_{2s},\leavevmode\nobreak\ \leavevmode\nobreak\
\omega_{\mu p}=\omega_{0p}+\mu D_{1p}+\tfrac{1}{2}\mu^{2}D_{2p},$ (3)

where, $D_{1s,1p}/2\pi$ are the free spectral ranges, FSRs, and $D_{2s,2p}$
are dispersions.

Coupled-mode equations governing the evolution of $a_{\mu}(t)$, $b_{\mu}(t)$
are Skryabin (2020)

$\displaystyle i\text{P}_{t}a_{\mu}=$ $\displaystyle(\omega_{\mu
s}-\tfrac{1}{2}\omega_{p})a_{\mu}-\frac{i\kappa_{s}}{2}a_{\mu}-\gamma_{s}\sum_{\mu_{1}\mu_{2}}\widehat{\delta}_{\mu,\mu_{1}-\mu_{2}}b_{\mu_{1}}a^{*}_{\mu_{2}},$
(4) $\displaystyle i\text{P}_{t}b_{\mu}=$ $\displaystyle(\omega_{\mu
p}-\omega_{p})b_{\mu}-\frac{i\kappa_{b}}{2}\big{(}b_{\mu}-\widehat{\delta}_{\mu,\mu^{\prime}}{\cal
H}_{\mu^{\prime}}\big{)}$
$\displaystyle-\gamma_{p}\sum_{\mu_{1}\mu_{2}}\widehat{\delta}_{\mu,\mu_{1}+\mu_{2}}a_{\mu_{1}}a_{\mu_{2}},$

where $\widehat{\delta}_{\mu,\mu^{\prime}}=1$ for $\mu=\mu^{\prime}$ and is
zero otherwise. ${\cal H}_{\mu^{\prime}}$ is the pump parameter, ${\cal
H}^{2}_{\mu^{\prime}}={\cal F}_{p}W/2\pi$, where $W$ is the laser power, and
${\cal F}_{p}=D_{1p}/\kappa_{p}$ is the pump finesse Skryabin (2020). Pumping
to the even mode number, $2M$, corresponds to $\mu^{\prime}=0$ and pumping to
the odd mode, $2M+1$, corresponds to $\mu^{\prime}=1$.

The frequency matching, i.e. phase matching, parameter for the non-degenerate
parametric process initiated by, e.g., the $\mu=0$ mode in the pump field is
defined as

$\varepsilon_{\mu}=\omega_{\mu s}+\omega_{-\mu
s}-\omega_{0p}=\frac{c}{R}\left[\frac{M+\mu}{n_{M+\mu}}+\frac{M-\mu}{n_{M-\mu}}-\frac{2M}{n_{2M}}\right].$
(5)

Here, $n_{m}$ is the effective refractive index taken for the frequencies of
the modes with the absolute numbers $m=M\pm\mu$ (signal and idler) and $2M$
(pump), $c$ is the vacuum speed of light and $R$ is the resonator radius.
Arranging for

$\varepsilon_{0}=2\omega_{0s}-\omega_{0p}=\frac{c}{R}\left[\frac{2M}{n_{M}}-\frac{2M}{n_{2M}}\right]=0,\text{\leavevmode\nobreak\
i.e.,\leavevmode\nobreak\ }n_{M}=n_{2M},$ (6)

corresponds to the exact matching for the degenerate parametric conversion. We
shall note, that Eq. (1) in the main text has been derived in Puzyrev and
Skryabin (07 1) for the pumping to the even mode only, $\mu^{\prime}=0$.

Fourier transforming the coupled mode equations, Eq. (4), to the physical
space one arrives to the following set of partial-differential equations
Skryabin (2020),

$\displaystyle i\text{P}_{t}A=$
$\displaystyle\left(\omega_{0s}-\tfrac{1}{2}\omega_{p}-iD_{1s}\text{P}_{\vartheta}-\frac{1}{2}D_{2s}\text{P}_{\vartheta}^{2}\right)A$
(7) $\displaystyle-\frac{i\kappa_{s}}{2}A-\gamma_{s}BA^{*}\leavevmode\nobreak\
,$ $\displaystyle i\text{P}_{t}B=$
$\displaystyle\left(\omega_{0p}-\omega_{p}-iD_{1p}\text{P}_{\vartheta}-\frac{1}{2}D_{2p}\text{P}_{\vartheta}^{2}\right)B$
$\displaystyle-\frac{i\kappa_{p}}{2}\left(B-{\cal
H}_{\mu^{\prime}}e^{i\mu^{\prime}\vartheta}\right)-\gamma_{p}A^{2}.$

The calculation of the resonator’s FSR, dispersion and phase-matching
temperatures are carried out by approximating the resonance frequencies as in
Gorodetsky and Fomin (2006), including the Sellmeier equationWei  _et al._
(2018) and thermal expansion coefficientsZawilski  _et al._ (2010) for CdSiP2
and assuming the fundamental transverse mode numbers for the pump and signal
waves. The transmittance in Fig. 1d is calculated as the power dissipated over
the roundtrip,

$T=1-\frac{2\pi\eta}{W}\sum_{\mu}\left(\frac{\left|b_{\mu}\right|^{2}}{\mathcal{F}_{p}}+\frac{\left|a_{\mu}\right|^{2}}{\mathcal{F}_{s}}\right),$
(8)

where $\eta=0.3$ is the fitting parameter.

Table 1:  Resonator parameters. |  Pump (1556 nm) |  Signal (3112 nm)
---|---|---
Repetition rate |  $D_{1p}/2\pi=$27.06 GHz |  $D_{1s}/2\pi=$27.57 GHz
Dispersion |  $D_{2p}/2\pi=-$378 kHz |  $D_{2s}/2\pi=-$164 kHz
Linewidth |  $\kappa_{p}/2\pi=$55 MHz |  $\kappa_{s}/2\pi=$64 MHz
Nonlinearity |  $\gamma_{p}/2\pi=$1 GHz $\text{W}^{-1/2}$ |  $\gamma_{s}/2\pi=$2 GHz $\text{W}^{-1/2}$

## References

  * Picqué and Hänsch (2019) N. Picqué and T. Hänsch, Nature Photon. 13, 146 (2019).
  * I.Coddington  _et al._ (2016) I.Coddington, N. Newbury, and W. Swann, Optica 3, 414 (2016).
  * Scalari  _et al._ (2019) G. Scalari, J. Faist, and N. Picqué, Appl. Phys. Lett. 114, 150401 (2019).
  * Gaeta  _et al._ (2019) A. L. Gaeta, M. Lipson, and T. J. Kippenberg, Nat. Photonics 13, 158 (2019).
  * Wang  _et al._ (2013) C. Y. Wang, T. Herr, P. Del’Haye, A. Schliesser, J. Hofer, R. Holzwarth, T. W. Hänsch, N. Picqué, and T. J. Kippenberg, Nat Commun 4, 1345 (2013).
  * Griffith  _et al._ (2015) A. G. Griffith, R. K. Lau, J. Cardenas, Y. Okawachi, A. Mohanty, R. Fain, Y. H. D. Lee, M. Yu, C. T. Phare, C. B. Poitras, A. L. Gaeta, and M. Lipson, Nat Commun 6, 6299 (2015).
  * Luke  _et al._ (2015) K. Luke, Y. Okawachi, M. R. E. Lamont, A. L. Gaeta, and M. Lipson, Opt. Lett. 40, 4823 (2015).
  * Yu  _et al._ (2016) M. Yu, Y. Okawachi, A. G. Griffith, M. Lipson, and A. L. Gaeta, Optica 3, 854 (2016).
  * Yu  _et al._ (2018) M. Yu, Y. Okawachi, A. G. Griffith, N. Picqué, M. Lipson, and A. L. Gaeta, Nat Commun. 9, 1869 (2018).
  * Savchenkov  _et al._ (2015) A. A. Savchenkov, V. S. Ilchenko, F. Teodoro, P. M. Belden, W. T. Lotshaw, A. B. Matsko, and L. Maleki, Opt. Lett. 40, 3460 (2015).
  * Bao  _et al._ (2020) C. Bao, Z. Yuan, H. Wang, L. Wu, B. Shen, K. Sung, S. Leifer, Q. Lin, and K. Vahala, Optica 7, 309 (2020).
  * Bao  _et al._ (2021) C. Bao, Z. Yuan, L. Wu, M. G. Suh, H. Wang, Q. Lin, and K. J. Vahala, Nat Commun. 12, 6573 (2021).
  * Herr  _et al._ (2018) S. J. Herr, V. Brasch, J. Szabados, E. Obrzud, Y. Jia, S. Lecomte, K. Buse, I. Breunig, and T. Herr, Opt. Lett. 43, 5745 (2018).
  * Hugi  _et al._ (2012) A. Hugi, G. Villares, S. Blaser, H. C. Liu, and J. Faist, Nature 492, 229–233 (2012).
  * Feng  _et al._ (2018) T. Feng, T. H. L. Shterengas and, A. Belyanin, and G. Kipshidze, ACS Photonics 12, 4978 (2018).
  * Bagheri  _et al._ (2018) M. Bagheri, C. Frez, L. A. Sterczewski, I. Gruidin, M. Fradet, I. Vurgaftman, C. L. Canedy, W. W. Bewley, C. D. Merritt, C. S. Kim, M. Kim, and J. R. Meyer, Sci Rep 8, 3322 (2018).
  * Sterczewski  _et al._ (2020) L. A. Sterczewski, M. Bagheri, C. Frez, C. L. Canedy, I. Vurgaftman, and J. R. Meyer, Appl. Phys. Lett. 116, 141102 (2020).
  * Sterczewski  _et al._ (2021) L. A. Sterczewski, M. Bagheri, C. Frez, C. L. Canedy, I. Vurgaftman, M. Kim, C. S. Kim, C. D. Merritt, W. W. Bewley, and J. R. Meyer, J. Phys. Photonics 3, 042003 (2021).
  * Ricciardi  _et al._ (2020) I. Ricciardi, S. Mosca, M. Parisi, F. Leo, T. Hansson, M. Erkintalo, P. Maddaloni, P. D. Natale, S. Wabnitz, and M. D. Rosa, Micromachines 11, 230 (2020).
  * Ulvila  _et al._ (2013) V. Ulvila, C. R. Phillips, L. Halonen, and M. Vainio, Opt. Lett. 38, 4281 (2013).
  * Ulvila  _et al._ (2014) V. Ulvila, C. R. Phillips, L. Halonen, and M. Vainio, Opt. Express 22, 10535 (2014).
  * Mosca  _et al._ (2018) S. Mosca, M. Parisi, I. Ricciardi, F. Leo, T. Hansson, M. Erkintalo, P. Maddaloni, P. De Natale, S. Wabnitz, and M. De Rosa, Phys. Rev. Lett. 121, 093903 (2018).
  * Amiune  _et al._ (2021) N. Amiune, D. N. Puzyrev, V. V. Pankratov, D. V. Skryabin, K. Buse, and I. Breunig, Opt. Express 29, 41378 (2021).
  * Bruch  _et al._ (2021) A. W. Bruch, X. Liu, Z. Gong, J. B. Surya, M. Li, C. Zou, and H. X. Tang, Nat. Photonics 15, 21 (2021).
  * Wang  _et al._ (2022) X. Wang, K. Jia, M. Chen, S. Cheng, X. Ni, J. Guo, Y. Li, H. Liu, L. Hao, J. Ning, G. Zhao, X. Lv, S. Huang, Z. Xie, and S. Zhu, Photon. Res. 10, 509 (2022).
  * Puzyrev and Skryabin (07 1) D. N. Puzyrev and D. V. Skryabin, Communications Physics (in press, 2022, doi: 10.1038/s42005-022-00907-1), 10.1038/s42005-022-00907-1.
  * Szabados  _et al._ (2020) J. Szabados, D. N. Puzyrev, Y. Minet, L. Reis, K. Buse, A. Villois, D. V. Skryabin, and I. Breunig, Phys. Rev. Lett. 124, 203902 (2020).
  * Puzyrev  _et al._ (2021) D. N. Puzyrev, V. V. Pankratov, A. Villois, and D. V. Skryabin, Phys. Rev. A 104, 013520 (2021).
  * Villois  _et al._ (2019) A. Villois, N. Kondratiev, I. Breunig, D. N. Puzyrev, and D. V. Skryabin, Opt. Lett. 44, 4443 (2019).
  * Smirnov  _et al._ (2021) S. Smirnov, V. Andryushkov, E. Podivilov, B. Sturman, and I. Breunig, Opt. Express 29, 27434 (2021).
  * Skryabin (2021) D. V. Skryabin, Opt. Express 29, 28521 (2021).
  * Skryabin (2020) D. V. Skryabin, J. Opt. Soc. Am. B 37, 2604 (2020).
  * Gorodetsky and Fomin (2006) M. L. Gorodetsky and A. E. Fomin, IEEE J. Sel. Top. Quantum Electron. 12, 33 (2006).
  * Wei  _et al._ (2018) J. Wei, J. M. Murray, F. K. Hopkins, D. M. Krein, K. T. Zawilski, P. G. Schunemann, and S. Guha, Opt. Mater. Express 8, 235 (2018).
  * Zawilski  _et al._ (2010) K. T. Zawilski, P. G. Schunemann, T. C. Pollak, D. E. Zelmon, N. C. Fernelius, and F. Kenneth Hopkins, Journal of Crystal Growth 312, 1127 (2010).


# Cross-Lingual Named Entity Recognition Using Parallel Corpus: A New Approach
Using XLM-RoBERTa Alignment

Bing Li
Microsoft
<EMAIL_ADDRESS>
&Yujie He
Microsoft
<EMAIL_ADDRESS>
&Wenjin Xu
Microsoft
<EMAIL_ADDRESS>

###### Abstract

We propose a novel approach for cross-lingual Named Entity Recognition (NER)
zero-shot transfer using parallel corpora. We built an entity alignment model
on top of XLM-RoBERTa to project the _entities_ detected on the English part
of the parallel data to the target language sentences, whose accuracy
surpasses all previous unsupervised models. With the alignment model we can
get pseudo-labeled NER data set in the target language to train task-specific
model. Unlike using translation methods, this approach benefits from natural
fluency and nuances in target-language original corpus. We also propose a
modified loss function similar to focal loss but assigns weights in the
opposite direction to further improve the model training on noisy pseudo-
labeled data set. We evaluated this proposed approach over 4 target languages
on benchmark data sets and got competitive F1 scores compared to most recent
SOTA models. We also gave extra discussions about the impact of parallel
corpus size and domain on the final transfer performance.

##  1 Introduction

Named entity recognition (NER) is a fundamental task in natural language
processing, which seeks to classify words in a sentence to predefined semantic
types. Due to its nature that the ground truth label exists at word level,
supervised training of NER models often requires large amount of human
annotation efforts. In real-world use cases where one needs to build multi-
lingual models, the required human labor scales at least linearly with number
of languages, or even worse for low resource languages. Cross-Lingual transfer
on Natural Language Processing(NLP) tasks has been widely studied in recent
years Conneau et al. (2018); Kim et al. (2017); Ni et al. (2017); Xie et al.
(2018); Ni and Florian (2019); Wu and Dredze (2019); Bari et al. (2019); Jain
et al. (2019), in particular zero-shot transfer which leverages the advances
in high resource language such as English to benefit other low resource
languages. In this paper, we focus on the cross-lingual transfer of NER task,
and more specifically using parallel corpus and pretrained multilingual
language models such as mBERT Devlin (2018) and XLM-RoBERTa (XLM-R) Lample and
Conneau (2019); Conneau et al. (2020).

Our motivations are threefold. (1) Parallel corpus is a great resource for
transfer learning and is rich between many language pairs. Some recent
research focus on using completely unsupervised machine translations (e.g.
word alignment Conneau et al. (2017)) for cross-lingual NER, however
inaccurate translations could harm the transfer performance. For example in
the word-to-word translation approach, word ordering may not be well
represented during translations, such gaps in translation quality may harm
model performance in down stream tasks. (2) A method could still provide
business value-add even if it only works for some major languages that have
sufficient parallel corpus as long as it has satisfying performance. It is a
common issue in industry practices where there is a heavily customized task
that need to be extended into major markets but you do not want to annotate
large amounts of data in other languages. (3) Previous attempts using parallel
corpus are mostly heuristics and statistical-model based Jain et al. (2019);
Xie et al. (2018); Ni et al. (2017). Recent breakthroughs in multilingual
language models have not been applied to such scenarios yet. Our work bridges
the gap and revisits this topic with new technologies.

We propose a novel semi-supervised method for the cross-lingual NER transfer,
bridged by parallel corpus. First we train an NER model on source-language
data set - in this case English - assuming that we have labeled task-specific
data. Second we label the English part of the parallel corpus with this model.
Then, we project those recognized entities onto the target language, i.e.
label the span of the same entity in target-language portion of the parallel
corpus. In this step we will leverage the most recent XLM-R model Lample and
Conneau (2019); Conneau et al. (2020), which makes a major distinction between
our work and previous attempts. Lastly, we use this pseudo-labeled data to
train the task-specific model in target language directly. For the last step
we explored the option of continue training from a multilingual model fine-
tuned on English NER data to maximize the benefits of model transfer. We also
tried a series of methods to mitigate the noisy label issue in this semi-
supervised approach.

The main contributions of this paper are as follows:

  * •

We leverage the powerful multilingual model XLM-R for entity alignment. It was
trained in a supervised manner with easy-to-collect data, which is in sharp
contrast to previous attempts that mainly rely on unsupervised methods and
human engineered features.

  * •

Pseudo-labeled data set typically contains lots of noise, we propose a novel
loss function inspired by the focal loss Lin et al. (2017). Instead of using
native focal loss, we went the opposite direction by weighting hard examples
less as those are more likely to be noise.

  * •

By leveraging existing natural parallel corpus we got competitive F1 scores of
NER transfer on multiple languages. We also tested that the domain of parallel
corpus is critical in an effective transfer.

##  2 Related Works

There are different ways to conduct zero-shot multilingual transfer. In
general, there are two categories, model-based transfer and data-based
transfer. Model-based transfer often use source language to train an NER model
with language independent features, then directly apply the model to target
language for inference Wu and Dredze (2019); Wu et al. (2020). Data-based
transfer focus on combining source language task specific model, translations,
and entity projection to create weakly-supervised training data in target
language. Some previous attempts includes using annotation projection on
aligned parallel corpora, translations between a source and a target langauge
Ni et al. (2017); Ehrmann et al. (2011), or to utilize Wikipedia hyperlink
structure to obtain anchor text and context as weak labels Al-Rfou et al.
(2015); Tsai et al. (2016). Different variants exist in annotation projection,
e.g. Ni et al. Ni et al. (2017) used maximum entropy alignment model and data
selection to project English annotated labels to parallel target language
sentences. Some other work used bilingual mapping combined with lexical
heuristics or used embedding approach to perform word-level translation with
which naturally comes the annotation projection Mayhew et al. (2017); Xie et
al. (2018); Jain et al. (2019). This kind of translation + projection approach
is used not just in NER, but in other NLP tasks as well such as relation
extraction Kumar (2015). There are obvious limitations to the translation +
projection approach, word or phrase-based translation makes annotation
projection easier, but sacrifices the native fluency and language nuances. In
addition, orthographic and phonetic based features for entity matching may
only be applicable to languages that are alike, and requires extensive human
engineered features. To address these limitations, we proposed a novel
approach which utilizes machine translation training data and combined with
pretrained multilingual language model for entity alignment and projection.

##  3 Model Design

We will demonstrate the entity alignment model component and the full training
pipeline of our work in following sections.

###  3.1 Entity Alignment Model

Translation from a source language to target language may break the word
ordering therefore an alignment model is needed to project entities from
source language sentence to target language, so that the labels from source
language can be zero-shot transferred. In this work, we use XLM-R Lample and
Conneau (2019); Conneau et al. (2020) series of models which introduced the
translation language model(TLM) pretraining task for the first time. TLM
trains the model to predict a masked word using information from both the
context and the parallel sentence in another language, making the model
acquire great cross-lingual and potentially alignment capability.

Our alignment model is constructed by concatenating the English name of the
entity and the target language sentence, as segment A and segment B of the
input sequence respectively. For token level outputs of segment B, we predict
1 if it is inside the translated entity, 0 otherwise. This formulation
transforms the entity alignment problem into a token classification task. An
implicit assumption is that the entity name will still be a consecutive phrase
after translation. The model structure is illustrated in Fig 1.

Figure 1: Entity alignment model. The query entity on the left is ’Cologne’,
and the German sentence on the right is ’Köln liegt in Deutschlands’, which is
’Cologne is located in Germany’ in English, and ’Köln’ is the German
translation of ’Cologne’. The model predicts the word span aligned with the
query entity.

###  3.2 Cross-Lingual Transfer Pipeline

Fig 2 shows the whole training/evaluation pipeline which includes 5 stages:
(1) Fine-tune pretrained language model on CoNLL2003 Sang and Meulder (2003)
to obtain English NER model; (2) Infer labels for English sentences in the
parallel corpus; (3) Run the entity alignment model from the previous
subsection and find corresponding detected English entities in the target
language, failed examples are filtered out during the alignment process; (4)
Fine-tune the multilingual model with data generated from step (3); (5)
Evaluate the new model on the target language test sets.

Figure 2: Training Pipeline Diagram. Yellow pages represent English documents
while light blue pages represent German documents. Step 1 and 5 used original
CoNLL data for train and test respectively; step 2, 3, and 4 used machine
translation data from OPUS website. Pretrained model is either mBert or XLM-R.
The final model was first fine-tuned on English NER data set then fine-tuned
on target language pseudo-labeled NER data set.

##  4 Experiments And Discussions

###  4.1 Parallel Corpus

In our method, we leveraged the availability of large-scale parallel corpus to
transfer the NER knowledge obtained in English to other languages. Existing
parallel corpora is easier to obtain than annotated NER data. We used parallel
corpus crawled from the OPUS website Tiedemann (2012). In our experiments, we
used the following data sets:

  * •

Ted2013: consists of volunteer transcriptions and translations from the TED
web site and was created as training data resource for the International
Workshop on Spoken Language Translation 2013.

  * •

OpenSubtitles: a new collection of translated movie subtitles that contains 62
languages. Lison and Tiedemann (2016)

  * •

WikiMatrix: Mined parallel sentences from Wikipedia in different languages.
Only pairs with scores above 1.05 are used. Schwenk et al. (2019)

  * •

UNPC: Manually translated United Nations documents from 1994 to 2014. Ziemski
et al. (2016)

  * •

Europarl: A parallel corpus extracted from the European Parliament web site.
Koehn (2005)

  * •

WMT-News: A parallel corpus of News Test Sets provided by WMT for training SMT
that contains 18 languages.111http://www.statmt.org/wmt19/translation-
task.html

  * •

NewsCommentary: A parallel corpus of News Commentaries provided by WMT for
training SMT, which contains 12 languages.222http://opus.nlpl.eu/News-
Commentary.php

  * •

JW300: Mined, parallel sentences from the magazines Awake! and Watchtower.
Agić and Vulić (2019)

In this work, we focus on 4 languages, German, Spanish, Dutch and Chinese. We
randomly select data points from all data sets above with equal weights. There
might be slight difference in data distribution between languages due to data
availability and relevance.

###  4.2 Alignment Model Training

The objective of alignment model is to find the entity from a foreign
paragraph given its English name. We feed the English name and the paragraph
as segment A and B into the XLM-R model Lample and Conneau (2019); Conneau et
al. (2020). Unlike NER task, the alignment task has no requirement for label
completeness, since we only need one entity to be labeled in one training
example. The training data set can be created from Wikipedia documents where
anchor text in hyperlinks naturally indicate the location of entities and one
can get the English entity name via linking by Wikipedia entity Id. An
alternative to get the English name for mentions in another language is
through state-of-the-art translation system. We took the latter approach to
make it simple and leveraged Microsoft’s Azure Cognitive Service to do the
translation.

During training, we also added negative examples with faked English entities
which did not appear in the other language’s sentence. The intuition is to
force model to focus on English entity (segment A) and its translation instead
of doing pure NER and picking out any entity in other language’s sentence
(segment B). We also added examples of noun phrases or nominal entities to
make the model more robust.

We generated a training set of 30K samples with 25$\%$ negatives for each
language and trained a XLM-R-large model Conneau et al. (2020) for 3 epochs
with batch size 64. The initial learning rate is 5e-5 and other
hyperparameters were defaults from HuggingFace Transformers library for the
token classification task. The precision/recall/F1 on the reserved test set
reached 98$\%$. The model training was done on 2 Tesla V100 and took about 20
minutes.

###  4.3 Cross-Lingual Transfer

|  | DE |  |  |  | ES |  |  |  | NL |  |  |  | ZH |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Model | P | R | F1 |  | P | R | F1 |  | P | R | F1 |  | P | R | F1
Bari et al. (2019) | - | - | 65.24 |  | - | - | 75.9 |  | - | - | 74.6 |  | - | - | -
Wu and Dredze (2019) | - | - | 71.1 |  | - | - | 74.5 |  | - | - | 79.5 |  | - | - | -
Moon et al. (2019) | - | - | 71.42 |  | - | - | 75.67 |  | - | - | 80.38 |  | - | - | -
Wu et al. (2020) | - | - | 73.16 |  | - | - | 76.75 |  | - | - | 80.44 |  | - | - | -
Wu et al. (2020) | - | - | 73.22 |  | - | - | 76.94 |  | - | - | 80.89 |  | - | - | -
Wu et al. (2020) | - | - | 74.82 |  | - | - | 79.31 |  | - | - | 82.90 |  | - | - | -
Our Models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
mBERT zero-transfer | 67.6 | 77.4 | 72.1 |  | 72.4 | 78.2 | 75.2 |  | 77.8 | 79.3 | 78.6 |  | 64.1 | 65.0 | 64.6
mBERT fine-tune | 73.1 | 76.2 | 74.6 |  | 77.7 | 77.6 | 77.6 |  | 80.5 | 76.7 | 78.6 |  | 80.8 | 63.3 | 71.0
|  |  | (+2.5) |  |  |  | (+2.4) |  |  |  | (+0.0) |  |  |  | (+6.4)
XLM-R zero-transfer | 67.9 | 79.8 | 73.4 |  | 79.8 | 81.9 | 80.8 |  | 82.2 | 80.3 | 81.2 |  | 68.7 | 65.5 | 67.1
XLM-R fine-tune | 75.5 | 78.4 | 76.9 |  | 76.9 | 81.0 | 78.9 |  | 81.2 | 78.4 | 79.7 |  | 77.8 | 65.9 | 71.3
|  |  | (+3.5) |  |  |  | (-1.9) |  |  |  | (-1.5) |  |  |  | (+4.2)
Table 1: Cross-Lingual Transfer Results on German, Spanish, Dutch and Chinese:
Experiments are done with both mBERT and XLM-RoBERTa model. For each of them
we compared zero-transfer result (trained on CoNLL2003 English only) and fine-
tune result using zero-transfer pseudo-labeled target language NER data. Test
sets for German, Spanish, Dutch are from CoNLL2003 and CoNLL2002, and People’s
Daily data set for Chinese.

We used the CoNLL2003 Sang and Meulder (2003) and CoNLL2002333http://lcg-
www.uia.ac.be/conll2002/ner/ data sets to test our cross-lingual transfer
method for German, Spanish and Dutch. We ignored the training sets in those
languages and only evaluated our model on test sets. For Chinese, we used
People’s Daily444https://github.com/zjy-ucas/ChineseNER as the major
evaluation set, and we also reported numbers on MSRA Levow (2020) and Weibo
Peng and Dredze (2015) data sets in the next section. One notable difference
for People’s Daily data set is that it only covers three entity types, LOC,
ORG and PER, so we suppressed the MISC type from English during the transfer
by training English NER model with ’MISC’ marked as ’O’.

To enable cross-lingual transfer, we first trained an English teacher model
using the CoNLL2003 EN training set with XLM-R-large as the base model. We
trained with focal loss Lin et al. (2017) for 5 epochs. We then ran inference
with this model on the English part of the parallel data. Finally, with the
alignment model, we projected entity labels onto other languages. To ensure
the quality of target language training data, we discarded examples if any
English entity failed to map to tokens in the target language. We also
discarded examples where there are overlapping target entities because it will
cause conflicts in token labels. Furthermore, when one entity is mapped to
multiple, we only keep the example if all the target mention phrases are the
same. This is to accommodate the situation where same entity is mentioned more
than once in one sentence.

As the last step, we fine-tuned the multilingual model pre-trained on English
data set with lower n(0, 3, 6, etc.) layers frozen, on the target language
pseudo-labeled data. We used both mBERT Devlin et al. (2019); Devlin (2018)
and XLM-RConneau et al. (2020) with about 40K training samples for 1 epoch.
The results are shown in Table 1. All the inference, entity projection and
model training experiments are done on 2 Tesla V100 32G gpus and the whole
pipeline takes about 1-2 hours. All numbers are reported as an average of 5
random runs with the same settings.

For loss function we used something similar to focal loss Lin et al. (2017)
but with opposite weight assignment. The focal loss was designed to weight
hard examples more. This intuition holds true only when training data is
clean. In some scenarios such as the cross-lingual transfer task, the pseudo-
labeled training data contains lots of noise propagated from upper-stream of
the pipeline, in which case, those ’hard’ examples are more likely to be just
errors or outliers and could hurt the training process. We went the opposite
direction and lowered their weights instead, so that the model could focus on
less noisy labels. More specifically, we added weight $(1+p_{t})^{\gamma}$
instead of $(1-p_{t})^{\gamma}$ on top of the regular cross entropy loss, and
for the hyper-parameter $\gamma$ we experimented with values from 1 to 5 and
the value of 4 it works best.

From Table 1, we can see for mBERT model, fine-tuning with pseudo-labeled data
has significant effects on all languages except NL. The largest improvement is
in Chinese, 6.4$\%$ increase in F1 on top of the zero-transfer result, this
number is 2.5$\%$ for German and 2.4$\%$ for Spanish. The same experiment with
XLM-R model shows a different pattern, F1 increased 3.5$\%$ for German but
dropped a little bit on Spanish and Dutch after fine-tuning. For Chinese, we
see a comparable improvement with mBERT of 4.2$\%$. The negative result on
Spanish and Dutch is probably because XLM-R has already had very good
pretraining and language alignment for those European languages, which can be
seen from the high zero-transfer numbers, therefore a relatively noisy data
set did not bring much gain. On the contrary, Chinese is a relatively distant
language from the perspective of linguistics so that the add-on value of task
specific fine-tuning with natural data is larger.

Another pattern we observed from Table 1 is that across all languages, the
fine-tuning step with pseudo-labeled data is more beneficial to precision
compared with recall. We observed a consistent improvement in precision but a
small drop in recall in most cases.

##  5 Discussions

###  5.1 Impact of the amount of parallel data

One advantage of the parallel corpora method is high data availability
compared to the supervised approach. Therefore a natural question next is
whether more data is beneficial for the cross-lingual transfer task. To answer
this question, we did a series of experiments with varying number of training
examples ranging from 5K to 200K, and the model F1 score increases with the
amount of data at the beginning and plateaued around 40K. All the numbers
displayed in Table 1 are reported for training on a generated data set of size
around 40K (sentences). One possible explanation for the plateaued performance
might be due to the propagated error in the pseudo-labeled data set. Domain
mismatch may also limit the effectiveness of transfer between languages. More
discussions on this topic in the next section.

###  5.2 Impact of the domain of parallel data

Figure 3: F1 scores evaluated on three data sets using different domains’
parallel data. Blue column on the left is the result of zero-shot model
transfer. To its right are F1 scores for 3 different domains and all domains
combined.

Learnings from machine translation community showed that the quality of neural
machine translation models usually strongly depends on the domain they are
trained on, and the performance of a model could drop significantly when
evaluated on a different domain Koehn and Knowles (2017). Similar observation
can be used to explain challenges of the NER cross-lingual transfer. In NER
transfer, the first domain mismatch comes from the natural gap in entity
distributions between different language corpus. Many entities only live
inside the ecosystem of a specific group of languages and may not be
translated naturally to others. The second domain mismatch is between the
parallel corpora and the NER data set. The English model might not have good
domain adaptation ability and could perform well on the CoNLL2003 data set but
poorly on the parallel data set.

Domain | PER | ORG | LOC | All
---|---|---|---|---
OpenSubtitles | 24,036 | 3,809 | 5,196 | 33,041
UN | 1,094 | 25,875 | 12,718 | 39,687
News | 10,977 | 9,568 | 28,168 | 48,713
All Domains Combined | 10,454 | 14,269 | 17,412 | 42,135
Table 2: Entity Count by type in the pseudo-labeled Chinese NER training data
set. We listed multiple domains that were extracted from different parallel
data source. And AllDomains is a combination of all resources.

To study the impact of the domain of parallel data on transfer performance, we
did an experiment on Chinese using parallel data from different domains. We
picked three representative data sets from OPUS Tiedemann (2012),
OpenSubtitles, UN(contains UN and UNPC), News(contains WMT and News-
Commentary) and another one with all these three combined. OpenSubtitles is
from movie subtitles and language style is informal and oral. UN is from
united nation reports and language style is more formal and political
flavored. News data is from newspaper and content is more diverse and closer
to CoNLL data sets. We evaluated the F1 on three Chinese testsets, Weibo, MSRA
and People’s Daily, where Weibo contains messages from social media, MSRA is
more formal and political, and People’s Daily data set are newspaper articles.

From Fig 3 we see OpenSubtitles performs best on Weibo but poorly on the other
two. On the contrary UN performs worst on Weibo but better on the other two.
News domain performs the best on People’s Daily, which is also consistent with
one’s intuition because they are both from newspaper articles. All domains
combined approach has a decent performance on all three testsets.

Different domains of data have quite large gap in the density and distribution
of entities, for example OpenSubtitles contains more sentences that does not
have any entity. In the experiments above we did filtering to keep the same
ratio of ’empty’ sentences among all domains. We also examined the difference
in type distributions. In Tab 2, we calculated entity counts by type and
domain. OpenSubtitles has very few ORG and LOC entities, whereas UN data has
very few PER data. News and All domain data are more balanced.

Figure 4: F1 score by type evaluated on People’s Daily data set. The same as
Fig 3, we compare the results using different domains of parallel data for the
NER transfer.

In Fig 4, we show evaluation on People’s Daily by type. We want to understand
how does parallel data domain impacts transfer performance for different
entity types. News data has the best performance on all types, and Subtitles
has very bad result on ORG. All these observations are consistent with the
type distribution in Table 2.

##  6 Ablation Study

To better understand the contribution of each stage in the training process,
we conducted an ablation study on German data set with mBERT model. We
compared 6 different settings: (1) approach proposed in this work, i.e. fine-
tune the English model with pseudo-labeled data with the new loss, denoted as
re-weighted (RW) loss; (2) zero-transfer is direct model transfer which was
trained only on English NER data; (3) fine-tune the English model with pseudo-
labeled data with regular cross-entropy (CE) loss; (4) Skip fine-tune on
English and directly fine-tune mBERT on pseudo-labeled data. (5)(6) fine-tune
the model directly with mixed English and pseudo-labeled data simultaneously
with RW and CE losses respectively.

| Test P | Test R | F1
---|---|---|---
Sequential fine-tune with RW | 73.1 | 76.2 | 74.6
Zero-transfer | 67.6 | 77.4 | 72.1
Sequential fine-tune with CE | 71.2 | 74.3 | 72.7
Skip fine-tune on English | 71.0 | 73.6 | 72.3
Fine-tune on En/De mixed (CE) | 73.1 | 75.9 | 74.5
Fine-tune on En/De mixed (RW) | 65.5 | 42.6 | 51.6
Table 3: Ablation study results evaluated on CoNLL2002 German NER data. All
experiments used the mBERT-base model. RW denotes the re-weighted loss we
proposed in the paper; CE denotes the regular cross entropy loss.

From Table 3, we see that both pretraining on English and fine-tuning with
pseudo German data are essential to get the best score. The RW loss performed
better in sequential fine-tuning than in simultaneous training with mixed
English and German data, this is probably because noise portion in English
training set is much smaller than in the German pseudo-labeled training set,
using RW loss on English data failed to exploit the fine-grained information
in some hard examples and results in insufficiently optimized model. Another
observation is that training the mBERT with a combination of English and
German using cross entropy loss, we can get almost the same score as our best
model, which is trained with two stages.

##  7 Conclusion

In this paper, we proposed a new method of doing NER cross-lingual transfer
with parallel corpus. By leveraging the XLM-R model for entity projection, we
are able to make the whole pipeline automatic and free from human engineered
feature or data, so that it could be applied to any other language that has a
rich resource of translation data without extra cost. This method also has the
potential to be extended to other NLP tasks, such as question answering. In
this paper, we thoroughly tested the new method in four languages, and it is
most effective in Chinese. We also discussed the impact of parallel data
domain on NER transfer performance and found that a combination of different
domain parallel corpora yielded the best average results. We also verified the
contribution of the pseudo-labeled parallel data by an ablation study. In the
future we will further improve the alignment model precision and also explore
alternative ways of transfer such as self teaching instead of direct fine-
tuning. We are also interested to see how the propose approach generalizes to
cross-lingual transfers for other NLP tasks.

## References

  * Ni et al. (2017) Jian Ni, Georgiana Dinu, and Radu Florian. 2017.  Weakly supervised cross-lingual named entity recognition via effective annotation and representation projection.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1470–1480. Association for Computational Linguistics.
  * Kim et al. (2017) Joo-Kyung Kim, Young-Bum Kim, Ruhi Sarikaya, and Eric Fosler-Lussier. 2017.  Cross-lingual transfer learning for POS tagging without cross-lingual resources.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2832–2838, Copenhagen, Denmark. Association for Computational Linguistics.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  XNLI: Evaluating cross-lingual sentence representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.
  * Xie et al. (2018) Jiateng Xie, Zhilin Yang, Graham Neubig, Noah A. Smith, and Jaime Carbonell. 2018\.  Neural cross-lingual named entity recognition with minimal resources.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 369–379, Brussels, Belgium. Association for Computational Linguistics.
  * Ni and Florian (2019) Jian Ni and Radu Florian. 2019.  Neural cross-lingual relation extraction based on bilingual word embedding mapping.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 399–409, Hong Kong, China. Association for Computational Linguistics.
  * Lin et al. (2017) T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár.  Focal loss for dense object detection.  arXiv preprint arXiv:1708.02002, 2017.
  * Wu and Dredze (2019) Shijie Wu and Mark Dredze. 2019.  Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 833–844, Hong Kong, China. Association for Computational Linguistics.
  * Jain et al. (2019) Alankar Jain, Bhargavi Paranjape, and Zachary C. Lipton.  Entity projection via machine translation for cross-lingual NER.  In EMNLP, pages 1083–1092, 2019.
  * Bari et al. (2019) M Saiful Bari, Shafiq Joty, and Prathyusha Jwalapuram.  Zero-resource cross-lingual named entity recognition.  arXiv preprint arXiv:1911.09812, 2019.
  * Sang and Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the conll-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning, CoNLL 2003, Held in cooperation with HLT-NAACL 2003, Edmonton, Canada, May 31 - June 1, 2003_ , pages 142–147.
  * Wu and Dredze (2019) Shijie Wu and Mark Dredze.  Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT.  arXiv preprint arXiv:1904.09077, 2019.
  * Devlin (2018) Jacob Devlin. 2018.  Multilingual bert readme document.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Conneau et al. (2017) Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017.  Word translation without parallel data.  _arXiv preprint arXiv:1710.04087_.
  * Lample and Conneau (2019) Guillaume Lample and Alexis Conneau. 2019.  Cross-lingual language model pretraining.  _arXiv preprint arXiv:1901.07291_.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel Data, Tools and Interfaces in OPUS.  In _Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12)_ , Istanbul, Turkey. European Language Resources Association (ELRA).
  * Lison and Tiedemann (2016) Pierre Lison and Jörg Tiedemann. 2016.  OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016)_ , Paris, France. European Language Resources Association (ELRA).
  * Schwenk et al. (2019) Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2019.  WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia.  _arXiv preprint arXiv:11907.05791_.
  * Ziemski et al. (2016) Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016.  The United Nations Parallel Corpus v1.0.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016)_ , Paris, France. European Language Resources Association (ELRA).
  * Koehn (2005) Philipp Koehn. 2005.  Europarl: A Parallel Corpus for Statistical Machine Translation.  In _Conference Proceedings: the tenth Machine Translation Summit_ , pages 79–86, Phuket, Thailand. AAMT, AAMT.
  * Agić and Vulić (2019) Željko Agić and Ivan Vulić. 2019.  JW300: A Wide-Coverage Parallel Corpus for Low-Resource Languages.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3204–3210, Florence, Italy. Association for Computational Linguistics.
  * Ehrmann et al. (2011) Maud Ehrmann, Marco Turchi, and Ralf Steinberger. 2011.  Building a multilingual named entity-annotated corpus using annotation projection.  In Proceedings of Recent Advances in Natural Language Processing. Association for Computational Linguistics, pages 118–124.  http://aclweb.org/anthology/R11-1017.
  * Al-Rfou et al. (2015) Rami Al-Rfou, Vivek Kulkarni, Bryan Perozzi, and Steven Skiena. 2015.  Polyglot-ner: Massive multilingual named entity recognition.  In Proceedings of the 2015 SIAM International Conference on Data Mining. SIAM, Vancouver, British Columbia, Canada.  https://doi.org/10.1137/1.9781611974010.66.
  * Tsai et al. (2016) Chen-Tse Tsai, Stephen Mayhew, and Dan Roth. 2016.  Cross-lingual named entity recognition via wikification.  In _CoNLL_ , pages 219–228.
  * Mayhew et al. (2017) Stephen Mayhew, Chen-Tse Tsai, and Dan Roth. 2017.  Cheap translation for cross-lingual named entity recognition.  In _EMNLP_ , pages 2526–2535.
  * Kumar (2015) Faruqui Kumar, S. 2015.  Multilingual open relation extraction using cross-lingual  In _Proceedings of NAACL-HLT_ , 1351–1356.
  * Wu et al. (2020) Qianhui Wu, Zijia Lin, Guoxin Wang, Hui Chen, Börje F Karlsson, Biqing Huang, and Chin-Yew Lin.  Enhanced meta-learning for cross-lingual named entity recognition with minimal resources.  In AAAI, 2020.
  * Moon et al. (2019) Taesun Moon, Parul Awasthy, Jian Ni, and Radu Florian. 2019.  Towards lingua franca named entity recognition with bert.  _arXiv preprint arXiv:1912.01389_.
  * Wu et al. (2020) Wu, Q.; Lin, Z.; Karlsson, B. F.; Lou, J.-G.; and Huang, B. 2020.  Single-/Multi-Source Cross-Lingual NER via Teacher-Student Learning on Unlabeled Data in Target Language.  In _Association for Computational Linguistics_.
  * Wu et al. (2020) Qianhui Wu and Zijia Lin and Börje F. Karlsson and Biqing Huang and Jian-Guang Lou.  UniTrans: Unifying Model Transfer and Data Transfer for Cross-Lingual Named Entity Recognition with Unlabeled Data.  In _IJCAI 2020_.
  * Levow (2020) Gina-Anne Levow.  The Third International Chinese Language Processing Bakeoff: Word Segmentation and Named Entity Recognition.  In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing, 2006.  .
  * Peng and Dredze (2015) Nanyun Peng and Mark Dredze.  Named Entity Recognition for Chinese Social Media with Jointly Trained Embeddings.  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015.  .
  * Koehn and Knowles (2017) Philipp Koehn and Rebecca Knowles.  Six Challenges for Neural Machine Translation.  In Proceedings of the First Workshop on Neural Machine Translation, pages 28–39. 2017.  .