††thanks: Equal contribution.††thanks: Equal contribution.††thanks: Equal
contribution.

# Twin-field quantum key distribution without phase locking

Wei Li  Hefei National Research Center for Physical Sciences at the Microscale
and School of Physical Sciences, University of Science and Technology of
China, Hefei 230026, China  Shanghai Research Center for Quantum Science and
CAS Center for Excellence in Quantum Information and Quantum Physics,
University of Science and Technology of China, Shanghai 201315, China  Likang
Zhang  Hefei National Research Center for Physical Sciences at the Microscale
and School of Physical Sciences, University of Science and Technology of
China, Hefei 230026, China  Shanghai Research Center for Quantum Science and
CAS Center for Excellence in Quantum Information and Quantum Physics,
University of Science and Technology of China, Shanghai 201315, China  Yichen
Lu  Hefei National Research Center for Physical Sciences at the Microscale and
School of Physical Sciences, University of Science and Technology of China,
Hefei 230026, China  Shanghai Research Center for Quantum Science and CAS
Center for Excellence in Quantum Information and Quantum Physics, University
of Science and Technology of China, Shanghai 201315, China  Zheng-Ping Li
Hefei National Research Center for Physical Sciences at the Microscale and
School of Physical Sciences, University of Science and Technology of China,
Hefei 230026, China  Shanghai Research Center for Quantum Science and CAS
Center for Excellence in Quantum Information and Quantum Physics, University
of Science and Technology of China, Shanghai 201315, China  Hefei National
Laboratory, University of Science and Technology of China, Hefei 230088, China
Cong Jiang  Yang Liu  Jinan Institute of Quantum Technology, Jinan, Shandong
250101, China  Jia Huang  Hao Li  Zhen Wang  State Key Laboratory of
Functional Materials for Informatics, Shanghai Institute of Microsystem and
Information Technology, Chinese Academy of Sciences, Shanghai 200050, China
Xiang-Bin Wang  Hefei National Laboratory, University of Science and
Technology of China, Hefei 230088, China  Jinan Institute of Quantum
Technology, Jinan, Shandong 250101, China  State Key Laboratory of Low
Dimensional Quantum Physics, Department of Physics, Tsinghua University,
Beijing, China  Qiang Zhang  Hefei National Research Center for Physical
Sciences at the Microscale and School of Physical Sciences, University of
Science and Technology of China, Hefei 230026, China  Shanghai Research Center
for Quantum Science and CAS Center for Excellence in Quantum Information and
Quantum Physics, University of Science and Technology of China, Shanghai
201315, China  Hefei National Laboratory, University of Science and Technology
of China, Hefei 230088, China  Jinan Institute of Quantum Technology, Jinan,
Shandong 250101, China  Lixing You  State Key Laboratory of Functional
Materials for Informatics, Shanghai Institute of Microsystem and Information
Technology, Chinese Academy of Sciences, Shanghai 200050, China  Feihu Xu
Jian-Wei Pan  Hefei National Research Center for Physical Sciences at the
Microscale and School of Physical Sciences, University of Science and
Technology of China, Hefei 230026, China  Shanghai Research Center for Quantum
Science and CAS Center for Excellence in Quantum Information and Quantum
Physics, University of Science and Technology of China, Shanghai 201315, China
Hefei National Laboratory, University of Science and Technology of China,
Hefei 230088, China

###### Abstract

Twin-field quantum key distribution (TF-QKD) has emerged as a promising
solution for practical quantum communication over long-haul fiber. However,
previous demonstrations on TF-QKD require the phase locking technique to
coherently control the twin light fields, inevitably complicating the system
with extra fiber channels and peripheral hardware. Here we propose and
demonstrate an approach to recover the single-photon interference pattern and
realize TF-QKD _without_ phase locking. Our approach separates the
communication time into reference frames and quantum frames, where the
reference frames serve as a flexible scheme for establishing the global phase
reference. To do so, we develop a tailored algorithm based on fast Fourier
transform to efficiently reconcile the phase reference via data post-
processing. We demonstrate no-phase-locking TF-QKD from short to long
distances over standard optical fibers. At 50-km standard fiber, we produce a
high secret key rate (SKR) of 1.27 Mbit/s, while at 504-km standard fiber, we
obtain the repeater-like key rate scaling with a SKR of 34 times higher than
the repeaterless secret key capacity. Our work provides a scalable and
practical solution to TF-QKD, thus representing an important step towards its
wide applications.

Introduction.—Quantum key distribution (QKD) can provide information-
theoretically secure keys among distant parties [1] and it has become an
indispensable cryptographic primitive in the upcoming quantum era [2, 3]. Due
to the loss of photons in their transmission, the point-to-point secret key
capacity (SKC0) without the quantum repeater scales linearly $O(\eta)$ with
the channel transmission [4, 5]. The twin-field (TF) QKD protocol [6], an
efficient version of measurement-device-independent QKD [7], can greatly
enhance the transmission distance by achieving a repeater-like rate-loss
scaling of $O(\sqrt{\eta})$ with current available technology. Consequently,
TF-QKD has been studied extensively in theory [8, 2, 10, 11] and experiment
[12, 13, 14, 15, 16, 17, 18, 19, 20]. These efforts make the long-haul fiber
network within reach. Moreover, its measurement-device-independent advantage
can remove trusted nodes from the networks, thus granting a security boost
over deployed quantum communication network [21, 22, 23].

In practice, however, TF-QKD is phase-sensitive [6], which normally requires
sufficiently long coherence time for two independent laser sources. In
previous TF-QKD realizations, such stringent requirement has been fulfilled by
phase-locking laser sources using the optical phase-lock loop [12, 14, 17,
20], the time-frequency dissemination [13, 18, 19] or the injection locking
[15] techniques. These approaches require extra servo channels to disseminate
the reference light and peripheral hardware to perform the locking, which
could potentially hinder the wide deployment of TF-QKD in network settings.
Furthermore, the future quantum network may involve users with free-space link
[24, 25] or with integrated photonic chip [26, 27, 28]. However, the servo
channel is hard to establish in the free-space link, and the phase-locking
components are challenging to be integrated on chip.

In this work, we propose and demonstrate an approach to realize TF-QKD
_without_ phase locking. We alternate the communication period into quantum
frame (Q-frame) and reference frame (R-frame), and use the R-frame to provide
a phase reference for the Q-frame by reconciling the signals via data post-
processing. We develop an algorithm based on the fast Fourier transform (FFT)
to efficiently track the frequency and the phase fluctuation. By doing so, we
are able to recover the interference pattern with 259 photon detections in a
duration of 7 $\upmu$s, yielding an interference error rate (ER) of 2.32%, or
an ER of 3.74% with 159 photon detections in a duration of 11 $\upmu$s.
Moreover, we derive an analytical model to study the sources of phase
fluctuation and provide a guideline to optimize the experimental parameters in
the aspect of the ratio of the Q-frame to the R-frame. To test our approach,
we build a TF-QKD setup without any phase locking (or phase compensating) at
the transmitter (or receiver), and demonstrate TF-QKD from 50- to 504-km
standard fiber channels. A secret key rate (SKR) of 2.05 bit/s is generated
over 504-km standard fiber (96.8 dB loss or equivalent to 605-km ultra low
loss fiber) in the finite-size regime, which is 34 times higher than SKC0. At
50-km standard fiber, we are able to produce 1.27 Mbit/s secret keys.

No-phase-locking scheme.—The phase difference of two light fields $\Phi(t)$
evolves as,

$\Phi(t)=2\pi\nu t+\phi_{0}+\Delta\phi(t),$ (1)

where $\nu$, $\phi_{0}$ and $\Delta\phi(t)$ denote the beat-note frequency,
the initial phase and the phase fluctuation respectively. Both $\nu$ and
$\phi_{0}$ can be estimated via the data post-processing, whereas
$\Delta\phi(t)$ includes the intrinsic phase noise of the laser sources and
the fluctuation introduced in the channel transmission.

Figure 1: No-phase-locking scheme. The interference pattern is evaluated
frequently in the R-frame, providing a phase reference for the Q-frame. The
detection events of D0 (D1) are mapped to the amplitude of +1 (-1) and the
detection probability is related to the interference pattern. Each R-frame is
used twice for the neighboring Q-frames.

To perform the parameter estimation, we propose to supplement the quantum
pulses (Q-frame) with strong R-frame. The R-frame is used to reconcile the
phase in the Q-frame through post-processing the photon detection events. As
shown in Fig. 1, the detection probabilities of two detectors are correlated
with the interference pattern. In light of this, we develop an FFT-based
algorithm to reconstruct the frequency spectrum (see Supplementary Section I).
From the frequency spectrum, we choose the component with the largest
amplitude within a certain frequency range. Then, $\nu$ is the frequency of
the component and $\phi_{0}$ can be obtained from the amplitude of the
component. Here to reduce the estimation error, we propose to extend the
detection series with padding zeros. This can greatly narrow the distribution
of the frequency estimation and decrease the frequency deviation, thus
enhancing the frequency estimation precision (see Supplementary Fig. 1).
Moreover, we duplex the detection events to use them more efficiently, i.e.,
each R-frame is used twice by the two neighbour Q-frame. This doubles the
available photon events and reduces the error rate when the photon count rate
is low.

According to the estimated $\hat{\nu}$ and $\hat{\phi}_{0}$, the ER can be
further evaluated. When single photon events arrival time $t$ satisfy (a)
$2n\pi-\pi/M<\Phi(t)<2n\pi+\pi/M$ or (b)
$2n\pi+(M-1)\pi/M<\Phi(t)<2n\pi+(M+1)\pi/M$, a valid event is counted, where
$\Phi(t)=2\pi\hat{\nu}t+\hat{\phi}_{0}$. If condition (a) ((b)) is satisfied
and D0 (D1) clicks, it is counted as a correct event. Otherwise, if condition
(a) ((b)) is satisfied and D1 (D0) clicks, it is counted as an error event.

After the estimation, the first two terms of Equation 1 reduce to the
estimation error in the analysis of the ER. While the third term (mainly
contributed by the phase noise of laser source and the fiber length
fluctuation) is a random noise, its contribution can be analyzed in theory and
characterized in experiment. To do so, we derive an analytical model to study
the phase noise/fluctuation and provide a guideline to optimize the
experimental parameters (see Supplementary Section II). In previous TF-QKD
experiments [6, 13, 15], the variance of fiber fluctuation of a fixed time
interval is normally specified using time-domain measurements. In contrast, we
use the _frequency-domain_ power spectral density (PSD) to characterize the
random noise. By doing so, we are able to calculate the fluctuation in
variable interval $\tau$ and relate it to the ER. Furthermore, the phase noise
of the laser sources is also taken into consideration in our model. This was
often ignored in the previous analysis for the phase-locked TF-QKD schemes [6,
13, 15]. Nonetheless, as the linewidth of the laser source increases, it will
dominate the phase fluctuation. Even for the phase-locked laser sources, their
phase noise replicate the one of the reference source at best, which can not
be ignored. By including the phase noise, our model can properly analyze the
linewidth requirement for the laser sources for both phase-locking and no-
phase-locking TF-QKD.

Setup.—To implement the no-phase-locking TF-QKD scheme, we build an
experimental setup as shown in Fig. 2. Alice and Bob transmit their quantum
light to Charlie’s measurement site via symmetric quantum channels,
constituted by standard telecom fiber spools (G.652). No servo link is used
for the dissemination of a third phase reference laser. Each user holds a
commercial external cavity laser diode (RIO PLANEX) with a Lorentzian
linewidth of 5 kHz and the wavelength is set at 1550.12 nm, but with a slight
frequency mismatch of about 100 MHz. The frequency drifts slowly to an extent
of 30 MHz over one day (see Supplementary Section VI), which can be tracked by
the frequency estimation algorithm.

The continuous light is encoded into two frames by three cascaded intensity
modulators. Due to the fact that two lasers are heterodyned, no modulation in
intensity and phase is required for the R-frame to resolve the phase ambiguity
as in ref. [13, 15]. The Q-frame is used to generate quantum signals following
the 3-intensity sending-or-not-sending (SNS) TF-QKD protocol [3] (see
Supplementary Section VII for full descriptions). Two cascaded phase
modulators are used for 16-level random phase modulation covering 2$\pi$
range. One intensity modulator creates the two frames and modulates the weak
decoy intensity, and the other two shape the pulses in Q-frame and extinct
vacuum pulses jointly. Such configuration allow us to apply bias control on
the first intensity modulator to stabilize the signal and decoy intensities.

Figure 2: Experimental setup. Alice (Bob) send their phase- and intensity-
modulated states to Charlie to perform single-photon measurements. D0(1) are
used to generate secure keys, while D2(3) are used for delay and polarization
compensation. PM, phase modulator; IM, intensity modulator; BS, beamsplitter;
VOA, variable optical attenuator; EPC, electrical polarization controller;
DWDM, dense wavelength-division multiplexer; PBS, polarization beamsplitter;
SNSPD, superconducting nanowire single photon detector; MBC, modulator bias
controller.

Light from Alice and Bob interfere at Charlie’s 50:50 beam splitter with the
same polarization as the output of the polarization beamsplitter is
polarization-maintaining. This transforms the polarization variation into
intensity variation. We use two electrical polarization controllers to rotate
the polarization at 5 Hz so as to keep the photon rates at the other port of
the polarization beamsplitter constant (5% of total photon rate). Moreover,
the photons detected by D2 (D3) are accumulated to track the delay drift of
the fiber every 5 s. The transition edge from the Q-frame to the R-frame is
used as the time mark. These are sufficient to compensate the polarization and
delay drift caused by the indoor fiber spools.

The interference output is detected by two superconducting nanowire single
photon detectors (SNSPD). We use two types of SNSPDs, SNSPD#1 and #2, to
perform the measurements (see Supplementary Section VIII for the
characterization). The detected photon events are registered by a time tagging
unit and then processed by a computer. The frequency $\hat{\nu}$ and initial
phase $\hat{\phi}_{0}$ of the beat note can be both estimated by the FFT-based
algorithm. The valid arriving time $t$ in X basis should satisfy:

$\left|\cos{\left(2\pi\hat{\nu}t+\hat{\phi}_{0}+(\phi_{a}-\phi_{b})\right)}\right|\geq\cos{(\pi/16)},$
(2)

where $\phi_{a}\ (\phi_{b})$ is phase modulated by the users and announced
publicly in the post-processing. We also use actively odd-parity pairing
method [30, 4] in the error rejection through two-way classical communication,
significantly reducing the bit-flip error rate.

Results.—We firstly analyze the system performance with the no-phase-locking
scheme quantitatively. The phase noise PSD of the laser has a -20 dB/decade
slope, contributed by the white frequency noise. The phase noise PSD
introduced by the fiber spool is also measured (see Supplementary Section III
and IV), which is mainly distributed below the frequency 100 kHz and increase
slightly with the fiber length. With these two results, the phase fluctuation
in the Q-frame can be calculated accordingly. The fluctuation is converted to
the ER and averaged over the duration of the Q-frame.

To characterize the ER with different $T_{\mathrm{Q}}$ and $T_{\mathrm{R}}$,
we use the same setup as in Fig. 2, except that no modulation is applied. We
plot measured ER as a function of $T_{\mathrm{R}}$ with different photon count
rates in Fig. 3a. A higher count rate enables accurate frequency estimation at
smaller $T_{\mathrm{R}}$, thus lowering the minimum ER. The simulation result
assuming accurate frequency estimation is also plotted, including only the
influence of the phase fluctuation. Based on these results and the constraint
of the experimental system, we choose $T_{\mathrm{R}}$ and $T_{\mathrm{Q}}$ to
be 4.9152 and 1.6384 $\upmu$s respectively. The ER contributes to the quantum
bit error rate (QBER) of phase (X) basis in SNS-TF-QKD protocol. As another
important implication of our theoretical model, we can determine the linewidth
requirement for different selected $T_{\mathrm{Q}}$ and plot the simulated
results in Fig. 3b. As an example, to achieve a maximum channel loss of 66 dB
in our system, the laser linewidth should be narrower than 35.5 kHz when
$T_{\mathrm{Q}}=1~{}\upmu$s, resulting in an ER of 11%.

Figure 3: a, Measured and simulated interference error rate as a function of
$T_{\mathrm{R}}$ with different count rates per detector and
$T_{\mathrm{Q}}=1~{}\upmu$s. The simulated results assume accurate
estimations. b, Interference error rate as functions of the laser linewidth
and $T_{\mathrm{R}}=5~{}\upmu$s, in which accurate estimation is also assumed.

With the performance analysis and parameter optimization, we perform TF-QKD
experiments from 50 to 504-km standard fiber spools with different detectors
and finite sizes (Supplementary Section VIII). The channel is symmetrical and
the total channel loss amounts to 9.6, 38.4, 56.8, 72.1 and 96.8 dB
respectively. The results are presented in Fig. 4. With SNSPD#1, we measure
the dynamic range to be 44 dB when the count rate of the R-frame is 24
Mcount/s and the duty cycle (i.e., the ratio of Q-frame) is 1/4. That is to
say the dynamic dark count rate [32, 33] is about 1000 count/s, which is the
dominant noise. To increase the signal-to-noise ratio, we apply a gating
window of 200 ps, resulting in a dark count probability of $2\times 10^{-7}$.
With less than one hour of continuous run, we achieve a finite-size SKR of
$1.25\times 10^{-7}$ bit/pulse, or 39 bit/s at 380-km standard fiber. At the
short end, we increase the duty cycle to 3/4. And it enable a secret key rate
of 1.27 Mbit/s at 50-km standard fiber channel with a small number of sending
pulses of $10^{10}$.

Figure 4: SKRs at different standard fiber distances. The solid line is the
simulated SKRs with different finite sizes and two sets of experimental
parameters. The fiber loss coefficient is 0.19 dB/km. The solid stars denote
the experimental results. The orange dashed line denotes SKC0.

To generate positive key rates at longer distance, we replace the detectors
with SNSPD#2, which has a dynamic range over 58 dB. Furthermore, we keep the
detected count rate in R-frame as low as possible (8 Mcounts/s per detector)
and use a narrower gating window of 100 ps. With these upgrades, the
scattering noise in fiber becomes dominant and results in a noise probability
of $1.6\times 10^{-8}$. This is an order of magnitude of improvement over
SNSPD#1, allowing us to achieve a SKR of $6.56\times 10^{-9}$ bit/pulse or
2.05 bit/s at 504-km standard optical fiber.

Discussion.—In summary, we have proposed and demonstrated the no-phase-locking
scheme for TF-QKD. Our scheme does not only remove the service channels for
the dissemination of the reference light, but also does not needs active phase
compensation setup at the measurement site. Such features greatly simplify the
setup to match the one of measurement-device-independent QKD systems [34, 35],
with the ability to establish global phase reference nonetheless. We also show
that commercial kilohertz linewidth semiconductor lasers are sufficient to
perform TF-QKD with our scheme. Arguably, the simplification of setup with our
scheme does not lead to performance degradation. With ultra-low-loss fiber
deployed, 96.8-dB channel loss corresponds to over 600-km total distance,
comparable to the state-of-art TF-QKD experiment with phase-locking sources
[19] or phase-compensating module [17]. To achieve further transmission
distance, one could increase the dynamic range of the SNSPD and insert a
cryogenic bandpass filter to reduce the noise floor introduced by the
background radiation. On the other hand, more advanced algorithm may require
less counts in the recovery of the carrier, which helps to reduce the strong
reference light intensity. Overall, we believe our scheme provides a practical
solution to TF-QKD network and the phase recovery algorithm is applicable to
other phase-sensitive applications [36, 37].

Note added.— While preparing the manuscript, we notice two related works which
demonstrate different approaches [38, 39] to solve the phase-locking issue.
Ref. [38] uses the mode-paring MDI-QKD protocol and Ref. [39] employs the
optical frequency comb. In contrast, we propose an efficient method to
reconcile the phase reference in TF-QKD via data post-processing, and
demonstrate high-rate TF-QKD and beat the repeaterless secret key capacity
with two unlocked, low-cost, chip-scale lasers.

Acknowledgement.— This work was supported by National Natural Science
Foundation of China (Grant No. 62031024, No. 12204467), Innovation Program for
Quantum Science and Technology (2021ZD0300300), Anhui Initiative in Quantum
Information Technologies, Shanghai Municipal Science and Technology Major
Project (Grant No. 2019SHZDZX01), Shanghai Science and Technology Development
Funds (22JC1402900) and Shanghai Academic/Technology Research Leader
(21XD1403800). Y.L. acknowledges support from the Taishan Scholar Program of
Shandong Province. F. Xu acknowledges the support from the Tencent Foundation.

## References

  * [1] Bennett, C. H. & Brassard, G.  Quantum cryptography: Public key distribution and coin tossing.  In _Proceedings of the IEEE International Conference on Computers, Systems, and Signal Processing_ , 175–179 (1984).
  * [2] Xu, F., Ma, X., Zhang, Q., Lo, H.-K. & Pan, J.-W.  Secure quantum key distribution with realistic devices.  _Rev. Mod. Phys._ 92, 025002 (2020).
  * [3] Pirandola, S. _et al._ Advances in quantum cryptography.  _Adv. Opt. Photon._ 12, 1012–1236 (2020).
  * [4] Takeoka, M., Guha, S. & Wilde, M. M.  Fundamental rate-loss tradeoff for optical quantum key distribution.  _Nat. Commun._ 5, 5235 (2014).
  * [5] Pirandola, S., Laurenza, R., Ottaviani, C. & Banchi, L.  Fundamental limits of repeaterless quantum communications.  _Nat. Commun._ 8, 15043 (2017).
  * [6] Lucamarini, M., Yuan, Z. L., Dynes, J. F. & Shields, A. J.  Overcoming the rate–distance limit of quantum key distribution without quantum repeaters.  _Nature_ 557, 400–403 (2018).
  * [7] Lo, H.-K., Curty, M. & Qi, B.  Measurement-device-independent quantum key distribution.  _Phys. Rev. Lett._ 108, 130503 (2012).
  * [8] Ma, X., Zeng, P. & Zhou, H.  Phase-matching quantum key distribution.  _Phys. Rev. X_ 8, 031043 (2018).
  * [9] Wang, X.-B., Yu, Z.-W. & Hu, X.-L.  Twin-field quantum key distribution with large misalignment error.  _Phys. Rev. A_ 98, 062323 (2018).
  * [10] Curty, M., Azuma, K. & Lo, H.-K.  Simple security proof of twin-field type quantum key distribution protocol.  _npj Quantum Inf._ 5, 64 (2019).
  * [11] Cui, C. _et al._ Twin-field quantum key distribution without phase postselection.  _Phys. Rev. Applied_ 11, 034053 (2019).
  * [12] Minder, M. _et al._ Experimental quantum key distribution beyond the repeaterless secret key capacity.  _Nat. Photonics_ 13, 334 (2019).
  * [13] Liu, Y. _et al._ Experimental twin-field quantum key distribution through sending or not sending.  _Phys. Rev. Lett._ 123, 100505 (2019).
  * [14] Wang, S. _et al._ Beating the fundamental rate-distance limit in a proof-of-principle quantum key distribution system.  _Phys. Rev. X_ 9, 021046 (2019).
  * [15] Fang, X.-T. _et al._ Implementation of quantum key distribution surpassing the linear rate-transmittance bound.  _Nat. Photonics_ 14, 422–425 (2020).
  * [16] Zhong, X., Hu, J., Curty, M., Qian, L. & Lo, H.-K.  Proof-of-principle experimental demonstration of twin-field type quantum key distribution.  _Phys. Rev. Lett._ 123, 100506 (2019).
  * [17] Pittaluga, M. _et al._ 600-km repeater-like quantum communications with dual-band stabilization.  _Nat. Photonics_ 15, 530–535 (2021).
  * [18] Chen, J.-P. _et al._ Twin-field quantum key distribution over 511 km optical fiber linking two distant metropolitans.  _Nat. Photonics_ 15, 570 (2021).
  * [19] Chen, J.-P. _et al._ Quantum key distribution over 658 km fiber with distributed vibration sensing.  _Phys. Rev. Lett._ 128, 180502 (2022).
  * [20] Wang, S. _et al._ Twin-field quantum key distribution over 830-km fibre.  _Nat. Photon._ 16, 154–161 (2022).
  * [21] Peev, M. _et al._ The SECOQC quantum key distribution network in Vienna.  _New J. Phys._ 11, 075001 (2009).
  * [22] Sasaki, M. _et al._ Field test of quantum key distribution in the Tokyo QKD network.  _Opt. Express_ 19, 10387–10409 (2011).
  * [23] Chen, Y.-A. _et al._ An integrated space-to-ground quantum communication network over 4,600 kilometres.  _Nature_ 589, 214–219 (2021).
  * [24] Vallone, G. _et al._ Experimental satellite quantum communications.  _Phys. Rev. Lett._ 115, 040502 (2015).
  * [25] Cao, Y. _et al._ Long-distance free-space measurement-device-independent quantum key distribution.  _Phys. Rev. Lett._ 125, 260503 (2020).
  * [26] Ma, C. _et al._ Silicon photonic transmitter for polarization-encoded quantum key distribution.  _Optica_ 3, 1274–1278 (2016).
  * [27] Sibson, P. _et al._ Chip-based quantum key distribution.  _Nat. Commun._ 8, 13984 (2017).
  * [28] Wei, K. _et al._ High-speed measurement-device-independent quantum key distribution with integrated silicon photonics.  _Phys. Rev. X_ 10, 031030 (2020).
  * [29] Yu, Z.-W., Hu, X.-L., Jiang, C., Xu, H. & Wang, X.-B.  Sending-or-not-sending twin-field quantum key distribution in practice.  _Sci. Rep._ 9, 3080 (2019).
  * [30] Xu, H., Yu, Z.-W., Jiang, C., Hu, X.-L. & Wang, X.-B.  Sending-or-not-sending twin-field quantum key distribution: Breaking the direct transmission key rate.  _Phys. Rev. A_ 101, 042330 (2020).
  * [31] Jiang, C., Hu, X.-L., Yu, Z.-W. & Wang, X.-B.  Composable security for practical quantum key distribution with two way classical communication.  _New J. Phys._ 23, 063038 (2021).
  * [32] Burenkov, V., Xu, H., Qi, B., Hadfield, R. H. & Lo, H.-K.  Investigations of afterpulsing and detection efficiency recovery in superconducting nanowire single-photon detectors.  _J. Appl. Phys._ 113, 213102 (2013).
  * [33] Chen, S. _et al._ Dark counts of superconducting nanowire single-photon detector under illumination.  _Opt. Express_ 23, 10786–10793 (2015).
  * [34] Liu, Y. _et al._ Experimental measurement-device-independent quantum key distribution.  _Phys. Rev. Lett._ 111, 130502 (2013).
  * [35] Comandar, L. C. _et al._ Quantum key distribution without detector vulnerabilities using optically seeded lasers.  _Nat. Photonics_ 10, 312–315 (2016).
  * [36] Duan, L.-M., Lukin, M. D., Cirac, J. I. & Zoller, P.  Long-distance quantum communication with atomic ensembles and linear optics.  _Nature_ 414, 413–418 (2001).
  * [37] Humphreys, P. C. _et al._ Deterministic delivery of remote entanglement on a quantum network.  _Nature_ 558, 268–273 (2018).
  * [38] Zhu, H.-T. _et al._ Experimental mode-pairing measurement-device-independent quantum key distribution without global phase-locking.  _arXiv:2208.05649_ (2022).
  * [39] Zhou, L., Lin, J., Jing, Y. & Yuan, Z.  Twin-field quantum key distribution without optical frequency dissemination.  _arXiv:2208.09347_ (2022).

Supplementary for “Twin-field quantum key distribution without phase locking”

##  I FFT-based algorithm

In this experiment, Alice and Bob send bright continuous-wave light to Charlie
during the R-frame, who will then broadcast the detection result including
which detector clicks and the arrival time. We estimate the frequency and
phase of the interference pattern in the post-processing procedure based on
these detection results. The estimation problem can be modeled as following.
Supposing that the frequency difference of the two independent laser source of
Alice and Bob is $\nu$, and the initial phase difference at time $t=t_{0}$ is
$\phi_{0}$, The detection probabilities of the two detector are:

$\left\\{\begin{aligned} &P_{0}(t)=\dfrac{1+\cos{\Phi(t)}}{2}\\\
&P_{1}(t)=\dfrac{1-\cos{\Phi(t)}}{2}\\\ \end{aligned}\right.\quad,\quad\\\ $
(1)

where $\Phi(t)=2\pi\nu(t-t_{0})+\phi_{0}$. To recover $\Phi(t)$ using the
detections of two detectors, we defined
$P(t)=P_{0}(t)-P_{1}(t)=\cos{\Phi(t)}$. The value of $i$-th detection event
$x_{i}$ is assigned to +1 (-1) if D0 (D1) clicks. And the value of the $j$-th
bin $X_{j}=\sum x$ for all the events in the bin. And we perform the FFT
algorithm on the sparse signal $X_{j}$. From the spectrum, we select the
frequency component with the largest amplitude as the estimate of $\hat{\nu}$
and its phase in the spectrum as the estimate of $\hat{\phi}_{0}$. We set the
bin width to 100 ps in the experiment.

Due to the sparsity of $X_{j}$, the signal to noise ratio is often limited.
And the frequency resolution of the spectrum is limited by the time duration
of the sample. As a consequence, when there are enough detection events within
the R-frame, the precision of the estimated frequency is mainly affected by
the resolution of the spectrum. In this case, further intensifying the
reference light to increase effective number of detection is no longer
beneficial. Another possible way to reduce the frequency estimation error is
to combine a number of neighboring R-frames for a single frequency estimation.
However, this method is not always useful as the estimation error of $\Phi(t)$
will accumulate with time under a fixed estimation error of $\hat{\nu}$, thus
increasing the interference error rate with more R-frame used.

We enhance the spectral resolution by padding $X_{n}$ with trailing zeros to a
suitable length. Fig. S1 shows the comparison of the frequency estimation
results using the FFT algorithm _with_ and _without_ padding zeros. By
improving the resolution from about 0.1 MHz to 0.01 MHz, the full width at
half maximum of the frequency distribution decreases from 0.13 MHz to 0.02
MHz. It is clear that padding zeros greatly narrows the distribution of the
frequency estimation and decreases frequency deviation. This in turn verifies
that the frequency estimation precision is mainly limited by the frequency
resolution of the algorithm instead of our system. Besides, in our
experimental implementation, we utilize two neighboring R-frames to obtain one
estimate of the differential phase of the R-frame. This doubles the available
photon events for the estimation while contributing little to the error
accumulation effect, which reduce the error rate with relatively low count
rate at longer distance. The improved algorithm can recover the spectrum with
higher frequency resolution and higher signal-to-noise ratio. As a result, the
interference error rate improves from 27.22% to 3.89%.

Figure S1: The histogram of frequency estimation results using FFT algorithm
with/without padding zeros. The results are based on a beat frequency of 80
MHz. The solid lines represent the corresponding Gaussian fit of the two
histograms.

##  II Interference error rate calculation

Here we present the theoretical model for the calculation of the interference
error rate introduced by the phase fluctuation. The emitted laser electric
field undergoing phase fluctuations at Alice’s (Bob’s) side can be written as:

$\displaystyle E_{\mathrm{A(B)}}(t)$
$\displaystyle=E_{0}\exp{\left\\{i\Phi_{\mathrm{A(B)}}(t)\right\\}},$ (2)
$\displaystyle\Phi_{\mathrm{A(B)}}(t)$
$\displaystyle=2\pi\left<\nu_{A(B)}\right>t+\phi_{\mathrm{A(B)}}(t),$

where $\Phi_{\mathrm{A(B)}}(t)$ and $\left<\nu_{\mathrm{A(B)}}\right>$ are the
phase and the average frequency of the laser, $\phi_{\mathrm{A(B)}}(t)$ is a
stochastic process representing the phase noise of the laser.

After transmitting through the fiber, the fluctuation of the fiber length and
refractive index changes the optical distance, and therefore changes the phase
of the light field. Note that the light right before interference at time $t$
is emitted at time $t-L_{\mathrm{A(B)}}(t)/c$, where $L_{\mathrm{A(B)}}(t)$ is
the optical distance through which the photon arriving at time $t$ passes and
$c$ is the speed of light in vacuum. We assume $L_{\mathrm{A(B)}}(t)$ to be a
Gaussian stationary process. The phase change of the laser is the integral of
frequency within the time $\tau$:

$\displaystyle\Phi_{\mathrm{A(B)}}(t+\tau)-\Phi_{\mathrm{A(B)}}(t)$ (3)
$\displaystyle=$ $\displaystyle\int_{t-L_{\mathrm{A(B)}}(t)/c}^{t+\tau-
L_{\mathrm{A(B)}}(t+\tau)/c}2\pi\nu_{\mathrm{A(B)}}(t^{\prime})\mathrm{d}t^{\prime}$
$\displaystyle=$ $\displaystyle\int_{t-L_{\mathrm{A(B)}}(t)/c}^{t+\tau-
L_{\mathrm{A(B)}}(t+\tau)/c}2\pi\delta\nu_{\mathrm{A(B)}}(t^{\prime})\mathrm{d}t^{\prime}+2\pi\left<\nu_{\mathrm{A(B)}}\right>\cdot(\tau-\frac{L_{\mathrm{A(B)}}(t+\tau)-L_{\mathrm{A(B)}}(t)}{c}),$

where
$\delta\nu_{\mathrm{A(B)}}(t)=\nu_{\mathrm{A(B)}}(t)-\left<\nu_{\mathrm{A(B)}}\right>$.
It is the result of treating the frequency modulation noise
$\delta\nu_{\mathrm{A(B)}}(t)$ as the source of the phase noise. By
substituting Equation 2 into above equation, we can get phase fluctuation at
one side:

$\displaystyle\Delta\phi_{\mathrm{A(B)}}(\tau)=$
$\displaystyle\phi_{\mathrm{A(B)}}(t+\tau)-\phi_{\mathrm{A(B)}}(t)$ (4)
$\displaystyle=$ $\displaystyle\int_{t-L_{\mathrm{A(B)}}(t)/c}^{t+\tau-
L_{\mathrm{A(B)}}(t+\tau)/c}2\pi\delta\nu_{\mathrm{A(B)}}(t^{\prime})dt^{\prime}-\frac{L_{\mathrm{A(B)}}(t+\tau)-L_{\mathrm{A(B)}}(t)}{c}\cdot
2\pi\left<\nu_{\mathrm{A(B)}}\right>.$

By using the Wiener–Khinchin theorem and the Gaussian stationary assumptions,
its second order moment has the following form:

$\displaystyle\left<\Delta\phi_{\mathrm{A(B)}}^{2}(\tau)\right>$ (5)
$\displaystyle=$ $\displaystyle
4\pi^{2}\tau^{2}\int_{0}^{\infty}S_{\delta\nu_{\mathrm{A(B)}}}(f)\operatorname{sinc}^{2}(\pi
f\tau)\mathrm{d}f$ $\displaystyle+$ $\displaystyle
4\pi^{2}\nu_{0}^{2}\sigma_{L_{\mathrm{A(B)}}}^{2}(\tau)$ $\displaystyle+$
$\displaystyle
2\int_{0}^{\infty}S_{\delta\nu_{\mathrm{A(B)}}}(f)\left[1-\exp{\left(-2\pi^{2}\sigma_{L_{\mathrm{A(B)}}}^{2}(\tau)f^{2}\right)}\right]\frac{\cos{\left(2\pi
f\tau\right)}}{f^{2}}\mathrm{d}f.$

In the following, we omit the subscript to represent the difference of two
sides. Since the noise of Alice’s side and Bob’s side are identical and
independent, so the variance of phase fluctuation at time $t+\tau$ can be
calculated:

$\left<\Delta\phi^{2}(\tau)\right>:=\left<\left(\phi(t+\tau)-\phi(t)\right)^{2}\right>=\left<\Delta\phi_{\mathrm{A}}^{2}(\tau)\right>+\left<\Delta\phi_{\mathrm{B}}^{2}(\tau)\right>$
(6)

In the experiment, we need the R-frame with time interval
$\mathcal{R}=[-T_{\mathrm{R}}-T_{\mathrm{Q}}/2,-T_{\mathrm{Q}}/2]\cup[T_{\mathrm{Q}}/2,T_{\mathrm{Q}}/2+T_{\mathrm{R}}]$
to estimate $\hat{\phi}$, and predict the relative phase in the Q-frame
$\mathcal{Q}=[-T_{\mathrm{Q}}/2,T_{\mathrm{Q}}/2]$. The estimation process can
be modeled as the following least square optimization problem assuming the
estimate of frequency difference is accurate $\hat{\nu}=\left<\nu\right>$:

$\displaystyle\frac{1}{2T_{\mathrm{R}}}\int_{\mathcal{R}}\left[\hat{\phi}-\phi(t)\right]^{2}\mathrm{d}t$
(7) $\displaystyle=$
$\displaystyle\frac{1}{2T_{\mathrm{R}}}\int_{\mathcal{R}}\left[\left(\phi(t)-\frac{1}{2T_{\mathrm{R}}}\int_{\mathcal{R}}\phi(t^{\prime})\mathrm{d}t^{\prime}\right)+\left(\frac{1}{2T_{\mathrm{R}}}\int_{\mathcal{R}}\phi(t^{\prime})\mathrm{d}t^{\prime}-\hat{\phi}\right)\right]^{2}\mathrm{d}t$
$\displaystyle=$
$\displaystyle\overline{\Delta\phi^{2}}+\left(\bar{\phi}-\hat{\phi}\right)^{2}$

where
$\overline{\Delta\phi^{2}}=\frac{1}{2T_{\mathrm{R}}}\int_{\mathcal{R}}\left(\phi(t^{\prime})-\bar{\phi}\right)^{2}\mathrm{d}t^{\prime}$
is the time variance over R-frame. Note that this is different from the
ensemble average $\left<\Delta\phi^{2}\right>$. To minimize the above formula,
$\hat{\phi}$ takes the following value.

$\hat{\phi}=\bar{\phi}=\frac{1}{2T_{\mathrm{R}}}\int_{\mathcal{R}}\phi(t^{\prime})\mathrm{d}t^{\prime}$
(8)

The phase fluctuation in the quantum frame (at time $t\in\mathcal{Q}$) is
derived as follow:

$\displaystyle\left<\left(\phi(t)-\hat{\phi}\right)^{2}\right>=$
$\displaystyle\frac{1}{(2T_{\mathrm{R}})}\int_{0}^{T}\left[\left<\Delta\phi^{2}(t^{\prime}+T_{\mathrm{Q}}/2+t)\right>+\left<\Delta\phi^{2}(t^{\prime}+T_{\mathrm{Q}}/2-t)\right>\right]\mathrm{d}t^{\prime}$
(9) $\displaystyle-$
$\displaystyle\frac{1}{(2T_{\mathrm{R}})^{2}}\int_{0}^{T}\left[\left<\Delta\phi^{2}(T_{\mathrm{Q}}+t^{\prime})\right>+\left<\Delta\phi^{2}(T_{\mathrm{Q}}+2T_{\mathrm{R}}-t^{\prime})\right>+2\left<\Delta\phi^{2}(T_{\mathrm{R}}-t^{\prime})\right>\right]t^{\prime}\mathrm{d}t^{\prime}$

To this end, we have obtained the phase fluctuation after the estimation of
the phase reference. And we can prove that by using neighboring R-frames,
better estimation can be obtained. The next step is to transform the phase
fluctuation to the interference error rate. The instantaneous interference
error rate at time $t\in\mathcal{Q}$ is

$\displaystyle
ER_{ins}(t)=\frac{M}{4\pi}\int_{\hat{\phi}=-\frac{2\pi}{M}}^{\frac{2\pi}{M}}\int_{\phi=-\infty}^{\infty}\mathrm{sin}^{2}\left(\frac{\phi}{2}\right)\cdot\rho(\hat{\phi}-\phi;t)\mathrm{d}\phi\mathrm{d}\hat{\phi}$
(10)

where $\rho(\hat{\phi}-\phi;t)$ is the Gaussian probability distribution
function of the phase fluctuation.

$\rho(\hat{\phi}-\phi;t)=\dfrac{1}{\sqrt{2\pi\left<\left(\phi(t)-\hat{\phi}\right)^{2}\right>}}\cdot\exp{\left[-\dfrac{(\hat{\phi}-\phi)^{2}}{2\left<\left(\phi(t)-\hat{\phi}\right)^{2}\right>}\right]}$
(11)

Finally, we can obtain the average interference error rate over the Q-frame
$\mathcal{Q}=[-T_{\mathrm{Q}}/2,T_{\mathrm{Q}}/2]$:

$ER(\mathcal{Q})=\frac{1}{T_{\mathrm{Q}}}\int_{\mathcal{Q}}\frac{M}{4\pi}\int_{\hat{\phi}=-\frac{2\pi}{M}}^{\frac{2\pi}{M}}\int_{\phi=-\infty}^{\infty}\mathrm{sin}^{2}\left(\frac{\phi}{2}\right)\cdot\rho(\hat{\phi}-\phi;t)\mathrm{d}\phi\mathrm{d}\hat{\phi}\mathrm{d}t$
(12)

##  III Laser phase noise characterization

We adopt the delayed self-heterodyne technique [1] to measure the laser’s
phase noise. The setup is shown in Fig. S2. The advantage of this method is
that no lengthy fiber (longer than the coherence length of the tested laser)
is needed. We use an acoustic optic modulator driven by a 80-MHz sinusoidal
signal as the frequency shifter. And the fiber length mismatch is chosen to be
16 m. Therefore a 80-ns delay is introduced, which is enough to ensure that
the measured phase noise is mainly contributed by the laser.

Figure S2: The phase noise measurement setup for lasers. AOM, acoustic optic
modulator; PD, photodiode with a trans-impedance amplifier; SSA, signal source
analyzer.

The measurement is performed on a Keysight E5052B signal source analyzer. The
numbers of correlation and trace average are both set to 10. The measured
results are shown in Fig. S3. The phase noise power spectral density (PSD) of
the laser $S_{\phi,\mathrm{LD}}(f)$ is related to measured PSD by the
following equation:

$S_{\phi,\mathrm{LD}}(f)=S_{\phi}(f)\cdot\frac{1}{4\sin^{2}(\pi f\tau)}$ (13)

The PSDs of the lasers are also plotted in Fig. S3. The PSD of the lasers have
a -20 dB/decade slope, indicating that the white frequency noise is the noise
source. Therefore, the linewidth of the laser can be simply calculated as
$\delta\nu=\pi h_{\mathrm{w}}$. $h_{\mathrm{w}}$ is the amplitude of the white
frequency noise. The linewidth of Alice’s and Bob’s lasers are 5.9 and 2.4 kHz
respectively.

Figure S3: The phase noise PSD for lasers. The PSD is calculated from the
measured PSD using the setup in Fig. S2. The white frequency noise PSD is used
to fit the data in the frequency range of 100 kHz to 10 MHz.

##  IV Fiber phase noise characterization

Also, we characterize the phase noise of the fiber channel using the setup as
shown in Fig. S2. The light source is a fiber laser with a narrow linewidth
$<$100 Hz. This ensure that the measured phase noise is mainly contributed by
the fiber spools. The measured PSDs are plotted in Fig. S5. The noise below
100 kHz is significantly larger than the one without fiber spools. The noise
is mainly contributed by the mechanical vibration of the fibers. As we tap the
spool, we observe the noise level below the frequency range of 100 kHz
bouncing up and down. However as the total distance increases from 202 km to
380 km. The increase of the noise is minor. This phenomenon can also be
observed in the measurement of the interference error rate, as we show in the
next section.

Figure S4: The phase noise measurement setup for fiber spools. The setup is
similar to the one for characterizing laser source, except that lengthy fiber
spools are inserted in the Mach-Zehnder interferometer. Figure S5: The phase
noise results for fiber spools.

##  V Interference error rate characterization

In order to analyze the sources of the interference error rate reported in the
Main Text and validate our theoretical analysis, we perform series of tests
under different conditions.

First, we use a self-heterodyne setup as in Fig. S6 to characterize the error
introduced by our fast-Fourier-transform-based algorithm as the phase
fluctuations contributed by the source and the fiber channel are negligible in
this case. Using the no-phase-locking scheme described in the Main Text, the
total error rate is measured to be 1.66% with a count rate of 24 Mcount/s per
detector.

Figure S6: The error rate measurement setup for a single laser. The VOA is
used to attenuate the light to single photon level and to balance the loss of
the Mach-Zehnder interferometer. VOA, variable optical attenuator; SNSPD,
superconducting nanowire single photon detector.

We use the same setup as in figure 2 in the Main Text to characterize the
error rate of two lasers, except that no modulation is applied. In the
experiment, $T_{\mathrm{R}}$ and $T_{\mathrm{Q}}$ are 4.9152 and 1.6384
$\upmu$s respectively. As shown in Tab. S1, the count rate is 24 Mcount/s for
each detector. The error rate of two lasers increases to 2.55%. And the fiber
channel with different lengths contribute little to the error rate. Therefore,
the phase noise of the sources play a bigger role than the fiber channel in
our experiment. And to replicate a more practical scenario, we place one of
the laser to a different room. The error rate in this case increases slightly.

Table S1: Interference error rate with different configurations. Configuration | Interference error rate
---|---
One laser + 0-km fiber | 1.66 $\pm$ 0.02%
Two lasers + 0-km fiber | 2.55 $\pm$ 0.06%
Two lasers + 302-km fiber | 2.64 $\pm$ 0.17%
Two lasers + 504-km fiber | 2.69 $\pm$ 0.18%

##  VI Source frequency stability test

The beat note frequency can be measured by a PD and a frequency counter. The
result is shown in Fig. S7. In the estimation of the beat note frequency, we
limit the frequency range in 50 to 200 MHz. The frequency is within the range
over days. Somehow, if the frequency drifts out of the range, Alice or Bob
could abort the QKD session and recalibrate their wavelength by discussing
over the public channel.

Figure S7: The frequency drift of two unlocked lasers used in our TF-QKD
experiment.

##  VII Three-intensity SNS-TF-QKD with active odd parity pairing protocol

The 3-intensity SNS protocol [2, 3] is implemented by our TF-QKD system, and
the actively odd-parity paring (AOPP) method is used to extract the final key
[4]. For the completeness of the paper, we simply review the content as
following.

In this protocol, Alice (Bob) has three sources $o,x,y$
($o^{\prime},x^{\prime},y^{\prime}$) with intensities
$\mu_{o}=0,\mu_{x},\mu_{y}$ respectively. Here the sources $o,o^{\prime}$ are
vacuum sources. In each time window, Alice (Bob) randomly decides whether it
is a window that emits pulses from source $o$ ($o^{\prime}$) with probability
$p_{o}$, or a window that emits pulses from source $x$ ($x^{\prime}$) with
probability $p_{x}$, or a signal window with probability
$p_{y}=1-p_{o}-p_{x}$. If it is a signal window, Alice (Bob) randomly chooses
the sources $o$ or $y$ ($o^{\prime}$ or $y^{\prime}$) with probabilities
$1-\epsilon$ and $\epsilon$. For the vacuum pulses in the signal windows,
Alice (Bob) denotes them as $0$ ($1$), and for the non-vacuum pulses in the
signal windows, Alice (Bob) denotes them as $1$ ($0$).

In the data post-processing, for the time windows that Alice and Bob choose
the sources $x$ and $x^{\prime}$ respectively, Alice and Bob would announce
the private phases of the pulse pair and take the following criterion to
perform the post-selection:

$1-|\cos(\theta_{A1}-\theta_{B1}-\psi_{AB})|\leq\lambda,$ (14)

where $\theta_{A1}$ and $\theta_{B1}$ are the private phases of Alice’s and
Bob’s pulses respectively; $\psi_{AB}$ can take an arbitrary value, which can
be different from time to time as Alice and Bob like; $\lambda$ is a small
positive value. Denote the number of total pulse pairs sent out that
satisfactory Eq. (14) by $N_{X}$ and the number of corresponding error
effective events as $m_{X}$.

We denote the number of pulse pairs of source
$\kappa\zeta(\kappa=o,x,y;\zeta=o^{\prime},x^{\prime},y^{\prime})$ sent out in
the whole protocol by $N_{\kappa\zeta}$, and the total number of one-detector
heralded events of source $\kappa\zeta$ by $n_{\kappa\zeta}$. We define the
counting rate of source $\kappa\zeta$ by
$S_{\kappa\zeta}=n_{\kappa\zeta}/N_{\kappa\zeta}$, and the corresponding
expected value by $\langle S_{\kappa\zeta}\rangle$. With all those
definitions, we have

$\begin{split}&N_{oo^{\prime}}=[p_{o}^{2}+2p_{o}p_{y}(1-\epsilon)]N\\\
&N_{ox^{\prime}}=N_{xo^{\prime}}=[p_{o}+p_{y}(1-\epsilon)]p_{x}N\\\
&N_{oy^{\prime}}=N_{yo^{\prime}}=p_{o}p_{y}\epsilon N\end{split}$ (15)

With the decoy-state analysis, we can get the lower bounds of the expected
values of the counting rate of states $|01\rangle\langle 01|$ and
$|10\rangle\langle 10|$, which are

$\displaystyle\langle\underline{s_{01}}\rangle$
$\displaystyle=\frac{\mu_{y}^{2}e^{\mu_{x}}\langle
S_{ox^{\prime}}\rangle-\mu_{x}^{2}e^{\mu_{y}}\langle
S_{oy^{\prime}}\rangle-(\mu_{y}^{2}-\mu_{x}^{2})\langle
S_{oo^{\prime}}\rangle}{\mu_{y}\mu_{x}(\mu_{y}-\mu_{x})},$ (16)
$\displaystyle\langle\underline{s_{10}}\rangle$
$\displaystyle=\frac{\mu_{y}^{2}e^{\mu_{x}}\langle
S_{xo^{\prime}}\rangle-\mu_{x}^{2}e^{\mu_{y}}\langle
S_{yo^{\prime}}\rangle-(\mu_{y}^{2}-\mu_{x}^{2})\langle
S_{oo^{\prime}}\rangle}{\mu_{y}\mu_{x}(\mu_{y}-\mu_{x})}.$ (17)

Then we can get the lower bound of the expected value of the counting rate of
untagged photons

$\langle\underline{s_{1}}\rangle=\frac{1}{2}(\langle\underline{s_{10}}\rangle+\langle\underline{s_{01}}\rangle),$
(18)

and the lower bound of the expected value of the untagged bits $1$,
$\langle\underline{n_{10}}\rangle$, and untagged bits $0$,
$\langle\underline{n_{01}}\rangle$

$\displaystyle\langle\underline{n_{10}}\rangle=Np_{y}^{2}\epsilon(1-\epsilon)\mu_{y}e^{-\mu_{y}}\langle\underline{s_{10}}\rangle,$
(19)
$\displaystyle\langle\underline{n_{01}}\rangle=Np_{y}^{2}\epsilon(1-\epsilon)\mu_{y}e^{-\mu_{y}}\langle\underline{s_{01}}\rangle.$
(20)

And the upper bound of the expected value of the phase-flip error rate satisfy

$\langle\overline{e_{1}^{ph}}\rangle=\frac{\langle
T_{X}\rangle-e^{-\mu_{A1}-\mu_{B1}}\langle
S_{oo^{\prime}}\rangle/2}{e^{-\mu_{A1}-\mu_{B1}}(\mu_{A1}+\mu_{B1})\langle\underline{s_{1}}\rangle},$
(21)

where $\langle T_{X}\rangle$ is the expected value of $T_{X}$, and
$T_{X}=m_{X}/N_{X}$.

With values above, we can calculate the lower bound of untagged bits and
phase-flip error rate after AOPP, $n_{1}^{\prime}$ and $e_{1}^{\prime ph}$ by
the method proposed in Refs. [4]. We have the related formulas of
$n_{1}^{\prime}$ as follows:

$\displaystyle u=\frac{n_{g}}{2n_{odd}},$ (22a)
$\displaystyle\underline{n_{10}}=\varphi^{L}(u\langle\underline{n_{10}}\rangle),$
(22b)
$\displaystyle\underline{n_{01}}=\varphi^{L}(u\langle\underline{n_{01}}\rangle),$
(22c) $\displaystyle\underline{n_{1}}=\underline{n_{10}}+\underline{n_{01}},$
(22d) $\displaystyle
n_{1}^{r}=\varphi^{L}\left(\frac{\underline{n_{1}}^{2}}{2un_{t}}\right),$
(22e) $\displaystyle
n_{01}^{\prime}=2n_{1}^{r}\left(\frac{\underline{n_{01}}}{\underline{n_{1}}}-\sqrt{-\frac{\ln\varepsilon}{2n_{1}^{r}}}\right)$
(22f) $\displaystyle
n_{10}^{\prime}=2n_{1}^{r}\left(\frac{\underline{n_{10}}}{\underline{n_{1}}}-\sqrt{-\frac{\ln\varepsilon}{2n_{1}^{r}}}\right)$
(22g) $\displaystyle n_{min}=\min(n_{01}^{\prime},n_{10}^{\prime}),$ (22h)
$\displaystyle
n_{1}^{\prime}=2\varphi^{L}\left(n_{min}(1-\frac{n_{min}}{2n_{1}^{r}})\right),$
(22i)

where $n_{t}$ is number of raw keys that Alice and Bob get in the experiment;
$n_{g}$ is the number of pair if Alice and Bob perform AOPP to their raw keys;
${n_{odd}}$ is the number of pairs with odd-parity if Bob randomly groups all
his raw key bits two by two, and $n_{g}$ and $n_{odd}$ are observed values;
$\epsilon$ is the failure probability of parameter estimation; and
$\varphi^{U}(x),\varphi^{L}(x)$ are the upper and lower bounds while using
Chernoff bound [5] to estimate the real values according to the expected
values.

And we have the the related formulas of $e_{1}^{\prime ph}$ as follows:

$\displaystyle
r=\frac{\underline{n_{1}}}{\underline{n_{1}}-2n_{1}^{r}}\ln\frac{3(\underline{n_{1}}-2n_{1}^{r})^{2}}{\varepsilon},$
(23a) $\displaystyle
e_{\tau}=\frac{\varphi^{U}(2n_{1}^{r}\langle\overline{e_{1}^{ph}}\rangle)}{2n_{1}^{r}-r}$
(23b)
$\displaystyle\bar{M}_{s}=\varphi^{U}[(n_{1}^{r}-r){e_{\tau}}(1-{e_{\tau}})]+r,$
(23c) $\displaystyle e_{1}^{\prime ph}=\frac{2\bar{M}_{s}}{n_{1}^{\prime}}$
(23d)

Finally, we can get the key rate $R$ by

$\begin{split}R=&\frac{1}{N}\left\\{n_{1}^{\prime}[1-h(e_{1}^{\prime
ph})]-fn_{t}^{\prime}h(E^{\prime})-2\log_{2}{\frac{2}{\varepsilon_{cor}}}\right.\\\
&\left.-4\log_{2}{\frac{1}{\sqrt{2}\varepsilon_{PA}\hat{\varepsilon}}}\right\\}.\end{split}$
(24)

where $N$ is the total number of pulse pairs sent out in the experiment;
$h(x)=-x\log_{2}x-(1-x)\log_{2}(1-x)$ is the Shannon entropy; $f$ is the error
correction inefficiency; $n_{t}^{\prime}$ is the number of survived bits after
AOPP; $E^{\prime}$ is the bit-flip error rate of the survived bits after AOPP;
$\varepsilon_{cor}$ is the failure probability of error correction,
$\varepsilon_{PA}$ is the failure probability of privacy amplification, and
$\hat{\varepsilon}$ is the coefficient while using the chain rules of smooth
min- and max- entropy [6].

##  VIII Experimental parameters and data

As Fig. S8 shows, the R-frame is time-multiplexed with the Q-frame, and the
ratio of them can be tuned flexibly according to the channel loss. In Tab. S2,
we present the frame configuration for different fiber distances in our
experiment. At the long end, the count rate of the reference frame is reduced
to lower the dark count noise related to the strong reference pulses. At the
short end, the quantum frame duty cycle can be improved to increase the secret
key rate (SKR).

Figure S8: Intensity and phase modulation format. A buffer zone between the two frames is used to reduce the modulation error from the transition of two frames with different dynamics. Table S2: Frame time and count rate in the reference frame for different fiber distances. $T_{\mathrm{Q}}$ ($T_{\mathrm{R}}$) is the frame time for the quantum (reference) frame. Fiber distance (km) |  $T_{\mathrm{R}}$ ($\upmu$s) |  $T_{\mathrm{Q}}$ ($\upmu$s) | Count rate (Mcount/s)
---|---|---|---
50 | 0.4096 | 1.2288 | 24
202 | 4.9152 | 1.6384 | 24
302 | 4.9152 | 1.6384 | 24
380 | 4.9152 | 1.6384 | 24
504 | 4.9152 | 1.6384 | 8

Tab. S3 summarizes the encoding information of our experiments using
3-intensity SNS protocol. The encoding parameters are optimized to generate
higher SKR according to the protocol presented in Section VII.

The insertion loss of the measurement setup is listed in Tab. S4. The EPC,
DWDM and PBS are spliced together to reduce the insertion loss. We use two
different SNSPDs for the experiments of 504-km-long fiber and other distances.
Their performances are summarized in Tab. S5. At 504 km, the noise in the key
generation basis is mainly from the scattering photons in the fiber induced by
the strong light in the R-frame. Therefore, the dense wavelength-division
multiplexer is used to reduce the Raman noise. However, double Rayleigh
scattering still contributes to the noise count of 160 count/s per detector.

In Tab. S6 we report the detailed experimental results for the different fiber
distances. Raw detection counts needed to calculate the SKR and and the AOPP
results are listed. The counts are labelled in the suffix as ‘AB$ab$’, ‘A’
(‘B’) and ‘$a$’ (‘$b$’) denotes the basis and decoy intensity chosen by Alice
(Bob) for a certain pulse, where ‘$y$’, ‘$x$’, ‘$o$’ are the signal, decoy and
vacuum states respectively.

Table S3: Parameters for the different fiber lengths, including the decoy intensities and probabilities. $\mu_{y},\ \mu_{x},\ \mu_{o}$ are the signal, decoy and vacuum intensities. $P_{Z}$ is the probability of choosing Z basis. $\epsilon$ is the sending probability when Z basis is chosen. $P_{\mu_{x}}$ is the probability of sending in $\mu_{x}$ decoy intensity. Fiber length (km) |  $\mu_{y}$ (ph/pulse) |  $\mu_{x}$ (ph/pulse) |  $\mu_{o}$ (ph/pulse) | $P_{Z}$ | $\epsilon$ | $P_{\mu_{x}}$
---|---|---|---|---|---|---
50 | 0.53 | 0.028 | 0.00005 | 0.94 | 0.28 | 0.055
202 | 0.49 | 0.044 | 0.00005 | 0.878 | 0.28 | 0.115
302 | 0.48 | 0.045 | 0.00005 | 0.87 | 0.28 | 0.12
380 | 0.45 | 0.050 | 0.00005 | 0.85 | 0.28 | 0.136
504 | 0.43 | 0.064 | 0.00005 | 0.77 | 0.28 | 0.2
Table S4: Insertion loss of the measurement setup. The results are specified in dB.  Component | Alice | Bob
---|---|---
EPC+DWDM+PBS | 0.87 | 0.92
BS-D0 | 3.51 | 3.39
BS-D1 | 3.41 | 3.45
PC | 0.21 | 0.12
Table S5: Performance of two sets of SNSPDs.  Detector | SNSPD#1 | SNSPD#2
---|---|---
D0 | D1 | D0 | D1
Efficiency | 74% | 65% | 70% | 72%
Dark count (count/s) | 7 | 6 | 7 | 5
Dynamic range (dB) | 44 | 48 | 58 | 60
Table S6: Experimental results and SKR results of finite-size SNS protocol with AOPP.  Detector | SNSPD#1 | SNSPD#2
---|---|---
Distance (km) | 50 | 202 | 302 | 380 | 504
Loss (dB) | 9.6 | 38.4 | 56.8 | 72.1 | 96.8
R (bit per pulse) | $1.35\times 10^{-3}$ | $2.15\times 10^{-5}$ | $1.18\times 10^{-6}$ | $1.25\times 10^{-7}$ | $6.56\times 10^{-9}$
K (bit per second) | $1.27\times 10^{6}$ | $6.72\times 10^{3}$ | $3.69\times 10^{2}$ | $3.92\times 10^{1}$ | $2.05$
$\mathrm{N}_{\mathrm{send}}$ | $10^{10}$ | $10^{10}$ | $10^{11}$ | $10^{12}$ | $10^{13}$
Detected-ZZ$yy$ | 91070688 | 3753305 | 4348062 | 6061368 | 2848762
Detected-ZZ$yo$ | 125590483 | 4799465 | 5625148 | 7930021 | 3744013
Detected-ZZ$oy$ | 119406794 | 4855778 | 5423145 | 7773735 | 3470979
Detected-ZZ$oo$ | 328365 | 13077 | 35858 | 123832 | 68929
Detected-ZX$yx$ | 10890616 | 950371 | 1162482 | 1957861 | 1529226
Detected-ZX$yo$ | 935187 | 55645 | 89416 | 613864 | 235751
Detected-ZX$ox$ | 1510180 | 215377 | 248739 | 194847 | 515998
Detected-ZX$oo$ | 2457 | 153 | 563 | 5898 | 4721
Detected-XZ$xy$ | 10228682 | 955528 | 1144738 | 1954830 | 1553976
Detected-XZ$xo$ | 1543109 | 202480 | 261767 | 199155 | 496252
Detected-XZ$oy$ | 889432 | 55600 | 86138 | 533252 | 216947
Detected-XZ$oo$ | 4981 | 303 | 1045 | 3235 | 6753
Detected-XX$xx$ | 239419 | 72844 | 84062 | 251290 | 337914
Detected-XX$xo$ | 11791 | 2406 | 4066 | 15654 | 31099
Detected-XX$ox$ | 11074 | 2572 | 3985 | 13337 | 32511
Detected-XX$oo$ | 16 | 2 | 9 | 158 | 178
Detected XX$xx$ matching | 30014 | 9055 | 10447 | 31442 | 42569
Correct XX$xx$ matching | 28560 | 8407 | 9549 | 28859 | 40621
Sifted key bits in Z-basis before AOPP | 336396330 | 13421625 | 15432213 | 21888956 | 10132683
$\mathrm{QBER}_{\mathrm{ZZ}}$ before AOPP |  27.17$\%$ |  28.06$\%$ |  28.41$\%$ |  28.26$\%$ |  28.79$\%$
Survived key bits in Z-basis after AOPP | 71375551 | 2707044 | 3137485 | 4522494 | 2111964
$\mathrm{QBER}_{\mathrm{ZZ}}$ after AOPP |  0.20$\%$ |  0.21$\%$ |  0.52$\%$ |  1.23$\%$ |  1.50$\%$

## References

  * [1] Camatel, S. & Ferrero, V.  Narrow linewidth CW laser phase noise characterization methods for coherent transmission system applications.  _Journal of Lightwave Technology_ 26, 3048–3055 (2008).
  * [2] Wang, X.-B., Yu, Z.-W. & Hu, X.-L.  Twin-field quantum key distribution with large misalignment error.  _Physical Review A_ 98, 062323 (2018).
  * [3] Yu, Z.-W., Hu, X.-L., Jiang, C., Xu, H. & Wang, X.-B.  Sending-or-not-sending twin-field quantum key distribution in practice.  _Scientific Reports_ 9, 3080 (2019).
  * [4] Jiang, C., Hu, X.-L., Yu, Z.-W. & Wang, X.-B.  Composable security for practical quantum key distribution with two way classical communication.  _New Journal of Physics_ 23, 063038 (2021).
  * [5] Chernoff, H.  A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations.  _The Annals of Mathematical Statistics_ 23, 493–507 (1952).
  * [6] Vitanov, A., Dupuis, F., Tomamichel, M. & Renner, R.  Chain rules for smooth min-and max-entropies.  _IEEE Transactions on Information Theory_ 59, 2603–2612 (2013).


# Improving Non-native Word-level Pronunciation Scoring with Phone-level Mixup
Data Augmentation and Multi-source Information

###### Abstract

Deep learning-based pronunciation scoring models highly rely on the
availability of the annotated non-native data, which is costly and has
scalability issues. To deal with the data scarcity problem, data augmentation
is commonly used for model pretraining. In this paper, we propose a phone-
level mixup, a simple yet effective data augmentation method, to improve the
performance of word-level pronunciation scoring. Specifically, given a phoneme
sequence from lexicon, the artificial augmented word sample can be generated
by randomly sampling from the corresponding phone-level features in training
data, while the word score is the average of their GOP scores. Benefit from
the arbitrary phone-level combination, the mixup is able to generate any word
with various pronunciation scores. Moreover, we utilize multi-source
information (e.g., MFCC and deep features) to further improve the scoring
system performance. The experiments conducted on the Speechocean762 show that
the proposed system outperforms the baseline by adding the mixup data for
pretraining, with Pearson correlation coefficients (PCC) increasing from 0.567
to 0.61. The results also indicate that proposed method achieves similar
performance by using 1/10 unlabeled data of baseline. In addition, the
experimental results also demonstrate the efficiency of our proposed multi-
source approach.

Index Terms: pronunciation scoring, mixup, data augmentation, computer
assisted pronunciation training (CAPT), multi-source and multi-view approach

##  1 Introduction

It is well-known that the L2 learning process is heavily affected by a well-
established habitual perception of phonemes and articulatory motions in the
learners’ primary language (L1) [1], which often cause mistakes and imprecise
articulation in speech productions by the L2 learners, e.g., a negative
language transfer [1, 2]. As a feasible tool, computer assisted pronunciation
training (CAPT) is often employed to automatically assess L2 learners’
pronunciation quality at different levels, e.g., phone-level [3, 4, 5, 6, 7,
8, 9, 10, 11, 12], word-level [13, 14, 15, 16, 17] and sentence-level [18, 19,
20, 21, 22].

Over the last few decades, many methods have been proposed to evaluate non-
native leaners’ pronunciation quality at the aforementioned levels. Most of
these methods can be divided into two major groups, namely one-step or two-
step approaches. Methods in the former group directly utilize confidence
scores [3, 4, 5, 6, 7] derived from automatic speech recognition (ASR) systems
to assess phone-level and sentence-level pronunciation quality. For example,
the HMM-based phone log-likelihood [3] or log-posterior [4, 5, 6, 7]. Among
these methods, Goodness of Pronunciation (GOP) [4] is widely used in
pronunciation scoring task. Given an acoustic segment of phoneme with multiple
frames, the corresponding GOP is defined as the normalized frame-level
posterior probability over the segment.

In addition, the output phone sequence from ASR, e.g. end-to-end models [10,
9, 11] or extended recognition network [13, 14], can be aligned with the given
prompts, and a phone or word is marked as mispronounced if there is some
mismatch between these two sequences. Alternatively, the two-step approaches
treat pronunciation scoring or mispronunciation detection as regression or
classification task. Specifically, phone, word and sentence boundaries are
first generated by forced-alignment, and then either frame-level or segmental-
level pronunciation features within each boundary are fed into task-dependent
classifiers or regressors (e.g., [6, 12, 8, 15, 16, 17, 18, 22, 21, 19, 20]).
Finally, the posterior probabilities or predicted values obtained from those
models are often used as pronunciation scores. Taking advantage of using extra
supervised training data, the two-step approach armed with deep learning
techniques was reported to achieve better scoring performance than its
counterparts [16].

Although the aforementioned two-step approaches have achieved satisfactory
pronunciation scoring or mispronunciation detection results, the performances
heavily rely on the size of labeled scoring samples, that is not always
practical in many real-world applications. Meanwhile, the non-native data
collection and labeling process is costly and has scalability issues [16].
Take the recently released public free dataset Speechocean762 [23] for
example, only 5,000 sentences, 31,816 words and 95,078 phones have been
assigned with human scores. Another public dataset L2-ARCTIC [24] has
collected 11,026 utterances including 97,000 words and 349,000 phone segments.
The comparison in [25] show that the largest non-native corpus contains 90,841
utterances, but it is not publicly available. To tackle the limited non-native
data problem, data augmentation techniques have been investigated in some
previous work [16, 26, 22]. In [26], Text-To-Speech (TTS) was used to
synthesize “incorrect stress” samples on top of modified lexical stress. In
[16], the author created negative training samples by modifying the canonical
text as the modified text forms a mismatched pair with its original word-level
speech. More recently, authors in [22] collected a large number of unlabeled
non-native data for pretraining sentence-level scorer with GOP scores. It
avoids time-consuming labeling, but non-native data collection is still a
challenge task.

Figure 1: The mixup data generation module.

Inspired by the recent success of mixup on image classifications [27], this
paper proposes a phone-level mixup to deal with limited word-level training
data. Different to image, word pronunciation can be explicitly defined as a
sequence of phones. Hence, we are able to build a phone-level dataset from the
limited word-level training samples. For each phone, a pool of phone-level
features and corresponding GOP scores is first constructed from the training
data. Then the artificial word-level augmented features are concatenated by
random sampling from the corresponding phone-level data, where the word score
is the average of the phone GOP scores. Leveraging the arbitrary combination
of phone-level information, we can generate any word with various
pronunciation scores to improve model’s generalization ability. As reported in
[8], deep features extracted from non-native acoustic model are not
discriminate enough due to the inconsistency of non-native phone label.
Therefore, we used both MFCC and deep features to further improve the scoring
system.

##  2 Pronunciation Scoring Framework with Mixup

Our proposed scoring framework contains two modules: the mixup data generation
module, which is able to create arbitrary words with various pronunciation
scores, and the pronunciation scorer module, which is used to map a sequence
of frame-level acoustic features into word-level pronunciation score.

###  2.1 Mixup Data Generation

Mixup was first proposed in [27], where interpolation between original
feature-target pairs were used to generate augmented data for image
classification, where the linear interpolation is applied on the pixel-based
feature space. Different to the image data used in classification task, which
has the same size and inherent ordering, for the word-level pronunciation
scoring, the speech signal generally consists of different phoneme sequences
and various duration, which is hard to perform interpolation on word-level
features directly. Alternatively, we apply mixup on a lower-level feature
space, phone-level features. Given a phone sequence from lexicon, it is easy
to generated the augmented word-level features by concatenating the
corresponding phone-level data, while the word score is the average of the
phone scores.

To perform mixup, we need to collect a pool of phone-level features and their
corresponding GOP scores. Specifically, an acoustic model, trained with paired
speech signal and transcription, is utilized for deep features and GOPs score
extraction. As shown in Figure 1 (a), given a collection of non-native audio
with canonical transcription, we first extract MFCC features from the speech
signal. An acoustic model is utilized to force-align all non-native audio
(pairs of MFCC features and canonical transcription) to get the start and end
time boundary for each phoneme. The MFCCs of each phoneme are then fed into
the acoustic model to extract corresponding deep features and frame-level
posterior probability from the bottleneck layer [22] and output layer,
respectively. Substantially, the GOP score of a specific phoneme can be
interpreted as the normalized posterior probability over the phoneme duration
obtained by the force-alignment. Finally, a pool of phone-level features and
corresponding GOP scores is collected for each phoneme.

The constructed pools are then used for mixup as shown in Figure 1 (b).
Firstly, a word and its phoneme sequence is sampled from lexicon based on the
word frequency in training set. For each phoneme in the sequence, we randomly
sample one quadruplet from the pool of the corresponding phoneme. As the
example shown in Figure 1, given a word $\mathrm{DAY}$ sampled from the
lexicon, we first obtain its phoneme sequence of /D EY/. Then for each phoneme
/D/ and /EY/, the GOP scores and corresponding features are randomly sampled
from the pool of /D/ and /EY/, respectively. Finally, the features of
generated word is formed by concatenating the features of /D/ and /EY/. We can
obtain the the word score of 0.6 by averaging the corresponding GOP scores,
which are 0.3 and 0.9 respectively.

Compared to [22], the proposed mixup is able to generate as many as artificial
word samples with a small amount of task-related unlabeled data. Different
from [16, 26], where the label of generated training samples is binary(e.g.,
correct or mispronounced), proposed mixup can generate continuous scores for
augmented data.

###  2.2 Pronunciation Scorer

Word pronunciation scorer is designed to mapping acoustic feature to the word
score. The Word pronunciation scorer consists of two training steps: (1)
pretraining with unlabeled and mixup augmented data and (2) fine-tuning with
human labeled data.

During pretraining, two sets of data were used for model training, including
task related unlabeled data and augmented data generated by mixup (as
described in Section 2.1). Given a word with sequence of frame-level deep
features {$\mathbf{x}_{1}^{\text{d}}$, $\mathbf{x}_{2}^{\text{d}}$,
…,$\mathbf{x}_{T}^{\text{d}}$}, we first apply nonlinear transformation on
each of them, and then add them with corresponding canonical phone embeddings
to obtain a sequence of frame-level phonetic features, which is subsequently
fed into the 1D-CNN [28] module. Then a mean-overtime pooling operation is
applied over the feature map obtained by 1D-CNN module, to get the word-level
representation $\mathbf{h}^{\text{d}}$. Similarly, $\mathbf{h}^{\text{m}}$ can
be obtained after doing the same operation on the MFCC features. Finally,
their concatenation is fed into Eq.(1) to predict word pronunciation score
$p$.

$\begin{split}p=\text{Sigmoid}(\mathbf{W}([\mathbf{h}^{\text{d}};\mathbf{h}^{\text{m}}])+\mathbf{b})\end{split}$
(1)

where the [;] denotes the concatenation of two vectors. The mean square error
(MSE) is used as the train objective for model updating, which is defined as:

$\begin{split}\mathcal{L}&=\frac{1}{n}\sum_{i=1}^{n}(y_{i}^{\text{GOP}}-p_{i})^{2}\end{split}$
(2)

where, $n$ denotes the total number of word samples in the pretraining data.
$y_{i}^{\text{GOP}}$ is the word GOP score of the $i$-th word.

At fine-tune stage, we adjust the pretrained pronunciation scorer by a small
amount of human labeled data. Similar to the process in pretraining, the
scorer input consists of MFCC, deep feature and corresponding phoneme
sequence. And the word pronunciation score is obtained by Eq. (1).

Different to pretaining, at fine-tune, the $i$-th word score
$y_{i}^{\text{Human}}$ assigned by human rater are used to supervise the
scorer updating. The Eq. (2) can be rewrite as:

$\begin{split}\mathcal{L}&=\frac{1}{m}\sum_{i=1}^{m}(y_{i}^{\text{Human}}-p_{i})^{2}\end{split}$
(3)

where $m$ denotes the total number of word samples in the fine-tuning data.

##  3 Experimental setup

###  3.1 Speech Corpus

The ASR training corpus consists of 5,000-hour speech data, including 960-hour
native speech from LibriSpeech corpus [29], and 4,000-hour non-native private
recordings from Bytedance. While the pronunciation scoring data consists of
5,000 English utterances spoken by 250 non-native learners from Speechocean762
111http://www.openslr.org/101/. The word pronunciation score in each of them
has been labeled by five experts, the median scores were adopted following the
score files coming with the database, ranging from 0 to 10. Following the
calculation method of previous study [30], the averaged inter-rater agreement
is 0.726. In addition, linguistic experts in Bytedance collected a small
amount of task-related unlabeled data (e.g., a group of Chinese adults are
required to read aloud given English prompts). It is used in proposed mixup
for word-level augmented data generation. The statistics of pronunciation
scoring data is detailed in Table 1.

Table 1: Data statistics for pronunciation scorer.

Task-related unlabeled data Speechocean762 Train Test # of words 50,000 15,849
15,967 # of phones 168,472 47,390 47,688

###  3.2 Experimental configuration

DFSMN-HMM [31] based ASR model was adopted, which consists of 2 convolutional
layers, 24 FSMN layers, a bottleneck layer and a feedforward layer.
39-dimensional MFCC feature was extracted using a 25 ms hamming window with 10
ms shift as the model input. While, the softmax output layer had 5,432 units,
representing the senone labels derived from forced-alignment with a GMM-HMM
system. The deep features were extracted from output of the bottleneck layer
with the dimension of 512. While, the phoneme GOP scores were calculated with
frame-level senones and their corresponding posteriors. The frame accuracy of
the ASR system is 73%.

Table 2: The architecture of the proposed 1D-CNN-MLP, where
$L_{i}=(L_{i-1}+2*padding-kernel\\_size)/stride+1$, and $L_{0}$ is the frame
length of one word.

Layer Description Output Size 1 Input a sequence of phonetic features
32$\times{L_{0}}$ 2 1D-Convolution kernel_size=3, stride=1, 32 filters
32$\times{L_{1}}$ 3 Batch normalization 32$\times{L_{1}}$ 4 ReLU
32$\times{L_{1}}$ 5 Dropout probability: 0.1 32$\times{L_{1}}$ 6
1D-Convolution kernel_size=3, stride=1, 32 filters 32$\times{L_{2}}$ 7 Batch
normalization 32$\times{L_{2}}$ 8 ReLU 32$\times{L_{2}}$ 9 Dropout
probability: 0.1 32$\times{L_{2}}$ 10 Max pooling kernel_size=2,stride=2
32$\times{L_{3}}$ 11 1D-Convolution kernel_size=1, stride=1, 32 filters
32$\times{L_{4}}$ 12 Batch normalization 32$\times{L_{4}}$ 13 ReLU
32$\times{L_{4}}$ 14 Dropout probability: 0.1 32$\times{L_{4}}$ 15 Mean
pooling average time 32$\times$1 16 Feature concatenation mfcc + deep feature
64$\times$1 17 Fully connected output size: 32 32$\times$1 18 Fully connected
output size: 1 1$\times$1 19 Sigmoid predicted word score 1$\times$1

The multi-source scoring model takes sequence of frame-level 39-dim MFCC and
512-dim deep feature as its input. Nonlinear transformations are applied to
these features, and output frame-level 32-dim feature vectors, which are added
to 32-dim reference phone embeddings to get phonetic features. Subsequently,
we use 1D-CNN and MLP module to map phonetic features into word score. The
detailed configuration is listed in Table 2. The Adam optimizing algorithm is
chosen to minimize the MSE, and learning rate is set to 0.002. The word scores
of both augmented and human-labeled data are scaled to the interval from 0 to
1 for model training and evaluation.

###  3.3 System Setup

To valid the proposed ideas, we considered the state-of-the-art method as our
baseline.

  * •

No-pretrain: the scoring model is trained on the human-labeled data directly.
No pretraining process is applied;

  * •

Real-pretrain: we calculate GOP score of each word in task-related unlabeled
data, and use it to pretrain word-level scorer, which is then fine-tuned with
the labeled data in training set shown in Table 1. This pretrain and fine-tune
strategy achieved the highest PCC for sentence scoring in the recent study
[22], hence we apply it for word-level scoring and treat it as a strong
baseline;

  * •

Mixup-pretrain: The pretraining is first preformed on the task-related
unlabeled data and the artificial data generated by mixup. The model is then
fine-tuned with the labeled data in training set.

For fair comparison, both baselines and proposed system shared the same ASR
and scoring model structures.

##  4 Results and analyses

To assess the system performance, the Pearson correlation coefficient (PCC)
between machine predicted scores and human scores was used in our experiments.
A higher PCC indicates a better system performance.

###  4.1 Validate the Configurations of Proposed Method

In this section, we examined two configurations of proposed method, including
the effects of mixup data size and multi-source information for predicting
word-level pronunciation quality.

Figure 2: Performance of different amount of augmented data. The augmented data consists of both real unlabeled data (50k words) and artificial data generated with mixup. Take augmented data size of 500k as an example, it is formed with 50k real unlabeled data and 450k mixup data. Augmented data size of 50k indicates no mixup data. Table 3: Comparison of different methods. Mixup-pretrain is pretrained with 50k real unlabeled data and 450k artificial augmented word created by mixup. Methods | Real unlabeled data | PCC
---|---|---
No-pretrain | - | 0.507
Real-pretrain | 50k | 0.567
100k | 0.570
200k | 0.586
500k | 0.612
Mixup-pretrain | 50k | 0.610

####  4.1.1 Effect of Mixup Data Size

We first compare the performance of the word-level scoring over different size
of augmented data, which consists of real unlabeled data (50k words) and
artificial data generated with mixup, e.g. 50k, 100k, 200k, 500k and 1,000k
words. Figure 2 presents the PCC results with different amount of augmented
data. It is observed that the system performance improves as the augmented
data size increases for all three features. When the augmented data size
reaches 500k, the best performance is achieved, with the PCC of 0.610. After
that, we observe the PCC plateaus out for multi-source and deep features.
Therefore, we set the augmented data size as 500k words (50k unlabeled data +
450k mixup data) in the rest of the experiments.

####  4.1.2 Effect of Multi-Source Information for Scoring

Then, we examined word-level pronunciation scoring performance over different
input, e.g. MFCC, deep and multi-source feature. As shown in Figure 2, it is
observed that deep feature consistently outperforms the MFCC for pronunciation
scoring in all the tests. This observation confirms that deep feature is more
discriminative than MFCC due to utilizing extra large amount of non-native
data for its feature extractor training. The performance is further improved
by multi-source feature, which utilizing both MFCC and deep features as system
input. The results imply that MFCC and deep features are complementary to each
other, and their combination in hidden space is helpful for pronunciation
scoring.

###  4.2 Comparison with Baselines

In this section, we validate the proposed mixup by comparing against
baselines. To focus on comparison of different pretraining methods, all the
systems take multi-source feature as their input. Table 3 presents the PCC
results from different methods. Benefit from the real unlabeled non-native
data, all the real- and mixup-pretrain methods outperform the no-pretrain
counterpart. This confirms the effectiveness of the data augmentation and
pretrainining method for pronunciation scoring.

Then we compare the proposed mixup-pretrain against real-pretrain baseline.
Since the task is defined in a low-resource scenario, we fix the size of real
unlabelled data as 50k for the proposed method. Experimental results indicate
that the proposed mixup-pretrain method performs better than the real-pretrain
baseline trained with the same amount of unlabelled data when using 50k real
unlabelled data. To achieve comparable performance of real-pretrain with the
proposed mixup-pretrain, we increase the amount of real unlabeled data from
50k to 500k. From Table 3, we clearly see that the system performance improves
as the unlabeled data size increases. The results show that, the real-pretrain
baseline achieves similar performance to the proposed method when using 500k
real unlabelled data, which is 10 times of proposed method. It suggests that
the proposed mixup method offers a clear advantage over baseline under limited
data condition.

##  5 Conclusion

In this paper, phone-level mixup is proposed to make full use of smaller unit
(e.g., phone) to creates arbitrary bigger unit (e.g., word) with various
pronunciation scores. Experiments conducted on Speechocean762 show that
proposed augmented data is complementary to the real non-native data for
training word-level pronunciation scorer. Moreover, feature combination under
the proposed training framework brings extra improvement. In the future, we
will extend current phone-level mixup to word-level mixup for improving
sentence-level scoring. Meanwhile, we will also investigate whether similar
improvement could be achieved, when the proposed data augmentation method is
applied to attention-based pronunciation scorers [17, 21].

## References

  * [1] R. Ellis and R. R. Ellis, _The study of second language acquisition_. Oxford University, 1994\.
  * [2] H. Meng, “Developing speech recognition and synthesis technologies to support computer-aided pronunciation training for Chinese learners of English,” in _Proceedings of the 23rd Pacific Asia Conference on Language, Information and Computation, Volume 1_ , 2009, pp. 40–42.
  * [3] H. Franco, L. Neumeyer, Y. Kim, and O. Ronen, “Automatic pronunciation scoring for language instruction,” in _ICASSP_. IEEE, 1997.
  * [4] S. M. Witt and S. J. Young, “Phone-level pronunciation scoring and assessment for interactive language learning,” _Speech communication_ , vol. 30, no. 2-3, pp. 95–108, 2000.
  * [5] J. Zheng, C. Huang, M. Chu, F. K. Soong, and W.-p. Ye, “Generalized segment posterior probability for automatic mandarin pronunciation evaluation,” in _2007 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , vol. 4. IEEE, 2007, pp. IV–201.
  * [6] W. Hu, Y. Qian, F. K. Soong, and Y. Wang, “Improved mispronunciation detection with deep neural network trained acoustic models and transfer learning based logistic regression classifiers,” _Speech Communication_ , vol. 67, pp. 154–166, 2015.
  * [7] J. Shi, N. Huo, and Q. Jin, “Context-aware goodness of pronunciation for computer-assisted pronunciation training,” in _Proc. Interspeech_ , 2020\.
  * [8] W. Li, N. F. Chen, S. M. Siniscalchi, and C.-H. Lee, “Improving Mispronunciation Detection for Non-Native Learners with Multisource Information and LSTM-Based Deep Models.” in _Proc. Interspeech_ , 2017, pp. 2759–2763.
  * [9] W.-K. Leung, X. Liu, and H. Meng, “CNN-RNN-CTC based end-to-end mispronunciation detection and diagnosis,” in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 8132–8136.
  * [10] B.-C. Yan, M.-C. Wu, H.-T. Hung, and B. Chen, “An end-to-end mispronunciation detection system for L2 English speech leveraging novel anti-phone modeling,” in _Proc. Interspeech_ , 2020, pp. 3032–3036.
  * [11] Y. Feng, G. Fu, Q. Chen, and K. Chen, “SED-MDD: Towards sentence dependent end-to-end mispronunciation detection and diagnosis,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 3492–3496.
  * [12] S. Wei, G. Hu, Y. Hu, and R.-H. Wang, “A new method for mispronunciation detection using support vector machine based on pronunciation space models,” _Speech Communication_ , vol. 51, no. 10, pp. 896–905, 2009.
  * [13] X. Qian, H. Meng, and F. K. Soong, “The use of DBN-HMMs for mispronunciation detection and diagnosis in L2 English to support computer-aided pronunciation training,” in _Thirteenth Annual Conference of the International Speech Communication Association_ , 2012.
  * [14] K. Kyriakopoulos, K. Knill, and M. Gales, “Automatic detection of accent and lexical pronunciation errors in spontaneous non-native English speech,” in _Proc. Interspeech_ , 2020.
  * [15] A. Lee and J. Glass, “Pronunciation assessment via a comparison-based system,” in _Speech and Language Technology in Education_ , 2013.
  * [16] A. Lee _et al._ , “Language-independent methods for computer-assisted pronunciation training,” Ph.D. dissertation, Massachusetts Institute of Technology, 2016.
  * [17] D. Korzekwa, J. Lorenzo-Trueba, T. Drugman, S. Calamaro, and B. Kostek, “Weakly-supervised word-level pronunciation error detection in non-native English speech,” pp. 4408–4412, 2021.
  * [18] T. Cincarek, R. Gruhn, C. Hacker, E. Nöth, and S. Nakamura, “Automatic pronunciation scoring of words and sentences independent from the non-native’s first language,” _Computer Speech & Language_, vol. 23, no. 1, pp. 65–88, 2009.
  * [19] Z. Yu, V. Ramanarayanan, D. Suendermann-Oeft, X. Wang, K. Zechner, L. Chen, J. Tao, A. Ivanou, and Y. Qian, “Using bidirectional LSTM recurrent neural networks to learn high-level abstractions of sequential features for automated scoring of non-native spontaneous speech,” in _ASRU_. IEEE, 2015, pp. 338–345.
  * [20] L. Chen, J. Tao, S. Ghaffarzadegan, and Y. Qian, “End-to-end neural network based automated speech scoring,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 6234–6238.
  * [21] B. Lin, L. Wang, X. Feng, and J. Zhang, “Automatic Scoring at Multi-Granularity for L2 Pronunciation.” in _Proc. Interspeech_ , 2020, pp. 3022–3026.
  * [22] B. Lin and L. Wang, “Deep Feature Transfer Learning for Automatic Pronunciation Assessment,” _Proc. Interspeech_ , pp. 4438–4442, 2021.
  * [23] J. Zhang, Z. Zhang, Y. Wang, Z. Yan, Q. Song, Y. Huang, K. Li, D. Povey, and Y. Wang, “speechocean762: An open-source non-native english speech corpus for pronunciation assessment,” in _Proc. Interspeech_ , 2021.
  * [24] G. Zhao, S. Sonsaat, A. Silpachai, I. Lucic, E. Chukharev-Hudilainen, J. Levis, and R. Gutierrez-Osuna, “L2-arctic: A non-native english speech corpus.” in _Proc. Interspeech_ , 2018, pp. 2783–2787.
  * [25] N. F. Chen, D. Wee, R. Tong, B. Ma, and H. Li, “Large-scale characterization of non-native Mandarin Chinese spoken by speakers of European origin: Analysis on iCALL,” _Speech Communication_ , vol. 84, pp. 46–56, 2016.
  * [26] D. Korzekwa, R. Barra-Chicote, S. Zaporowski, G. Beringer, J. Lorenzo-Trueba, A. Serafinowicz, J. Droppo, T. Drugman, and B. Kostek, “Detection of lexical stress errors in non-native (l2) english with data augmentation and attention,” in _Proc. Interspeech_ , 2021.
  * [27] H. Zhang, M. Cissé, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” in _6th International Conference on Learning Representations, ICLR_ , 2018.
  * [28] Y. Kim, “Convolutional Neural Networks for Sentence Classification,” _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_ , 08 2014.
  * [29] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in _2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2015, pp. 5206–5210.
  * [30] H. Zhang, K. Shi, and N. Chen, “Multilingual speech evaluation: Case studies on english, malay and tamil,” _Proc. Interspeech_ , pp. 4443–4447, 2021\.
  * [31] S. Zhang, M. Lei, Z. Yan, and L. Dai, “Deep-FSMN for large vocabulary continuous speech recognition,” in _ICASSP 2018-2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 5869–5873.