# Indirect Control of the 29SiV- Nuclear Spin in Diamond

Hyma H. Vallabhapurapu<EMAIL_ADDRESS>School of Electrical
Engineering and Telecommunications, The University of New South Wales, Sydney,
NSW 2052, Australia  Chris Adambukulam  School of Electrical Engineering and
Telecommunications, The University of New South Wales, Sydney, NSW 2052,
Australia  Andre Saraiva  School of Electrical Engineering and
Telecommunications, The University of New South Wales, Sydney, NSW 2052,
Australia  Arne Laucht<EMAIL_ADDRESS>School of Electrical Engineering
and Telecommunications, The University of New South Wales, Sydney, NSW 2052,
Australia

###### Abstract

Coherent control and optical readout of the electron spin of the 29SiV- center
in diamond has been demonstrated in literature, with exciting prospects for
implementations as memory nodes and spin qubits. Nuclear spins may be even
better suited for many applications in quantum information processing due to
their long coherence times. Control of the 29SiV- nuclear spin using
conventional NMR techniques is feasible, albeit at slow kilohertz rates due to
the nuclear spin’s low gyromagnetic ratio. In this work we theoretically
demonstrate how indirect control using the electron spin-orbit effect can be
employed for high-speed, megahertz control of the 29Si nuclear spin. We
discuss the impact of the nuclear spin precession frequency on gate times and
the exciting possibility of all optical nuclear spin control.

##  I  Introduction

FIG. 1: Nuclear spin quantization field. (a) Ground state energy levels
showing Zeeman-split electron spin states. Relevant electron spin transitions
for the zero-strain and moderately strained cases are shown as red and blue
arrows, respectively. (b) The primary $\boldsymbol{\rm{B_{\rm\alpha}}}$ and
secondary $\boldsymbol{\rm{B_{\rm\beta}}}$ quantization fields represent the
net fields ($\boldsymbol{\rm{B_{\rm hf}}}$ \+ $\boldsymbol{\rm{B_{0}}}$)
experienced by the nuclear spin as a result of the electron spin orientations.
Here, $\Delta\theta$ quantifies the change in the net quantization field
direction of the nuclear spin as a result of electron spin $\pi$-rotations.
(c) Plot of $|90^{\circ{}}-\Delta\theta|$ as a function of the
$\boldsymbol{\rm{B_{0}}}$ vector for the zero-strain case, illustrating the
deviation from orthogonal nuclear spin eigenstate orientations. The dark
region represents the ideal operating parameters for the
$\boldsymbol{\rm{B_{0}}}$ field so as to maximize $\Delta\theta$. (d) Same as
(c) but for the moderately strained case. The labels ‘A’ and ‘B’ indicate the
$\boldsymbol{\rm{B_{0}}}$ set-point parameters used for the simulations
described in Table 2 in Sec. III.1.

The state of the art in diamond-based spin qubit implementations includes low
temperature nanophotonics and coherent control of negatively charged group IV
vacancies, as a result of their favourable optical and coherence properties at
low temperatures ($\leq$4 K) [1, 2, 3, 4]. In particular, much progress has
been made with the negatively charged silicon vacancy (SiV-) color center in
diamond, which has emerged as a promising candidate for the implementation of
a spin qubit [5, 6, 7, 8, 9, 10].

Both electron and nuclear spins have long been proposed for applications as
qubits for quantum information processing [11, 12, 13, 14]. Nuclear spins are
particularly suited for such applications due to their excellent isolation
from the environment, resulting in long coherence times. Typical approaches to
access the nuclear spins in diamond color centers involve finding a vacancy
that is either hyperfine coupled to a nearby randomly occurring 13C atom [15,
16, 17, 18] or to the host nuclear spin intrinsic to the vacancy center [19,
5]. A relevant example is the 29SiV- center consisting of a 29Si isotope with
a nuclear spin $I=1/2$ [8, 5], which is the focus of this work. Such intrinsic
nuclear spins are preferred for scalability [20] and take advantage of
available precision defect implantation techniques in diamond [21, 22].

Fast, coherent control of the 29Si nuclear spin combined with the favourable
optical readout properties of the SiV- centers are envisioned to present
robust implementations of spin qubit platforms for information processing.
Multiple methods exist for coherent nuclear control, including conventional
nuclear magnetic resonance (NMR) and recently nuclear electrical resonance
(NER) [23]. However, these control methods are usually slow, and fast nuclear
spin control of the 29SiV- system presents a practical problem worth
addressing. Firstly, electrical control is only possible for nuclei with spin
$I>1/2$, eliminating the possibility for use with 29SiV-. Secondly, the low
gyromagnetic ratio of the host nuclear spin is roughly three orders of
magnitude smaller than that of the electron, restricting the achievable
nuclear Rabi frequencies using direct RF control to the kilohertz range. This
is especially true for low temperature operation, where the usable microwave
power is limited due to the maximally permissible thermal load of the
cryostat. In contrast, electron spins can be driven at megahertz Rabi
frequencies [24, 25, 26, 27], and approaches exist to exploit the relative
ease of high speed electron spin control to actuate on the nuclear spin
towards a desired state via the hyperfine interaction, in a technique called
indirect control (IC) [28, 29, 30].

In the past, indirect control has relied on the anisotropic hyperfine
interaction as a result of electron-nuclear dipolar interaction with nearby
13C nuclei [31, 32, 33, 34]. However, the hyperfine coupling in the 29SiV-
defect is known to be dominated by the isotropic contact hyperfine term [8];
the anisotropic contribution to the hyperfine coupling is $\sim$2.7 MHz, which
is relatively minor when compared to the isotropic contribution of $\sim$75.3
MHz [35, 36]. Fortunately, the 29SiV- center possesses a significant spin-
orbit coupling ($\sim$46 GHz [37]), due to which the electron-nuclear spin
system exhibits similar anisotropic behaviour under an applied magnetic field
$\boldsymbol{\rm{B_{0}}}$ – making indirect control a possibility.

In this work, we show via simulation of the spin Hamiltonian that fast control
of the 29SiV- host nuclear spin is possible using the IC technique. We first
explore the parameters of the 29SiV- Hamiltonian that enable indirect control
of the nuclear spin in Sec. II. Since strain has a particularly significant
impact on how the SiV qubit is implemented [38, 5], we explore both the
unstrained and strained cases. In Sec. III we use simulations to explicitly
demonstrate several single qubit IC gates. Lastly, in Sec. IV we discuss the
effect of the $\boldsymbol{\rm{B_{0}}}$ field intensity and angle on the gate
durations, along with possible applications of IC implemented using all-
optical spin control methods.

FIG. 2: Eigenstate orientations for the electron $\ket{\uparrow,\downarrow}$
and nuclear $\ket{\Updownarrow}$ spins. The $\ket{\Updownarrow}$ symbol
indicates the presence of both the $\ket{\Uparrow}$ and $\ket{\Downarrow}$
nuclear spin states, as they are always oriented anti-parallel. (a) Plot of
the energy levels and spin eigenstate orientations for the zero-strain case,
as a function of the $\boldsymbol{\rm{B_{0}}}$ field applied at an angle of
$54.7^{\circ{}}$ away from the SiV axis. (b) Same as (a) but for the 150 GHz
strain case. Note that the hyperfine splitting is not visible due to the scale
of the vertical axes.  FIG. 3: Change in nuclear spin quantization direction.
$\Delta\theta$ values are plotted as a function of the
$\boldsymbol{\rm{B_{0}}}$ field with the vector oriented $54.7^{\circ}$ away
from the SiV axis.

##  II The 29SiV-Hamiltonian

###  II.1 Interaction Terms

We begin by defining the optical ground state of 29SiV- Hamiltonian in the lab
frame. The combined Hilbert space is a product of the ground state orbitals,
and the electron and nuclear spin basis states. The total Hamiltonian is
expressed in Eq. 1, and the relevant interaction terms for the electron
($\textbf{H}^{\rm e}$) & nuclear ($\textbf{H}^{\rm n}$) Zeeman, hyperfine
coupling ($\textbf{H}^{\rm hf}$), spin-orbit coupling ($\textbf{H}^{\rm SO}$)
and strain ($\textbf{H}^{\rm str}$ defined in the SiV coordinate frame [39])
are expressed in Eqs. (2) to (6). The z axis is defined along the main
symmetry axis of the SiV defect $\langle 111\rangle$. Here,
$\boldsymbol{\rm{S}}=[S_{x}~{}S_{y}~{}S_{z}]^{\intercal}$ and
$\boldsymbol{\rm{I}}=[I_{x}~{}I_{y}~{}I_{z}]^{\intercal}$ are proportional to
the Pauli operators for the electron and nuclear spins in the combined Hilbert
space. The operators $\boldsymbol{\rm{L}}=[L_{x}~{}L_{y}~{}L_{z}]^{\intercal}$
are the orbital angular momentum operators and are expressed in the spin-orbit
basis of $e_{\pm}$; the operators $L_{x}$ & $L_{y}$ vanish as they only couple
the basis states to energy levels much higher in energy [40]. The static
magnetic field vector is defined as
$\boldsymbol{\rm{B_{0}}}=[B_{x}~{}B_{y}~{}B_{z}]^{\intercal}$. The strain
Hamiltonian is also expressed in the spin-orbit basis using the transformation
operator $\mathds{T}$ as shown in Eq. (6) [39]. The relevant parameters of the
Hamiltonian are summarized in Table 1.

$\textbf{H}=\textbf{H}^{\rm e}+\textbf{H}^{\rm n}+\textbf{H}^{\rm
hf}+\textbf{H}^{\rm SO}+\textbf{H}^{\rm str},$ (1)

where:

$\textbf{H}^{\rm
e}=\gamma_{e}(\boldsymbol{\rm{B_{0}}}\cdot{}\boldsymbol{\rm{S}})+\gamma_{L}qL_{z}B_{z},$
(2) $\textbf{H}^{\rm n}=-\gamma_{n}(\boldsymbol{\rm{B_{0}}}\cdot{}\bf{I}),$
(3) $\textbf{H}^{\rm
hf}=A_{||}{S_{z}}{I_{z}}+A_{\perp}({S_{x}}{I_{x}}+{S_{y}}{I_{y}}),$ (4)
$\textbf{H}^{\rm SO}=-\lambda_{\rm SO}L_{z}S_{z},$ (5) $\textbf{H}^{\rm
str}=\mathds{T}\left(\begin{bmatrix}\alpha&\beta\\\
\beta&-\alpha\end{bmatrix}\otimes\mathds{1}_{4}\right)\mathds{T}^{-1},\\\
\mathds{T}=\begin{bmatrix}-1&-i\\\ 1&-i\end{bmatrix}\otimes\mathds{1}_{4}$ (6)

Table 1: Hamiltonian physical constants for 29SiV-. Parameter | Value | Description
---|---|---
$\gamma_{e}$ | 28 GHz/T | Electron gyromagnetic ratio
$\gamma_{n}$ | -8.46 MHz/T | Nuclear gyromagnetic ratio
$A_{||}$ | 70 MHz | Hyperfine $||$ SiV--axis [8]
$A_{\perp{}}$ | 78 MHz | Hyperfine $\perp$ SiV--axis [35, 36]
$q$ | 0.1 | Orbital quenching factor
$\gamma_{L}$ | $\gamma_{e}/2$ | Orbital gyromagnetic ratio
$\lambda_{\rm SO}$ | 46 GHz | Ground state SO coupling [37]
$\alpha$ | 150 GHz |  $\frac{1}{2}(\epsilon_{xx}-\epsilon_{yy})$ (in-plane
|  | strain anisotropy)
$\beta$ | 150 GHz |  $\epsilon_{xy}$ (shearing strain)

In this work we investigate both the unstrained ($\alpha=\beta$= 0 GHz) and
strained ($\alpha=\beta$= 150 GHz) regimes of the 29SiV- Hamiltonian, as they
can lead to different implementations of the spin qubit, i.e., by either
driving the spin-orbit separated transition $\omega_{1}$ or the mixed orbital
spin transitions $\omega_{2}$ [see Fig. 1(a)] [5]. This is a relevant
consideration as it is possible to strain engineer [38] the silicon vacancy
color centers, or fabricate them with naturally unstrained (or minimally
strained) [40] properties. The values of strain were chosen such that there is
sufficient mixing between orbitals, allowing electron spin transitions between
spin states. Such values of strain allow the electron spin to undergo a full
$\pi$-rotation for an optimised $\boldsymbol{\rm{B_{0}}}$ field, which need
not be re-optimised for higher strain values. The relevant electron spin
transitions of interest are illustrated in Fig. 1(a) for the unstrained and
strained Hamiltonians, with transition labeled $\rm\omega_{1}$ and
$\rm\omega_{2}$, respectively. These transitions represent electron
$\pi$-rotations between eigenstates of the Hamiltonian. Lastly, the
contribution of the Jahn-Teller interaction strength is not considered due a
relatively negligible strength compared to the $\lambda_{SO}$ and strain [40].

###  II.2 Spin Quantization Axis

The quantization axis of the electron spin depends primarily on the net
magnetic field vector $\boldsymbol{\rm{B_{net}}}$[41] – the sum of the applied
field $\boldsymbol{\rm{B_{0}}}$, the field induced by spin-orbit coupling
$\boldsymbol{\rm{B_{\rm SO}}}$, and the hyperfine field via coupling to the
29Si nucleus. Thus, with $|$$\boldsymbol{\rm{B_{0}}}$$|=0$ T or with
$\boldsymbol{\rm{B_{0}}}$ $\parallel$ [1 1 1], the effective quantization
field for the electron and nucleus remains parallel to the defect axis.
However, when misalignment of the $\boldsymbol{\rm{B_{0}}}$ field is
introduced, the effective quantization field is no longer parallel to
$\boldsymbol{\rm{B_{0}}}$ due to the competition between the
$\boldsymbol{\rm{B_{0}}}$ and $\boldsymbol{\rm{B_{\rm SO}}}$ vectors [40, 5].
The quantization field of the electron with respect to the defect symmetry
axis is thus expected to vary as a function of the applied
$\boldsymbol{\rm{B_{0}}}$ field. In comparison, the electron is not
significantly affected by the magnetic moment of the nucleus as it results in
a negligible hyperfine field of $\sim 3$ mT, as felt by the electron.

On the other hand, the electron spin orientation has a significant effect on
the nuclear spin orientation due to the electron’s larger magnetic moment,
resulting in a hyperfine field of $|$$\boldsymbol{\rm{B_{\rm hf}}}$$|\approx$
4.14 T, as felt by the 29Si nucleus. Thus, any change in the orientation of
the electron spin also significantly affects the quantization field of the
nuclear spin. Moreover, as the spin-orbit coupling is a relativistic effect
experienced by the orbiting electron, the nuclear spin only experiences the
applied $\boldsymbol{\rm{B_{0}}}$ field and the hyperfine field
$\boldsymbol{\rm{B_{\rm hf}}}$. In summary, the main effect is that the
nuclear spin eigenstate orientations are no longer strictly parallel to each
other when the electron spin is flipped. This is the origin of the anisotropy-
like effect of the nuclear spin quantization fields, captured by the angle
$\Delta\theta$, which enables indirect control to become possible even in the
absence of explicit hyperfine anisotropy. The relevant static field vectors of
interest are illustrated in Fig. 1(b).

We investigate the change in orientation of the quantization fields of the
nuclear spin represented by $\Delta\theta$ as a result of the relevant
electron spin flipping transitions
$\ket{\uparrow}\leftrightarrow\ket{\downarrow}$, illustrated in Fig. 1(a). We
plot $\Delta\theta$ as a function of the $\boldsymbol{\rm{B_{0}}}$ field
intensity and orientation for these transitions in Fig. 1(c) & (d), for the
unstrained and strained Hamiltonians, respectively. Closer inspection of these
plots shows that there are ideal parameters for the $\boldsymbol{\rm{B_{0}}}$
field that result in $\Delta\theta$ close to $90^{\circ}$. This is favourable
as the efficiency of the indirect control technique is optimised by maximising
this angle [28].

As an example we illustrate the exact spin eigenstates of the Hamiltonian,
showing the quantization axes orientations in Fig. 2, with respect to the
29SiV- main symmetry axis. These orientations are plotted with respect to the
defect axis, for both cases of strain and with the $\boldsymbol{\rm{B_{0}}}$
field orientated at $54.7^{\circ{}}$ (corresponding to [1 0 0] vector
orientation). Lastly, the $\Delta\theta$ values for the nuclear spin
corresponding to the relevant electron transitions [illustrated in Fig. 1(a)]
are plotted in Fig. 3 as a function of $|$$\boldsymbol{\rm{B_{0}}}$$|$ applied
at an angle $54^{\circ{}}$ away from the 29SiV- symmetry axis.

FIG. 4: Indirect control (IC) of the nuclear spin. (a) Series of electron
$\pi$-rotations separated by delays $\tau$ constituting a typical IC-gate
sequence. (b) (i-iii) Bloch sphere representations of the nuclear spin
trajectories, showing the action of the X gate on the nuclear spin initialised
in the $\ket{\Uparrow}$, $\ket{+X}$ & $\ket{+Y}$ states, respectively. The
time evolved trajectories of the spin in the rotating frame are plotted in
solid red on the Bloch sphere. Green arrows represent the initial state of the
nuclear spin before the gate sequence is applied, whereas blue arrows
represent the final state. (iv-viii) Demonstrations of Y, Z, Hadamard, S-phase
and T-phase gates.

##  III Indirect Control

For a proof-of-principle demonstration we assume an unstrained SiV center in a
[1 0 0] diamond sample, resulting in a $\boldsymbol{\rm{B_{0}}}$ angle of
$54.7^{\circ}$ relative to the defect symmetry axis. Informed by Fig. 3, we
choose $|$$\boldsymbol{\rm{B_{0}}}$$|=3.5$ T such that the change in
quantization field orientation $\Delta\theta$ is $120^{\circ{}}$, which is
equivalent to $\Delta\theta=60^{\circ{}}$. Such an angle allows for complete
control of the nuclear spin on the Bloch-sphere with just four electron
$\pi$-rotations [28], enabling us to construct the required IC gate. The
advantage of limiting IC gates to using electron $\pi$-rotations is twofold:
(i) the gate construction and optimisation is less computationally intensive
and (ii) the electron spin is mostly in an eigenstate and is thus not greatly
subject to dephasing, which would ultimately propagate to the nuclear spin
through the hyperfine coupling. Although there are methods to implement self-
protected IC gates using dynamical decoupling techniques in the presence of
noise [42], this would introduce additional complexity which is not required
for our demonstration.

A sequence of electron $\pi$-rotations and free precession delays
($\tau_{1-4}$) [illustrated in Fig. 4(a)] is used to actuate on the nuclear
spin, evolving the system towards a deterministic target spin state. The gate
sequence optimisation is performed with Matlab’s ‘Surrogate Optimisation’
solver, using an optimisation routine that considers the overlap of the
system’s final time-evolved optimised state $|\psi_{\rm opt}\rangle$ and the
target state $|\psi_{\rm T}\rangle$ given as $|\langle\psi_{\rm opt}|\psi_{\rm
T}\rangle|^{2}$, along with the total gate time as the optimisation goal. We
track the time evolution of the nuclear spin in its rotating reference frame,
in order to properly derive the correct phase manipulation of the qubit gates.

###  III.1 Single Qubit Gates

Two-axes control of a qubit requires control of both the zenith and azimuthal
angles over the Bloch sphere. Conventionally, phase control of a spin qubit is
performed using IQ modulation of the MW or RF pulses [43]– which is not
possible in the absence of such oscillating fields. Importantly, simple
precession of the nuclear spin around the primary quantization axis merely
results in a global phase accumulation which is not relevant for quantum
information processing. Fortunately, as we show here, phase control of the
nuclear spin does not impose any further requirements on the electron spin
control apart from simple $\pi$-rotations. In this way, the phase is
accumulated as the spin precesses around the secondary quantization axis which
detunes the spin from the resonance frequency.

We consider the application of a standard set of single qubit gates on the
nuclear spin which include phase gates, implemented in simulation as
instantaneous electron $\pi$-rotations. The nuclear spin IC gates are
numerically optimised by enforcing that two seperate qubits each initialised
in a different state correctly evolve to their respective final states in
their rotating frames. Examples of gate actions on a pair of initial states
are $\ket{+X}\leftrightarrow\ket{+X}$ and
$\ket{\Uparrow}\leftrightarrow\ket{\Downarrow}$ for the X gate, where
$\ket{\pm{}X}=(\ket{\Uparrow}\pm\ket{\Downarrow})/\sqrt{2}$.

Further, it is enforced that gate operations may only be performed when the
lab frame is in phase with the rotating frame. This is to account for the fact
that the quantization fields remain stationary in the lab frame. To achieve
this, an appropriate delay is padded to the end of the optimised gate
sequence, such that the rotating frame returns to phase with the lab frame at
the end of the sequence. This padding delay is given as
$\tau_{4}=\frac{\left\lceil T_{\rm gate}f_{\rm RF}\right\rceil}{f_{\rm
RF}}-T_{\rm gate}$, where $f_{\rm RF}$ is the nuclear spin precession
frequency, $T_{\rm gate}$ is the total optimised gate time and the
$\left\lceil\right\rceil$ brackets represent a ceiling function. This ensures
that there is a unique optimised gate sequence for a given gate operation.

We demonstrate the following gates: X, Y, Z, H, S & T, with the gate durations
summarised in Table 2 for the unstrained (‘A’) and strained (‘B’) Hamiltonian
parameters, also indicated in Fig. 3. The gates are simulated with high
fidelity (F $\geq 0.98$), with diminishing returns expected for longer
optimisation routines.

We find the total gate times to be at the order of hundreds of nanoseconds,
significantly faster than those that can be achieved with NMR techniques. For
example, the length of the IC Y gate for the unstrained Hamiltonian (‘A’) is
equivalent to that achieved via an NMR Rabi frequency of $\sim 1.6$ MHz. Such
high speed control would typically require an applied radio-frequency $B_{1}$
field of $\sim 400$ mT, which is impossible to achieve in millikelvin
cryostats. Note that the gate times are generally higher for the strained case
despite $\Delta\theta=90^{\circ{}}$. This is because the gate durations are
limited by lower precession frequencies of the nuclear spin as discussed in
Sec. IV.2.

Lastly, we can find the inverse gates S-1 & T-1 for uncomputation operations
[44] using the same method described in Sec. III.1.

Table 2: Simulated single qubit IC gate parameters and durations for the
nuclear spin in the unstrained and strained Hamiltonians corresponding to
labels A & B, respectively, indicated in Fig. 3 & Fig. 1(c),(d). The average
simulated gate fidelity F is over 0.98.

Strain | Gate | $\tau_{1}$ (ns) | $\tau_{2}~{}(ns)$ | $\tau_{3}~{}(ns)$ | $\tau_{4}~{}(ns)$ | Total | F
---|---|---|---|---|---|---|---
Case | (ns)
Unstrained (A) | X | 2.99 | 172.96 | 22.66 | 10.46 | 209.06 |  $\geq$0.99
Y | 11.73 | 197.03 | 84.09 | 20.73 | 313.59 |  $\geq$0.99
Z | 21.34 | 96.27 | 21.29 | 17.89 | 156.79 |  $\geq$0.99
H | 29.33 | 111.23 | 10.55 | 5.69 | 156.79 |  $\geq$0.99
S | 5.18 | 90.85 | 35.24 | 25.52 | 156.79 |  $\geq$0.99
S-1 | 31.73 | 238.11 | 1.62 | 42.13 | 313.59 |  $\geq$0.99
T | 4.49 | 88.29 | 34.56 | 29.47 | 156.79 |  $\geq$0.99
T-1 | 90.26 | 184.23 | 60.14 | 31.22 | 365.86 |  $\geq$0.99
Strained (B) | X | 245.39 | 1.49 | 401.84 | 647.73 | 1296.50 |  $\geq$0.99
Y | 119.27 | 329.26 | 192.64 | 7.06 | 648.23 |  $\geq$0.98
Z | 86.42 | 1.77 | 233.88 | 326.16 | 648.23 |  $\geq$0.99
H | 28.76 | 162.03 | 196.84 | 260.61 | 648.23 |  $\geq$0.98
S | 128.51 | 1.05 | 31.65 | 487.02 | 648.23 |  $\geq$0.99
S-1 | 60.16 | 195.47 | 33.49 | 358.1 | 648.23 |  $\geq$0.99
T | 79.09 | 1.63 | 1.00 | 566.50 | 648.23 |  $\geq$0.98
T-1 | 60.14 | 223 | 60.14 | 31.22 | 648.23 |  $\geq$0.99

FIG. 5: Nuclear spin precession. (a) Nuclear spin precession periods of the
primary $\alpha$ and secondary $\beta$ quantization axes plotted as a function
of the $\boldsymbol{\rm{B_{0}}}$ field for the unstrained case. (b) Same plots
as (a) but for the strained case. These plots are to be used as reference
together with Fig. 1(c) & (d).

##  IV Discussion

###  IV.1 High-speed electron $\pi$-rotations

In this work we have shown simulations wherein instantaneous electron
$\pi$-rotations are used to actuate on the nuclear spin orientation. Fast
electron switching is required to minimally disturb both the electron and
nuclear spins during the switching period, and to keep the electron switching
fast compared to the nuclear spin timescales. Practical realisations of such
high-speed electron Rabi frequencies involve high microwave frequency $B_{1}$
fields, which though achievable, would require high-Q factor resonators with
contradictorily long ring-down times (order of 100 ns). This limits the
suitability of magnetic resonance techniques for coherent electron spin
control. Alternatively, high electron Rabi frequencies can be achieved using
coherent acoustic control via surface acoustic waves [26], or using coherent
all-optical control of the electron spin state [27, 45].

All-optical control is particularly favourable, as it does not involve the
need for fabrication of an on-chip antenna or acoustic transducer. This would
allow for precise, targeted nuclear spin qubit control, taking advantage of
the state-of-the-art optical confocal microscopy methods [46]. The use of
nanowatt scale optical power renders this method readily compatible with ultra
low temperature cryostats such as dilution refrigerators with a cooling power
of 10-500 $\mu{}$W [47]. Lastly, optical manipulation of the electron spin
offers the potential of picosecond timescale $\pi$-rotations [48], which
justifies the instantaneous spin rotations employed in our simulations.

###  IV.2 Nuclear gate timescale

The efficiency of indirect control directly depends on the delay times between
switching of the quantization axes, which are determined by the precession
frequency of the nuclear spin under the influence of its quantization field. A
$\boldsymbol{\rm{B_{0}}}$ field in a non-optimal orientation may negate the
benefits of indirect control. We plot the nuclear spin precession period as a
function of magnetic field orientation, for the primary and secondary
quantization axes as well as for the unstrained and strained cases in Fig. 5.
These plots are to be used as reference for optimising the
$\boldsymbol{\rm{B_{0}}}$ field parameters while also considering the
resulting change in quantization axes as shown in Fig. 1(c)&(d). For the
unstrained 29SiV- the precession times can range up to few microseconds for
both the primary and secondary quantization fields [cf. Fig. 5(a)]. Whereas,
for the strained 29SiV- the precession period is increased up to tens of
microseconds with a $\boldsymbol{\rm{B_{0}}}$ field aligned either parallel or
perpendicular to the 29SiV- defect axis [cf. Fig. 5(b)]. For both the strain
and unstrained cases, it is comparatively favourable to align the
$\boldsymbol{\rm{B_{0}}}$ field at $54.7^{\circ{}}$ as used in this work,
given that an appropriate $|$$\boldsymbol{\rm{B_{0}}}$$|$ is chosen.

##  V Conclusion

We have proposed a path towards fast and coherent control of the host nuclear
spin in the 29SiV- defect in diamond through indirect control methods using
simulations of time-optimised single qubit gates. We have shown that the large
spin-orbit interaction in the 29SiV- Hamiltonian replaces the need for an
appreciable hyperfine anisotropy required for indirect control shown in
literature. We demonstrate various simulated single qubit IC gate times for
both the unstrained and strained Hamiltonians as proof of principle,
considering that both strain cases of the silicon vacancy centers can be found
naturally or through strain engineering. By using instantaneous electron
$\pi$-rotation sequences we demonstrate one-qubit nuclear spin gates at
megahertz rates. We envision the exciting possibility of all-optical nuclear
spin control in SiV defects in diamond. Finally, we expect indirect control
techniques to be carried over to other spin systems that exhibit similar
electronic properties such as the negatively-charged 73GeV centers in diamond
[49, 50].

## Acknowledgements

We thank V.K. Sewani for useful discussions. We acknowledge support from the
Australian Research Council (CE170100012, FL190100167). H.H.V. acknowledges
support from the Sydney Quantum Academy. C.A. and A.L. acknowledge support
from the University of New South Wales Scientia program.

## References

  * Bradac  _et al._ [2019] C. Bradac, W. Gao, J. Forneris, M. E. Trusheim, and I. Aharonovich, Quantum nanophotonics with group IV defects in diamond, Nature Communications 2019 10:1 10, 1 (2019).
  * Siyushev  _et al._ [2017a] P. Siyushev, M. H. Metsch, A. Ijaz, J. M. Binder, M. K. Bhaskar, D. D. Sukachev, A. Sipahigil, R. E. Evans, C. T. Nguyen, M. D. Lukin, P. R. Hemmer, Y. N. Palyanov, I. N. Kupriyanov, Y. M. Borzdov, L. J. Rogers, and F. Jelezko, Optical and microwave control of germanium-vacancy center spins in diamond, Physical Review B 96, 081201 (2017a).
  * Debroux  _et al._ [2021] R. Debroux, C. P. Michaels, C. M. Purser, N. Wan, M. E. Trusheim, J. A. Martínez, R. A. Parker, A. M. Stramma, K. C. Chen, L. de Santis, E. M. Alexeev, A. C. Ferrari, D. Englund, D. A. Gangloff, and M. Atatüre, Quantum control of the tin-vacancy spin qubit in diamond (2021), arXiv:2106.00723 [quant-ph] .
  * Aghaeimeibodi  _et al._ [2021] S. Aghaeimeibodi, D. Riedel, A. E. Rugar, C. Dory, and J. Vučković, Electrical tuning of tin-vacancy centers in diamond, Physical Review Applied 15, 10.1103/physrevapplied.15.064010 (2021).
  * Rogers  _et al._ [2014] L. J. Rogers, K. D. Jahnke, M. H. Metsch, A. Sipahigil, J. M. Binder, T. Teraji, H. Sumiya, J. Isoya, M. D. Lukin, P. Hemmer, and F. Jelezko, All-Optical Initialization, Readout, and Coherent Preparation of Single Silicon-Vacancy Spins in Diamond, Physical Review Letters 113, 263602 (2014).
  * Sukachev  _et al._ [2017] D. D. Sukachev, A. Sipahigil, C. T. Nguyen, M. K. Bhaskar, R. E. Evans, F. Jelezko, and M. D. Lukin, Silicon-Vacancy Spin Qubit in Diamond: A Quantum Memory Exceeding 10 ms with Single-Shot State Readout, Physical Review Letters 119, 223602 (2017).
  * Becker and Becher [2017] J. N. Becker and C. Becher, Coherence Properties and Quantum Control of Silicon Vacancy Color Centers in Diamond, physica status solidi (a) 214, 1700586 (2017).
  * Pingault  _et al._ [2017] B. Pingault, D.-D. D. Jarausch, C. Hepp, L. Klintberg, J. N. Becker, M. Markham, C. Becher, and M. Atatüre, Coherent control of the silicon-vacancy spin in diamond, Nature Communications 8, 1 (2017).
  * Nguyen  _et al._ [2019a] C. T. Nguyen, D. D. Sukachev, M. K. Bhaskar, B. Machielse, D. S. Levonian, E. N. Knall, P. Stroganov, C. Chia, M. J. Burek, R. Riedinger, H. Park, M. Lončar, and M. D. Lukin, An integrated nanophotonic quantum register based on silicon-vacancy spins in diamond, Phys. Rev. B 100, 165428 (2019a).
  * Bhaskar  _et al._ [2020] M. K. Bhaskar, R. Riedinger, B. Machielse, D. S. Levonian, C. T. Nguyen, E. N. Knall, H. Park, D. Englund, M. Lončar, D. D. Sukachev, and M. D. Lukin, Experimental demonstration of memory-enhanced quantum communication, Nature 580, 60 (2020).
  * Loss and DiVincenzo [1998] D. Loss and D. P. DiVincenzo, Quantum computation with quantum dots, Physical Review A 57, 120 (1998).
  * Kane [1998] B. E. Kane, A silicon-based nuclear spin quantum computer, Nature 393, 133 (1998).
  * Wrachtrup  _et al._ [2001] J. Wrachtrup, S. Y. Kilin, and A. P. Nizovtsev, Quantum computation using the 13C nuclear spins near the single NV defect center in diamond, Optics and Spectroscopy 91, 429 (2001).
  * Wrachtrup and Jelezko [2006] J. Wrachtrup and F. Jelezko, Processing quantum information in diamond, Journal of Physics: Condensed Matter 18, 807 (2006).
  * Jelezko  _et al._ [2004] F. Jelezko, T. Gaebel, I. Popa, M. Domhan, A. Gruber, and J. Wrachtrup, Observation of Coherent Oscillation of a Single Nuclear Spin and Realization of a Two-Qubit Conditional Quantum Gate, Physical Review Letters 93, 130501 (2004).
  * Smeltzer  _et al._ [2011] B. Smeltzer, L. Childress, and A. Gali, 13C hyperfine interactions in the nitrogen-vacancy centre in diamond, New Journal of Physics 13, 025021 (2011).
  * Metsch  _et al._ [2019] M. H. Metsch, K. Senkalla, B. Tratzmiller, J. Scheuer, M. Kern, J. Achard, A. Tallaire, M. B. Plenio, P. Siyushev, and F. Jelezko, Initialization and Readout of Nuclear Spins via a Negatively Charged Silicon-Vacancy Center in Diamond, Physical Review Letters 122, 190503 (2019).
  * Abobeih  _et al._ [2019] M. Abobeih, J. Randall, C. Bradley, H. Bartling, M. Bakker, M. Degen, M. Markham, D. Twitchen, and T. Taminiau, Atomic-scale imaging of a 27-nuclear-spin cluster using a quantum sensor, Nature 576, 411 (2019).
  * Scharfenberger  _et al._ [2014] B. Scharfenberger, W. J. Munro, and K. Nemoto, Coherent control of an NV- center with one adjacent 13C, New Journal of Physics 16, 093043 (2014).
  * Fuchs  _et al._ [2011] G. D. Fuchs, G. Burkard, P. V. Klimov, and D. D. Awschalom, A quantum memory intrinsic to single nitrogen–vacancy centres in diamond, Nature Physics 7, 789 (2011).
  * Pezzagna  _et al._ [2010] S. Pezzagna, D. Wildanger, P. Mazarov, A. D. Wieck, Y. Sarov, I. Rangelow, B. Naydenov, F. Jelezko, S. W. Hell, and J. Meijer, Nanoscale Engineering and Optical Addressing of Single Spins in Diamond, Small 6, 2117 (2010).
  * Schukraft  _et al._ [2016] M. Schukraft, J. Zheng, T. Schröder, S. L. Mouradian, M. Walsh, M. E. Trusheim, H. Bakhru, and D. R. Englund, Invited Article: Precision nanoimplantation of nitrogen vacancy centers into diamond photonic crystal cavities and waveguides, APL Photonics 1, 020801 (2016).
  * Asaad  _et al._ [2020] S. Asaad, V. Mourik, B. Joecker, M. A. I. Johnson, A. D. Baczewski, H. R. Firgau, M. T. Ma̧dzik, V. Schmitt, J. J. Pla, F. E. Hudson, K. M. Itoh, J. C. McCallum, A. S. Dzurak, A. Laucht, and A. Morello, Coherent electrical control of a single high-spin nucleus in silicon, Nature 579, 205 (2020).
  * Fuchs  _et al._ [2009] G. D. Fuchs, V. V. Dobrovitski, D. M. Toyli, F. J. Heremans, and D. D. Awschalom, Gigahertz dynamics of a strongly driven single quantum spin, Science 326, 1520 (2009).
  * Vallabhapurapu  _et al._ [2021] H. H. Vallabhapurapu, J. P. Slack-Smith, V. K. Sewani, C. Adambukulam, A. Morello, J. J. Pla, and A. Laucht, Fast Coherent Control of a Nitrogen-Vacancy-Center Spin Ensemble Using a ${\mathrm{K}\mathrm{Ta}\mathrm{O}}_{3}$ Dielectric Resonator at Cryogenic Temperatures, Phys. Rev. Applied 16, 044051 (2021).
  * Maity  _et al._ [2020] S. Maity, L. Shao, S. Bogdanović, S. Meesala, Y.-I. Sohn, N. Sinclair, B. Pingault, M. Chalupnik, C. Chia, L. Zheng, K. Lai, and M. Lončar, Coherent acoustic control of a single silicon vacancy spin in diamond, Nature Communications 11, 193 (2020).
  * Becker  _et al._ [2018] J. N. Becker, B. Pingault, D. Groß, M. Gündoğan, N. Kukharchyk, M. Markham, A. Edmonds, M. Atatüre, P. Bushev, and C. Becher, All-Optical Control of the Silicon-Vacancy Spin in Diamond at Millikelvin Temperatures, Physical Review Letters 120, 053603 (2018).
  * Khaneja [2007] N. Khaneja, Switched control of electron nuclear spin systems, Physical Review A 76, 032326 (2007).
  * Zhang  _et al._ [2019] J. Zhang, S. S. Hegde, and D. Suter, Improved Indirect Control of Nuclear Spins in Diamond N-V Centers, Physical Review Applied 12, 064047 (2019).
  * Hegde  _et al._ [2020] S. S. Hegde, J. Zhang, and D. Suter, Efficient Quantum Gates for Individual Nuclear Spin Qubits by Indirect Control, Physical Review Letters 124, 220501 (2020).
  * Hodges  _et al._ [2008] J. S. Hodges, J. C. Yang, C. Ramanathan, and D. G. Cory, Universal control of nuclear spins via anisotropic hyperfine interactions, Physical Review A - Atomic, Molecular, and Optical Physics 78, 010303 (2008).
  * Nguyen  _et al._ [2019b] C. T. Nguyen, D. D. Sukachev, M. K. Bhaskar, B. Machielse, D. S. Levonian, E. N. Knall, P. Stroganov, R. Riedinger, H. Park, M. Lončar, and M. D. Lukin, Quantum Network Nodes Based on Diamond Qubits with an Efficient Nanophotonic Interface, Phys. Rev. Lett. 123, 183602 (2019b).
  * Maity  _et al._ [2022] S. Maity, B. Pingault, G. Joe, M. Chalupnik, D. Assump ç ao, E. Cornell, L. Shao, and M. Lončar, Mechanical control of a single nuclear spin, Phys. Rev. X 12, 011056 (2022).
  * Takou  _et al._ [2022] E. Takou, E. Barnes, and S. E. Economou, Multipartite electron-nuclear gates in defect systems through dynamical decoupling sequences, arXiv preprint 10.48550/ARXIV.2203.09459 (2022).
  * Nizovtsev  _et al._ [2020] A. P. Nizovtsev, S. Y. Kilin, A. L. Pushkarchuk, S. A. Kuten, N. A. Poklonski, D. Michels, D. Lyakhov, and F. Jelezko, Spatial and Hyperfine Characteristics of SiV– and SiV0 Color Centers in Diamond: DFT Simulation, Semiconductors 2020 54:12 54, 1685 (2020).
  * Goss  _et al._ [2007] J. P. Goss, P. R. Briddon, and M. J. Shaw, Density functional simulations of silicon-containing point defects in diamond, Physical Review B 76, 075204 (2007).
  * Rogers  _et al._ [2019] L. J. Rogers, O. Wang, Y. Liu, L. Antoniuk, C. Osterkamp, V. A. Davydov, V. N. Agafonov, A. B. Filipovski, F. Jelezko, and A. Kubanek, Single Si-V- Centers in Low-Strain Nanodiamonds with Bulklike Spectral Properties and Nanomanipulation Capabilities, Physical Review Applied 11, 24073 (2019).
  * Meesala  _et al._ [2018] S. Meesala, Y.-I. Sohn, B. Pingault, L. Shao, H. A. Atikian, J. Holzgrafe, M. Gündoğan, C. Stavrakas, A. Sipahigil, C. Chia, R. Evans, M. J. Burek, M. Zhang, L. Wu, J. L. Pacheco, J. Abraham, E. Bielejec, M. D. Lukin, M. Atatüre, and M. Lončar, Strain engineering of the silicon-vacancy center in diamond, Physical Review B 97, 205444 (2018).
  * Hepp [2014] C. Hepp,  _Electronic Structure of the Silicon Vacancy Color Center in Diamond_ , Ph.D. thesis, Universität des Saarlandes (2014).
  * Hepp  _et al._ [2014] C. Hepp, T. Müller, V. Waselowski, J. N. Becker, B. Pingault, H. Sternschulte, D. Steinmüller-Nethl, A. Gali, J. R. Maze, M. Atatüre, and C. Becher, Electronic Structure of the Silicon Vacancy Color Center in Diamond, Physical Review Letters 112, 036405 (2014).
  * Slichter [1996] C. P. Slichter,  _Principles of Magnetic Resonance (3rd Enlarged and Updated Edition)_ (Springer, 1996).
  * Liu  _et al._ [2013] G.-Q. Liu, H. C. Po, J. Du, R.-B. Liu, and X.-Y. Pan, Noise-resilient quantum evolution steered by dynamical decoupling, Nature Communications 2013 4:1 4, 1 (2013).
  * Sewani  _et al._ [2020] V. K. Sewani, H. H. Vallabhapurapu, Y. Yang, H. R. Firgau, C. Adambukulam, B. C. Johnson, J. J. Pla, and A. Laucht, Coherent control of NV - centers in diamond in a quantum teaching lab, American Journal of Physics 88, 10.1119/10.0001905 (2020).
  * Nielsen and Chuang [2010] M. A. Nielsen and I. L. Chuang,  _Quantum Computation and Quantum Information: 10th Anniversary Edition_ (Cambridge University Press, 2010).
  * Takou and Economou [2021] E. Takou and S. E. Economou, Optical control protocols for high-fidelity spin rotations of single $\mathrm{Si}{\mathrm{V}}^{-}$ and $\mathrm{Sn}{\mathrm{V}}^{-}$ centers in diamond, Phys. Rev. B 104, 115302 (2021).
  * Kuhlmann  _et al._ [2013] A. V. Kuhlmann, J. Houel, D. Brunner, A. Ludwig, D. Reuter, A. D. Wieck, and R. J. Warburton, A dark-field microscope for background-free detection of resonance fluorescence from single semiconductor quantum dots operating in a set-and-forget mode, Review of Scientific Instruments 84, 073905 (2013).
  * blu [2020] Bluefors dilution refrigerator, https://bluefors.com/products/ld-dilution-refrigerator/ (2020), accessed: 2020-10-19.
  * Berezovsky  _et al._ [2008] J. Berezovsky, M. H. Mikkelsen, N. G. Stoltz, L. A. Coldren, and D. D. Awschalom, Picosecond coherent optical manipulation of a single electron spin in a quantum dot, Science 320, 349 (2008).
  * Siyushev  _et al._ [2017b] P. Siyushev, M. H. Metsch, A. Ijaz, J. M. Binder, M. K. Bhaskar, D. D. Sukachev, A. Sipahigil, R. E. Evans, C. T. Nguyen, M. D. Lukin, P. R. Hemmer, Y. N. Palyanov, I. N. Kupriyanov, Y. M. Borzdov, L. J. Rogers, and F. Jelezko, Optical and microwave control of germanium-vacancy center spins in diamond, Phys. Rev. B 96, 081201 (2017b).
  * Nadolinny  _et al._ [2016] V. Nadolinny, A. Komarovskikh, Y. Palyanov, I. Kupriyanov, Y. Borzdov, M. Rakhmanova, O. Yuryeva, and S. Veber, EPR study of Si- and Ge-related defects in HPHT diamonds synthesized from Mg-based solvent-catalysts, physica status solidi (a) 213, 2623 (2016).


# Syntax-guided Localized Self-attention by Constituency Syntactic Distance

Shengyuan Hou1 Jushi Kai1∗ Haotian Xue1∗
Bingyu Zhu2 Bo Yuan2 Longtao Huang2 Xinbing Wang1 Zhouhan Lin1
1 Shanghai Jiao Tong University 2 Alibaba Group
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>∗ Equal contribution. Zhouhan Lin is the corresponding
author.

###### Abstract

Recent works have revealed that Transformers are implicitly learning the
syntactic information in its lower layers from data, albeit is highly
dependent on the quality and scale of the training data. However, learning
syntactic information from data is not necessary if we can leverage an
external syntactic parser, which provides better parsing quality with well-
defined syntactic structures. This could potentially improve Transformer’s
performance and sample efficiency. In this work, we propose a syntax-guided
localized self-attention for Transformer that allows directly incorporating
grammar structures from an external constituency parser. It prohibits the
attention mechanism to overweight the grammatically distant tokens over close
ones. Experimental results show that our model could consistently improve
translation performance on a variety of machine translation datasets, ranging
from small to large dataset sizes, and with different source languages. 111Our
code is available at https://github.com/LUMIA-Group/distance_transformer

(a) Constituency Grammar Tree

(b) Syntactic Distance

(c) Syntactic Local Range

Figure 1: (a) The constituency tree for the example sentence "I swim across
the river.". (b) Its syntactic distances. (c) The attention mask reflecting
the syntactic local ranges of each word For example, rather than attending to
the whole sequence, "across" encourages attention towards swim, the and river
while suppresses the others.

##  1 Introduction

Although Transformer doesn’t have any inductive bias on syntactic structures,
some studies have shown that it tends to learn syntactic information from data
in its lower layers Tenney et al. (2019); Goldberg (2019); Jawahar et al.
(2019). Given the pervasiveness of syntactic parsers that provides high
quality parsing results with well-defined syntactic structures, Transformers
may not need to re-invent this wheel if grammar structures could be directly
incorporated into it.

Prior to Transformer Vaswani et al. (2017), earlier works have demonstrated
that syntactic information could be helpful for various NLP tasks. For
example, Levy and Goldberg (2014) introduced dependency structure into word
embeddings, and Chen et al. (2017) uses Tree-LSTMs to process the grammar
trees for machine translation.

More recently, dependency grammar has been successfully integrated into
Transformer in various forms. Strubell et al. (2018) improves semantic role
labelling by restricting tokens to only attend to its dependency parent. Zhang
et al. (2020) modifies BERT Devlin et al. (2019) for named entity recognition
as well as GLUE tasks Wang et al. (2018) by adding an additional attention
layer that allows every token to only attend to its ancestral tokens in the
dependency parse tree. Bugliarello and Okazaki (2020) improves machine
translation by constructing the attention weights from dependency tree, while
Li et al. (2021) masks out distant nodes in the dependency tree from
attention.

While the dependency grammar demonstrates the relation between nodes, the
constituency grammar focuses more on how a sentence is formed in a merging way
block by block. Constituency grammar contains more information about the
global structure of a sentence in a hierarchical way, which we think will
greatly improve global attention mechanism like self-attention in
Transformers. Since constituency grammar doesn’t directly reflect grammatical
relations between words and introduces new constituent nodes, integrating it
into Transformer becomes less obvious. Ma et al. (2019) explores different
ways of utilizing constituency syntax information in the Transformer model,
including positional embedding, output sequence, etc. Yang et al. (2020) uses
dual encoders to encode both source text and template yielded by constituency
grammar, at a cost of introducing a large amount of parameters.

In this work, we propose a syntax-guided localized self-attention that
effectively incorporates constituency grammar into Transformers, without
introducing additional parameters. We first serialize constituency trees
through syntactic distance Shen et al. (2018), and then select several
attention heads as grammar-aware heads in which the attention ranges of each
token are individually modulated according to their grammatical roles. the
modulated attention ranges are named as syntactic local ranges, which
prohibits the attention mechanism to overweight the grammatically distant
tokens over close ones. Experimental results show that our model could
consistently improve translation performance on a variety of machine
translation datasets, ranging from small to large dataset sizes, and with
different source languages.

##  2 Preliminary: Syntactic Distance

###  2.1 Definition

Syntactic distance Shen et al. (2018) is a serialized vector representation of
constituency grammar tree (Fig 1(a)) that is defined as:

######  Definition 2.1.

(Syntactic Distance) Given a length $n$ sentence $S=(t_{1},...,t_{n})$ and its
constituency grammar tree $T$, in which the height of the lowest common
ancestors of any pair of tokens $t_{i},t_{j}$ is noted as $h_{j}^{i}$. The
syntactic distance $D=(d_{1},...,d_{n-1})$ of this sentence could be any
vector of scalars with length $n-1$, which satisfies: $\forall i,j\in[1,n-1],$

$\text{sign}(d_{i}-d_{j})=\text{sign}(h_{i+1}^{i}-h_{j+1}^{j})$ (1)

Intuitively, syntactic distance $D$ keeps the same ranking order as the
sequence of $(h_{2}^{1},h_{3}^{2},...,h_{n}^{n-1})$, in which $h_{i+1}^{i}$ is
the height of the lowest common ancestors between pairs of consecutive words
in the sentence (See Fig. 1(b)).

###  2.2 Generation of Syntactic Distance

The syntactic distance could be generated by recursively spliting the
constituency tree in a top-down manner. According to the merging order of
constituency syntactic tree, for any subtree T, the subtrees rooted by every
child node of T must be constructed at first. Therefore, the merging of all of
T’s child nodes must take place afterwards. The syntax distance in all the
subtrees of T can be calculated first, and then the maximum distance value
plus 1 is the current merging distance order.

During preprocessing, the syntactic distance is calculated on different
datasets respectively. For each sentence, we first concatenate all BPE word
segmentations, then analyze the syntax tree structure using the Stanford
corenlp toolkit Manning et al. (2014), and calculate the syntactic distance
according to the algorithm in Algorithm 1. When filling in the syntactic
distance between words in BPE word segmentation, the lowest value is 0, and
finally all syntactic distances are added by 1 to become a syntactic distance
vector with a minimum value of 1. In the case of multiple sentences, we
generate the syntactic distance of each sentence respectively, and then fill
in the maximum 999 between different sentences, indicating that all sentences
are merged at last.

Algorithm 1 Syntactic Distance Generation

1:Constituency Grammar Tree $T$

2:Syntactic Distance $D=(d_{1},...,d_{n})$

3:function distance(root)

4: $\bm{d}\leftarrow[\ ]$

5: if root is not leaf then //Node can be split

6: $d\\_set\leftarrow\\{\ \\}$

7: $maxd\leftarrow 0$

8: for c in root’s child do

9: $\bm{c_{d}}\leftarrow Distance(c)$

10: $maxc\leftarrow\max(c_{d,1},...,c_{d,|\bm{c_{d}}|})$

11: $d\\_set\leftarrow d\\_set\cup\\{\bm{c_{d}}\\}$

12: $maxd\leftarrow\max(maxd,maxc)$

13: end for

14: $nd\leftarrow maxd+1$

15: $\bm{d}\leftarrow[\bm{c_{d1}};nd;\bm{c_{d2}};...;nd;\bm{c_{d|d\\_set|}}]$

16: end if

17: return $\bm{d}$;

18:end function

##  3 Method

We are going to present a form of local self-attention that dynamically
controls each word’s attention range according to its syntactic role in the
sentence (See Fig. 1(c)). Attention heads that incorporate this localized
self-attention could significantly outweight the grammatically close tokens
over distant ones, thus incorporate the syntax information as prior knowledge.

###  3.1 Syntactic Local Range

It is widely believed that in a grammar tree, sibling words within a certain
constituent is more correlated than words that are more distant Klein and
Manning (2003). For example, in Fig. 1(a), for the word the, the word river
locates to its right is more related than the word I. More generally, we
define a syntactic local range for each of the words to mark a set of adjacent
words as more syntactically correlated to it. For a given token $t$, we
consider the words that are 1) direct siblings of $t$; or 2) left/right
siblings of the constituent that has $t$ on its left/right boundary
respectively, as syntactically more correlated to $t$. Note that, depending on
the structure of the grammar tree, the syntactic local range to the left of
the word could be different with that to the right. Formally, we have the
following definitions.

######  Definition 3.1.

(Syntactic Local Range) For a given token $t$, let $f$ be its parent node in
the constituency tree. Syntactic local range is defined as the concatenation
of two ranges residing on $t$’s left and right sides, corresponding to its
pre-text and post-text directions, respectively.

For the pre-text direction,

  * •

If $t$ is not the leftmost child of $f$, then $t$’s syntactic local range on
its left side starts at $f$’s leftmost child, and stops at $t$.

  * •

If $t$ is the leftmost child of $f$, back-trace its ancestors to find a
nearest constituent $v$ where $v$ is not the leftmost child of its parent $w$.
Then $t$’s syntactic local range on its left side starts at $w$’s leftmost
child and stops at $t$.

Similar rules apply for the pos-text direction except that the left/right
directions are altered accordingly.

###  3.2 Inducing Syntactic Local Range from Syntactic distance

Intuitively, for a given token $t$, its syntactic local range keeps stretching
on both pre-text and post-text directions until it hits a distance that is
larger than $t$’s on the corresponding side. We can represent all the
syntactic local ranges for every words in a sentence as a masking matrix
consisting of 0s and 1s (c.f. Fig. 1(c)), and compute it through Algorithm 2.
It can be proved that the ranges generated by Algorithm 2 is identical to the
syntactic local ranges defined by Definition 3.1:

Algorithm 2 Inducing Syntactic Local Range

1:Syntactic Distance $D=d_{i}(i\in[1,n-1])$

2:Syntactic Local Range $G$

3:$G=\textbf{0}\in\\{0,1\\}^{n\times n}$

4:for i=$1$ to $n$ do

5: $b_{l}=\operatorname*{arg\,max}\limits_{j<i-1,d_{j}>d_{i-1}}\\{j+1\\}$ $//$
left boundary

6: $b_{r}=\operatorname*{arg\,min}\limits_{j>i,d_{j}>d_{i}}\\{j\\}$ $//$ right
boundary

7: $G[i,b_{l}:b_{r}]=1$

8:end for

9:return $G$

###### Proof.

Consider the token $t_{i}$ in the sequence $(t_{1},t_{2},...,t_{n})$ with the
syntactic distances $(d_{1},d_{2},...,d_{n-1})$.

For the pre-text direction, let $t_{i}$’s syntactic local range starts at
$t_{l}$. According to Definition 3.1, $t_{l}$ and $t_{i}$ share a lowest
common ancestor $w$. Let the height of $w$ be $h_{w}$.

  * 1.

Since $t_{l}$ is the leftmost child of $w$, we have $h_{l}^{l-1}>=h_{w}$.

  * 2.

Since $t_{i}$ is either the immediate child of $w$, or the leftmost child of a
constituent $v$ who is the immediate child of $w$, we have
$h_{i}^{i-1}=h_{w}-1$.

  * 3.

For all tokens between $t_{l}$ and $t_{i}$ inclusively, $w$ should also be
their common ancestor. Thus we have $\forall
j\in[l+1,i],h_{j}^{j-1}<=h_{w}-1$.

Comprehensively, we have

$\forall j\in[l+1,i-1],h_{l}^{l-1}>h_{i}^{i-1}>=h_{j}^{j-1}$ (2)

According to Eq. 1, we could derive that

$\forall j\in[l,i-2],d_{l-1}>d_{i-1}>=d_{j}$ (3)

Thus validates line 3 in Algorithm 2.

Symmetrically, the proof for the post-text direction is obvious. ∎

Specifically, we first define the syntactic local pattern in a constituency
tree in Definition 3.1, which is explicit and easy to understand. However,
using syntactic distance can facilitate the computing of syntactic local
pattern since it can be well paralleled. Therefore we turn to an alternative
expression of Definition 3.1 using syntactic distance, and propose Algorithm 2
to generate mask from syntactic distance, which is also proved to be identical
to the definition of syntactic local pattern.

###  3.3 Syntax-guided Self-Attention

Given the masking matrix $G$, we can incorporate it into self-attention
through masked softmax to get a syntax-guided self-attention:

$\text{Attn}(Q,K,V,G)=\text{S}_{m}\left(G,\frac{QK^{T}}{\sqrt{d_{k}}}\right)V$
(4)
$\text{S}_{m}\left(M,X\right)={\left[\text{S}\left(\frac{m_{ji}e^{x_{ji}}}{\sum_{k=1}^{n}m_{jk}e^{x_{jk}}}\right)\right]}_{j=[1,...,n]}$
(5)

where $\text{S}_{m}(\cdot)$ and $\text{S}(\cdot)$ are masked softmax and
softmax, respectively. And $m_{ji}$ and $x_{ji}$ are the element in the $j$-th
row and $i$-th column of the $M$ and $X$ matrices.
$\left[\cdot\right]_{j=[1,...,n]}$ corresponds to vertically stack the
elements yielded by the function inside it, whose input $j$ being $1,...,n$
respectively.

Moreover, since we want to encourage the attention within the syntactic local
range and suppress those outside of it, rather than zeroing out the attention
weights outside of the syntactic local range. To this end, we would still like
the model to attend on tokens outside of the range. Thus, rather than using a
crispy boundary, i.e., elements in $G$ are either $0$ or $1$, we smooth it
through the following steps.

First we define the soft comparison between $d_{i}$ and $d_{j}$ as

$\alpha_{i}^{j}=\frac{\text{tanh}((d_{i}-d_{j})/\tau)+1}{2}$ (6)

which yields $0$ when $d_{i}\ll d_{j}$ and gradually increases to $1$ as
$d_{i}$ increases, with $\tau$ controlling the softness.

We can simulate the attention range found by Algorithm 2 by multiplying
$\alpha$ from the central token to both sides. i.e., entries in $G$ can be
approximated by

$G_{ij}=\prod_{j\leq t\leq i-1}\alpha_{t}^{i-1},j<i-1$ (7)

To initiate the continued product, mask values on the two secondary diagonals
in $G$ are set to 1.

We then use this syntactic-guided attention mask to augment some of the
attention heads in the vanilla Transformer model, thus incorporates
constituency grammar information into Transformer.

##  4 Experiments

###  4.1 Datasets and Experiment Settings

We implement our model based on Fairseq Ott et al. (2019) toolkit The Stanford
CoreNLP parser is utilized to generate constituency grammar tree, which is
further used to calculate syntactic distance. All distance values are natural
numbers with a minimum value of 1 and distances between sentences are set to
999 by default.

We evaluate our model on machine translation tasks of different languages. We
use IWSLT-14 German-English on both directions (De2En and En2De), NC11 German-
English on both directions, ASPEC from Chinese to Japanese (Ch2Ja), and WMT14
from English to German. Unless otherwise noted, we only modify the first
encoder layer, with 3 heads for IWSLT14-En2De and NC11-En2De, 4 heads for
NC11-De2En and IWSLT14-De2En, 2 heads for ASPEC-Ch2Ja and WMT14-En2De.
Detailed settings and available in Appendix A.

###  4.2 Experimental Results

datasets | Transformer | Ours
---|---|---
IWSLT14-De2En | 34.56 | 35.74
IWSLT14-En2De | 28.17 | 29.28
NC11-De2En | 26.83 | 27.67
NC11-En2De | 25.26 | 26.19
ASPEC-Ch2Ja | 47.77 | 48.34
WMT14-En2De | 27.3 | 28.48
Table 1: Test BLEU score on five datasets.

We compare our syntactic-guided model with Transformer Vaswani et al. (2017)
measuring by BLEU score222https://pypi.org/project/sacrebleu/. All of the
results are averaged over 5 independent runs on different seeds and the
improvements have statistical significance (p<0.01). For the large WMT
dataset, we average the test results over the last 5 checkpoints in a single
run due to computation limits. Our model outperforms Transformer baseline
model on all six machine translation datasets. Our model can achieve up to
1.18 improvements on these test beds. Detailed results are shown in table 1.

###  4.3 Ablation Study

Figure 2: Layer-wise for IWSLT14-De2En models | IWSLT14 | NC11
---|---|---
De2En | En2De | De2En | En2De
baseline | 34.56 | 28.17 | 26.83 | 25.26
1-head | 35.65 | 29.20 | 27.58 | 25.87
2-head | 35.68 | 29.21 | 27.67 | 25.99
3-head | 35.71 | 29.28 | 27.56 | 26.19
4-head | 35.74 | 29.23 | 27.67 | 26.08
Table 2: Head-wise for IWSLT14 and NC11.

We apply syntactic distance in different encoder layers of Transformer to get
best utilization of syntactic distance. The results are shown in Fig. 2, where
"x-layer" means that only 3 attention heads of x-th encoder layer use
syntactic information. The model performs best with syntactic distance used in
the first layer. Moreover, the deeper layer we add syntactic distance, the
worse performance the model has. It can be concluded that top layers focus
more on long range dependency of data.

We also studied effects of different heads in using syntactic distance, as
shown in table 2. "x-head" means we only use syntactic distance in x heads of
the first encoder layer. We found that the utilization of different extent
does not lead to great difference in performance.

##  5 Conclusion

In this work, we introduce constituency grammar into Transformer by using
syntactic distance to guide the self-attention mechanism. We implement the
complete pipeline of fusion by converting constituency tree to syntactic
distance and finally generating syntactic local pattern. Experiments show that
our method could consistently improve the performance of Transformer on a
variety of machine translation tasks. We also excavate that the syntactic-
guided attention achieve the best effectiveness in the bottom layer. Future
work could expand to other NLU tasks and larger-scale pretrained models.

## Limitations

Although the constituency-based syntactic local patterns implemented by
syntactic distance appear to be advantageous for self-attention learning in
seq2seq architecture, there are still some directions for further improvement.

Firstly, the constituency grammar information is generated explicitly using
external parser, whereas most low-resource languages like Southeast Asian
family lack sophisticated parsing toolkit. Specially consituency syntax tree
bank is far rarer compared with dependency tree bank. Due to this, our
experiments do not include low resource languages. To solve this problem,
incorporating syntactic information implicitly without any annotation is a
heated topic with great prospect. We will further explore this direction.

Also, most parsers are based on statistical methods so that generated grammar
tree often entails noise and uncertainty. These hinder the wide applications
of our method. A more modern parser like the Berkeley Neural parser with Bert
or Roberta etc. embeddings would do better probably. It would be interesting
to see whether the effect on the BLUE scores of using a parser has a higher
accuracy that could presumably then provide the basis for an improved
incorporation of syntactic representation.

Secondly, the utilization position of syntactic mask is a tough problem when
it is manually tuned, especially on large model configurations and large
datasets. One alternative is to learn the usage of syntactic attention mask
adaptively and softly, by which every attention head in every layer is a
combination of original global attention and syntactic masked attention. The
combination weight of every head is parameterized and learned throughout the
training process.

Thirdly, the pretraining and finetuning architecture has been around in recent
years for its generality and excellent performance compared with the task-
specific model training from scratch. It’s intuitive to explore whether our
method could still achieve good performance on pretraining architecture such
as BERT, even though some studies find that trained on a large corpus the
pretraining model has inherently entailed syntactic structural information.

Fourthly, many studies focus on the usage of dependency syntax information to
enhance the self-attention learning. The constituency grammar structure is
naturally suitable for languages with strong local structure, while dependency
grammar is suitable for languages with high flexibility of permutation. The
difference between two kinds of syntax structure on different source languages
are left for further study.

## Acknowledgements

This work is sponsored by the National Natural Science Foundation of China
(NSFC) grant (No. 62106143), and Shanghai Pujiang Program (No. 21PJ1405700).
We would also like to thank the anonymous reviewers for their constructive
comments and suggestions.

## References

  * Bugliarello and Okazaki (2020) Emanuele Bugliarello and Naoaki Okazaki. 2020.  Enhancing machine translation with dependency-aware self-attention.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1618–1627, Online. Association for Computational Linguistics.
  * Chen et al. (2017) Huadong Chen, Shujian Huang, David Chiang, and Jiajun Chen. 2017.  Improved neural machine translation with a syntax-aware encoder and decoder.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1936–1945, Vancouver, Canada. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Goldberg (2019) Yoav Goldberg. 2019.  Assessing bert’s syntactic abilities.  _arXiv preprint arXiv:1901.05287_.
  * Jawahar et al. (2019) Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019.  What does BERT learn about the structure of language? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3651–3657, Florence, Italy. Association for Computational Linguistics.
  * Klein and Manning (2003) Dan Klein and Christopher D. Manning. 2003.  Accurate unlexicalized parsing.  In _Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics_ , pages 423–430, Sapporo, Japan. Association for Computational Linguistics.
  * Levy and Goldberg (2014) Omer Levy and Yoav Goldberg. 2014.  Dependency-based word embeddings.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, June 22-27, 2014, Baltimore, MD, USA, Volume 2: Short Papers_ , pages 302–308. The Association for Computer Linguistics.
  * Li et al. (2021) Zhongli Li, Qingyu Zhou, Chao Li, Ke Xu, and Yunbo Cao. 2021.  Improving BERT with syntax-aware local attention.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 645–653, Online. Association for Computational Linguistics.
  * Ma et al. (2019) Chunpeng Ma, Akihiro Tamura, Masao Utiyama, Eiichiro Sumita, and Tiejun Zhao. 2019\.  Improving neural machine translation with neural syntactic distance.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2032–2037, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Manning et al. (2014) Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014.  The Stanford CoreNLP natural language processing toolkit.  In _Association for Computational Linguistics (ACL) System Demonstrations_ , pages 55–60.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Shen et al. (2018) Yikang Shen, Zhouhan Lin, Athul Paul Jacob, Alessandro Sordoni, Aaron Courville, and Yoshua Bengio. 2018.  Straight to the tree: Constituency parsing with neural syntactic distance.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1171–1180, Melbourne, Australia. Association for Computational Linguistics.
  * Strubell et al. (2018) Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018\.  Linguistically-informed self-attention for semantic role labeling.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 5027–5038, Brussels, Belgium. Association for Computational Linguistics.
  * Tenney et al. (2019) Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019.  BERT rediscovers the classical NLP pipeline.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4593–4601, Florence, Italy. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Yang et al. (2020) Jian Yang, Shuming Ma, Dongdong Zhang, Zhoujun Li, and Ming Zhou. 2020.  Improving neural machine translation with soft template prediction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5979–5989, Online. Association for Computational Linguistics.
  * Zhang et al. (2020) Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, Hai Zhao, and Rui Wang. 2020\.  Sg-net: Syntax-guided machine reading comprehension.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 9636–9643. AAAI Press.

##  Appendix A Training Details

We perform experiments on six machine translation datasets and use the BLEU
score to evaluate our model’s performance. ISWLT14 EN/DE dataset is stemmed
from TED talks. NC11 EN/DE is stemmed from news commentary. ASPEC CH/JA is
stemmed from scientific paper excerpt corpus. WMT14 EN/DE is stemmed from
European news. Sizes of different datasets are listed in Table3.

datasets | Train | Valid | Test
---|---|---|---
IWSLT14-De/En | 160239 | 7283 | 6750
NC11-En/De | 238843 | 2169 | 2999
ASPEC-Ch/Ja | 672315 | 2090 | 2107
WMT14-En/De | 4528223 | 3000 | 3003
Table 3: Sizes of Experimental Datasets

For Iwslt14 dataset preprocessing, we use Moses
toolkit333https://github.com/moses-smt/mosesdecoder for uncased word
tokenization. Then we clean up the dataset, and only keep the pairs whose
source-target length ratio is within 1.5 and the total length does not exceed
175. Next we use the subword NMT
toolkit444https://github.com/rsennrich/subword-nmt for BPE subword
tokenization with a shared dictionary of size 10k. Finally we randomly select
5% of training as the validation set, and merge dev2010, dev2012, tst2010,
tst2011 and tst2012 into the test set. The size of the training, validation
and test set are 160k/7.3k/6.7k respectively.

For NC11 dataset preprocessing, we still use Moses toolkit for uncased word
tokenization. Then we clean up the dataset, and only keep the pairs whose
source-target length ratio is within 1 and the total length does not exceed
80. Next we use the subword NMT toolkit for BPE subword tokenization with a
shared dictionary of size 16k. Otherwise, we dropout the pairs which do not
have a correct language. Validation and test set are newstest2015 and
newstest2016 respectively. The size of the training, validation and test set
are 239k/2.2k/3k respectively.

For ASPEC dataset preprocessing, we use Moses to extract statements. Then we
use
StanfordNLP-2014-01-01-segmenter555https://nlp.stanford.edu/software/segmenter.shtml
for Chinese and juman-7.0666https://nlp.ist.i.kyoto-u.ac.jp for Japanese in
word tokenization. Next we use the subword NMT toolkit for BPE subword
tokenization with a Chinese dictionary of size 61k and a Japanese dictionary
of size 46k. The size of the training, validation and test set are
672.3k/2k/2.1k respectively.

For WMT dataset preprocessing, subword-NMT and Moses are still ustilized for
tokenization and dictionary built-up. The size of BPE shared dictionary is
40000. Non-printing characters are removed and all punctuations are
normalized. The source-target length ratio and the total length are limited
within 1 and 250 respectively. We sample 1% of training data as validation set
and test sets are directly downloaded.The size of training, validation and
test set are 4528k/3k/3k respectively.

Manual Tuning is performed for hyper-parameters tuning on the basis of test
BLEU score for the best checkpoint. We use Adam optimizer with
$\beta_{1}=0.9$, $\beta_{2}=0.98$, and $\epsilon=10^{-8}$. We use label
smoothing of value $\epsilon_{ls}=0.1$ and weight decay of 0.0001. By default,
the attention-dropout is set to 0.2, dropout is set to 0.3, except that we set
the attention-dropout to 0.1 in ASPEC and set the attention-dropout and
dropout both to 0.1 in WMT. We take the inverse square root learning rate
scheduler and set the peak learning rate as 1e-3 with 4000 steps linear warm-
up and 1e-7 initial learning rate. The scaling parameter $\tau$ is set to 10.
The model has a hidden dimension of 512, 6 encoder layers and 6 decoder layers
with 4 attention heads per layer, except for WMT dataset with 8 attention
heads per layer. Hidden layer size for FFN is set to 2048 in ASPEC and 1024 in
other datasets. For IWSLT14 and ASPEC, each update has up to $8192\times
2\times 1$ tokens(max_tokens$\times$gpu_num$\times$num_updates ). For NC11,
each update has up to $8192\times 2\times 2$ tokens. We use 8 80G-A100 GPUs
from distributed clusters to train our model on WMT14 dataset with CUDA
version 11.1 and 2 40G-A100 GPUs for other datasets with CUDA version 11.2.

datasets | Ours | Transformer
---|---|---
IWSLT14-De2En | 80min | 70min
IWSLT14-En2De | 80min | 80min
NC11-En2De | 80min | 80min
NC11-De2En | 80min | 80min
ASPEC-Ch2Ja | 12h | 12h
WMT14-En2De | 9h | 8.5h
Table 4: Training time of Different Datasets

Since our incorporation of syntactic information is based on explicit
calculation of syntactic distance and local attention pattern mask by external
syntactic parser, we do not introduce extra parameters compared with original
Transformer model. The number of parameters is 50M for configuration of WMT
dataset, 99M for ASPEC dataset and 30M for the other datasets.

The training time of Transformer and our distance enhanced model are listed as
in table 4. Basically, our model costs about the same time for training as
Transformer.