# High-fidelity quantum control via Autler-Townes splitting

Michele Delvecchio  Dipartimento di Matematica, Fisica e Informatica,
Universitá di Parma, Parco Area delle Scienze 7/A, 43124 Parma, Italy
Istituto Nazionale di Fisica Nucleare (INFN), Sezione Milano Bicocca, Gruppo
di Parma, Parco Area delle Scienze 7/A, 43124 Parma, Italy  Teodora Kirova
<EMAIL_ADDRESS>Institute of Atomic Physics and Spectroscopy, University
of Latvia, Jelgavas street 4, Riga, LV-1004, Latvia  Ennio Arimondo
<EMAIL_ADDRESS>Dipartimento di Fisica, Università di Pisa, Largo
Pontecorvo 3, 56127 Pisa, Italy  Istituto Nazionale di Ottica - Consiglio
Nazionale delle Ricerche, Università di Pisa, Largo Pontecorvo 3, 56127 Pisa,
Italy  Donatella Ciampini<EMAIL_ADDRESS>Dipartimento di Fisica,
Università di Pisa, Largo Pontecorvo 3, 56127 Pisa, Italy  Sandro Wimberger
<EMAIL_ADDRESS>Dipartimento di Matematica, Fisica e
Informatica, Universitá di Parma, Parco Area delle Scienze 7/A, 43124 Parma,
Italy  Istituto Nazionale di Fisica Nucleare (INFN), Sezione Milano Bicocca,
Gruppo di Parma, Parco Area delle Scienze 7/A, 43124 Parma, Italy

###### Abstract

We propose quantum control protocols for the high-fidelity preparation of
target-states in systems with Autler-Townes splitting. We investigate an
approximated three-level system obtained from a four-level one by
adiabatically eliminating a state that does not participate in the evolution.
In our work we use linear, arctan and Roland-Cerf functions for transferring
population between two eigenstates of the system obtaining a high fidelity for
long evolution times. Additionally, in order to overcome the restriction given
by the lifetimes of the experimental setup, we propose an accelerated
adiabatic evolution with a shortcut to adiabaticity protocol, which allows us
to reach fidelities close to one but much faster.

Quantum control, Autler-Townes splitting, Shortcut-to-Adiabaticity,
Counterdiabatic Driving

##  I Introduction

Quantum control is based on the application of unitary transformations to
simple or complex quantum systems to drive their evolution into a target
quantum state [1, 2, 3, 4]. Quantum control schemes should be classified on
the basis of two elements, i.e., the experimental handle producing the quantum
evolution, and the specific protocol describing the temporal modification of
the system Hamiltonian. The detuning of a laser exciting the quantum system,
for instance, a two-level system, is a convenient handle for preparing the
initial quantum state and generating a Rabi evolution into the final state.
The protocol determines the time dependence of the handle, optimized to
enhance fidelity, robustness and speed of the unitary transformation.

In a multilevel system with an intrinsic interaction between the internal
states, the laser excitation may play a different indirect role. The non-
resonant control photons do not produce directly the quantum transfer: they
alter the energy levels of the controlled system. Following the initial
quantum preparation the final state transfer is driven by the internal
Hamiltonian. Within this not-resonant approach the quantum control based on
laser-induced potentials, i.e., on AC Stark shift, in some context known as
Autler-Townes splitting, [5], has received a widespread interest, from atomic-
molecular physics [6, 7, 8, 9] to solid state systems [10, 11] and chemistry
[12, 13]. Within the atomic-molecular physics area such an approach has been
applied to different processes as molecular alignment, photodissociation
reactions, such as reviewed in [9]. Our attention is focused on the quantum
control of the two-electron singlet-triplet transitions in molecular systems,
see e.g. Refs. [14, 15, 16, 17, 18, 19, 20, 21]. The perturbed singlet-triplet
states, arising from the level mixing due to the spin-orbit coupling,
represent a gateway to gain access to the otherwise dark triplet states in
these systems.

In most cases, for instance, in ion strings [19, 20], the Autler-Townes
manipulation of the singlet-triplet transition is produced by properly shaped
short laser pulses. These pulses decouple the singlet and triplet electronic
states, as well as minimize absorption to excited singlets and ionization of
the absorbing species. Refs. [17, 18] introduce an alternative approach based
on longer time scales, but leading to a more efficient singlet-triplet
transfer. Controlling the spin character of a spin-orbit coupled pair of
levels, the Autler-Townes effect acts as an all-optical spin switch between
singlet and triplet manifolds. In experiments on molecular states in lithium
dimers this tuning produces an optical control of the singlet-triplet
probability distribution. Based on levels experiencing a weak singlet-triplet
coupling the Autler-Townes effect leads to a well controlled transfer to
triplet states. In an equivalent quantum control experiment on perovskite
nanocrystals exhibiting a spin-orbit induced multi-band structure, the optical
absorption is finely tuned using the Autler-Townes effect [22]. In addition
Ref. [23] applies an electrical control over the detuning energy of the
quantum states of the two-electron singlet-triplet qubit across a chain of
three quantum dots.

Next to the experimental handle, the second control element is the specific
time-dependent driving protocol. Detuning protocols such as the Landau-Zener-
Majorana-Stückelberg tunneling (LZMS) [24, 25, 26] have acquired a significant
role in different research areas. Relying on a linear dependence of a quantum
handle it represents an easy to implement protocol for the preparation of the
target quantum state. Both the asymptotic transition probability and its time
dependence have been examined extensively, theoretically in Refs. [27, 28] and
experimentally in cold atoms and solid state qubits [29, 30, 31, 32, 33, 34].
Ref. [35] applied this protocol to a spin-orbit-coupled Bose-Einstein
condensate. The LZMS tunneling is equivalent to the adiabatic passage of
light-induced potentials introduced in [6] for controlling molecular
reactions. Improvements in LZMS fidelity and speed are obtained applying a
nonlinear temporal dependence of the control parameter [36, 37, 38, 39]. In
order to speed up the operation, more recently alternative superadiabatic
protocols, referred to as ”shortcuts to adiabaticity”, have been introduced
theoretically and experimentally as reviewed in [40]. In order to avoid the
additional requests on the parameter resources, an effective instantaneous
following of the adiabatic states is achieved by introducing fast oscillations
in the Hamilton parameters already present [41, 42].

While the experiments of [16, 17, 18] with lithium molecules are basically in
a continuous-wave regime, the present target is to modify the Autler-Townes
quantum control in order to realize a time-dependent quantum transfer
satisfying high-level requests for fidelity, robustness and speed. This scheme
realizes a triggered and fast all-optical spin switch. The time dependence of
the transfer protocol is based on the control of the Autler-Townes effect
parameters. A similar time-dependent control of the Hamiltonian parameters can
be applied to other qubit systems experiencing a spin-orbit coupling.

Our scheme based on the control of a probe laser frequency detuning, contains
some features of the three-level Stimulated Raman Adiabatic Passage operation
[43]. While the probe laser produces the excitation from the initial state to
an intermediate excited one, the second step for the transfer to the target
state is produced by the spin-orbit Hamiltonian. This second step not directly
driven by a coupling laser, is controlled by the Autler-Townes effect of a
second laser connecting the target state to a fourth level. Therefore an
original feature of our theoretical analysis is the application of LZMS-like
protocol to such an effective four-level system. The conditions for the
implementation of linear, non-linear and superadiabatic protocols are imposed
on a reduced three-level system derived through an adiabatic elimination. The
requests for such an elimination and the superadiabatic realization in the
full four-level system are carefully investigated. Even if our analysis is
based on the experimental parameters of Ref. [16, 17], the four-level LZMS
protocol can be applied to a large variety of experimental schemes.

The paper is organized as follows: Section II introduces the level scheme and
presents in detail the role of the spin-orbit coupling. In the following, the
adiabatic protocol, the reduction of the four-level system to a three-level
one, and the time-dependence of eigenvectors-eigenvalues of the full
Hamitonian are presented. Section III examines the application of different
adiabatic protocols to the four-level system and introduces the
counterdiabatic approach as a tool to enhance the driving speed. A short
summary concludes our work.

Figure 1: Four-level system with pump and coupling lasers, driving the
transitions $|1\rangle\to|S\rangle$, with Rabi frequency $\Omega_{p}$ and
detuning $\delta_{p}$, and $|T\rangle\to|2\rangle$, with Rabi frequency
$\Omega_{c}$ and detuning $\delta_{c}$, respectively. A spontaneous decay with
rate $\Gamma_{T}$ from the $|T\rangle$ state, represented by the dashed arrow,
prepares the final state $|T_{g}\rangle$ of the all-optical switch. While in
the experiment of Ref. [17] the coupling laser connects the $\left|S\right>$
and $\left|2\right>$ states, our equivalent choice allows an adiabatic
elimination of the $\left|2\right>$ state. The dressed states $|\pm\rangle$,
arising due to the Autler-Townes mixing discussed below in Eq. (II.3), are
shown for a blue detuned transition with $\delta_{c}>0$ as sketched in the
figure.

##  II Hamiltonian and driving schemes

In this section, we aim at providing the basis of the system setups that we
used in our study. We first introduce the theoretical model of the four-level
system we considered, then we show how and under what conditions it can be
reduced to a three-level one, and finally we discuss the Autler-Townes effect
in our context.

###  II.1 Singlet-triplet Hamiltonian

The singlet-triplet transfer experiment, as those in refs. [16, 17, 18], are
described by the four levels introduced in [44] and presented in Fig. 1. Their
original structure for a lithium dimer is shown in Fig. 2 in Ref. [17]. In
particular, our initial ground state $|1\rangle$ would correspond in Li2 to
the rotational-vibrational state $X^{1}\Sigma_{g}^{+}(\nu=2,J=22)$, and the
spin-orbit coupled states $\left|T\right>$ and $\left|S\right>$ to
$1^{3}\Sigma_{g}^{-}(\nu=1,J=21,f)$ and $G^{1}\Pi_{g}(\nu=12,J=21,f)$,
respectively. In the experiment reported in [17] the first transition from the
ground to the excited state is a two-photon one. Our target is to transfer the
population from the singlet initial state $|1\rangle$ to the triplet excited
state $|T\rangle$, and from the latter finally to the state $|T_{g}\rangle$
either automatically by spontaneous decay or by the application of an
additional $\pi$-pulse. The gateway key is the $(|S\rangle,|T\rangle)$
manifold of states with mixed singlet-triplet symmetries owing to a spin-orbit
coupling. They are originated from the $(|S_{0}\rangle,|T_{0}\rangle)$ states
corresponding to unperturbed singlet and triplet states with original energy
separation $\Delta_{0}$. They experience a spin-orbit perturbation mixing with
amplitude $V$ described by the Hamiltonian $H_{so}=V|S_{0}\rangle\langle
T_{0}|+H.c$, (assuming $\hbar=1$). The mixed states $|S\rangle$ and
$|T\rangle$ are given by

$|S\rangle=\alpha|S_{0}\rangle-\beta|T_{0}\rangle,\qquad|T\rangle=\beta|S_{0}\rangle+\alpha|T_{0}\rangle,$
(1)

where the $(\alpha,\beta)$ coefficients are normalized to one, i.e.,
$|\alpha|^{2}+|\beta|^{2}=1$, while in the following these parameters are
assumed to be real without loss of generality. The unperturbed energy
separation $\Delta_{0}$ and the perturbation $V$ are linked to the effective
energy splitting $\Delta_{so}$ of the $(|S\rangle,|T\rangle)$ levels and to
the mixing coefficients by (see [44]):

$\Delta_{0}=(\alpha^{2}-\beta^{2})\Delta_{so}\qquad V=\alpha\beta\Delta_{so}.$
(2)

The Lithium molecular states of Ref. [16] have mixing coefficients
$\alpha^{2}=0.87$ and $\beta^{2}=0.13$, and spin-orbit splitting
$\Delta_{so}=2\pi\cdot 0.75$ GHz equivalent to to $4.71$ ns-1. In the
following all parameters are given in the latter units, while the conversion
into GHz is obtained dividing those by $2\pi$.

The access to the singlet-triplet manifold is provided by the pump ($p$) laser
connecting the $\left|1\right>$ singlet to the $|S\rangle$ state with detuning
$\delta_{p}$. The singlet component of both $|S\rangle,|T\rangle$ eigenstates
of Eq. (1) determines their excitation by the $p$ laser characterized by Rabi
frequencies $\alpha\Omega_{p}$ and $\beta\Omega_{p}$, respectively. The
effective laser excitation is controlled by the pump detuning from those
states. Due to the mixed singlet-triplet character a fraction of molecules in
the $|T\rangle$ excited state decays to the pure lower energy $|T_{g}\rangle$
triplet state. This process representing the basis of the optical pumping in
triplet states for ultracold molecules is not very efficient for the rather
small $\beta^{2}$ value reported above.

The quantum control of the singlet-triplet transfer introduced by [16, 17] is
based on the modification of the energy separation of the spin-orbit
$|S\rangle,|T\rangle$ manifold. Such a modification is produced by a control
($c$) off-resonant laser linking the $|T\rangle$ state to an additional
$|2\rangle$ triplet one with detuning $\delta_{c}$. The Rabi frequencies for
the mixed manifold components are $-\beta\Omega_{c}$ and $\alpha\Omega_{c}$,
respectively.

The temporal evolution of the optically driven levels
$(|1\rangle,|S\rangle,|T\rangle,|2\rangle)$ is described by a four-level
Hamiltonian. For the $(|S\rangle,|T\rangle)$ basis we obtain

$H^{(4)}=\begin{pmatrix}\delta_{p}+\Delta_{so}&\alpha\Omega_{p}/2&-\beta\Omega_{p}/2&0\\\
\alpha\Omega_{p}/2&\Delta_{so}&0&\beta\Omega_{c}/2\\\
-\beta\Omega_{p}/2&0&0&\alpha\Omega_{c}/2\\\
0&\beta\Omega_{c}/2&\alpha\Omega_{c}/2&-\delta_{c}\end{pmatrix},$ (3)

where we assume the zero energy at the position of the $|T\rangle$ state, and
define $\delta_{p}=\omega_{p}-(E_{S}-E_{1})$,
$\delta_{c}=\omega_{c}-(E_{2}-E_{T})$, with $E_{i}$ the energy of the states
$(i=1,2,S,T)$. In the experiment of [16], the Rabi frequencies were
$\Omega_{p}=0.24$, $\Omega_{c}=3.8$, all in units of ns-1.

While the lower $|1\rangle$ level is stable and the $|2\rangle$ state is off-
resonantly excited, the $|S\rangle,|T\rangle$ mixed levels suffer from
spontaneous emission decay to levels outside the manifold of the above
Hamiltonian, with decay rates of $\Gamma_{S}=0.06$ and $\Gamma_{T}=0.10$, both
in ns-1 for the experiments of ref. [16]. That experiment monitored the
spontaneous decay to the $|T_{g}\rangle$ triplet-ground state representing the
final product of the all-optical switch, as denoted by the dashed line in Fig.
1.

In our analysis we consider two regimes for $\delta_{c}$, supposed to be fixed
during the protocol. The first is $\delta_{c}\approx 0$, in which the Autler-
Townes effect must be considered. In the $|\delta_{c}|\gg\Omega_{c}$ second
case, the problem can be reduced to a three-level model by adiabatically
eliminating the state $\left|2\right>$ from the beginning.

###  II.2 Reduced three-level System

For $|\delta_{c}|\gg\Omega_{c}$, corresponding to a virtual excitation of the
$|2\rangle$ state, the $\left|2\right>$ state can be adiabatically eliminated.
The corresponding reduced Hamiltonian reads

$H^{(3)}=\begin{pmatrix}\delta_{p}+\Delta_{so}&\frac{\alpha\Omega_{p}}{2}&-\frac{\beta\Omega_{p}}{2}\\\
\frac{\alpha\Omega_{p}}{2}&\Delta_{so}+\frac{\beta^{2}\Omega_{c}^{2}}{4\delta_{c}}&\frac{\alpha\beta\Omega_{c}^{2}}{4\delta_{c}}\\\
-\frac{\beta\Omega_{p}}{2}&\frac{\alpha\beta\Omega_{c}^{2}}{4\delta_{c}}&\frac{\alpha^{2}\Omega_{c}^{2}}{4\delta_{c}}\\\
\end{pmatrix}\,,$ (4)

now in the basis $\\{\left|1\right>,\left|S\right>,\left|T\right>\\}$. Notice
that within the effective energy of the state $|T\rangle$ the light-shift or
ac shift $\delta_{ls}$ given by

$\delta_{ls}=\frac{\alpha^{2}\Omega_{c}^{2}}{4\delta_{c}},$ (5)

associated to the Autler-Townes process. That term may produce a degeneracy
between the $|S\rangle$ and $|T\rangle$ for a blue detuning of the control
laser and a proper choice of the control laser parameters. If we neglect the
“second order” Hamiltonian matrix elements proportional to
$\beta^{2},\alpha^{2}\text{ and }\alpha\beta$, we obtain a matrix equivalent
to Eq. (5) of Ref. [44].

###  II.3 Autler-Townes effect

The driving of a two-level system by an intense resonant laser produces the
Autler-Townes effect, i.e., the splitting of an optical transition [5]. Within
a dressed-atom analysis [45] the degeneracy between the ground and the excited
state is broken by the strong laser. The resulting energy separation is given
by the Rabi frequency of the driving laser. Translated to our Fig. 1 level
scheme, the Autler-Townes configuration of Refs. [16, 17] is created by the
coupling laser tuned at $\delta_{c}\approx 0$ modifying the energies of the
states $|T\rangle$ and $|2\rangle$. That driving produces as eigenstates the
following $|\pm\rangle$ superpositions:

$\displaystyle|+\rangle=$ $\displaystyle a_{+}|T\rangle+b_{+}|2\rangle,$
$\displaystyle|-\rangle=$ $\displaystyle a_{-}|T\rangle+b_{-}|2\rangle,$ (6)

with eigenvalues $\lambda_{\pm}$ given by

$\lambda_{\pm}=-\frac{-\delta_{c}\pm\sqrt{\delta_{c}^{2}+\Omega_{c}^{2}}}{2}.$
(7)

and eigenstate coefficients given by

$\frac{b_{\pm}}{a_{\pm}}=\frac{\delta_{c}\pm\sqrt{\delta_{c}^{2}+\alpha^{2}\Omega_{c}^{2}}}{\alpha\Omega_{c}}.$
(8)

Because the Autler-Townes control is realized at low $\delta_{c}$ values, in
this regime we usually cannot apply the adiabatic elimination reducing the
four-level Hamiltonian of Eq. (3) to the three-level of Eq. (4). However, in
the next section we show that, using the states in Eq. (II.3), we obtain a
matrix description in which $\left|T\right>$ and $\left|2\right>$ are not
coupled. This will be useful for reaching a high fidelity in the Autler-Townes
regime.

Because the energy separation between the states $|S\rangle$ and $|T\rangle$
may be modified by the Autler-Townes energy shift of the $|T\rangle$ state,
their singlet-triplet coupling is also modified. In the experiment of Refs.
[16, 17] an increased triplet excitation reached approximately twice the value
$\beta^{2}=0.13$ for the molecular states of interest.

###  II.4 Fidelity

To measure the performance of the triplet transfer, we define the fidelity as
the probability of being in the target state $\left|T\right>$ at final time of
the evolution $t=t_{f}$. In formula,

$\mathcal{F}(t_{f})=\left|\langle\psi\left(t=t_{f}\right)|T\rangle\right|^{2}\,$
(9)

where $\psi(t=t_{f})$ is the state of the system at the end of the protocol.
As long as $t_{f}<\Gamma_{T}^{-1}$, our target state decays after the protocol
into the experimentally relevant one $\left|T_{g}\right>$, and the fidelities
of both states will practically be identical. Alternatively, the population of
the state $\left|T\right>$ can be fast transferred to the final target
$\left|T_{g}\right>$ by a $\pi$ pulse. We will also use the infidelity
$\mathcal{I}\equiv 1-\mathcal{F}$ if necessary for a better visualization.

##  III Laser driving protocols

This section introduces the adiabatic driving functions that we adopted in our
schemes and then the protocol we chose to accelerate the adiabatic evolution
of the mentioned driving functions.

###  III.1 Adiabatic evolution

Generalizing the idea of the LZMS problem [36, 37, 38, 46, 39], in order to
realize the above goal we used specific functions such that the detuning
$\delta_{p}$ is given by the following dimensionless form

$\delta_{p}(\tau)=sf(\tau)\,,$ (10)

where $\tau=t/t_{f}$, $s$ is a scaling parameter and $f(\tau)$ a dimensionless
function which we call sweep function. Exploiting the adiabatic theorem of
quantum mechanics, which states that if the system changes slowly enough it
follows the evolution of one of its initially prepared eigenstates
adiabatically in time, we can design specific sweep functions in order to
maximize the transfer to our target state. In particular, we use three
exemplary sweep functions: a linear one from the classical LZMS problem, an
arctan function and the Roland-Cerf [36, 47]. They explicitly read

  * •

Linear

$\delta_{p}(\tau)=a(2\tau-1)\,,$ (11)

  * •

Arctan

$\delta_{p}(\tau)=a\arctan(b\tau)-c,$ (12)

  * •

Roland-Cerf

$\delta_{p}(\tau)=\frac{\beta\Omega_{p}(1-2\tau)}{2\sqrt{4\tau(1-\tau)+\left(\frac{\beta\Omega_{p}}{2a}\right)^{2}}}-d\,.$
(13)

In our simulations we set $a=10$ ns-1 producing the linear scan of Fig.2(a).
The dimensionless parameter $b$ of the arctan scan controls the shape of the
$f(\tau)$ function and, consequently, the distance between the states
$\left|1\right>$ and $\left|T\right>$ at the end of the protocol, as in Fig.
2(b). The Roland-Cerf function is based on the procedure described in [46] for
the tangent protocol. The bias value $d$ is optimized to create the wanted
avoided crossing, as in Fig. 2(c). An optimal choice of the parameters can
improve the adiabatic evolution, but may degrade the evolution with
counterdiabatic pulses, as seen later on in Sec. IV.

Figure 2: Instantaneous eigenvalues vs. rescaled time for the three sweep
functions (a) linear; (b) arctan with $b=20$ and $c=19.2$ ns-1; (c) Roland-
Cerf with $d=4.68$ ns-1. In these plots $\delta_{c}=100$ ns-1 leading to
$\delta_{ls}=0.03$ ns-1 for the light-shift defined in Eq. (5). On the left
the position of the three initial eigenstates
$\left|1\right>,\left|S\right>,\left|T\right>$ and on the right their final
energies are marked. In (a) two avoided crossing are created, but we are only
interested in the one between the states $\left|1\right>$ and
$\left|T\right>$. In (b), thanks to the arctan function shape, only the
avoided crossing of interest, observed in the inset, is generated. Also in (c)
although with a different sweep function, a single avoided crossing is
obtained.

For the three-level Hamiltonian in Eq. (4), we compare the performance of the
above sweep functions. Fig. 2 reports the instantaneous eigenvalues of the
Hamiltonian for the three functions when varying the detuning $\delta_{p}$
with time. Because the energy of the triplet state $\left|T\right>$ is lower
than the $\left|S\right>$ one, a scan of $\delta_{p}$ from a negative value
with a positive slope reaches in time the $\left|1\right>\to\left|T\right>$
anti-crossing before the $\left|1\right>\to\left|S\right>$ one, as for the
linear sweep in (a). We are interested only in the first anticrossing where,
adiabatically following the lowest eigenstate (blue/black solid line), a
$\left|1\right>\rightarrow\left|T\right>$ population transfer is realized.
Therefore the linear sweep is terminated before reaching the
$\left|1\right>\to\left|S\right>$ anticrossing. In Fig. 2(b) or the arctan
case, the coefficients $b$ and $c$ of Eq. (12) governing the $\delta_{p}$ non-
constant rate are chosen in order to avoid the second anticrossing. This
generates temporal oscillations in the fidelity as shown later in the
corresponding section. Also here an adiabatic following of the lowest
eigenstate (blue/black solid line) produces the required population transfer.
In Fig. 2(c), such a transfer occurs in the instantaneous energies scheme for
the optimized Roland-Cerf sweep, too.

###  III.2 Accelerated evolution

The final time of transfer protocols must be compared to the spontaneous
emission decay of the molecular states of interest. Therefore we accelerate
the evolution in order to reach the required fidelity in a shorter time.
However, an accelerated evolution typically generates unwanted non-adiabatic
transitions between the eigenstates of the system, reducing the final
fidelity. The approach introduced in [48, 49, 50] drives the system in a
perfectly adiabatic manner in an arbitrary short time. In the next sections,
we show the result for the reduced three-level model, but the same procedure
can be applied to the complete system as well.

Given a generic time-dependent system Hamiltonian

$\hat{H}_{0}(t)=\sum_{k}E_{k}(t)\left|k(t)\right>\left<k(t)\right|\,,$ (14)

where $E_{k}(t)$ and $\left|k(t)\right>$ are, respectively, the instantaneous
eigenvalues and eigenvectors of the system, the counter-adiabatic (CD)
Hamiltonian $H_{CD}(t)$ is determined by:

$H_{CD}(t)=i\hbar\sum_{n\neq
k}\sum_{k}\frac{\left|n(t)\right>\left<n(t)\right|\partial_{t}H_{0}(t)\left|k(t)\right>\left<k(t)\right|}{E_{k}(t)-E_{n}(t)}\,.$
(15)

The system evolution is driven, thus, by the new Hamiltonian
$\hat{H}(t)=\hat{H}_{0}+H_{CD}(t)$. In this way, if the system starts in an
eigenstate $\left|k(t)\right>$, it remains in that instantaneous eigenstate
for the entire evolution. While for a two-level system the expression of
$H_{CD}$ can be found analytically, for more levels it is generally more
convenient to resort to numerical tools. Identifying $\hat{H}_{0}$ with the
matrices in Eqs. (3) and (4), the Schrödinger equation with the total
Hamiltonian $\hat{H}(t)$ is then numerically evolved using the function
sesolve of the python library qutip [51], ensuring the convergence of the
results.

Figure 3: Instantaneous eigenvalues of the four-level system vs the
$\tau=t/t_{f}$ rescaled time at $\delta_{c}=1$ ns-1 for the sweep functions:
(a) linear; (b) arctan with $b=10$ and $c=18$ ns-1; (c) Roland-Cerf with
$d=3.41$ ns-1. The energies of the four states
$\left|1\right>,\left|S\right>,\left|T\right>,\left|2\right>$ evolve with time
from the initial to final values. The impact of the final
$\left|1\right>$-$\left|S\right>$ avoided crossing is handled as described in
the caption of Fig. 2. Note the presence of the
$\left|1\right>$-$\left|2\right>$ additional avoided crossing with a small
coupling strength.

##  IV Autler-Townes regime

For the four-level system described by the Eq. (3), we examine the population
transfer between the states $\left|T\right>$ and $\left|1\right>$ at low
$\delta_{c}$ values where the Autler-Townes effect plays a key role. The
instantaneous eigenvalues are depicted in Fig. 3 for the protocols of Sec
III.1 with the parameters reported in the figure caption. The transfer of our
interest is associated to the avoided crossing between the states
$\left|T\right>$ and $\left|1\right>$. However in all the protocols of Fig. 3,
before reaching the level $\left|T\right>$ the scanning reaches an additional
$\left|1\right>$-$\left|2\right>$ avoided crossing. Even with a reduced
strength this crossing would block the realization of a very high fidelity in
the final $\left|T\right>$ occupation. In order to bypass this loss, for both
linear and arctan sweeps we split the $\delta_{p}$ evolution in two pieces. At
first, using a short evolution time we scan diabatically the first avoided
crossing. Next we impose to $\delta_{p}$ the counter-diabatic evolution of
Sec. III.2. The transition occurs at $\tau=t/t_{f}=0.2$. The temporal
evolution of the Roland-Cerf sweep is automatically optimized for such fast-
slow scanning.

As an example of the fidelity results, Fig. 4(a) shows the populations of the
four levels, of the original basis, in the case of the arctan protocol. The
$\left|T\right>$ state fidelity reaches a final value around sixty-five
percent. However, even with a perfectly adiabatic evolution, a large part of
the final population, $\approx 35\%$ in this case, occupies the state
$\left|2\right>$. This result appears because the counter-adiabatic protocol
imposes an evolution linking initial and final superposition of states. For
the present case, the Autler-Townes effect produces a close degeneracy between
the states $\left|1\right>$ and $\left|+\right>$. Figure 4(b) represents the
populations of the dressed states obtained from the evolution of the dressed
basis $\left|\pm\right>$.

Therefore, we have applied the counter-diabatic evolution to the new four
level system where the $\left|T\right>,\left|2\right>$ states are now replaced
by the dressed basis $\left|\pm\right>$ of Eqs. (II.3). Figure 4(b) shows the
populations of the four states
$\left|+\right>,\left|S\right>,\left|1\right>,\left|-\right>$ obtained from
the evolution of the CD Hamiltonian in the dressed representation. Figure 4(c)
reports the populations of the four states
$\left|T\right>,\left|S\right>,\left|1\right>,\left|2\right>$ derived from
that evolution by performing a projective measurement of the occupations of
the states $\left|T\right>$ and $\left|2\right>$. The states
$\left|\pm\right>$ are superpositions of $\left|T\right>,\left|2\right>$.
Thus, the measured $\left|T\right>$ population contains contributions from
both dressed states leading to interference effects. The CD dressed
Hamiltonian with a maximized fidelity produces a final 97% occupation of
$\left|T\right>$, higher than that in the case (a). This population can be
transferred by a $\pi$ pulse on a short time scale to the final experimental
target $\left|T_{g}\right>$. The spontaneous emission
$\left|T\right>\to\left|T_{g}\right>$ with $1/\Gamma_{T}\gg t_{f}$ represents
an alternative for the accumulation into the target state.

Figure 4: Populations of the four states obtained from the different
evolutions in (a) with the four-level Hamiltonian of Eq. (3) in the
original/bare basis, and in (b) and (c) in the dressed representation
introduced in Sec. IV. The populations are measured in (a) and (c) in the bare
basis $\\{\left|T\right>,\left|2\right>\\}$ and in (b) in the dressed basis
$\\{\left|\pm\right>\\}$. The population of $\left|T\right>$ can then be
transferred to $\left|T_{g}\right>$ either by spontaneous decay or by applying
an additional $\pi$ pulse. All evolutions are performed using the arctan sweep
together with the shortcut to adiabaticity protocol presented in Sec. III.2
for $\tau=t/t_{f}>0.2$. We choose a very fast evolution with $t_{f}=1$ ns in
order to diabatically pass through the first avoided crossing. At $t_{f}=1$
ns, the measured $\left|T\right>$ population is 97% in (c), much higher than
for the different evolution in (a). (d) shows a zoom of Fig. 3(b) into the
interested avoided crossing. Please notice that the population transfer in
(a),(b) and (c) occurs at the position of the avoided crossing in (d).

##  V Three-level performances

In this section, we return to the three-level system introduced above by Eq.
(4). We first analyze the fidelities of the driving protocols when no CD
correction is applied, then we accelerate the evolution introducing the CD
term presented in Sec. III.2.

Figure 5: Comparison between the fidelity, at $t=t_{f}$, as defined in Eq.
(9), of the three adiabatic protocols denoted as LZ, AT and RC, at
$\delta_{c}=30$ ns-1. The dashed lines represent the reduced three-level
system, while the solid lines describe the complete four-level system. The two
systems are identified by the numbers $3$ and $4$ in the legends. For such a
large $\delta_{c}$, the four and three-level systems produce very similar
fidelities.

###  V.1 Linear, arctan and Roland-Cerf protocols

Figure 5 reports a comparison in fidelity of the three protocols presented in
Sec. III, without the counter-diabatic term and in the regime of large values
of $\delta_{c}$, such that the $|2\rangle$ can be eliminated and the system is
effectively reduced to a three-level one. The well designed Roland-Cerf
driving reaches a better performance with $\mathcal{F}=93\%$ at $\tau_{f}=50$
ns for $\delta_{c}=30$ ns-1, as shown in Fig. 5 by the dashed line with square
symbols. Good fidelities are reached at longer final times with the arctan
protocol. However, in this case the population of the level $\left|T\right>$
experiences temporal oscillations, as shown Fig. 6(b). Similar oscillations
are observed also in Fig 5. The amplitude of these oscillations is reduced
using larger $t_{f}$ values. Lower fidelities are obtained with the linear
protocol, with large oscillations in the populations. These are generated by
changing the linear protocol to a constant after $t/t_{f}=0.3$. As above, this
change is necessary if one wants to avoid to create an anticrossing with
$\left|S\right>$ and transfer, in this way, part of the population to it.

Figure 6: Populations of the three levels at $t_{f}=1000$ ns and
$\delta_{c}=30$ ns-1, for the (a) linear, (b) arctan and (c) Roland-Cerf sweep
functions. The populations in all the three cases are represented by the
orange dashed line for the state $\left|1\right>$, the blue solid line for
$\left|T\right>$ and green dash-dot line for the state $\left|S\right>$. The
legend in (b) is valid also for (a) and (c). For the same evolution time, the
latter two driving protocols produce better results. The population of the
state $\left|T\right>$ represents the fidelity as a function of the rescaled
time. Oscillations in (a) are caused by the fact that the linear protocol is
terminated before reaching the level $\left|S\right>$ (at $t/t_{f}=0.3$ in
this case) and, therefore, it assumes a constant shape for the rest of the
evolution time.

###  V.2 Counter-diabatic evolution

Figure 7: In (a) imaginary parts of the $H_{CD}^{(i,j)}$ elements with
$i,j=1,2,3$ for the arctan protocol, with $H_{CD}^{(1,3)}$ the largest
required pulse. Parameters $\delta_{c}=30$ ns-1, $t_{f}=1$ ns, $c=19.2$ ns-1,
$b=20$. In (b) infidelities, at the $10^{-3}$ level, for different CD
corrections. Blue dots: for full $H_{CD}$ elements; orange squares:
eliminating the $H_{CD}^{(2,3)}$ correction only; green triangles: eliminating
both $H_{CD}^{(2,3)}$ and $H_{CD}^{(1,2)}$ corrections.

We report here the performance of the arctan $\left|1\right>\to\left|T\right>$
protocol as corrected by the Berry counter-diabatic Hamiltonian of Eq. (15).
For the three-level system three additional pulses are required in order to
eliminate all the non-adiabatic transitions. The imaginary part of those
$H_{CD}^{(i,j)}$ matrix elements are shown in Fig. 7(a). Applying these
correction pulses, we obtain infidelities of the order of $10^{-3}$ as in the
blue solid line in Fig. 7(b) for an arbitrary evolution time.
The application of the Berry correction pulses is costly in terms of energy
and of new matrix elements in the Hamiltonian. Towards a simplified
experimental realization, the number of $H_{CD}$ corrections may be reduced to
the most relevant ones, as analyzed in [46]. Fig. 7(a) evidences that the
$H_{CD}^{(1,3)}$ correcting peak is greater than the remaining ones. Fig. 7(b)
compares the infidelities applying the full $H_{CD}$, and eliminating either
one or both the small amplitude corrections. The elimination of the two terms
in the CD Hamiltonian produces only a maximum change at the $5\cdot 10^{-4}$
level, but simplifies a lot the experimental realizations. Therefore the
single $H_{CD}^{(1,3)}$ implementation produces very high fidelities on
timescales much shorter than without the acceleration, bringing experimental
times down to the coherence time of the system. A similar reduction of the
effect number of matrix elements in $H_{CD}$ is possible in the protocol shown
in Sec. IV, but will not be shown here in full detail.

##  VI Conclusions

We have explored the Autler-Townes control of spin-orbit coupling in an
original four-level system with the target of a large transfer efficiency to a
final triplet state. We extended the flexibility explored by [21] for the
nonlinear response in such a system to population transfer. Our work confirms
the interesting feature of a probe-coupling excitation laser scheme, assisted
by the intermediate spin-orbit coupling. However the present system is not
equivalent to a STIRAP configuration [43] which realizes a population transfer
in, e.g., a three-level system without populating the intermediate state. Our
target, instead, is to populate the final triplet state modifying the spin-
orbit coupling of the intermediate and final states. The quantum control is
based on the Autler-Townes produced a modified energy separation between the
singlet and triplet states. The detuning and the intensity of the control
laser represent the key parameters of the control operation.

We have applied the counter-diabatic control in order to speed up the transfer
process with a high transfer fidelity. Our population transfers are all larger
than those reached in the experiments of [16]. The use of accelerated
adiabatic driving produces the required transfer in a time short compared to
the decay of the molecular states of interest. These features make our
protocols very relevant for future experiments.

For the present population transfer the coupling between initial and final
state is based on an intermediate singlet-triplet coupling Hamiltonian.
However the present analysis may be applied as well, e.g., to configurations
where that coupling is produced by an applied magnetic field, static or
oscillating, or by a two-photon laser interaction. Thus, our quantum transfer
may be useful to a large class of quantum simulations. In heavy alkali dimer
molecules such as Rb2 and Cs2, the spin-orbit interaction is much stronger
than for the lighter ones. Such perturbed levels are used as intermediate
levels in the transfer of cold molecules formed at long range in the triplet
state to deeply bound levels of the singlet ground state. Our control
approaches could be applied also to those systems.

###### Acknowledgements.

TK acknowledges support from the Visiting Fellow Program of the University of
Pisa. TK and EA thank A. H. Ahmed and A. M. Lyyra for extended stimulating
discussions. We are grateful to Francesco Petiziol for comments on the
manuscript.

## References

  * Nielsen and Chuang [2010] M. A. Nielsen and I. L. Chuang,  _Quantum Computation and Quantum Information_ (Cambridge University Press, 2010).
  * D’Alessandro [2007] D. D’Alessandro,  _Introduction to Quantum Control and Dynamics_, Chapman & Hall/CRC Applied Mathematics & Nonlinear Science (CRC Press, 2007).
  * Wiseman and Milburn [2009] H. M. Wiseman and G. J. Milburn,  _Quantum Measurement and Control_ (Cambridge University Press, 2009).
  * Glaser  _et al._ [2015] S. J. Glaser, U. Boscain, T. Calarco, C. P. Koch, W. Köckenberger, R. Kosloff, I. Kuprov, B. Luy, S. Schirmer, T. Schulte-Herbrüggen, D. Sugny, and F. K. Wilhelm, Training Schrödinger´s cat: quantum optimal control - strategic report on current status, visions and goals for research in europe, Eur. Phys. J. D 69, 279 (2015).
  * Autler and Townes [1955] S. H. Autler and C. H. Townes, Stark effect in rapidly varying fields, Phys. Rev. 100, 703 (1955).
  * Garraway and Suominen [1998] B. M. Garraway and K.-A. Suominen, Adiabatic passage by light-induced potentials in molecules, Phys. Rev. Lett. 80, 932 (1998).
  * Brumer and Shapiro [2003] P. W. Brumer and M. Shapiro,  _Principles of the Quantum Control of Molecular Processes_ (Wiley Hoboken, 2003).
  * Sussman  _et al._ [2006] B. J. Sussman, D. Townsend, M. Y. Ivanov, and A. Stolow, Dynamic Stark control of photochemical processes, Science 314, 278 (2006), https://www.science.org/doi/pdf/10.1126/science.1132289 .
  * Sola  _et al._ [2018] I. R. Sola, B. Y. Chang, S. A. Malinovskaya, and V. S. Malinovsky, Quantum control in multilevel systems, in  _Advances Atomic Molecular Optical Physics_ , Vol. 67 (Elsevier, 2018) p. 151.
  * Boyle  _et al._ [2009] S. J. Boyle, A. J. Ramsay, A. M. Fox, M. S. Skolnick, A. P. Heberle, and M. Hopkinson, Beating of exciton-dressed states in a single semiconductor $\mathrm{InGaAs}/\mathrm{GaAs}$ quantum dot, Phys. Rev. Lett. 102, 207401 (2009).
  * Astapenko  _et al._ [2015] V. Astapenko, P. Golovinski, and A. Yakovets, Control of excitation transfer in coupled quantum dots by a nonresonant laser pulse, Optics and Laser Technology 71, 103 (2015).
  * Stranius  _et al._ [2018] K. Stranius, M. Hertzog, and B. Karl, Selective manipulation of electronically excited states through strong light-matter interactions, Nature Comm. 9, 043418 (2018).
  * Eizner  _et al._ [2019] E. Eizner, L. A. Martìnez-Martìnez, J. Yuen-Zhou, and S. Kèna-Cohen, Inverting singlet and triplet excited states using strong light-matter coupling, Science Advances 5, eaax4482 (2019).
  * Sola  _et al._ [2006] I. R. Sola, J. González-Vázquez, and V. S. Malinovsky, Optical control of a spin switch in the weak spin-orbit coupling limit, Phys. Rev. A 74, 043418 (2006).
  * González-Vázquez  _et al._ [2006] J. González-Vázquez, I. R. Sola, J. Santamaria, and V. S. Malinovsky, Optical control of the singlet-triplet transition in Rb2, The Journal of Chemical Physics 125, 124315 (2006).
  * Ahmed  _et al._ [2011] E. H. Ahmed, S. Ingram, T. Kirova, O. Salihoglu, J. Huennekens, J. Qi, Y. Guan, and A. M. Lyyra, Quantum Control of the Spin-Orbit Interaction Using the Autler-Townes Effect, Phys. Rev. Lett. 107, 163601 (2011).
  * Ahmed and Lyyra [2014] E. H. Ahmed and A. M. Lyyra, Frequency domain control of quantum state singlet/triplet character and prospects for an all-optical spin switch, Journal of Modern Optics 61, 7 (2014).
  * Ahmed  _et al._ [2014] E. H. Ahmed, X. Pan, J. Huennekens, and A. M. Lyyra, Optical control of collisional population flow between molecular electronic states of different spin multiplicity, Phys. Rev. A 89, 061401 (2014).
  * Vindel-Zandbergen  _et al._ [2013] P. Vindel-Zandbergen, M. Falge, B. Y. Chang, V. Engel, and I. R. Sola, Manipulating the singlet–triplet transition in ion strings by nonresonant dynamic Stark effect, Theoretical Chemistry Accounts 132, 1359 (2013).
  * Falge  _et al._ [2014] M. Falge, P. Vindel-Zandbergen, V. Engel, M. Lein, B. Y. Chang, and I. R. Sola, The time-scale of nonlinear events driven by strong fields: can one control the spin coupling before ionization runs over?, Journal of Physics B: Atomic, Molecular and Optical Physics 47, 124027 (2014).
  * Jamshidi-Ghaleh  _et al._ [2017] K. Jamshidi-Ghaleh, Z. Ebrahimi-hamed, and M. Sahrai, Controlling nonlinear optical response in an open four-level molecular system using quantum control of spin-orbit interaction, The European Physical Journal Plus 132, 424 (2017).
  * Yumoto  _et al._ [2021] G. Yumoto, H. Hirori, F. Sekiguchi, R. Sato, M. Saruyama, T. Teranishi, and Y. Kanemitsu, Strong spin-orbit coupling inducing Autler-Townes effect in lead halide perovskite nanocrystals, Nature Commun 12, 3026 (2021).
  * Feng  _et al._ [2018] M. Feng, C. J. Kwong, T. S. Koh, and L. C. Kwek, Coherent transfer of singlet-triplet qubit states in an architecture of triple quantum dots, Phys. Rev. B 97, 245428 (2018).
  * Landau [1932] L. D. Landau, Zur Theorie der Energieübertragung II, Phys. Z. Sowjetunion 2, 46 (1932).
  * Zener [1932] C. Zener, Non-adiabatic crossing of energy levels, Proc. R. Soc. A 137, 696 (1932).
  * E.C.G. Stückelberg [1932] E.C.G. Stückelberg, Theory of inelastic collisions between atoms, Helv. Phys. Acta 5, 369 (1932).
  * Vitanov and Garraway [1996] N. V. Vitanov and B. M. Garraway, Landau-Zener model: Effects of finite coupling duration, Phys. Rev. A 53, 4288 (1996).
  * Vitanov [1999] N. V. Vitanov, Transition times in the Landau-Zener model, Phys. Rev. A 59, 988 (1999).
  * Wilkinson  _et al._ [1997] S. R. Wilkinson, C. F. Bharucha, M. C. Fischer, K. W. Madison, P. R. Morrow, Q. Niu, B. Sundaram, and M. G. Raizen, Experimental evidence for non-exponential decay in quantum tunnelling, Nature(London) 387, 575 (1997).
  * Sillanpää  _et al._ [2006] M. Sillanpää, T. Lehtinen, A. Paila, Y. Makhlin, and P. Hakonen, Continuous-time monitoring of landau-zener interference in a cooper-pair box, Phys. Rev. Lett. 96, 187002 (2006).
  * Zenesini  _et al._ [2009] A. Zenesini, H. Lignier, G. Tayebirad, J. Radogostowicz, D. Ciampini, R. Mannella, S. Wimberger, O. Morsch, and E. Arimondo, Time-Resolved Measurement of Landau-Zener Tunneling in Periodic Potentials, Phys. Rev. Lett. 103, 090403 (2009).
  * Kling  _et al._ [2010] S. Kling, T. Salger, C. Grossert, and M. Weitz, Atomic Bloch-Zener Oscillations and Stückelberg Interferometry in Optical Lattices, Phys. Rev. Lett. 105, 215301 (2010).
  * Zhou  _et al._ [2014] J. Zhou, P. Huang, Q. Zhang, Z. Wang, T. Tan, X. Xu, F. Shi, X. Rong, S. Ashhab, and J. Du, Observation of Time-Domain Rabi Oscillations in the Landau-Zener Regime with a Single Electronic Spin, Phys. Rev. Lett. 112, 010503 (2014).
  * Sun  _et al._ [2015] G. Sun, X. Wen, M. Gong, D.-W. Zhang, Y. Yu, S.-L. Zhu, J. Chen, P. Wu, and S. Han, Observation of coherent oscillation in single-passage Landau-Zener transitions, Scientific Reports 5, 8463 (2015).
  * Olson  _et al._ [2014] A. J. Olson, S.-J. Wang, R. J. Niffenegger, C.-H. Li, C. H. Greene, and Y. P. Chen, Tunable Landau-Zener transitions in a spin-orbit-coupled Bose-Einstein condensate, Phys. Rev. A 90, 013616 (2014).
  * Roland and Cerf [2002] J. Roland and N. J. Cerf, Quantum search by local adiabatic evolution, Phys. Rev. A 65, 042308 (2002).
  * Garanin and Schilling [2002] D. A. Garanin and R. Schilling, Effects of nonlinear sweep in the Landau-Zener-Stueckelberg effect, Phys. Rev. B 66, 174438 (2002).
  * Malossi  _et al._ [2013] N. Malossi, M. G. Bason, M. Viteau, E. Arimondo, R. Mannella, O. Morsch, and D. Ciampini, Quantum driving protocols for a two-level system: From generalized Landau-Zener sweeps to transitionless control, Phys. Rev. A 87, 012116 (2013).
  * Stefanatos and Paspalakis [2020] D. Stefanatos and E. Paspalakis, Speeding up adiabatic passage with an optimal modified roland–cerf protocol, Journal of Physics A: Mathematical and Theoretical 53, 115304 (2020).
  * Guéry-Odelin  _et al._ [2019] D. Guéry-Odelin, A. Ruschhaupt, A. Kiely, E. Torrontegui, S. Martínez-Garaot, and J. G. Muga, Shortcuts to adiabaticity: Concepts, methods, and applications, Rev. Mod. Phys. 91, 045001 (2019).
  * Suqing  _et al._ [2005] D. Suqing, L.-B. Fu, J. Liu, and X.-G. Zhao, Effects of periodic modulation on the Landau–Zener transition, Physics Letters A 346, 315 (2005).
  * Petiziol  _et al._ [2018] F. Petiziol, B. Dive, F. Mintert, and S. Wimberger, Fast adiabatic evolution by oscillating initial Hamiltonians, Phys. Rev. A 98, 043436 (2018).
  * Vitanov  _et al._ [2017] N. V. Vitanov, A. A. Rangelov, B. W. Shore, and K. Bergmann, Stimulated Raman adiabatic passage in physics, chemistry, and beyond, Rev. Mod. Phys. 89, 015006 (2017).
  * Kirova and Spano [2005] T. Kirova and F. C. Spano, Designing molecular eigenstates in a four-level $\Lambda$ system, Phys. Rev. A 71, 063816 (2005).
  * Cohen-Tannoudji [1996] C. N. Cohen-Tannoudji, The Autler-Townes Effect Revisited, in  _Amazing Light: A Volume Dedicated To Charles Hard Townes On His 80th Birthday_, edited by R. Y. Chiao (Springer New York, New York, NY, 1996) pp. 109–123.
  * Petiziol  _et al._ [2019] F. Petiziol, B. Dive, S. Carretta, R. Mannella, F. Mintert, and S. Wimberger, Accelerating adiabatic protocols for entangling two qubits in circuit QED, Phys. Rev. A 99, 042315 (2019).
  * Petiziol and Wimberger [2019] F. Petiziol and S. Wimberger, Effect of phase errors on a quantum control protocol using fast oscillations, Condens. Matter 4, 34 (2019).
  * Berry [2009] M. V. Berry, Transitionless quantum driving, J. Phys. A 42, 365303 (2009).
  * Demirplak and Rice [2003] M. Demirplak and S. A. Rice, Adiabatic population transfer with control fields, J. Phys. Chem. A 107, 9937 (2003).
  * Fleischhauer  _et al._ [1999] M. Fleischhauer, R. Unanyan, B. Shore, and K. Bergmann, Coherent population transfer beyond the adiabatic limit: Generalized matched pulses and higher-order trapping states, Phys. Rev. A 59, 3751 (1999).
  * Johansson  _et al._ [2013] J. Johansson, P. Nation, and F. Nori, Qutip 2: A python framework for the dynamics of open quantum systems, Computer Physics Communications 184, 1234 (2013).


# Novel Slot Detection: A Benchmark for Discovering Unknown Slot Types in the
Task-Oriented Dialogue System

Yanan Wu1∗, Zhiyuan Zeng1∗, Keqing He2∗, Hong Xu1
Yuanmeng Yan1, Huixing Jiang2, Weiran Xu1
1Pattern Recognition & Intelligent System Laboratory
1Beijing University of Posts and Telecommunications, Beijing, China
2Meituan Group, Beijing, China
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>The first three authors contribute
equally. Weiran Xu is the corresponding author.

###### Abstract

Existing slot filling models can only recognize pre-defined in-domain slot
types from a limited slot set. In the practical application, a reliable
dialogue system should know what it does not know. In this paper, we introduce
a new task, Novel Slot Detection (NSD), in the task-oriented dialogue system.
NSD aims to discover unknown or out-of-domain slot types to strengthen the
capability of a dialogue system based on in-domain training data. Besides, we
construct two public NSD datasets, propose several strong NSD baselines, and
establish a benchmark for future work. Finally, we conduct exhaustive
experiments and qualitative analysis to comprehend key challenges and provide
new guidance for future
directions111https://github.com/ChestnutWYN/ACL2021-Novel-Slot-Detection.

##  1 Introduction

Slot filling plays a vital role to understand user queries in personal
assistants such as Amazon Alexa, Apple Siri, Google Assistant, etc. It aims at
identifying a sequence of tokens and extracting semantic constituents from the
user queries. Given a large scale pre-collected training corpus, existing
neural-based models Mesnil et al. (2015); Liu and Lane (2015, 2016); Goo et
al. (2018); Haihong et al. (2019); Chen et al. (2019); He et al. (2020b, d);
Yan et al. (2020); Louvan and Magnini (2020); He et al. (2020a) have been
actively applied to slot filling and achieved promising results.

Existing slot filling models can only recognize pre-defined entity types from
a limited slot set, which is insufficient in the practical application
scenario. A reliable slot filling model should not only predict the pre-
defined slots but also detect potential unknown slot types to know what it
doesn’t know, which we call Novel Slot Detection (NSD) in this paper. NSD is
particularly crucial in deployed systems—both to avoid performing the wrong
action and to discover potential new entity types for future development and
improvement. We display an example as Fig 1 shows.

Figure 1: An example of Novel Slot Detection in the task-oriented dialogue
system. Without NSD, the dialogue system gives the wrong response since it
misunderstands the unknown slot “is this my world” as the in-domain _playlist_
type. In contrast, NSD recognizes “is this my world” as _NS_ and the system
gives a fallback response. Meanwhile, with human-in-the-loop annotation, the
system can increase its functions or skills.

Utterance | play | is | this | my | world | by | leo | arnaud
---|---|---|---|---|---|---|---|---
Slot Filling Labels | O | B-album | I-album | I-album | I-album | O | B-artist | I-artist
Novel Slot Detection Labels | O | NS | NS | NS | NS | O | B-artist | I-artist

Table 1: Comparison between slot filling and novel slot detection. In the
novel slot detection labels, we consider “album” as an unknown slot type that
is out of the scope of the pre-defined slot set. Meanwhile, “artist” belonging
to in-domain slot types still needs to be recognized as the original slot
filling task.

In this paper, we define _Novel Slot_ (NS) as new slot types that are not
included in the pre-defined slot set. NSD aims to discover potential new or
out-of-domain entity types to strengthen the capability of a dialogue system
based on in-domain pre-collected training data. There are two aspects in the
previous work related to NSD, out-of-vocabulary (OOV) recognition Liang et al.
(2017a); Zhao and Feng (2018); Hu et al. (2019); He et al. (2020c, d); Yan et
al. (2020); He et al. (2020e) and out-of-domain (OOD) intent detection Lin and
Xu (2019); Larson et al. (2019); Xu et al. (2020a); Zeng et al. (2021b, a).
OOV means many slot types can have a large number of new slot values while the
training set only obtains a tiny part of slot values. OOV aims to recognize
unseen slot values in training set for pre-defined slot types, using character
embedding Liang et al. (2017a), copy mechanism Zhao and Feng (2018), few/zero-
shot learning Hu et al. (2019); He et al. (2020e); Shah et al. (2019),
transfer learning Chen and Moschitti (2019); He et al. (2020c, b) and
background knowledge Yang and Mitchell (2017); He et al. (2020d), etc.
Compared to OOV recognition, our proposed novel slot detection task focuses on
detecting unknown slot types, not just unseen values. NSD faces the challenges
of both OOV and no sufficient context semantics (see analysis in Section 6.2),
greatly increasing the complexity of the task. Another line of related work is
OOD intent detection Hendrycks and Gimpel (2017); Lee et al. (2018); Lin and
Xu (2019); Ren et al. (2019); Zheng et al. (2020); Xu et al. (2020a) which
aims to know when a query falls outside the range of predefined supported
intents. The main difference is that NSD detects unknown slot types in the
token level while OOD intent detection identifies out-of-domain intent
queries. NSD requires a deep understanding of the query context and is prone
to label bias of _O_ (see analysis in Section 5.3.1), making it challenging to
identify unknown slot types in the task-oriented dialog system.

In this paper, we first introduce a new and important task, Novel Slot
Detection (NSD), in the task-oriented dialogue system (Section 2). NSD plays a
vital role in avoiding performing the wrong action and discovering potential
new entity types for the future development of dialogue systems. Then, we
construct two public NSD datasets, Snips-NSD and ATIS-NSD, based on the
original slot filling datasets, Snips Coucke et al. (2018) and ATIS Hemphill
et al. (1990) (Section 2.2). From the perspective of practical application, we
consider three kinds of dataset construction strategies, Replace, Mask and
Remove. Replace denotes we label the novel slot values with all _O_ in the
training set. Mask is to label with all _O_ and mask the novel slot values.
Remove is the most strict strategy where all the queries containing novel
slots are removed. We dive into the details of the three different
construction strategies in Section 3.2 and perform a qualitative analysis in
Section 5.3.1. Besides, we propose two kinds of evaluation metrics, span-level
F1 and token-level F1 in Section 3.4, following the slot filling task. Span F1
considers the exact matching of a novel slot span while Token F1 focuses on
prediction accuracy on each word of a novel slot span. We discuss performance
comparison between the two metrics and propose a new metric, restriction-
oriented span evaluation (ROSE), to combine the advantages of both in Section
5.3.3. Then, we establish a fair benchmark and propose extensive strong
baselines for NSD in Section 4. Finally, we perform exhaustive experiments and
qualitative analysis to shed light on the challenges that current approaches
faced with NSD in Section 5.3 and 6.

Our contributions are three-fold: (1) We introduce a Novel Slot Detection
(NSD) task in the task-oriented dialogue system. NSD helps avoid performing
the wrong action and discovering potential new entity types for increasing
functions of dialogue systems. (2) We construct two public NSD datasets and
establish a benchmark for future work. (3) We conduct exhaustive experiments
and qualitative analysis to comprehend key challenges and provide new guidance
for future NSD work.

##  2 Problem Formulation

###  2.1 Slot Filling

Given a sentence $X=\\{x_{1},...,x_{n}\\}$ with $n$ tokens, the slot filling
task is to predict a corresponding tag sequence $Y=\\{y_{1},...,y_{n}\\}$ in
BIO format, where each $y_{i}$ can take three types of values: B-slot_type,
I-slot_type and _O_ , where “B” and “I” stand for the beginning and
intermediate word of a slot and “ _O_ ” means the word does not belong to any
slot. Here, slot filling assumes $y_{i}\in\mathbf{y}$, where $\mathbf{y}$
denotes a pre-defined slot set of size $\mathcal{M}$. Current approaches
typically model slot filling as a sequence labeling problem using RNN Liu and
Lane (2015, 2016); Goo et al. (2018) or pre-trained language models Chen et
al. (2019).

###  2.2 Novel Slot Detection

Original Utterance | play | is | this | my | world | by | leo | arnaud
---|---|---|---|---|---|---|---|---
Original Slot Filling Labels | O | B-album | I-album | I-album | I-album | O | B-artist | I-artist
Strategy | Replace | play | is | this | my | world | by | leo | arnaud
O | O | O | O | O | O | B-artist | I-artist
Mask | play | MASK | MASK | MASK | MASK | by | leo | arnaud
O | O | O | O | O | O | B-artist | I-artist
Remove | - | - | - | - | - | - | - | -
- | - | - | - | - | - | - | -

Table 2: Comparison between three processing strategies in the training set.
We consider “album” as an unknown slot type and “-” denotes the sentence is
removed from the training data.

We refer to the above training data $D$ as in-domain (IND) data. Novel slot
detection aims to identify unknown or out-of-domain (OOD) slot types via IND
data while correctly labeling in-domain data. We denote unknown slot type as
_NS_ and in-domain slot types as IND in the following sections. Note that we
don’t distinguish between B-NS and I-NS and unify them as _NS_ because we
empirically find existing models hardly discriminate B and I for an unknown
slot type. We provide a detailed analysis in Section 5.3.3. We show an example
of NSD in Table 1. The challenges of recognizing NSD come from two aspects,
_O_ tags and in-domain slots. On the one hand, models need to learn entity
information for distinguishing _NS_ from _O_ tags. On the other hand, they
require discriminating _NS_ from other slot types in the pre-defined slot set.
We provide a detailed error analysis in Section 6.1.

##  3 Dataset

Since there are not existing NSD datasets, we construct two new datasets based
on the two widely used slot filling datasets, Snips Coucke et al. (2018) and
ATIS Hemphill et al. (1990). We first briefly introduce Snips and ATIS, then
elaborate on data construction and processing in detail, and display the
statistic of our NSD datasets, Snips-NSD and ATIS-NSD. Finally, we define two
evaluation metrics for the NSD task, Span F1 and Token F1.

###  3.1 Original Slot Filling Datasets

Snips222https://github.com/sonos/nlu-benchmark/tree/master/2017-06-custom-
intent-engines is a custom intent engine dataset. It originally has 13,084
train utterances, 700 and 700 test utterances.
ATIS333https://github.com/yvchen/JointSLU/tree/master/data contains audio
recordings of people making flight reservations. It originally has 4,478 train
utterances, 500 dev and 893 test utterances. The full statistic is shown in
Table 3. Note that the vocabulary only contains words in the training set, and
test set words that do not exist in the vocabulary are referred to OOV words.
The percentage of OOV words represents the portion of OOV words in the test
set.

###  3.2 Data Construction and Processing

For Snips and ATIS datasets, we keep some slot classes in training as unknown
and integrate them back during testing, following Fei and Liu (2016); Shu et
al. (2017); Lin and Xu (2019). We randomly select part of slot types in Snips
and ATIS as unknown slots(5%, 15%, and 30% in this paper). Note that the
original train/val/test split is fixed. Considering class imbalance, we
perform weighted sampling where the chosen probability is relevant to the
number of class examples similar to Lin and Xu (2019). To avoid randomness of
experiment results, we report the average result over 10 runs.

| Snips | ATIS
---|---|---
Vocabulary Size | 11,241 | 722
Percentage of OOV words | 5.95% | 0.77%
Number of Slots | 39 | 79
Training Set Size | 13,084 | 4,478
Development Set Size | 700 | 500
Testing Set Size | 700 | 893
Table 3: Statistics of ATIS and Snips datasets.

After we choose the unknown slot types, a critical problem is how to handle
sentences including these unknown slot types in training set. For OOD intent
detection, we just need to remove these sentences in training and validation
set. However, for Novel Slot Detection, a sentence perhaps contains both in-
domain slots and unknown slots, which is nontrivial for tackling unknown slots
at the token level. We need to balance the performance of recognizing unknown
slots and in-domain slots. Therefore, we propose three different processing
strategies as follows: (1) Replace: We label the unknown slot values with all
_O_ in the training set while the original values remain unchanged. (2) Mask:
We label the unknown slot values with all _O_ and mask these slot values with
a special token MASK. (3) Remove: All the sentences containing unknown slots
are directly removed.

We display examples of the above three strategies in Table 2. For the val and
test set, we just label the unknown slot values with all _NS_ while keeping
the in-domain labeling fixed. Note that _NS_ tags only exist in the val and
test set, not in the training set. Besides, we keep original in-domain slots
fixed to evaluate the performance of both _NS_ and in-domain slots. We aim to
simulate the practical scenario where we can hardly know what unknown slots
are. These three strategies all have its practical significance. Compared with
others, Remove is the most suitable strategies for real-world scenarios. In
practical scenario, dialog systems first train in the data set labeled by
human annotators, and then applied to the actual application. In the process
of interaction with the real users, novel slot types appear gradually.
Therefore, we consider that the training set doesn’t contain potential novel
slots sentences. In other words, Remove is the most suitable strategy for NSD
in real applications. What’s more, Section 5.3.1 demonstrates Remove performs
best while the others suffer from severe model bias by _O_ tags. Therefore, we
adopt Remove as the main strategy in this paper.

###  3.3 Statistic of New NSD Datasets

Table 4 shows the detailed statistics of Snips-NSD-15% constructed by Remove
strategy, where we choose 15% classes in the training data as unknown slots.
444Since different proportions of unknown slots have different statistics,
here we only display the results of Snips-NSD-15% for brevity. Combining Table
3 and Table 4, we can find Remove strategy removes 28.70% of queries in the
original Snips training set, hence increases the percentage of OOV word from
5.95% to 8.51%. And unknown slot values account for 12.29% of total slot
values in the test set.

Snips-NSD-15% | Train | Val | Test
---|---|---|---
number of in-domain slots | 33 | 33 | 33
number of unknown slots | 6 | 6 | 6
percentage of OOV words | - | - | 8.51%
number of queries | 9,329 | 700 | 700
| number of queries
---
including unknown slots
0 | 192 | 202
number of slot values | 23,176 | 1,794 | 1,790
number of unknown slot values | 0 | 210 | 220

Table 4: The detailed statistics of Snips-NSD-15%.

###  3.4 Metrics

The traditional slot filling task uses Span F1
555https://www.clips.uantwerpen.be/conll2000/chunking/conlleval.txt for
evaluation. Span F1 considers the exact span matching of an unknown slot span.
However, we find in Section 5.3.3 that this metric is too strict to NSD
models. In the practical application, we only need to coarsely mine parts of
words of unknown slots, then send these queries containing potential unknown
slot tokens to human annotators, which has effectively reduced extensive labor
and improved efficiency. Therefore, we define a more reasonable metric, Token
F1 which focuses on the word-level matching of a novel slot span. We also
propose a new metric, Restriction-Oriented Span Evaluation (ROSE), for a fair
comparison in Section 5.3.3.

Figure 2: The overall architecture of our approach.

##  4 Methodology

In this section, we introduce the NSD models proposed in this paper and
illustrate the differences between the various parallel approaches during the
training and test stage.

###  4.1 Overall Framework

The overall structure of model is shown in Fig 2. In the training stage, we
either train a multiple-class classifier or binary classifier using different
training objectives. We use public BERT-large Devlin et al. (2019) embedding
layer and BiLSTM-CRF Huang et al. (2015) for token level feature extraction.
Then, in the test stage, we use the typical neural multiple classifier to
predict the in-domain slot labels. Meanwhile, we use the detection algorithm,
MSP or GDA to figure out novel slot tokens. Finally, we override the slot
token labels which are detected as NS. In terms of training objectives,
detection algorithms, and distance strategies, we compare different variants
as follows.

Models | 5% | 15% | 30%
---|---|---|---
IND | NSD | IND | NSD | IND | NSD
detection method | objective | distance strategy | Span F1 | Span F1 | Token F1 | Span F1 | Span F1 | Token F1 | Span F1 | Span F1 | Token F1
MSP | binary | - | 87.21 | 12.34 | 25.16 | 71.44 | 12.31 | 39.50 | 58.88 | 8.73 | 40.38
multiple | - | 88.05 | 14.04 | 30.50 | 79.71 | 20.97 | 40.02 | 78.52 | 25.26 | 46.91
binary+multiple | - | 89.59 | 23.58 | 37.55 | 83.72 | 24.70 | 45.32 | 79.08 | 30.66 | 52.10
GDA | binary | difference | 87.95 | 23.83 | 35.83 | 83.65 | 22.06 | 43.99 | 78.72 | 32.50 | 44.13
binary | minimum | 61.29 | 10.36 | 17.08 | 49.11 | 16.91 | 31.10 | 48.07 | 15.56 | 33.78
multiple | difference | 93.14 | 29.73 | 45.99 | 90.07 | 31.96 | 53.02 | 85.56 | 36.16 | 54.55
multiple | minimum | 93.10 |  31.67* |  46.97* | 90.18 | 32.19 |  53.75* |  86.26* |  38.64* |  55.24*

Table 5: IND and NSD results with different proportions (5%, 15% and 30%) of
classes are treated as unknown slots on Snips-NSD. * indicates the significant
improvement over all baselines (p $<$ 0.05).

Models | 5% | 15% | 30%
---|---|---|---
IND | NSD | IND | NSD | IND | NSD
detection method | objective | distance strategy | Span F1 | Span F1 | Token F1 | Span F1 | Span F1 | Token F1 | Span F1 | Span F1 | Token F1
MSP | binary | - | 92.04 | 19.73 | 29.63 | 91.74 | 23.40 | 33.89 | 80.49 | 21.88 | 39.17
multiple | - | 94.33 | 27.15 | 31.16 | 92.54 | 39.88 | 42.29 | 87.63 | 40.42 | 47.64
binary+multiple | - | 94.41 | 32.49 | 43.48 | 93.29 | 41.23 | 43.13 | 90.14 | 41.76 | 51.87
GDA | binary | difference | 93.69 | 27.02 | 34.21 | 92.13 | 30.51 | 36.30 | 88.73 | 30.91 | 45.64
binary | minimum | 93.57 | 15.90 | 20.96 | 90.98 | 24.53 | 27.26 | 88.21 | 26.40 | 39.83
multiple | difference | 95.20 |  47.78* |  51.54* | 93.92 |  50.92* |  52.24* | 92.02 |  51.26* |  56.59*
multiple | minimum |  95.31* | 41.74 | 45.91 | 93.88 | 43.78 | 46.18 | 91.67 | 45.44 | 52.37

Table 6: IND and NSD results with different proportions (5%, 15% and 30%) of
classes are treated as unknown slots on ATIS-NSD. * indicates the significant
improvement over all baselines (p $<$ 0.05).

Training objective. For in-domain slots, we propose two training objectives.
Multiple classifier refers to the traditional slot filling objective setting,
which performs token-level multiple classifications on the BIO tags Ratinov
and Roth (2009) combined with different slots. Binary classifier unifies all
non-O tags into one class, and the model makes a token-level binary
classification of O or non-O on the sequence. Note that in the test stage, for
in-domain prediction, we both use the multiple classifier. While, for novel
slot detection, we use the multiple classifier, or the binary classifier, or
both of them. In Table 5 and Table 6, binary+multiple means the token will be
labeled as NS only if both classifiers predict it as NS.

Detection algorithm. MSP and GDA are detection algorithms in the test stage.
MSP (Maximum Softmax Probability) Hendrycks and Gimpel (2017) applies a
threshold on the maximum softmax probability, if the maximum falls below the
threshold, the token will be predicted to be a novel slot token. GDA (Gaussian
Discriminant Analysis) Xu et al. (2020a) is a generative distance-based
classifier for out-of-domain detection with Euclidean space. We treat tokens
not belonging to any in-domain slots (including O) as novel slot tokens for
both methods. For example, with a binary classifier, if the softmax
probabilities belonging to O or non-O are both lower than an MSP threshold,
then the token is labeled as NS.

Distance strategy. The GDA detection is based on the distances between a
target and each slot representation cluster. In original GDA, when the minimum
distance is greater than a certain threshold, it is predicted to be novel
slots. We propose a novel strategy named Difference, which uses the maximum
distance minus the minimum distance, when the difference value of a target is
less than a threshold, it is predicted as novel slots. Both of their
thresholds are obtained by optimizing the NSD metrics on the validation set.

##  5 Experiment and Analysis

###  5.1 Implementation Details

We use the public pre-trained Bert-large-uncased model to embed tokens which
has 24 layers, 1024 hidden states, 16 heads and 336M parameters. The hidden
size for the BiLSTM layer is set to 128. Adam is used for optimization with an
initial learning rate of 2e-5. The dropout value is fixed as 0.5, and the
batch size is 64. We train the model only on in-domain labeled data. The
training stage has an early stopping setting with patience equal to 10. We use
the best F1 scores on the validation set to calculate the MSP and GDA
thresholds adaptively. Each result of the experiments is tested for 10 times
under the same setting and reports the average value. The training stage of
our model lasts about 28 minutes on single Tesla T4 GPU(16 GB of memory).

###  5.2 Main Results

Table 5 and 6 show the experiment results with seven different models on two
benchmark slot filling datasets Snips-NSD and ATIS-NSD constructed by Remove
strategy. We both report NSD and IND results using Span F1 and Token F1. We
compare these models from three perspectives, detection method, objective and
distance strategy in the following. The analysis of effect of the proportion
of unknown slot types is described in 5.3.2.

Detection Method: MSP vs GDA. Under the same setting of objective, GDA
performs better than MSP in both IND and NSD, especially in NSD. We argue that
GDA models the posterior distribution on representation spaces of the feature
extractor and avoids the issue of overconfident predictions Guo et al. (2017);
Liang et al. (2017b, 2018). Besides, comparing Snips-NSD and ATIS-NSD, NSD
Token F1 scores on ATIS-NSD are much higher than Snips-NSD but no significant
difference exists for NSD Span F1 scores. The reason is that Snips-NSD has a
higher average entity length (1.83) than ATIS-NSD (1.29), making it harder to
detect the exact _NS_ span.

Objective: Binary vs Multiple. Under all settings, Multiple outperforms Binary
with a large margin on two datasets in both IND and NSD metrics. For MSP,
combining Multiple and Binary get higher F1 scores. Specifically, the Binary
classifier is used to calculate the confidence of a token belonging to non-_O_
type, which can judge whether the token belongs to entities and distinguish
_NS_ from type _O_. On the other hand, we use the Multiple classifier to
calculate the confidence for tokens that are of type _NS_ , to distinguish
_NS_ from all predefined non-_O_ slot types. For GDA, we do not combine
Multiple and Binary because of poor performance. Multiple achieves the best
results for all the IND and NSD F1 scores. We suppose multi-class
classification can better capture semantic features than binary
classification.

Strategy | 5% | 15% | 30%
---|---|---|---
IND | NSD | IND | NSD | IND | NSD
Span | Span | Token | Span | Span | Token | Span | Span | Token
Replace | 94.52 | 1.93 | 5.27 | 94.33 | 0.66 | 2.29 | 94.02 | 0.27 | 0.82
Mask | 90.08 | 23.10 | 37.91 | 86.52 | 25.07 | 45.92 | 83.37 | 32.14 | 50.68
Remove | 93.10 | 31.67 | 46.97 | 90.18 | 32.19 | 53.75 | 86.26 | 38.64 | 55.24

Table 7: Comparison between different data processing strategies on Snips-NSD
using GDA+Multiple+Minimum.

Distance Strategy: Minimum vs Difference. We find under the same setting of
Binary, Difference strategy outperforms Minimum on both datasets for NSD
metrics. But under the same setting of Multiple, there is no consistent
superiority between the two distance strategies. For example, Difference
outperforms Minimum for NSD metrics on ATIS-NSD, opposite to the results on
Snips-NSD. We argue different distance strategies are closely related to
objective settings and dataset complexity. We will leave the theoretical
analysis to the future.

###  5.3 Qualitative Analysis

####  5.3.1 Effect of Different Data Processing Strategies

Table 7 displays IND and NSD metrics of three different dataset processing
strategies on Snips-NSD using the same model GDA+Multiple+Minimum. In this
section, we will dive into the analysis of the effects of different data
processing strategies. Results show the Replace strategy gets poor performance
in NSD, which proves labeling unknown slots as _O_ tags will severely mislead
the model. The Mask and Remove strategies are more reasonable since they
remove unknown slots from the training data. Their main difference is that
Mask only deletes token-level information, while Remove even eliminates the
contextual information. For NSD in all datasets, Remove gains significantly
better performance on both Token F1 and Span F1 than Mask by 9.06%(5%),
7.83%(15%) and 4.56%(30%) on Token F1, and 8.57%(5%), 7.12%(15%) and 6.5%(30%)
on Span F1. We argue the remaining context is still misleading even if the
novel slot tokens are not directly trained in the Mask strategy. Besides, Mask
does not conform to the real NSD scenario. Generally, Remove is the most
suitable strategy for NSD in real applications and can achieve the best
performance.

####  5.3.2 Effect of the Proportion of Unknown Slot Types

Fig LABEL:fig:propotion displays the effect of the proportion of unknown slot
types using the Remove strategy in GDA+Multiple+Minimum. Results show that
with the increase of the proportion of unknown slot types, the NSD F1 scores
get improvements while IND F1 scores decrease. We suppose fewer in-domain slot
types help the model distinguish unknown slots from IND slots, thus NSD F1
scores get improvements. However, for in-domain slot detection, since Remove
deletes all the sentences containing unknown slots in the training data, our
models suffer from the lack of sufficient context to recognize IND slots so
IND F1 scores decrease.

Figure 4: Effect of varying degrees of restrictions

| GDA+mul.+min. | MSP+bin.+mul.
---|---|---
ROSE-mean | 40.73 | 34.71
ROSE-100% | 40.39 | 33.74
ROSE-50% | 41.00 | 35.46

Table 8: ROSE metrics on Snips-NSD using GDA+Multiple+Minimum and
MSP+Binary+Multiple

####  5.3.3 New Metric: ROSE

The previous results have shown Span F1 is much lower than the token F1. The
reason is that Span F1 is a strict metric, where the model needs to correctly
predict all _NS_ tokens and the correct boundary. This is difficult for NSD
models due to the lack of supervised information. In fact, NSD models only
need to mark some tokens in the span of novel slots and send the total
sequence containing the _NS_ tokens back to the humans. A small number of
token omissions or misjudgments are acceptable. Therefore, to meet a
reasonable NSD scenario, we propose a new metric, restriction-oriented span
evaluation (ROSE), to evaluate the span prediction performance under different
restrictions. First, we do not punish the situation where tokens prediction
exceeds the span. Then, we consider a span is correct when the number of
correctly predicted tokens is greater than a settable proportion $p$ of the
span length. We take the average of the ROSE score and the original span F1 to
avoid the model obtaining an outstanding result through over-long prediction.
The results using Snips with 15% of novel slots are shown in Figure 4. As the
degree of restriction increases, the metrics tend to decline. It indicates
that the model can mostly identify more than half of the tokens in spans. To
make a comprehensive evaluation, we defined the ROSE-mean, namely the mean of
ROSE-25%, ROSE-50%, ROSE-75%, and ROSE-100%. We present results on part of
proposed models in Table 8.

####  5.3.4 Analysis of Single Unknown Slot

To analyze the relationship between NSD performance and a single specific
slot, we calculate the token and span metrics treating each single slot type
as an unknown slot and show the results of the top five and bottom five for
Token F1 scores in Table 9. We find that the slots with better performance
often account for a larger percentage of the data set, such as Object_name or
Entity_name. They also tend to have a larger value space, such as TimeRange,
Music_item, or Artist. These characteristics allow the semantic representation
of these slots to be distributed over a large area rather than clustered
tightly together. We consider that this distribution is more reasonable
because in a real application scenario, novel slots are diverse and its
distribution tends to be diffuse. Performance on these types also proves that
the NSD models we propose can be better generalized to a reasonable data
setting.

Type | Proportion(%) | Span Length | Token F1 | Span F1
---|---|---|---|---
top 5 | Object_name | 21.42 | 3.71 | 55.64 | 20.82
TimeRange | 15.29 | 2.35 | 53.65 | 30.15
Entity_name | 23.14 | 3.09 | 48.56 | 22.83
Music_item | 14.86 | 1.05 | 46.23 | 34.59
Artist | 15.29 | 2.05 | 45.26 | 26.36
bottom 5 | City | 8.57 | 1.32 | 18.72 | 15.85
Country | 6.29 | 1.57 | 14.19 | 11.11
State | 5.54 | 1.10 | 13.55 | 10.83
Best_rating | 6.14 | 1.00 | 11.04 | 11.04
Year | 3.43 | 1.00 | 10.24 | 10.24

Table 9: Results of single unknown slot.

Type 1 | Type 2 | Token F1 | Span F1
---|---|---|---
Object_name | - | 55.64 | 20.82
TimeRange | - | 53.65 | 30.15
Party_size_number | - | 33.44 | 28.57
City | - | 18.72 | 15.85
State | - | 13.55 | 10.83
Object_name | TimeRange | 53.88 | 23.37
Object_name | Party_size_number | 52.81 | 22.35
Object_name | City | 57.92 | 21.42
Object_name | State | 56.32 | 19.27
TimeRange | Party_size_number | 71.27∗ | 51.03∗
City | State | 29.33∗ | 27.14∗

Table 10: Results of combining multiple unknown slots. * denotes that NSD
performance of the combination of two unknown slots is significantly better
than each single slot.

####  5.3.5 Analysis for Relationship of Multiple Unknown Slots

In order to explore the effect of inter-slot relationships on NSD, we
conducted experiments in which two types are mixed as novel slots. Some of the
results are shown in Table 10. In the five types shown in the table,
Object_name is an open vocabulary slot with a wide range of values and
contains many OOV tokens, TimeRange and Party_size_number often contain
numbers, City and State are usually similar in semantics and context. We found
that when the other types combined with Object_name, NSD performance is often
maintained close to treat Object_name as a novel slot alone. The reason, on
the one hand, is that the proportion of other types in the dataset is
relatively small, so the overall impact on the metrics is smaller. On the
other hand, due to the large semantic distribution range of the open
vocabulary slot, there is a latent inclusion relationship for other types, so
the mixing of a single type tends to have a slight impact on the NSD
performance. We also found that the appropriate combination can significantly
improve the efficiency of NSD. Such as TimeRange with Party_size_number, or
City with State. This indicates that when the novel slot is similar to the in-
domain slot, the model tends to predict the novel slot as a similar slot,
which leads to errors. When both are treated as novel slots, these errors can
be mitigated.

NSD error proportion(%) | O | Open vocabulary slots | Other slots | Sum
---|---|---|---|---
Prediction is NS | 17.79 | 18.84 | 9.07 | 45.70
Target is NS | 18.47 | 7.54 | 28.29 | 54.30
Sum | 36.26 | 26.38 | 37.36 | 100.00

Table 11: Relative proportions of several types of errors.

Error type | NS | Example
---|---|---
NS to O |  | movie_name
---
(m_name)
| text: when will paris by night aired
---
true: O O B-m_name I-m_name I-m_name O
predict: O O NS O NS O
| NS to
---
open slot
album |  | text: play the insoc ep
---
true: O B-album I-album I-album
predict: O B-object_name I-object_name NS
| NS to
---
other slot
artist |  | text: play kurt cobain ballad tunes
---
true: O B-artist I-artist B-music_item O
predict: O B-genre I-genre B-music_item O
O to NS | artist |  | text: the workout playlist needs more chris cross
---
true: O B-playlist O O O B-artist I-artist
predict: O B-playlist O O NS NS NS
| open slots
---
to NS
object_type |  | text: tell me the actors of the saga awards
---
true: O O O B-object_name O O B-object_type O
predict: O O O NS O O NS O
| other slots
---
to NS
city |  | text: what is the weather of east portal ks
---
true: O O O O O B-city I-city B-state
predict: O O O O O NS NS NS

Table 12: Error case from NSD prediction.

##  6 Discussion

In this section, we empirically divide all the error samples into three
categories. Each type of problem contains two aspects, corresponding to NSD
precision and recall, respectively. We present the relative proportions of
several types of errors in Table 11, which using Snips dataset with 5% novel
slots on GDA+multiple+minimum model. For each error type, we present an
example in Table 12 to describe the characteristics and analyze the causes.
Then, we dive into identifying the key challenges and finally proposed
possible solutions for future work.

###  6.1 Error Analysis

Tag _O_. Tag _O_ is the largest and most widely distributed type in the
dataset, and it generally refers to the independent function tokens.
Therefore, when identifying, it is easy to be confused with other types, and
the confusion is more serious for novel slots without supervised learning. We
observed that tokens with _O_ label detected as novel slots usually exist near
spans, and the function words in the span labeled as a novel slot have a
probability of being predicted as _O_. We consider that this kind of problem
is related to the context. Although the processing strategy of Remove can
effectively reduce the misleading of _O_ for the novel slots, tag _O_ will
still be affected by context information of other in-domain slots.

Open Vocabulary Slots. We observe that a large number of novel slot tokens are
mispredicted as open vocabulary slots, while the reverse situation is much
less likely to happen. This indicates that in Snips, open vocabulary slots
tend to overlap or contain most other slots semantically. Even in traditional
slot filling tasks, open vocabulary slots are often confused with other slots.
We demonstrate this hypothesis in the analysis. Section 5.3.5 shows that NSD
performs better when open vocabulary slots are treated as novel slots, and
Section 5.3.4 shows that there is no significant performance change when open
vocabulary slots are mixed with some semantically concentrated slots. The
reason for this problem is that the definition of the dataset is not
reasonable. Slots with a large value range can hardly help the personal
assistant to give an appropriate reply, and the supervised information of
these slots is usually incomplete.

Similar Slots. Except for the two cases mentioned above, predicting novel
slots as other in-domain slots is the most common type of error, in which
similar slots account for a large part of it. Due to the overlap between
vocabulary or shared similar context, the model often tend to be overconfident
to predict similar slot labels, we analyze the phenomenon in Table 10, when
similar types is treated as a new slot at the same time, NSD efficiency will
rise significantly. We employ a generative classification method GDA, compared
with the traditional MSP method, to make full use of data features and
alleviate the problem.

###  6.2 Challenges

Based on the above analysis, we summarize the current challenges faced by the
NSD task:

Function tokens. Articles, prepositions, and so on that act as connective
words in a sequence. It is usually labeled with type _O_ , but also found in
some long-span slots, such as Movie_name. It can lead to confusion between _O_
and novel slot when this kind of slot is the target of NSD.

Insufficient context. Correct slot detection often depends on the context, and
this supervised information is missing for novel slots. Models can only
conduct NSD to tokens using the original embeddings or representations trained
in other contexts, which can lead to bias in the semantic modeling of the
novel slot.

Dependencies between slots. There are some semantic overlaps or inclusion
relationships in the slot definition of the current benchmark slot filling
datasets. As a result, the semantic features are not sufficiently
discriminative, and thus some outliers tokens in in-domain slots are easily
confused with the novel slots.

Open vocabulary slots. Open vocabulary slots is a special kind of slot, its
definition is usually macroscopic and can be further divided, the value range
is broad. The representation distribution for Open vocabulary slots tends to
be diffuse and uneven, which can be misleading to NSD.

###  6.3 Future Directions

For tag _O_ , a possible solution is to use a binary model to assist
identification between _O_ and non-_O_ function tokens, we provide a simple
method in this paper and leave further optimizing to future work. Then, to
decouple the dependencies between slots, it is critical to learn more
discriminative features for in-domain data, using contrastive learning or
prototypical network is expected to help. Besides, in the traditional slot
filling task, the open vocabulary slot problem has been researched for a long
time, and accumulate many achievements. Adaptive combination and improvement
of relevant methods with NSD tasks is also an important direction of our
future research.

##  7 Related Work

OOV Recognition OOV aims to recognize unseen slot values in training set for
pre-defined slot types, using character embedding Liang et al. (2017a), copy
mechanism Zhao and Feng (2018), few/zero-shot learning Hu et al. (2019); Shah
et al. (2019), transfer learning Chen and Moschitti (2019); He et al. (2020c)
and background knowledge Yang and Mitchell (2017); He et al. (2020d), etc. Our
proposed NSD task focuses on detecting unknown slot types, not just unseen
values.

OOD Intent Detection Lee et al. (2018); Lin and Xu (2019); Xu et al. (2020a)
aim to know when a query falls outside the range of predefined supported
intents. Generally, they first learn discriminative intent representations via
in-domain (IND) data, then employs detecting algorithms, such as Maximum
Softmax Probability (MSP) Hendrycks and Gimpel (2017), Local Outlier Factor
(LOF) Lin and Xu (2019), Gaussian Discriminant Analysis (GDA) Xu et al.
(2020b) to compute the similarity of features between OOD samples and IND
samples. Compared to our proposed NSD, the main difference is that NSD detects
unknown slot types in the token level while OOD intent detection identifies
sentence-level OOD intent queries.

##  8 Conclusion

In this paper, we defined a new task, Novel Slot Detection(NSD), then provide
two public datasets and establish a benchmark for it. Further, we analyze the
problems of NSD through multi-angle experiments and extract the key challenges
of the task. We provide some strong models for these problems and offer
possible solutions for future work.

## Acknowledgements

This work was partially supported by National Key R&D Program of China No.
2019YFF0303300 and Subject II No. 2019YFF0303302, DOCOMO Beijing
Communications Laboratories Co., Ltd, MoE-CMCC ”Artifical Intelligence”
Project No. MCM20190701.

## Broader Impact

Dialog systems have demonstrated remarkable performance across a wide range of
applications, with the promise of a significant positive impact on human
production mode and lifeway. The first step of the dialog system is to
identify users’ key points. In practical industrial scenario, users may make
unreasonable queries which fall outside of the scope of the system-supported
slot types. Previous dialogue systems will ignore this problem, which will
lead to wrong operations and limit the system’s development. In this paper, we
firstly propose to detect not only pre-defined slot types but also potential
unknown or out-of-domain slot types using MSP and GDA methods. According to
exhaustive experiments and qualitative analysis, we also discuss several major
challenges in Novel Slot Detection for future work. The effectiveness and
robustness of the model are significantly improved by adding Novel Slot
Detection, which takes a step towards the ultimate goal of enabling the safe
real-world deployment of dialog systems in safety-critical domains. The
experimental results have been reported on standard benchmark datasets for
considerations of reproducible research.

## References

  * Chen and Moschitti (2019) Lingzhen Chen and Alessandro Moschitti. 2019.  Transfer learning for sequence labeling using source model and target data.  _ArXiv_ , abs/1902.05309.
  * Chen et al. (2019) Qian Chen, Zhu Zhuo, and Wen Wang. 2019.  Bert for joint intent classification and slot filling.  _arXiv preprint arXiv:1902.10909_.
  * Coucke et al. (2018) A. Coucke, A. Saade, Adrien Ball, Théodore Bluche, A. Caulier, D. Leroy, Clément Doumouro, Thibault Gisselbrecht, F. Caltagirone, Thibaut Lavril, Maël Primet, and J. Dureau. 2018.  Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces.  _ArXiv_ , abs/1805.10190.
  * Devlin et al. (2019) J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL-HLT_.
  * Fei and Liu (2016) Geli Fei and B. Liu. 2016.  Breaking the closed world assumption in text classification.  In _HLT-NAACL_.
  * Goo et al. (2018) Chih-Wen Goo, Guang Gao, Yun-Kai Hsu, Chih-Li Huo, Tsung-Chieh Chen, Keng-Wei Hsu, and Yun-Nung Chen. 2018.  Slot-gated modeling for joint slot filling and intent prediction.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 753–757.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017.  On calibration of modern neural networks.  In _ICML_.
  * Haihong et al. (2019) E Haihong, Peiqing Niu, Zhongfu Chen, and Meina Song. 2019.  A novel bi-directional interrelated model for joint intent detection and slot filling.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5467–5471.
  * He et al. (2020a) Keqing He, Shuyu Lei, Yushu Yang, Huixing Jiang, and Zhongyuan Wang. 2020a.  Syntactic graph convolutional network for spoken language understanding.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 2728–2738, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * He et al. (2020b) Keqing He, Weiran Xu, and Yuanmeng Yan. 2020b.  Multi-level cross-lingual transfer learning with language shared and specific knowledge for spoken language understanding.  _IEEE Access_ , 8:29407–29416.
  * He et al. (2020c) Keqing He, Yuanmeng Yan, Si hong Liu, Z. Liu, and Weiran Xu. 2020c.  Learning label-relational output structure for adaptive sequence labeling.  _2020 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–8.
  * He et al. (2020d) Keqing He, Yuanmeng Yan, and Weiran Xu. 2020d.  Learning to tag OOV tokens by integrating contextual representation and background knowledge.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 619–624, Online. Association for Computational Linguistics.
  * He et al. (2020e) Keqing He, Jinchao Zhang, Yuanmeng Yan, Weiran Xu, Cheng Niu, and Jie Zhou. 2020e.  Contrastive zero-shot learning for cross-domain slot filling with adversarial attack.  In _COLING_.
  * Hemphill et al. (1990) C. T. Hemphill, J. J. Godfrey, and G. Doddington. 1990.  The atis spoken language systems pilot corpus.  In _HLT_.
  * Hendrycks and Gimpel (2017) Dan Hendrycks and Kevin Gimpel. 2017.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  _ArXiv_ , abs/1610.02136.
  * Hu et al. (2019) Ziniu Hu, Ting Chen, Kai-Wei Chang, and Yizhou Sun. 2019.  Few-shot representation learning for out-of-vocabulary words.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4102–4112, Florence, Italy. Association for Computational Linguistics.
  * Huang et al. (2015) Zhiheng Huang, W. Xu, and Kai Yu. 2015.  Bidirectional lstm-crf models for sequence tagging.  _ArXiv_ , abs/1508.01991.
  * Larson et al. (2019) Stefan Larson, Anish Mahendran, Joseph Peper, Christopher Clarke, Andrew Lee, P. Hill, Jonathan K. Kummerfeld, Kevin Leach, M. Laurenzano, L. Tang, and J. Mars. 2019.  An evaluation dataset for intent classification and out-of-scope prediction.  _ArXiv_ , abs/1909.02027.
  * Lee et al. (2018) Kimin Lee, Kibok Lee, H. Lee, and Jinwoo Shin. 2018.  A simple unified framework for detecting out-of-distribution samples and adversarial attacks.  In _NeurIPS_.
  * Liang et al. (2017a) Dongyun Liang, Weiran Xu, and Yinge Zhao. 2017a.  Combining word-level and character-level representations for relation classification of informal text.  In _Proceedings of the 2nd Workshop on Representation Learning for NLP_ , pages 43–47, Vancouver, Canada. Association for Computational Linguistics.
  * Liang et al. (2017b) Shiyu Liang, Yixuan Li, and R. Srikant. 2017b.  Principled detection of out-of-distribution examples in neural networks.  _ArXiv_ , abs/1706.02690.
  * Liang et al. (2018) Shiyu Liang, Yixuan Li, and R. Srikant. 2018.  Enhancing the reliability of out-of-distribution image detection in neural networks.  _arXiv: Learning_.
  * Lin and Xu (2019) Ting-En Lin and H. Xu. 2019.  Deep unknown intent detection with margin loss.  _ArXiv_ , abs/1906.00434.
  * Liu and Lane (2015) Bing Liu and Ian Lane. 2015.  Recurrent neural network structured output prediction for spoken language understanding.  In _Proc. NIPS Workshop on Machine Learning for Spoken Language Understanding and Interactions_.
  * Liu and Lane (2016) Bing Liu and Ian Lane. 2016.  Attention-based recurrent neural network models for joint intent detection and slot filling.  _arXiv preprint arXiv:1609.01454_.
  * Louvan and Magnini (2020) Samuel Louvan and B. Magnini. 2020.  Recent neural methods on slot filling and intent classification for task-oriented dialogue systems: A survey.  In _COLING_.
  * Mesnil et al. (2015) Grégoire Mesnil, Yann Dauphin, Kaisheng Yao, Yoshua Bengio, Li Deng, Dilek Z. Hakkani-Tur, Xiaodong He, Larry Heck, Gokhan Tur, Dong Yu, and Geoffrey Zweig. 2015.  Using recurrent neural networks for slot filling in spoken language understanding.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 23:530–539.
  * Ratinov and Roth (2009) Lev Ratinov and Dan Roth. 2009.  Design challenges and misconceptions in named entity recognition.  In _Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009)_ , pages 147–155.
  * Ren et al. (2019) J. Ren, Peter J. Liu, E. Fertig, Jasper Snoek, Ryan Poplin, Mark A. DePristo, Joshua V. Dillon, and Balaji Lakshminarayanan. 2019.  Likelihood ratios for out-of-distribution detection.  In _NeurIPS_.
  * Shah et al. (2019) Darsh J. Shah, Raghav Gupta, A. Fayazi, and Dilek Z. Hakkani-Tür. 2019.  Robust zero-shot cross-domain slot filling with example values.  _ArXiv_ , abs/1906.06870.
  * Shu et al. (2017) Lei Shu, Hu Xu, and Bing Liu. 2017.  Doc: Deep open classification of text documents.  _ArXiv_ , abs/1709.08716.
  * Xu et al. (2020a) H. Xu, Keqing He, Yuanmeng Yan, Si hong Liu, Z. Liu, and Weiran Xu. 2020a.  A deep generative distance-based classifier for out-of-domain detection with mahalanobis space.  In _COLING_.
  * Xu et al. (2020b) Hong Xu, Keqing He, Yuanmeng Yan, Sihong Liu, Zijun Liu, and Weiran Xu. 2020b.  A deep generative distance-based classifier for out-of-domain detection with mahalanobis space.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 1452–1460, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Yan et al. (2020) Yuanmeng Yan, Keqing He, Hong Xu, Sihong Liu, Fanyu Meng, Min Hu, and Weiran Xu. 2020.  Adversarial semantic decoupling for recognizing open-vocabulary slots.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6070–6075, Online. Association for Computational Linguistics.
  * Yang and Mitchell (2017) B. Yang and Tom Michael Mitchell. 2017.  Leveraging knowledge bases in lstms for improving machine reading.  In _ACL_.
  * Zeng et al. (2021a) Zhiyuan Zeng, Keqing He, Yuanmeng Yan, Hong Xu, and Weiran Xu. 2021a.  Adversarial self-supervised learning for out-of-domain detection.  In _NAACL_.
  * Zeng et al. (2021b) Zhiyuan Zeng, Hong Xu, Keqing He, Yuanmeng Yan, Sihong Liu, Zijun Liu, and Weiran Xu. 2021b.  Adversarial generative distance-based classifier for robust out-of-domain detection.  In _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 7658–7662.
  * Zhao and Feng (2018) Lin Zhao and Zhe Feng. 2018.  Improving slot filling in spoken language understanding with joint pointer and attention.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 426–431, Melbourne, Australia. Association for Computational Linguistics.
  * Zheng et al. (2020) Yinhe Zheng, Guanyi Chen, and Minlie Huang. 2020.  Out-of-domain detection for natural language understanding in dialog systems.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 28:1198–1209.