Further author information: (Send correspondence to Sergio Fabiani)
Sergio Fabiani: E-mail<EMAIL_ADDRESS>Telephone: +39 06 4993 4450

# CUSP: a two cubesats constellation for Space Weather and solar flares X-ray
polarimetry

Sergio Fabiani  INAF-IAPS, via del Fosso del Cavaliere 100, 00133 Rome, Italy
Ilaria Baffo  DEIM, Universitá degli studi della Tuscia, Largo
dell’Universitá, 01100 Viterbo, Italy  Sergio Bonomo  IMT s.r.l., via Carlo
Bartolomeo Piazza 30, 00161 Rome, Italy  Gessica Contini  SCAI Connect s.r.l.,
Via Francesco Gentile 135, 00173 Rome, Italy  Enrico Costa  INAF-IAPS, via del
Fosso del Cavaliere 100, 00133 Rome, Italy  Giovanni Cucinella  IMT s.r.l.,
via Carlo Bartolomeo Piazza 30, 00161 Rome, Italy  Giovanni De Cesare  INAF-
OAS Bologna, via Piero Gobetti 93/3, 40129 Bologna, Italy  Ettore Del Monte
INAF-IAPS, via del Fosso del Cavaliere 100, 00133 Rome, Italy  Andrea Del Re
SCAI Connect s.r.l., Via Francesco Gentile 135, 00173 Rome, Italy  Sergio Di
Cosimo  INAF-IAPS, via del Fosso del Cavaliere 100, 00133 Rome, Italy  Simone
Di Filippo  IMT s.r.l., via Carlo Bartolomeo Piazza 30, 00161 Rome, Italy
Alessandro Di Marco  INAF-IAPS, via del Fosso del Cavaliere 100, 00133 Rome,
Italy  Pierluigi Fanelli  DEIM, Universitá degli studi della Tuscia, Largo
dell’Universitá, 01100 Viterbo, Italy  Fabio La Monaca  INAF-IAPS, via del
Fosso del Cavaliere 100, 00133 Rome, Italy  Alfredo Locarini  Alma Mater
Studiorum Universitá di Bologna - Department of Industrial Engineering and
Interdepartmental Center for Industrial Aerospace Research, Via Fontanelle 40,
47121 Forlí, Italy  Pasqualino Loffredo  INAF-IAPS, via del Fosso del
Cavaliere 100, 00133 Rome, Italy  Giovanni Lombardi  INAF-IAPS, via del Fosso
del Cavaliere 100, 00133 Rome, Italy  Gabriele Minervini  INAF-IAPS, via del
Fosso del Cavaliere 100, 00133 Rome, Italy  Dario Modenini  Alma Mater
Studiorum Universitá di Bologna - Department of Industrial Engineering and
Interdepartmental Center for Industrial Aerospace Research, Via Fontanelle 40,
47121 Forlí, Italy  Fabio Muleri  INAF-IAPS, via del Fosso del Cavaliere 100,
00133 Rome, Italy  Andrea Negri  IMT s.r.l., via Carlo Bartolomeo Piazza 30,
00161 Rome, Italy  Massimo Perelli  IMT s.r.l., via Carlo Bartolomeo Piazza
30, 00161 Rome, Italy  John Rankin  INAF-IAPS, via del Fosso del Cavaliere
100, 00133 Rome, Italy  Alda Rubini  INAF-IAPS, via del Fosso del Cavaliere
100, 00133 Rome, Italy  Paolo Soffitta  INAF-IAPS, via del Fosso del Cavaliere
100, 00133 Rome, Italy  Eliseo Strollo  Alma Mater Studiorum Universitá di
Bologna - Department of Industrial Engineering and Interdepartmental Center
for Industrial Aerospace Research, Via Fontanelle 40, 47121 Forlí, Italy
Paolo Tortora  Alma Mater Studiorum Universitá di Bologna - Department of
Industrial Engineering and Interdepartmental Center for Industrial Aerospace
Research, Via Fontanelle 40, 47121 Forlí, Italy  Alessandro Zambardi  SCAI
Connect s.r.l., Via Francesco Gentile 135, 00173 Rome, Italy

###### Abstract

The CUbesat Solar Polarimeter (CUSP) project aims to develop a constellation
of two CubeSats orbiting the Earth to measure the linear polarisation of solar
flares in the hard X-ray band by means of a Compton scattering polarimeter on
board of each satellite. CUSP will allow to study the magnetic reconnection
and particle acceleration in the flaring magnetic structures. CUSP is a
project approved for a Phase A study by the Italian Space Agency in the
framework of the Alcor program aimed to develop CubeSat technologies and
missions.

###### keywords:

X-ray polarimetry, solar flares, space weather, detectors, solar physics,
CubeSat

##  1 INTRODUCTION

Solar flares (SFs) are violent energetic phenomena taking place on our Sun
that can have a strong impact on the human activities both on ground and in
space. Solar activity, including SFs, can degrade radio communications, can
cause radio blackouts and interference with GPS and satellite communications.
Moreover, high-energy particles (protons and electrons) can release their
energy in the satellite electronics producing malfunctions and also the loss
of the satellite. The occurrence of SFs is also linked to Coronal Mass
Ejection (CME) and Solar Energetic Particle (SEPs) events on the ground [1].
CUSP outcomes are intended to contribute to the present and future networks
for Space Weather, including the future ASI SPace weather InfraStructure
(ASPIS) [2]. Polarimetry gives instantaneous clues about the flaring event. As
soon as photons are detected, they can be downloaded and promptly analysed.
CUSP is currently approved for a phase A study by the Italian Space Agency in
the framework of the Alcor program aimed to develop CubeSat technologies and
missions. CUSP is one of the 20 selected missions among 49 proposals submitted
to a call for CubeSat missions that involved 22 participants from Research
Institutes and Universities and 78 companies, mainly Small and medium-sized
enterprises (SMEs) Over the next years, the Agency plans to deploy them in
orbit. INAF-IAPS is the Prime Contractor and the responsible for the
scientific payload development which front-end and back-end electronics will
be designed and realized by SCAI Connect s.r.l company. The platform,
compatible with the CubeSat 6U standard, will be designed and produced by IMT
s.r.l. The Interdepartmental Center for Aerospace Industrial Research (CIRI-
AERO) of the University of Bologna is responsible for the mission analysis,
while the University of Viterbo “La Tuscia” will take care of the ground
segment with the ground station located in the University Campus.

##  2 CUSP science objective and scientific requirements

In the classical picture of a SF, magnetic reconnection originates the huge
release of energy at the top of a magnetic loop. Particles are accelerated
along the magnetic field lines towards the lower layers of the solar
atmosphere and the interplanetary space. The dominant components of the energy
spectrum are:

  * •

thermal Bremsstrahlung (expected weakly polarised) [3] and emission lines
below 10 keV

  * •

non-thermal Bremsstrahlung from about 10–20 keV, that is expected to be highly
polarised [4]

Theoretical models predict high linear polarisation in the X-rays, depending
on the particle beaming and magnetic field properties [4, 5]. Moreover, the
directivity of accelerated particles can be derived from polarisation
measurements. Until nowadays, few measurements with low significance have been
performed [6, 7, 8, 9, 10, 11]. High significance measurements would allow to
overcome degeneracies in particle beaming models resulting from other
observables like energy spectra [5]. To perform a step forward in the
understanding of the SF physics, CUSP will measure the linear polarisation in
the 20-100 keV energy band with a Minimum Detectable polarisation $<10\%$ at
least for SFs most relevant in terms of Space Weather (X class). Moreover, due
to the fact that SFs are dynamical events (with time scales from minutes to
hours), CUSP will have the capability to study polarisation as a function of
time. A detailed assessment of mission requirements will be carried out during
phase A and B. CUSP science program also inlcude to perform some ancillary
science. CUSP can detect intense X-ray astrophysical sources falling in the
Field of View during the year while observing the Sun (F.O.V. $\pm 21^{\circ}$
at the Sun):

  * •

Gamma Ray Bursts: based on the SWIFT/BAT catalog, CUSP is expected to detect
16 GRBs/yrs with a peak flux over the absorber background

  * •

other sources as for example Crab Nebula (PWN), Sco X-1 (LMXB) and A0535+26
(HMXB)

##  3 The payload: The hard X-ray polarimeter

The payload hosts a dual-phase Compton scattering polarimeter (operating in
the 20-100 keV energy band) that exploits coincidence measurements between
plastic and inorganic scintillator rods. Fig. 1 shows the compact and exploded
view of the polarimeter. The low atomic number of the plastic scintillator
allows to maximise the scattering probability with respect to the heavier
inorganic crystal made of GaGG (Gd3Al2Ga3O12) that maximises the photoelectric
absorption of the scattered photon. Polarimetry is performed by measuring the
azimuthal angular distribution of plastic/GaGG. If radiation is polarised, the
preferential angular direction of polarisation produces a preferential
response of the detector in the azimuthal direction (normal to the incident
beam axis) as described by the Klain-Nishina cross section for Compton
scattering [12]:

$\frac{d\sigma}{d\Omega}=\frac{{r_{0}}^{2}}{2}\frac{{E^{\prime}}^{2}}{{E}^{2}}\Biggr{[}\frac{E}{E^{\prime}}+\frac{E^{\prime}}{E}-2\sin^{2}\theta\cos^{2}\phi\Biggl{]}$
(1)

where

$\frac{E^{\prime}}{E}=\frac{1}{1+\frac{E}{m_{e}c^{2}}(1-\cos\theta)}$ (2)

The energies of the incident and scattered photons are $E$ and $E^{\prime}$,
respectively. The angle $\theta$ is the scattering angle measured from the
incident photon direction and $\phi$ is the azimuthal angle measured from the
plane containing both the incoming direction and the electric vector of the
incident photon. Linearly polarised photons are preferentially scattered
perpendicularly to their polarisation direction. Thus, the response of $\phi$
emission directions for a polarised beam is modulated. The higher the
modulated response, the higher the sensitivity of the polarimeter. Therefore,
the modulation factor $\mu(\theta)$ (fraction of modulated signal
corresponding to 100$\%$ polarised radiation) is:

$\mu(\theta)=\frac{N_{\mathrm{max}}(\theta)-N_{\mathrm{min}}(\theta)}{N_{\mathrm{max}}(\theta)+N_{\mathrm{min}}(\theta)}=\frac{(\frac{d\sigma}{d\Omega})_{\phi=\frac{\pi}{2}}-(\frac{d\sigma}{d\Omega})_{\phi=0}}{(\frac{d\sigma}{d\Omega})_{\phi=\frac{\pi}{2}}+(\frac{d\sigma}{d\Omega})_{\phi=0}}=\frac{\sin^{2}\theta}{\frac{E}{E^{\prime}}+\frac{E^{\prime}}{E}-\sin^{2}\theta}$
(3)

In the limit of coherent scattering $E=E^{\prime}$, the maximum modulation
factor is for $\theta=90^{\circ}$ (orthogonally to the incident photons
direction). By increasing the energy it occurs at narrower scattering angles
(forward folding effect). However, at 100 keV it is still $\theta\simeq
90^{\circ}$ [13].

The payload of CUSP comprises a W collimator for limiting the field of view
about $\pm 21^{\circ}$ around the solar direction, the plastic and the
inorganic scintillators assemblies which are readout by means of 4 Multi-
anodes Photomultiplier Tubes (MAPMTs) for a total of 64 channels and 36
Avalanche Photo-diodes (APDs), respectively. Readout sensors have been
selected to have a high heritage due to the very short implementation time
required for the project. They are the MAPMT R7600 (rugged version to survive
launch vibration) and an SMD version of APD derived from the S8664-55, both by
Hamamatsu. MAPMTs and APDs are readout by MAROC 3A and SKIROC 2A ASICs by
WEEROC. The payload also comprises A/D conversion, Micro HVs and a payload
computer to handle House Keepings.

---
Figure 1: Compact (on the left) and exploded view of the hard X-ray
polarimeter of the CUSP payload.

In Fig. 2 are shown the current best estimates of the modulation factor $\mu$
(response in terms of modulation for 100$\%$ polarised radiation), efficiency
$\epsilon$ (Compton interaction and tagging efficiency) and quality factor
$Q$. Tagging efficiency is the probability to detect an event in the scatterer
after a detection of an event in the absorber [13]. The quality factor is:

$Q=\mu\sqrt{\epsilon}$ (4)

It identifies the energy range in which the polarimeter is effective in
measuring polarisation. This parameter is derived from the Minimum Detectable
Popularization (MDP)[14] by assuming a source dominated observation (if
background is negligible). This is true especially for brighter SFs.

---
Figure 2: Current best estimates of the modulation factor, efficiency (Compton
interaction and tagging efficiency) and quality factor.

Table1 reports the current best estimate of the MDP based on benchmark solar
flares from Saint-Hilaire et al. (2008)[15]. Few minutes of integration time
allow to measure the polarisation of solar flares that is expected to be well
above the MDP (at a level of some tens of per cent).

Table 1: Current best estimate of the MDP based on benchmark solar flares from Saint-Hilaire et al. (2008)[15] Flare Class | Integration Time (s) | MDP ($\%$)
---|---|---
M 5.2 | 284 | 9.2
X 1.2 | 240 | 4.8
X 10 | 351 | 0.9

##  4 The mission concept

We foresee a constellation consisting of 1 orbital plane and 2 CubeSats at
180∘ of phase difference along the orbit (see Fig. 3). This configuration
allows to have always at least 1 satellite of the constellation in daylight
for any $\beta$ angle (orientation of the orbital plane with respect to the
Sun) for observing the Sun. Depending on the specific orbit, there is also a
partial overlap of different duration between the two observation phases of
the two satellites. A preliminary mission analysis identifies as a target a
sun-synchronous (SSO) orbit at an altitude between 500 and 600 km with local
time ascending node LTAN variable between a Mid-Morning scenario (LTAN = 9:30)
and a Noon-Midnight scenario (LTAN = 12:00). This range of orbits offers the
largest number of launch opportunities and foresees different orientations of
the orbital plane with respect to the Sun with different duration of the
eclipse period. Both satellites will be at the same inclination and altitude
to have a common orbital period and node precession. The maximum contemporary
daylight for both satellites is about 16 minutes per orbit for an SSO Mid-
Morning (LTAN = 9:30) at 600 km of altitude.

---
Figure 3: Constellation of two CubeSats at 180∘ phase difference along the
orbit.

The time scale of the project, until launch, and then the operational phase of
the mission, are compatible with the observation of the Sun during the next
cycle maximum. Fig. 4 reports the prediction of the 25th solar cycle activity
by NOAA111https://www.swpc.noaa.gov/products/solar-cycle-progression in terms
of sunspot number and over-imposed the 3 years nominal operative life of the
CUSP mission. The peak of the solar activity and, therefore, the maximum
occurrence of solar flares, is expected between 2024 and 2028.

---
Figure 4: Prediction of the 25th solar cycle activity in terms of sunspot
number in relation with the 3 years nominal operative life of the CUSP
mission.

The CUSP mission foresees two main operative modes. The observation mode
requires that the each satellite rotates around the direction of pointing
towards the Sun, a direction aligned with the axis of symmetry of the
sensitive elements of the polarimeter. The rotation reduces the systematic
effects (spurious modulation) induced by the polarimeter geometry. The
duration of peak intensity in the hard X-rays of a solar flare is of the order
of a few minutes, thus the rotation speed of 1RPM is sufficient for sampling
the modulation curve induced by polarisation on a time scale of 30 seconds
(half of the rotation period). The rotation of the polarimeter is obtained
with the rotation of the entire satellite. The payload foresees a data storage
and pre-processing system on board to archive data before their transmission
to the ground. During the downlink phase, the satellite will change its
attitude, optimizing the transmission link, thus interrupting the observation
phase. The downlink of the data will take place by ensuring the continuity of
observation of the solar phenomena of interest, in the context of the
constellation of two satellites.

##  5 The platform

The platform foreseen for the CUSP project (see Fig. 5) is a standard 6U
CubeSat architecture developed by IMT s.r.l., that has a consolidated
experience in the realization of different CubeSats, also in the institutional
framework. The 6U CubeSat is based on the heritage of the HORTA and EOSS
platforms (6U CubeSat platforms funded by Italian regional POR / FESR 2014-20
projects of Lazio and Puglia regions, respectively). The preliminary
performance of the CUSP platform are reported in Table 2

Table 2: Preliminary performance of the CUSP platform. Peak Power | 30W with deployable panels
---|---
Battery |  $>$75Wh
Pointing accuracy |  $>$ 1∘ (down to 0.1∘ )
Operative frequencies | S / UHF
Downlink throughput | Up to 5 Mbps
Available interfaces | CAN Bus, I2C, UART, RS422
Regulated bus | 3,3V, 5V e 12V
Not regulated bus | 32V (24V-33.6V) or 16V (12V-16.8V)
Available volume for the payload | 2.5U
Nominal life time | 3 years in LEO

The platform hosts an On Board Computer that underwent a qualification for
radiation hardness above 20 krad and implements mitigation strategies for
latch-up errors (SEU / SEL) by architecture, adopting anti-latchup and TMR
(Triple Modular Redundancy) circuit. It can also offer memory storage for some
Payload data (thanks to NAND memories) with the possibility to manage
independently the Payload. During the 1RPM rotation required by the payload,
the satellite will provide the determination of the attitude with an accuracy
better than 0.5∘, needed for processing the scientific data. This will be
achieved thanks to a high-precision gyroscopes and star trackers. During the
non-observational phases of the Payload, the platform returns to the three
axes stabilized mode to perform the downlink of the collected data towards the
ground station. The telemetry and remote controls are realized through UHF
band uplink and downlink communication with omnidirectional turnstile
antennas. UHF antenna deployment will be based on the consolidated design by
IMT s.r.l. which has already implemented several deployment mechanisms, both
of antennas and solar panels. The S-band communication subsystem, in support
of the UHF one, provides a higher link speed to allow scientific data
transfer. The power needed by the on-board equipment is guaranteed by body
mounted and deployable solar panels up to about 30W. The PDU subsystem will
provide for the regulation and distribution of on-board energy.

---
Figure 5: CUSP platform. The top images show the external view. The bottom
image shows the allocation of subsystems including the payload.

Fig. 6 shows the scheme of a CUSP satellite subsystems.

---
Figure 6: Scheme of subsystems of a CUSP satellite.

##  6 The ground station

The Ground Station (see Fig. 7) is located at the “La Tuscia” University of
Viterbo, on the building F of the “Riello” Campus at the following coordinates
(see left panel of Fig. 8):

  * •

Latitude = 42.413∘ N

  * •

Longitude = 12.113∘ E

  * •

Minimum elevation $\epsilon$ min = 10∘

It was built in 2019 as part of the HORTA project (Italian regional funds POR-
FESR 2014-2020 of Lazio region). The Station allows for autonomous satellite
tracking (using TLE satellite data - Two Lines Elements) and satellite
communication. In addition, it can be controlled remotely. Available antennas
and bands are:

  * •

VHF: Uplink and Downlink

  * •

UHF: Uplink and Downlink

  * •

S-band: Downlink

The UHF/VHF bandwidth are 9.6 kbps as default for downlink (available also
1.2/ 2.4 / 4.8 kbps) and 1.2 kbps as default for uplink (available also 2.4 /
4.8/ 9.6 kbps). The S-band bandwidth is up to 1 Mbps for downlink. The
pointing accuracy of the ground station is 0.1° (both azimuth and elevation)
with a minimum tracking speed of 2∘/sec in azimuth, 1.8∘/sec in elevation.
Data received from the ground station is transferred via fibre optics cable to
dedicated workstations in the Mission Control Center that allows to schedule
the passage of the satellite, the TT&C and Payload data transceiving
activities. Moreover, it provides a Network Server service for data delivery
to third parties. From the preliminary Mission Analysis, about 2-4 contacts
per day will be possible during mission operations (see right panel of Fig. 8)

---
Figure 7: The ground station at the “La Tuscia” University of Viterbo.
---
Figure 8: Left panel: location of the ground station on the Earth. Right
panel: preliminary estimation of the number of contacts with the ground
station.

##  7 The planning

The CUSP project is based on subsystems with high TRL. The Platform and the
Ground Station can exploit a large heritage that allow them to guarantee a TRL
7. The Payload is based on elements with a high TRL (MAPMT, APD,
scintillators, ASICs, coincidence technique), but the polarimeter as whole
needs to be implemented. Thus, a TRL 3 is quoted. The Model Philosophy is
based on the production of 1 detector prototype at the end of Phase B. It will
be representative of the detector front-end to enhance detector TRL from 3 to
4. Then 1 payload EQM will be designed during phase B to be produced and
tested during phase C. it will be representative of the payload to enhance
detector TRL from 4 to 7. From the satellite point of view, 2 CubeSats will be
produced:

  * •

1 Proto-flight Model (PFM). To be qualified at proto-qualification level

  * •

1 Flight Model (FM). To be qualified at acceptance level.

The Calibration of the Hard X-ray Polarimeter of each CubeSat will be carried
out at INAF-IAPS calibration facility (already employed for calibrating the
IXPE Detector Units)[16]. A 3 months phase A (starting in September 2022) plus
possibly a 12 months phase B were approved.

##  8 Conclusions

Our Sun is still an astrophysical source with many aspects not well
understood. Solar Flare mechanisms of production and particle acceleration are
still debated. Moreover, solar flares can represent a threat for human
technological activities in space and on ground, because they are usually
correlated to Solar Energetic Particles Events (SEPs) at the Earth and Coronal
Mass Ejections (CMEs). CUSP mission is aimed to measure the linear
polarisation of solar flares in the 20-100 keV energy band to probe of
particle acceleration and magnetic field behaviour during such energetic
events. X-ray polarimetry allows to measure particle directivity and magnetic
filed structure of the flaring loop to assess also magnetic reconnection that
is thought to be at the origin of solar flares. CUSP will contribute to the
understanding of these solar violent phenomena also participating in the
present and future networks for Space Weather, including the ASI SPace weather
InfraStructure (ASPIS).

## References

  * [1] Papaioannou, A., Sandberg, I., Anastasiadis, A., Kouloumvakos, A., Georgoulis, M. K., Tziotziou, K., Tsiropoula, G., Jiggens, P., and Hilgers, A., “Solar flares, coronal mass ejections and solar energetic particle event characteristics,” Journal of Space Weather and Space Climate 6, A42 (Dec. 2016).
  * [2] Plainaki, C., Negri, B., Castronuovo, M., and Antonelli, A., “Towards an Italian Space Weather Infrastructure: the ASPIS project,” in [EGU General Assembly Conference Abstracts ], EGU General Assembly Conference Abstracts, 5239 (Apr. 2018).
  * [3] Emslie, A. G. and Brown, J. C., “The polarization and directivity of solar-flare hard X-ray bremsstrahlung from a thermal source,” ApJ 237, 1015–1023 (May 1980).
  * [4] Zharkova, V. V., Kuznetsov, A. A., and Siversky, T. V., “Diagnostics of energetic electrons with anisotropic distributions in solar flares. I. Hard X-rays bremsstrahlung emission,” A&A 512, A8 (Mar. 2010).
  * [5] Jeffrey, N. L. S., Saint-Hilaire, P., and Kontar, E. P., “Probing solar flare accelerated electron distributions with prospective X-ray polarimetry missions,” A&A 642, A79 (Oct. 2020).
  * [6] Tindo, I. P., Ivanov, V. D., Mandel’Stam, S. L., and Shuryghin, A. I., “On the Polarization of the Emission of X-Ray Solar Flares,” Sol. Phys. 14, 204–207 (Sept. 1970).
  * [7] Tindo, I. P., Ivanov, V. D., Mandel’Stam, S. L., and Shuryghin, A. I., “New Measurements of the Polarization of X-Ray Solar Flares,” Sol. Phys. 24, 429–433 (June 1972).
  * [8] Tindo, I. P., Ivanov, V. D., Valníček, B., and Livshits, M. A., “Preliminary Interpretation of the Polarization Measurements Performed on ’Intercosmos-4’ during Three X-Ray Solar Flares,” Sol. Phys. 27, 426–435 (Dec. 1972).
  * [9] Tramiel, L. J., Novick, R., and Chanan, G. A., “Polarization evidence for the isotropy of electrons responsible for the production of 5-20 keV X-rays in solar flares,” ApJ 280, 440–447 (May 1984).
  * [10] Suarez-Garcia, E., Hajdas, W., Wigger, C., Arzner, K., Güdel, M., Zehnder, A., and Grigis, P., “X-Ray Polarization of Solar Flares Measured with Rhessi,” Sol. Phys. 239, 149–172 (Dec. 2006).
  * [11] Boggs, S. E., Coburn, W., and Kalemci, E., “Gamma-Ray Polarimetry of Two X-Class Solar Flares,” ApJ 638, 1129–1139 (Feb. 2006).
  * [12] Heitler, W., [Quantum theory of radiation ], International Series of Monographs on Physics, Oxford: Clarendon, 1954, 3rd ed. (1954).
  * [13] Fabiani, S., Campana, R., Costa, E., Del Monte, E., Muleri, F., Rubini, A., and Soffitta, P., “Characterization of scatterers for an active focal plane Compton polarimeter,” Astroparticle Physics 44, 91–101 (Apr. 2013).
  * [14] Weisskopf, M., Elsner, R., and O’Dell, S., “On understanding the figures of merit for detection and measurementof x-ray polarization,” Proceedings of the SPIE, 7732, 77320E–77320E–5 (2010).
  * [15] Saint-Hilaire, P., Krucker, S., and Lin, R. P., “A Statistical Survey of Hard X-ray Spectral Characteristics of Solar Flares with Two Footpoints,” Sol. Phys. 250, 53–73 (July 2008).
  * [16] Muleri, L. and et al., “The IXPE Instrument Calibration Equipment,” In Preparation 00, 00–00 (Dec. 2021).


# Don’t Parse, Generate! A Sequence to Sequence
Architecture for Task-Oriented Semantic Parsing

Subendhu Rongali  University of Massachusetts AmherstAmherstMAUSA01002
<EMAIL_ADDRESS>,  Luca Soldaini  Amazon Alexa SearchManhattan
BeachCAUSA<EMAIL_ADDRESS>,  Emilio Monti  Amazon AlexaCambridgeUK
<EMAIL_ADDRESS>and  Wael Hamza  Amazon Alexa AINew YorkNYUSA
<EMAIL_ADDRESS>
(2020)

###### Abstract.

Virtual assistants such as Amazon Alexa, Apple Siri, and Google Assistant
often rely on a semantic parsing component to understand which action(s) to
execute for an utterance spoken by its users.Traditionally, rule-based or
statistical slot-filling systems have been used to parse “simple” queries;
that is, queries that contain a single action and can be decomposed into a set
of non-overlapping entities.More recently, shift-reduce parsers have been
proposed to process more complex utterances.These methods, while powerful,
impose specific limitations on the type of queries that can be parsed; namely,
they require a query to be representable as a parse tree.In this work, we
propose a unified architecture based on Sequence to Sequence models and
Pointer Generator Network to handle both simple and complex queries.Unlike
other works, our approach does not impose any restriction on the semantic
parse schema.Furthermore, experiments show that it achieves state of the art
performance on three publicly available datasets (ATIS, SNIPS, Facebook TOP),
relatively improving between 3.3% and 7.7% in exact match accuracy over
previous systems.Finally, we show the effectiveness of our approach on two
internal datasets.

Natural Language Understanding, Semantic Parsing, Voice Assistants, Sequence
to Sequence models

††copyright: acmcopyright††journalyear: 2020††copyright: iw3c2w3††conference:
Proceedings of The Web Conference 2020; April 20–24, 2020; Taipei,
Taiwan††booktitle: Proceedings of The Web Conference 2020 (WWW ’20), April
20–24, 2020, Taipei, Taiwan††doi: 10.1145/3366423.3380064††isbn:
978-1-4503-7023-3/20/04

##  1\. Introduction

Adoption of intelligent voice assistants such as Amazon Alexa, Apple Siri, and
Google Assistant has increased dramatically among consumers in the past few
years:as of early 2019, it is estimated that 21% of U.S. adults own a smart
speaker, a 78% year-over-year growth (National PublicRadio, 2019).These
systems are built to process user dialog and perform tasks such as media
playback and online shopping.A major part of any voice assistant is a semantic
parsing component designed to understand the action requested by its
users:given the transcription of an utterance, a voice assistant must identify
the action requested by a user (play music, turn on lights, etc.), as well as
parse any entities that further refine the action to perform (which song to
play? which lights to turn on?)Despite huge advances in the field of Natural
Language Processing (NLP), this task still remains challenging due to the
sheer number of possible combinations a user can use to express a command.

Figure 1. Semantic parsing of a “simple” query.Simple queries define single
action (intent) and can be decomposed into a set of non-overlapping entities
(slots).

Traditional approaches for task-oriented semantic dialog parsing frame the
problem as a slot filling task.For example, given the query _Play the song
don’t stop believin by Journey_ , a traditional slot filling system parses it
in two independent steps:(i) It first classifies the _intent_ of the user
utterance as PlaySongIntent, and then (ii) identifies relevant _named
entities_ and tags those slots, such as _don’t stop believin_ as a SongName
and _Journey_ as an ArtistName.Traditional semantic parsing can therefore be
reduced to a text classification and a sequence tagging problem, which is a
standard architecture for many proposed approaches in literature (Liu and
Lane, 2016; Mesnilet al., 2013; Laffertyet al., 2001). This is shown in Figure
1.With increasing expectations of users from virtual assistants, there is a
need for the systems to handle more complex queries – ones that are composed
of multiple intents and nested slots or contain conditional logic.For example,
the query _Are there any movie in the park events nearby?_ involves first
finding the location of _parks_ that are _nearby_ and then finding relevant
_movie_ events in them.This is not straightforward in traditional slot filling
systems. Guptaet al. (2018) and Einolghozati et al. (2019) proposed multiple
approaches for this using a Shift-reduce parser based on Recurrent Neural
Network Grammars (Dyeret al., 2016) that performs the tagging.In this paper,
we propose a unified approach to tackle semantic parsing for natural language
understanding based on Transformer Sequence to Sequence models (Vaswani et
al., 2017) and a Pointer Generator Network (Vinyalset al., 2015; Seeet al.,
2017).Furthermore, we demonstrate how our approach can leverage pre-trained
resources, such as neural language models, to achieve state of the art
performance on several datasets.In particular, we obtain relative improvements
between 3.3% and 7.7% over the best single systems on three public datasets
(SNIPS (Couckeet al., 2018), ATIS (Price, 1990) and TOP (Guptaet al., 2018),
the last consisting of complex queries); on two internal datasets, we show
relative improvements of up to 4.9%.Furthermore, our architecture can be
easily used to parse queries that do not conform to the grammar of either the
slot filling or RNNG systems. Some examples include semantic entities that
correspond to overlapping spans in the query, and entities comprising of non-
consecutive spans. We do not report any results on these kinds of datasets but
we explain how to formulate the problems using our architecture.In summary,
our contributions are as follows.

  * •

We propose a new architecture based on Sequence to Sequence models and a
Pointer Generator Network to solve the task of semantic parsing for
understanding user queries.

  * •

We describe how to formulate different kinds of queries in our architecture.
Our formulation is unified across queries with different kinds of tagging.

  * •

We achieve state-of-the-art results on three public datasets and two internal
datasets.

Figure 2. Semantic parse for a “complex” query in the Facebook TOP dataset.
This complex query is represented as a tree containing two nested intents and
slots.

##  2\. Methodology

Figure 3. Our architecture - Sequence to Sequence model with Pointer Generator
Network (Seq2Seq-Ptr). The model is currently decoding the symbol after
MediaType( by looking at the scores over the tagging vocabulary and the
attentions over the source pointers. It generates $@ptr_{2}$ since it has the
highest overall score.

We propose a unified architecture to solve the task of semantic parsing for
both simple and complex queries. This architecture can also be adapted to
handle queries containing slots with overlapping spans. It consists of a
Sequence to Sequence model and a Pointer Generator Network. We choose a
pretrained BERT (Devlinet al., 2018) model as our encoder. Our decoder is
modeled after the transformer decoder described in Vaswani et al. (2017) and
is augmented with a Pointer Generator Network (Vinyalset al., 2015; Jia and
Liang, 2016) which allows us to learn to generate pointers to the source
sequence in our target sequence.Figure 3 shows this architecture parsing an
example query. We train the model using a cross-entropy loss function with
label smoothing.In this section, we first describe how we formulate queries
and their semantic parses as sequences with pointers for our architecture. We
then describe our encoder and decoder components.

###  2.1. Query Formulation

A Sequence to Sequence architecture is trained on samples with a source
sequence and a target sequence. When some words in the target sequence are
contained in the source sequence, they can be replaced with a separate
_pointer_ token that points to that word in the source to be able to apply the
Pointer Generator Network.Take the example query from Figure 1. In our
architecture, we use the query as our source sequence. The target sequence is
constructed by combining the intent with all the slots, in order, with each
slot also containing its source words. The source and target sequences now
look as follows.

⬇

Target: PlaySongIntent SongName( $@ptr_{3}$ $@ptr_{4}$ $@ptr_{5}$

)SongName ArtistName( $@ptr_{7}$ )ArtistName

Here, each token $@ptr_{i}$ is a pointer to the $i^{th}$ word in the source
sequence. So $@ptr_{3}$, $@ptr_{4}$ and $@ptr_{5}$ point to the song words
_don’t stop believin_ , and $@ptr_{7}$ points to the artist word _journey_.
The slots have open and close tags since they are enclosing a consecutive span
of source tokens. The intent is just represented as a single tag at the
beginning of the target sequence. We can do this for simple queries since they
consist of just one intent. The target vocabulary hence consists of all the
available intents, two times the number of different slots, and the
pointers.Complex queries with multiple intents and nested slots can also be
transformed easily into this formulation. Figure 2 shows an example from the
Facebook TOP dataset along with its parse tree. This query _How far is the
coffee shop_ can be converted into our formulation as follows.

⬇

Target: [IN:GET_DISTANCE $@ptr_{0}$ $@ptr_{1}$ $@ptr_{2}$ [SL:DESTINATION

[IN:GET_RESTAURANT_LOCATION $@ptr_{3}$ [SL:TYPE_FOOD

$@ptr_{4}$ SL:TYPE_FOOD] $@ptr_{5}$ IN:GET_RESTAURANT_LOCATION]

SL:DESTINATION] IN:GET_DISTANCE]

We made a minor modification to the reference parses from the TOP dataset for
our formulation. We replaced the end-brackets with custom end-brackets
corresponding to the intent or slot they close. We found that this formulation
helped our models perform better.Finally, we show how we can express queries
from datasets that don’t conform to either the slot-filling or Shift-reduce
systems. Take the following example from the healthcare domain, where the task
is to extract a patient diagnosis and related information from a clinician’s
notes.

⬇

Annotations: Bleeding_Event (GI bleed),

Anatomical_Site (upper)

A traditional slot filling system wouldn’t know which non consecutive slots to
combine, while a shift-reduce parser cannot split the middle word into a
separate tag. In our architecture, we simply formulate the target sequence as
follows.

⬇

Anatomical_Site( $@ptr_{6}$ )Anatomical_Site

###  2.2. BERT Encoder

Language model pretraining has been shown to improve the downstream
performance on many NLP tasks (Peters et al., 2018; Radford et al., 2019;
Devlinet al., 2018). The idea is to train a language model on a large amount
of text using a next word prediction objective to learn good representations
for each of the words. These representations can then be fine-tuned on a given
NLP task to improve the performance of an existing model. Pretrained models
improve the performance of task models since they already contain a lot of
useful semantic information learned through the pretraining phase. This has
even more significance when the task-specific dataset is fairly small. Some
examples of pretrained models in literature include word embeddings such as
Word2Vec (Mikolov et al., 2013) and Glove (Penningtonet al., 2014), and
contextualized representations such as ELMo (Peters et al., 2018), OpenAI-GPT
(Radford et al., 2019), and BERT (Devlinet al., 2018).We choose BERT to encode
the source sequence in our architecture. BERT (Bidirectional Encoder
Representations from Transformers) is a language representation model
architecture based on Transformers (Vaswani et al., 2017).The original
publicly available model was pretrained on a millions of lines of text from
BooksCorpus and English Wikipedia.Unlike other language models (ELMo, OpenAI-
GPT), which are trained to predict the next token given the previous sequence
of words, BERT uses a composite objective that combines masked word prediction
and next sentence prediction.BERT’s architecture is based on a multi-layer
bidirectional Transformer, originally implemented in Vaswani et al. (2017).
The detailed implementation of this architecture can be found in Devlinet al.
(2018). For our experiments, we use three different variants of BERT.For the
three public datasets, we used the checkpoint released by Devlinet al. (2018).
Experiments on the two internal datasets were carried out using a model we
pretrained over a large sample of queries from the live traffic of Amazon
Alexa.We also experimented with a publicly-available variant of BERT called
RoBERTa (Liu et al., 2019). RoBERTa (A Robustly Optimized BERT Pretraining
Approach) uses the same architecture as BERT but changes the pretraining
process. The next sentence prediction objective is removed and a dynamic
masking scheme is used instead of a static one like the original BERT
implementation. RoBERTa was also trained with longer sequences, higher batch-
sizes, and for a longer time, and was reported to match or exceed the
performance of BERT in several NLP benchmarks. The detailed implementation can
be found in Liu et al. (2019).Finally as an ablation study, we experimented
with an encoder with no pretrained weights.

###  2.3. Decoder with Pointer Generator Network

We use the transformer decoder proposed in Vaswani et al. (2017) in our
architecture. The self-attention mechanism in the decoder learns to attend to
target words before the current step, as well as all the source words in the
encoder.We set up the decoder with different numbers of units, layers, and
attention-heads for different tasks based on the size and complexity of the
queries. These details are provided in the experiments section.In a
traditional Sequence to Sequence model, the target words are generated from
the decoder hidden states through a feed-forward layer that obtains
unnormalized scores over a target vocabulary distribution. In our
architecture, we use a Pointer Generator Network to generate two different
kinds of target words: words from the target vocabulary consisting of parse
symbols (the intent and slot delimiters), and words that are simply pointers
to the source sequence. Our Pointer Generator Network is based on the models
in Vinyalset al. (2015) and Seeet al. (2017), and is closest in implementation
to Jia and Liang (2016).We now describe our decoding process. For each input
source sequence $[x_{1}\dots x_{n}]$, we use the BERT encoder to encode it
into a sequence of encoder hidden states $[e_{1}\dots e_{n}]$. Having
generated the first $t-1$ output tokens, the transformer decoder generates the
token at step $t$ as follows.First, the decoder produces the decoder hidden
state at time $t$, $d_{t}$ by building multi-layer multi-head self-attention
on the encoded output as well as the embeddings of the previously generated
output sequence as described in Vaswani et al. (2017). We feed $d_{t}$ through
a dense layer to produce scores $[s_{1},\dots s_{|V|}]$ for each word in the
vocabulary $V$. $V$ contains all symbols necessary for the parse (intents,
slots) but not regular words appearing on the source side.We also use $d_{t}$
as a query and compute unnormalized attention scores $[a_{1}\dots a_{n}]$ with
the encoded sequence. Concatenating the unnormalized attention scores (size
$n$) and the output of the dense layer (size $|V|$), we obtain an unnormalized
distribution over $|V|+n$ tokens, the first $|V|$ of which are the output
parsing vocabulary and the last $n$ of which are the $@ptr_{i}$ ($0<i<n$)
words pointing to the source tokens. We then feed this through a softmax layer
to obtain the final probability distribution. This probability is used in the
loss function during training and will be used to choose the next token to
generate during inference. Since the transformer decoder uses embeddings of
previously generated tokens, we use a set of special embeddings to represent
$@ptr_{i}$ tokens.In the example in Figure 3, we are trying to predict a
target word after the token ’MediaType(’ at step 5. As shown in the figure, we
compute the scores $[a_{1}\dots a_{4}]$ (blue, left) over each of the source
tokens, and the scores $[s_{1}\dots s_{|V|}]$ (green, right) over the parsing
vocabulary. We expect the model to produce the highest score for $a_{3}$,
which corresponds to $@ptr_{2}$, representing the word _hits_.

##  3\. Datasets

We test our approach on five different datasets (three publicly available, two
interal), which we describe in this section.

###  3.1. Facebook TOP

The Task Oriented Parsing (TOP) (Guptaet al., 2018) dataset contains complex
hierarchical and nested queries that make the task of semantic parsing more
challenging. It contains around 45k annotations with 25 intents and 36 slots,
randomly split into 31k training, 5k validation and 9k test utterances. The
dataset mainly consists of user queries about navigation and various public
events. An example from this dataset can be seen in Figure 2.The IN: prefix
stands for intent while SL: is for slot. We can see how there are multiple
intents and nested slots in the semantic interpretation. This makes the query
much harder to interpret and parse using a simple slot tagging model that tags
each word with a single slot.

###  3.2. SNIPS

The SNIPS dataset (Couckeet al., 2018) is a public dataset that is used for
training and testing semantic parsing models for voice assistants. It consists
of utterances that belong to seven different intents: SearchCreativeWork,
GetWeather, BookRestaurant, PlayMusic, AddToPlaylist, RateBook, and
SearchScreeningEvent. Each intent contains around 2000 examples to train and
100 to test.This dataset contains only simple queries with single intents and
flat slots. An example is _Will there be fog in Tahquamenon Falls State Park_
, where the intent is GetWeather and the slots are condition_description for
_fog_ and geographic_poi for _Tahquamenon Falls State Park_.The dataset was
originally used to evaluate models in the Snips Voice Platform. It has since
been a widely used dataset to benchmark the performance of various task-
oriented parsing models.

###  3.3. ATIS

The Airline Travel Information System (ATIS) (Price, 1990) corpus is a widely
used dataset in spoken language understanding. It was built by collecting and
transcribing audio recordings of people making flight reservations in the
early 90s. It consists of simple queries.There are seventeen different goals
or intents such as Flight or Aircraft capacity. This distribution is however
skewed, with the Flight intent covering about 70% of the total queries. An
example from this dataset consists of the query _How much is the cheapest
flight from Boston to New York tomorrow morning?_ The intent is Airfare, while
the slots tag important information like the departure and arrival cities, and
the departure times.The ATIS corpus has supported research in the field of
spoken language understanding for more than twenty years. Some researchers
have performed extensive error analysis on the state of the art discriminative
models for this dataset and reported that despite really low error rates,
there exist many unseen categories and sequences in the dataset that can
benefit from incorporating linguistically motivated features (Turet al.,
2010). This supports the continued utility of ATIS as a research corpus.

###  3.4. Internal Datasets

Our internal datasets consist of millions of user utterances that are used to
train and test Amazon Alexa.For our experiments, we sampled two datasets of
utterances, one from the music domain and the other from video
domain.Utterances in these domains naturally included a large amount of
entities (e.g. artists and albums names, movie and video titles), and thus
represent a good benchmark for the ability of any neural model to generalize
over a diverse set of queries. The example in Figure 3 is from the music
domain.The sampled music domain dataset contains 6.2M training and 200k test
utterances, with 23 intents and 100 slots.The video domain dataset contains 1M
training and just 5k test utterances;parses in this dataset are comprised of
24 distinct intents and 59 slots.

##  4\. Baseline Models

We benchmark our performance on the internal datasets by comparing it to a
well tuned RNN based model.The model learns to perform joint intent and slot
tagging using a bidirectional LSTM and a Conditional Random Field (CRF)
(Huanget al., 2015).We further enhanced this baseline by replacing its
embedding and encoder layers with a language model pretrained on a subset of
Alexa’s live traffic.These components were fine-tuned on the two datasets
described in Section 3.4.For the ATIS and SNIPS datasets, we use the top four
performing methods reported by Zhanget al. (2019) as baselines. All these
models perform joint intent and slot tagging. There are two variants that use
RNNs: a simple RNN based model, and an RNN model augmented with attention.
There is also a model that works completely with just attention, the slot
gated full attention model. The final baseline, CapsuleNLU, uses Capsule
Networks (Sabouret al., 2017).For the TOP dataset, we pick a model based on
Recurrent Neural Network Grammars (RNNG) (Dyeret al., 2016), the Shift Reduce
Parser. We provide a brief overview of this model as described in Guptaet al.
(2018) \- the parse tree is constructed using a sequence of transitions, or
_actions_. The transitions are defined as a set of SHIFT, REDUCE, and the
generation of intent and slot labels. SHIFT action consumes an input token
(that is, adds the token as a child of the right most _open_ sub-tree node)
and REDUCE closes a sub-tree. The third set of actions is generating non-
terminals: the slot and intent labels. The model learns to perform one of
these actions at each step in time.We report scores of three experimental
setups with the shift reduce parser from Einolghozati et al. (2019): a simple
shift reduce parser, a shift reduce parser augmented with ELMo embeddings, and
an ensemble of these models augmented with ELMo and an SVM language model
reranker.

Method | Accuracy
---|---
exact match | intent
Shift Reduce (SR) Parser (Einolghozati et al., 2019) | 80.86 | –
SR with ELMo embeddings (Einolghozati et al., 2019) | 83.93 | –
SR ensemble + ELMo + SVMRank (Einolghozati et al., 2019) | 87.25 | –
Seq2Seq-Ptr (no pretraining) | 79.25 | 97.43
Seq2Seq-Ptr (BERT encoder) | 83.13 | 97.91
Seq2Seq-Ptr (RoBERTa encoder) | 86.67 | 98.13
Table 1. Results on TOP (Guptaet al., 2018).Our system (Seq2Seq-Ptr) outperforms the best single method (SR + ELMo) by 3.3%. It is close to the best ensemble approach (SR + ELMo + SVRank). Method | Accuracy
---|---
exact match | intent
Joint BiRNN (Hakkani-Tür et al., 2016) | 73.20 | 96.90
Attention BiRNN (Liu and Lane, 2016) | 74.10 | 96.70
Slot Gated Full Attention (Gooet al., 2018) | 75.50 | 97.00
CapsuleNLU (Zhanget al., 2019) | 80.90 | 97.30
Seq2Seq-Ptr (no pretraining) | 85.43 | 97.00
Seq2Seq-Ptr (BERT encoder) | 86.29 | 98.29
Seq2Seq-Ptr (RoBERTa encoder) | 87.14 | 98.00
Table 2. Results on SNIPS (Couckeet al., 2018).Our system (Seq2Seq-Ptr) outperforms the previous state of the art by 7.7%. Method | Accuracy
---|---
exact match | intent
Joint BiRNN (Hakkani-Tür et al., 2016) | 80.70 | 92.60
Attention BiRNN (Liu and Lane, 2016) | 78.90 | 91.10
Slot Gated Full Attention (Gooet al., 2018) | 82.20 | 93.60
CapsuleNLU (Zhanget al., 2019) | 83.40 | 95.00
Seq2Seq-Ptr (no pretraining) | 81.08 | 95.18
Seq2Seq-Ptr (BERT encoder) | 86.37 | 97.42
Seq2Seq-Ptr (RoBERTa encoder) | 87.12 | 97.42
Table 3. Results on ATIS (Price, 1990).Our system (Seq2Seq-Ptr) outperforms
the previous best method by 4.5%.

##  5\. Experimental Setup

All our models were trained on a machine with 8 NVIDIA Tesla V100 GPUs, each
with 16GB of memory.When using pretrained encoders, we leveraged gradual
unfreezing to effectively tune the language model layers on our datasets.We
used the ”Base” variant of BERT and RoBERTa encoders, which uses
768-dimensional embeddings, 12 layers, 12 heads, and 3072 hidden units.When
training from scratch, we used a smaller encoder consisting of 512-dimensional
embeddings, 6 layers, 8 heads, and 1024 hidden units.Depending on the dataset,
we used either a 128 units, 4 layers, 3 heads, and 512 hidden units decoder
(Facebook TOP, ATIS, SNIPS) or a larger 512 units, 6 layers, 8 heads, and 1024
hidden units decoder (internal Music and Video datasets).We used bi-linear
product attention to score the source words in the Pointer Network.While
training, the cross entropy loss function was modified with label smoothing
with $\epsilon=0.1$. We used the Adam (Kingma and Ba, 2014) optimizer with
noam learning rate schedule (Vaswani et al., 2017), each adjusted differently
for different datasets.At inference time, we used beam search decoding with a
beam size of 4.

##  6\. Results and Discussion

We use exact match (EM) accuracy as the main metric to measure the performance
of our models across all datasets.Under this metric, the entire semantic parse
for a query has to match the reference parse to be counted as correct.Because
EM is generally more challenging than slot-level precision and recall or
semantic error rate (Thomson et al., 2012), it is better suited to compare
high performing systems like the ones studied in this work.For completeness,
we also report the intent classification accuracy for our models.The results
from our experiments are documented in Tables 1-5. Our models match or beat
the baselines across all datasets on both exact match and intent
classification accuracies.We see significant improvements on both simple and
complex datasets.

###  6.1. Complex Queries

We achieve an improvement of 2.7 (+3.3%) EM accuracy points on the TOP dataset
over the state-of-the-art single model on this dataset (Table 1).Our Seq2Seq-
Ptr model with RoBERTa encoder is only surpassed by the ensemble model
reported in Einolghozati et al. (2019) (+0.6% EM accuracy points.)In addition,
we find that even without specifying any hard requirements for the grammar of
the parse trees in the complex queries, 98% of the generated parses are well
formatted. For the simple query datasets, it was greater than 99% but
difference is expected since the grammar is easier to learn there.During error
analysis, we found an interesting example in the TOP dataset where we believe
our model generates a valid, more meaningful parse than the reference
annotation. For the query _What time do I need to leave to get to Helen by
8pm_ , our model parses _Helen_ as [SL:DESTINATION [IN:GET_ LOCATION_HOME
[SL:CONTACT Helen ] ] ], while it is annotated as [SL:DESTINATION Helen ].Our
parse resolves the query as finding the estimated departure time to get to a
location that is the home location of a contact named Helen, while the
reference annotation suggests that the correct interpretation is tofind the
estimated departure time to get to a destination named _Helen_.We believe our
parse is more likely to be correct given that Helen is most likely the name of
a person.

###  6.2. Simple Queries

We report results of our sequence to sequence model (Seq2Seq-Ptr) on four
datasets (SNIPS, ATIS, internal music, internal video) that contain simple
queries in Tables 2, 3, 4, and 5.On the SNIPS and ATIS datasets, we note that
the best version of our method (Seq2Seq-Ptr with RoBERTa encoder) achieves a
significant improvement in EM accuracy over existing baselines (+7.7% and
+4.5% respectively.)Using a BERT encoder causes a slight decrease in
performance, but still achieves a meaningful improvement over the previous
state of the art (Zhanget al., 2019);this is consistent with what has been
observed on other NLP tasks (Liu et al., 2019).If no pretraining is used,
performance is further reduced but it is notable that this variant still beats
all the baselines on the SNIPS dataset.For our internal Alexa datasets, we
note that the proposed Seq2Seq-Ptr method obtains comparable results to a
BiLSTM-CRF tagger on the music domain, and slightly better EM accuracy (+1.9%)
on the video domain.We believe our model wasn’t able to outperform the
baseline on the music domain because the entities in this domain are very
diverse, especially song or album names. Sequence tagging methods therefore
benefit from having to solve a simpler task of having to tag each word in the
sequence, as opposed to our unconstrained model.We would however like to note
that our from-scratch variants beat the from-scratch baselines on both
domains.Also curiously, the performance of Seq2Seq-Ptr with a BERT encoder
fell behind that of a sequence to sequence model trained from scratch.Since
the scratch model uses a smaller transformer encoder (6 layers with 8 heads
per layer instead of 12/12), we believe it was able to converge more
effectively than the BERT encoder.

Method | Accuracy
---|---
exact match | intent
BiLSTM-CRF (no pretraining) | baseline
BiLSTM-CRF (pretrained LM) | +3.0% | +0.1%
Seq2Seq-Ptr (no pretraining) | -0.3% | -0.7%
Seq2Seq-Ptr (BERT encoder) | -2.2% | -0.8%
Seq2Seq-Ptr (RoBERTa encoder) | -3.5% | -0.7%
Table 4. Results on an internal dataset (music). The best configuration of our method (Seq2Seq-Ptr) is comparable to a BiLSTM-CRF pretrained on a large conversational dataset. Method | Accuracy
---|---
exact match | intent
BiLSTM-CRF (no pretraining) | baseline
BiLSTM-CRF (pretrained LM) | +3.0% | +0.1%
Seq2Seq-Ptr (no pretraining) | +2.9% | -0.1%
Seq2Seq-Ptr (BERT encoder) | +0.1% | -0.2%
Seq2Seq-Ptr (RoBERTa encoder) | +4.9% | -0.2%
Table 5. Results on an internal dataset (video). The best configuration of our
method (Seq2Seq-Ptr) outperforms a pretrained BiLSTM-CRF network by 1.9%.

##  7\. Related Work

The task of semantic parsing for intent and slot detection is well established
in literature. Traditionally, this was done with slot filling systems that
classify the query and then label each word in the query. There were a few
approaches that followed this system, using Recurrent Neural Networks (Liu and
Lane, 2016; Mesnilet al., 2013). Researchers have also experimented with
Convolutional Neural Networks and showed good results (Kim, 2014) and more
recently, Capsule Networks (Sabouret al., 2017; Zhanget al., 2019).Prior to
the advent of deep learning models, the task of sequence labeling was tackled
with the use of Conditional Random Fields (CRF) (Laffertyet al., 2001; Jiaoet
al., 2006; Penget al., 2004). CRFs learn pairwise potentials on labeling
subsequent words which allow models to find more probable label sequences for
a given query.Most of this work is valid for semantic parsing for simple
queries which boils down to a sequence labeling task. To handle more complex
cases with hierarchical slots such as the example in Figure 2, researchers
have experimented with Sequence to Sequence models and models based on
Recurrent Neural Network Grammars (RNNG) (Dyeret al., 2016). RNNGs were shown
to perform better on complex queries than RNN or Transformer-based Sequence to
Sequence models (Guptaet al., 2018). Researchers have also explored models
involving logical forms and discourse for language representation (Liang,
2016; Zettlemoyer andCollins, 2012; vanNoord et al., 2018).The Pointer
Generator Network in our architecture was introduced in Vinyalset al. (2015).
It was used in NLP applications where some words from the source sequence
reappeared in the target sequence such as text summarization and style
transfer (Seeet al., 2017; Pauluset al., 2017; Prabhumoye et al., 2018). They
were also used to copy out of vocabulary words from the source to target in
machine translation (Kleinet al., 2017). Our implementation of the Pointer
Network is closest to the architecture in Jia and Liang (2016). By using
pointers to represent the source tokens and imposing no particular logical
form over our target sequence, we can handle any kind of queries for parsing.
This makes our architecture as expressive as logical forms, while also being
able to learn as easily as simple slot tagging systems.

##  8\. Conclusion

We propose a unified architecture for the task of semantic parsing for
different kinds of queries. We show that our architecture matches or
outperforms existing approaches across multiple datasets : internal music and
video datasets, SNIPS, ATIS, and Facebook TOP. We significantly outperform the
current state of the art models on the public datasets TOP (3.3%), SNIPS
(7.7%), and ATIS (4.5%).We describe how to apply this architecture to both
simple queries and complex queries with hierarchical and nested slots. We also
describe how to formulate any set of queries with non-conforming grammars to
work with our architecture, making this model applicable to many different
types of semantic parsing. We leave the non-conforming grammar task to future
work.

## References

  * Couckeet al. (2018) Alice Coucke, Alaa Saade,Adrien Ball, Théodore Bluche,Alexandre Caulier, David Leroy,Clément Doumouro, ThibaultGisselbrecht, Francesco Caltagirone,Thibaut Lavril, et al.2018\.  Snips voice platform: an embedded spoken languageunderstanding system for private-by-design voice interfaces.  _arXiv preprint arXiv:1805.10190_(2018).
  * Devlinet al. (2018) Jacob Devlin, Ming-WeiChang, Kenton Lee, and KristinaToutanova. 2018.  Bert: Pre-training of deep bidirectionaltransformers for language understanding.  _arXiv preprint arXiv:1810.04805_(2018).
  * Dyeret al. (2016) Chris Dyer, AdhigunaKuncoro, Miguel Ballesteros, and Noah ASmith. 2016.  Recurrent neural network grammars.  _arXiv preprint arXiv:1602.07776_(2016).
  * Einolghozati et al. (2019) Arash Einolghozati,Panupong Pasupat, Sonal Gupta,Rushin Shah, Mrinal Mohit,Mike Lewis, and Luke Zettlemoyer.2019.  Improving Semantic Parsing for Task OrientedDialog.  _arXiv preprint arXiv:1902.06000_(2019).
  * Gooet al. (2018) Chih-Wen Goo, Guang Gao,Yun-Kai Hsu, Chih-Li Huo,Tsung-Chieh Chen, Keng-Wei Hsu, andYun-Nung Chen. 2018.  Slot-Gated Modeling for Joint Slot Filling andIntent Prediction. In _Proceedings of the 2018Conference of the North American Chapter of the Association forComputational Linguistics: Human Language Technologies, Volume 2 (ShortPapers)_. Association for Computational Linguistics,New Orleans, Louisiana, 753–757.  https://doi.org/10.18653/v1/N18-2118
  * Guptaet al. (2018) Sonal Gupta, Rushin Shah,Mrinal Mohit, Anuj Kumar, andMike Lewis. 2018.  Semantic Parsing for Task Oriented Dialog usingHierarchical Representations. In _Proceedings ofthe 2018 Conference on Empirical Methods in Natural Language Processing_.2787–2792.
  * Hakkani-Tür et al. (2016) Dilek Hakkani-Tür,Asli Celikyilmaz, Yun-Nung Chen,Jianfeng Gao, Li Deng, andYe-Yi Wang. 2016.  Multi-domain joint semantic frame parsing usingbi-directional rnn-lstm.. In _Interspeech_.
  * Huanget al. (2015) Zhiheng Huang, Wei Xu,and Kai Yu. 2015.  Bidirectional LSTM-CRF models for sequencetagging.  _arXiv preprint arXiv:1508.01991_(2015).
  * Jia and Liang (2016) Robin Jia and PercyLiang. 2016.  Data recombination for neural semantic parsing.  _arXiv preprint arXiv:1606.03622_(2016).
  * Jiaoet al. (2006) Feng Jiao, Shaojun Wang,Chi-Hoon Lee, Russell Greiner, andDale Schuurmans. 2006.  Semi-supervised conditional random fields forimproved sequence segmentation and labeling. In _Proceedings of the 21st International Conference onComputational Linguistics and the 44th annual meeting of the Association forComputational Linguistics_. Association for Computational Linguistics,209–216.
  * Kim (2014) Yoon Kim. 2014.  Convolutional neural networks for sentenceclassification.  _arXiv preprint arXiv:1408.5882_(2014).
  * Kingma and Ba (2014) Diederik P Kingma andJimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_(2014).
  * Kleinet al. (2017) Guillaume Klein, YoonKim, Yuntian Deng, Jean Senellart, andAlexander M Rush. 2017.  Opennmt: Open-source toolkit for neural machinetranslation.  _arXiv preprint arXiv:1701.02810_(2017).
  * Laffertyet al. (2001) John Lafferty, AndrewMcCallum, and Fernando CN Pereira.2001.  Conditional random fields: Probabilistic models forsegmenting and labeling sequence data.  (2001).
  * Liang (2016) Percy Liang.2016.  Learning executable semantic parsers for naturallanguage understanding.  _arXiv preprint arXiv:1603.06677_(2016).
  * Liu and Lane (2016) Bing Liu and Ian Lane.2016.  Attention-based recurrent neural network models forjoint intent detection and slot filling.  _arXiv preprint arXiv:1609.01454_(2016).
  * Liu et al. (2019) Yinhan Liu, Myle Ott,Naman Goyal, Jingfei Du,Mandar Joshi, Danqi Chen,Omer Levy, Mike Lewis,Luke Zettlemoyer, and VeselinStoyanov. 2019.  Roberta: A robustly optimized BERT pretrainingapproach.  _arXiv preprint arXiv:1907.11692_(2019).
  * Mesnilet al. (2013) Grégoire Mesnil,Xiaodong He, Li Deng, andYoshua Bengio. 2013.  Investigation of recurrent-neural-networkarchitectures and learning methods for spoken language understanding.. In _Interspeech_. 3771–3775.
  * Mikolov et al. (2013) Tomas Mikolov, IlyaSutskever, Kai Chen, Greg S Corrado,and Jeff Dean. 2013.  Distributed representations of words and phrasesand their compositionality. In _Advances in neuralinformation processing systems_. 3111–3119.
  * National PublicRadio (2019) National Public Radio.2019.  Smart Speakers See 78% Increase YOY.  https://www.npr.org/about-npr/682946406/npr-report-smart-speakers-see-78-increase-yoy
  * Pauluset al. (2017) Romain Paulus, CaimingXiong, and Richard Socher.2017.  A deep reinforced model for abstractivesummarization.  _arXiv preprint arXiv:1705.04304_(2017).
  * Penget al. (2004) Fuchun Peng, FangfangFeng, and Andrew McCallum.2004.  Chinese segmentation and new word detection usingconditional random fields. In _Proceedings of the20th international conference on Computational Linguistics_. Association forComputational Linguistics, 562.
  * Penningtonet al. (2014) Jeffrey Pennington,Richard Socher, and ChristopherManning. 2014.  Glove: Global vectors for word representation. In _Proceedings of the 2014 conference on empiricalmethods in natural language processing (EMNLP)_.1532–1543.
  * Peters et al. (2018) Matthew E Peters, MarkNeumann, Mohit Iyyer, Matt Gardner,Christopher Clark, Kenton Lee, andLuke Zettlemoyer. 2018.  Deep contextualized word representations.  _arXiv preprint arXiv:1802.05365_(2018).
  * Prabhumoye et al. (2018) Shrimai Prabhumoye, YuliaTsvetkov, Ruslan Salakhutdinov, andAlan W Black. 2018.  Style transfer through back-translation.  _arXiv preprint arXiv:1804.09000_(2018).
  * Price (1990) Patti J Price.1990.  Evaluation of spoken language systems: The ATISdomain. In _Speech and Natural Language:Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27,1990_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu,Rewon Child, David Luan,Dario Amodei, and Ilya Sutskever.2019.  Language models are unsupervised multitasklearners.  _OpenAI Blog_ 1,8 (2019).
  * Sabouret al. (2017) Sara Sabour, NicholasFrosst, and Geoffrey E Hinton.2017.  Dynamic routing between capsules. In _Advances in neural information processingsystems_. 3856–3866.
  * Seeet al. (2017) Abigail See, Peter J Liu,and Christopher D Manning.2017.  Get to the point: Summarization withpointer-generator networks.  _arXiv preprint arXiv:1704.04368_(2017).
  * Thomson et al. (2012) B. Thomson, M. Gasic,M. Henderson, P. Tsiakoulis, andS. Young. 2012.  N-best error simulation for training spokendialogue systems. In _2012 IEEE Spoken LanguageTechnology Workshop (SLT)_. 37–42.  https://doi.org/10.1109/SLT.2012.6424194
  * Turet al. (2010) Gokhan Tur, DilekHakkani-Tür, and Larry Heck.2010.  What is left to be understood in ATIS?. In _2010 IEEE Spoken Language Technology Workshop_.IEEE, 19–24.
  * vanNoord et al. (2018) Rik van Noord, LashaAbzianidze, Antonio Toral, and JohanBos. 2018.  Exploring neural methods for parsing discourserepresentation structures.  _Transactions of the Association forComputational Linguistics_ 6 (2018),619–633.
  * Vaswani et al. (2017) Ashish Vaswani, NoamShazeer, Niki Parmar, Jakob Uszkoreit,Llion Jones, Aidan N Gomez,Łukasz Kaiser, and IlliaPolosukhin. 2017.  Attention is all you need. In _Advances in neural information processingsystems_. 5998–6008.
  * Vinyalset al. (2015) Oriol Vinyals, MeireFortunato, and Navdeep Jaitly.2015.  Pointer networks. In _Advances in Neural Information ProcessingSystems_. 2692–2700.
  * Zettlemoyer andCollins (2012) Luke S Zettlemoyer andMichael Collins. 2012.  Learning to map sentences to logical form:Structured classification with probabilistic categorial grammars.  _arXiv preprint arXiv:1207.1420_(2012).
  * Zhanget al. (2019) Chenwei Zhang, YaliangLi, Nan Du, Wei Fan, andPhilip Yu. 2019.  Joint Slot Filling and Intent Detection via CapsuleNeural Networks. In _Proceedings of the 57th AnnualMeeting of the Association for Computational Linguistics_.Association for Computational Linguistics,Florence, Italy, 5259–5267.  https://doi.org/10.18653/v1/P19-1519