# Magnetoresistive Sensor Detectivity: A Comparative Analysis

J. E. Davies<EMAIL_ADDRESS>J. D. Watts  J. Novotny  D. Huang  P. G. Eames
NVE Corporation, Eden Prairie, MN 55344, USA

###### Abstract

We report on the noise performance characteristics of magnetic sensors using
both magnetic tunnel junction (MTJ) and giant magnetoresistance (GMR)
elements. Each sensor studied has a notably different noise and detectivity.
Of the sensors we measured, those based on GMR multilayers have the lowest
noise and detectivity. However, the GMR sensor also has a significantly
smaller linear range. To make a direct comparison between sensors we scale the
linear operating ranges of each sensor to be the same. This is the
phenomenological equivalent of modifying the flux concentration. Upon scaling
the low frequency detectivity of the TMR sensors becomes essentially equal to
that of the GMR sensor. Using the scaling approach we are able to place the
detectivity in the context of other key parameters, namely size and power
consumption. Lastly, we use this technique to examine the upper limit for
magnetoresistive sensor performance based on a notional MTJ sensor using
present record setting TMR values.

††preprint: AIP/123-QED

Magnetoresistive (MR) technologies have been the fundamental building block
for spintronic devices over the last three decades.Baibich  _et al._ (1988);
Binasch  _et al._ (1989); Dieny  _et al._ (1991); Parkin  _et al._ (1999) This
is due to their ability to serve as highly sensitive transducers that readily
respond to magnetic fields and spin currents.Slonczewski (2002); Ralph and
Stiles (2008); Sankey  _et al._ (2008); Wang, Anderson, and Daughton (1997);
Ikeda  _et al._ (2008); Wang  _et al._ (2009); Wolf  _et al._ (2001) Their
small size and high sensitivity are utilized as sensor components across
several industries including the biomedical, navigation and industrial
automation markets.Graham, Ferreira, and Freitas (2004); Caruso (1997); Schewe
and Schelter (1997); Fujiwara  _et al._ (2018) They are also at the core of
the magnetic data storage industry, being utilized as read heads in hard disk
drives and the storage medium for magnetic random access memory (MRAM)
bits.McFadyen, Fullerton, and Carey (2006); Parkin  _et al._ (1999) From the
sensor perspective, the MR sensor’s CMOS compatible fabrication process,
robust performance, low power operation and low cost are clear advantages over
other technologies. These advantages poise MR sensors to play a key role in
technologies benefiting from advanced, compact, and cost effective sensing,
such as the internet of things (IoT), smart grids and electric cars. Tahoori
_et al._ (2018); Liu  _et al._ (2019); Ouyang  _et al._ (2019)

With many types of MR sensors available it is often difficult to determine the
best sensor for a particular application. Typically, application engineers
have used maximization of the % MR and the minimization of noise as key
metrics.Nowak  _et al._ (1998); Ikeda  _et al._ (2008); Stutzke  _et al._
(2005) These metrics are straightforward for the researcher to focus on with
much work having gone into the development of MR sensor noise
phenomenology.Nowak  _et al._ (1998); Ingvarsson  _et al._ (2000); Klaassen,
Xinzhi Xing, and van Peppen (2004); Zheng  _et al._ (2019) Experimental noise
characterization has shown that there can be substantial variation in the
noise characteristics among sensors.Stutzke  _et al._ (2005); Egelhoff  _et
al._ (2009); Deak, Zhou, and Shen (2017) It is important to realize that such
comparisons between specific sensors can be misleading in that they do not
place the comparison in the context of other important specifications such as
sensor size, hysteresis, power consumption, linear range and cost. Performing
this more rigorous contextual comparison is non-trivial, but more telling.

This work places magnetoresistive sensor noise characterization in a more
parameterized context. We start with a noise and performance study of three
sensor variations. Each sample has a different field response and noise
spectrum. Normalizing the noise data to a key parameter, namely the linear
range, we obtain a more contextual picture of each sensor’s performance and
demonstrate that such normalization enables a more fundamental comparison
between different sensors. Finally, we look ahead to what is likely the
ultimate limit of MR sensor detectivity.

The noise measurement setup is shown in Fig.1. The sensors, configured as
Wheatstone bridges, are soldered to circuit boards and placed in a triple
layered $\mu$-metal container for shielding from environmental magnetic
fields. The sensors were all battery powered to 3 V and, in the case of the
unipolar sensor, a 0.5 mT field was applied along its sense direction to put
the sensor in the middle of its operate range. The outputs of the bridges,
$+V_{out}$ and $-V_{out}$ were input to a two-channel preamplifier. The
differential output was then captured by a spectrum analyzer.

Figure 1:  Illustration of the noise measurement setup. The sensor resides in
a mu-metal can and is battery powered to 3 V. Helmholtz coils are available
for field biasing of the sensor. Sensor outputs are fed to a low-noise
preamplifier. The differential signal is then captured on a spectrum Analyzer.

We limit this study to three often utilized sensor compositions (layer
thicknesses are in nm):

  1. 1.

Bipolar MTJ sensor with superparamagnetic freelayer (SPMTJ) with layer
composition: Ta(5)/Ru(5)/IrMn(11)/CoFe(2)/Ru(0.9)/CoFeB(2.5)/
MgO(2)/CoFeB(1.2)/Ta(2)/Ru(10)

  2. 2.

Bipolar MTJ sensor with full film free layer (FFMTJ) with layer composition:
Ta(5)/Ru(5)/IrMn(11)/CoFe(2)/Ru(0.9)/CoFeB(2.5)/
MgO(2)/CoFeB(2)/NiFe(6)/Ta(2)/Ru(10)

  3. 3.

Omnipolar GMR multilayer sensorZou  _et al._ (2001) (GMRMLs) with layers grown
on a Ta(5) seed and comprised of four CoFe(1)/NiFeCo(2)/CoFe(1) ferromagnetic
trilayers separated by CuAgAu(1.5) GMR spacers and capped with Ta(20)

All films were grown by magnetron sputtering onto Si wafers coated with 200 nm
vapor deposited SiNx. The SPMTJ and FFMTJ films were annealed ex-situ in a 400
mT field at 350C in a forming gas environment to both crystalize the MgO layer
and set the pinning layer magnetization. No annealing was performed on the
GMRML films.

The sensor elements were fabricated using standard photolithography processes.
SPMTJ and FFMTJ resistor elements are series connected arrays of forty 5
$\mu$m diameter MTJs. Individual resistors were then rotated and co-packaged
to form a push-pull sensor configuration.

The GMRML sensor was monlithically fabricated with four serpentine-style
resistors. Thick NiFe layers were then used in dual purpose to shield two
resistors and provide flux concentration for the active resistors.

Table 1: Nominal sensor parameters Sensor | MR | $R_{0}$ | $B_{sat}$ |  $Sens_{max}$1111 mV/V/mT = 0.1 mV/V/Oe | Area222Area is circuit board area occupied by the sensor. SPMTJ and FFMTJ sensors are in TDFN-6 packages, GMRML is in SOIC-8.
---|---|---|---|---|---
Type | (%) | $(k\Omega)$ | (mT) | (mV/V/mT) | ($mm^{2}$)
SPMTJ | 60 | 50 | 5 | 25 | 6.25
FFMTJ | 120 | 10 | 15 | 25 | 6.25
GMRML | 15 | 5 | 1 | 12 | 20

The TMR ratio, bridge resistance (at $\mu_{o}H$ = 0 mT), saturation field
($B_{sat}$) and sensitivity for each sensor are listed in Table 1. Sensor
outputs and sensitivities as a function of applied field for the three sensors
are shown in Fig. 2 as black and red curves, respectively.

Figure 2:  (left axes, black symbols) Bridge outputs and (right axes, red
symbols) sensitivities for (a) SPMTJ sensor, (b) FFMTJ and (c) GMRML sensors.
Noise measurements were performed and detectivity evaluated at H = 0 mT for
the SPMTJ and FFMTJ and H = 0.5 mT for the GMRML.

Fig. 2a shows the typical performance of SPMTJ sensors. The bridge output has
near zero hysteresis due to the thermal demagnetizing effects of the
superparamagnetic free layer. There is no offset in the output. This is due to
the fast randomization of the free layer’s magnetization and granular
structure of the film negating any of the traditional coupling effects. The
peak sensitivity of 25 mV/V/mT (Fig. 2a, red) occurs at $H$ = 0 mT. The
sensitivity decreases precipitously to either side of $H$ = 0 mT. The full
width at half maximum (FWHM) of the sensitivity "peak" is approximately 10 mT.
The sharpness of the sensitivity peak limits the linearity to +/- 2 mT.

The FFMTJ sensor response is shown in Fig. 2b. The FFMTJ was tailored to
minimize hysteresis and offset while extending the linear range through the
use of shape anisotropy. Interestingly, the maximum sensitivity is comparable
to the SPMTJ sensor (Fig.2b, red), however the sensitivity "peak" is much
broader with a FWHM of 30 mT. This results in the linear range of the sensor
being +/-10 mT, a factor of five increase over the SPMTJ films while
preserving the sensitivity.

The GMRML sensor response (Fig. 2c) is exotic compared to the bipolar films.
The multilayer structure results in a unipolar sensor response. Permalloy flux
concentrators are used in order to maximize the sensitivity and tailor the
linear range without modifying the underlying film properties. In this case,
the flux concentrators employed allow 10x reduction in the $B_{sat}$ (note the
field axis compared to the MTJ sensors). The flux concentration allows the
maximum sensitivity (Fig. 2c, red) to be comparable to the MTJ sensors (1).
This peak sensitivity occurs near $H$ = 0 mT. Without a bias field or magnet
in place, the omnipolar device produces a discontinuity in the bridge output
at $H$ = 0 mT. The usable range of the unbiased sensor shown is between 0.1 mT
and 1.5 mT, which has an average sensitivity of 12 mV/V/T.

Figure 3:  Noise spectra for the GMRML (black), SPMTJ (red) and FFMTJ (blue)
sensor elements in Fig. 2. Each sensor was biased to 3 V for the measurement
as illustrated in Fig. 1.

Noise data from for the three sensors at their average operating fields (0 mT
for the SPMTJ and FFMTJ and 0.5 mT for the GMRML) is shown in Fig. 3. At low
frequencies the noise is dominated by the magnetic $1/f$ contribution. The
SPMTJ (Fig. 3, red) and FFMTJ (Fig. 3, blue) sensors have a similar $1/f$
character (slope). However, the SPMTJ sensor is roughly 10x less noisy. The
GMRML sensor’s spectrum (Fig. 3, black) has a different character with more
curvature and overall lower noise throughout the frequency range.

Around 1 kHz, the SPMTJ spectra (Fig. 3, red) flattens out as the output noise
approaches the Johnson noise floor. The frequency at which this transition
takes place is increased for lower resistance devices. Thus, by 20 kHz the
difference in noise between the high resistance SPMTJ and the FFMTJ has been
reduced, and at still higher frequencies, the noise of the SPMTJ likely
surpasses that of the FFMTJ.

The GMRML sensor also begins to saturate at high frequency, but does so more
gradually (Fig. 3, black). Previous noise measurements of this particular make
of GMRML sensor (an NVE Corp. AA002) by Stutzke, et al. show the noise floor
is reached around 100 Hz. The GMRML sensor from this study appears to have
pushed the floor to higher frequencies and is likely due to our sensors being
biased to 3 V compared to the 1.2 V in the Stutzke study, resulting in a
larger electric $1/f$ contribution.Stutzke  _et al._ (2005); Jiang  _et al._
(2004)

Figure 4:  Detectivity versus frequency for the GMRML (black), SPMTJ (red)and
FFMTJ (blue) sensor elements in Fig. 2 The 2.5x difference between the SPMTJ
and GMRML sensors allows for the SPMTJ sensor to have the lowest detectivity
at 10 kHz.

Dividing the noise spectra by the sensitivity yields the detectivity, i.e. the
lowest detectable field. This is shown in Fig. 4 for each sensor. The GMRML
sensor (Fig. 4, black triangles) has the lowest detectivity. The SPMTJ film
has the second lowest detectivity; still 67% larger than the GMRML’s
throughout the frequency range. The detectivity for both the SPMTJ and GMRMR
drops below 1 nT/$\sqrt{Hz}$ at 10 kHz with values of 0.8 nT/$\sqrt{Hz}$ and
0.53 nT/$\sqrt{Hz}$, for the SPMTJ and GMRML sensors, respectively.

It is important to note that while the GMRML sensor has the lowest
detectivity, it also has the smallest linear range with $B_{sat}$ = 1 mT.
According to the $B_{sat}$ values in Table I, a flux concentration of 5x and
15x for the SPMTJ and FFMTJ films, respectively, would result in sensors with
the same linear range. This provides for a more direct comparison of the
sensors.

Figure 5:  Detectivity versus frequency for the GMRML (black), SPMTJ (red) and
FFMTJ (blue) sensor elements. This time the detectivity curves are normalized
to have the same linear range. The normalization allows for a more direct
comparison of the sensor materials.

Fig. 5 shows the detectivity of each of the three sensors as if they were flux
concentrated to have the same $B_{sat}$. The GMRML and SPMTJ films now have
comparable detectivities of 6 nT/$\sqrt{Hz}$ and 8 nT/$\sqrt{Hz}$ at 1 Hz,
respectively. The SPMTJ detectivity drops below that of the GMRML for $f$ >
100 Hz. In contrast, the FFMTJ still has a detectivity of 30 nT/$\sqrt{Hz}$ at
1 Hz, only becoming comparable to the other sensors above 1 kHz.

This comparative analysis relies on the ability to manipulate the flux
concentration to create equivalent sensors. In practice, adding flux
concentration results in two main performance trade-offs. First, even modest
amounts of flux concentration (e.g. 5x or higher) will drastically increase
the sensor size. Second, adding flux concentration reduces and limits the
operating field range of the sensor.

The second trade-off presents the dilemma for magnetoresistive sensors of
choosing detectivity minimization versus having a robust operating field
range. This is an important problem to address in applications such as
surgical navigation and precision automation where fields on the order of 1 mT
are used, but sub 1 nT/$\sqrt{Hz}$ resolution are required. Surgical
navigation sensors also need to be very small; ideally less than 2 mm in any
direction.

Figure 6:  Plot of sensitivity versus $B_{sat}$ for the (red) SPMTJ, (blue)
FFMTJ and (black) GMRML sensors. (purple) A notional sensor, assuming the
ability to create a device with record MgO TMR experimentally achieved by
Ikeda, et al. is also shownIkeda  _et al._ (2008); Zheng  _et al._ (2019).
Lines take each sensor type from 1x to 20x flux concentration. The coincident
dashed line represents the potential ultimate limit of TMR sensor performance.

One option to address such requirements is to maximize the TMR ratio. It has
been shown that TMR ratios beyond 100 % won’t significantly improve sensor
noise, but will allow for a reduced need for flux concentration, and hence a
higher linear range.Egelhoff  _et al._ (2009) Fig. 6 shows the possible linear
ranges, i.e. $(B_{sat})$ versus sensitivities for various GMR and TMR sensors.
Each line shows when the sensor is flux concentrated between 1x (highest
$B_{sat}$ value) and 20x (lowest $B_{sat}$ value). Sensors at higher $B_{sat}$
and sensitivity correlate with larger TMR ratios.

The record room temperature TMR ratio is 604%, demonstrated by Ikeda et
al.Ikeda  _et al._ (2008) The long standing of this record makes it something
of an upper limit for MTJ devices. That particular MTJ is not practical for a
sensor for several reasons, most specifically it being in a pseudo-spin-valve
configuration and having large hysteresis. However, it can notionally be used
to show the limits of TMR sensor detectivity. Thus, included in Fig. 6 is a
hypothetical sensor with 604% TMR. The dashed line is used to delineate the
ultimate limits of TMR sensitivity versus $B_{sat}$. Comparing measured
literature values from the MR Sensor roadmap, it appears that MR sensor
detectivity will minimize around 10 pT/$\sqrt{Hz}$ at 1 Hz without significant
material changes or technology shifts.Zheng  _et al._ (2019)

Fig. 7 shows models of the detectivity versus frequency for the SPMTJ, FFMTJ,
GMRML and notional film. The models constrain parameters such that the sensor
footprint and magnetic response are as identical as possible. We assume an
"earth’s field anomaly" application where $B_{sat}$ = 0.1 mT and there is a
reasonably small component with an active resistor area of 3 mm2. The small
$B_{sat}$ and large resistor area allows for a significant drop in the
detectivity compared to the measured values in Fig. 4.

At low frequencies (1 Hz), where $1/f$ noise dominates, the detectivity of the
MTJ materials is comparable with the FFMTJ and notional films having the
lowest detectivity at roughly 10 pT/$\sqrt{Hz}$. This reaffirms the
diminishing impact of TMR > 100 % as shown by other groups. Egelhoff  _et al._
(2009). The SPMTJ sensor has the next lowest detectivity of 20 pT/$\sqrt{Hz}$.
This performance degradation is primarily attributed to the lower TMR ratio,
although this could also be hampered by active area (which is generally an
inaccessible parameter to an end user). The GMRML sensor has the highest
detectivity at 70 pT/$\sqrt{Hz}$.

Figure 7:  Modeled ultimate detectivity versus frequency for the (red) SPMTJ,
(blue) FFMTJ, (black) and (purple) notional sensor reaching 604% TMR.Ikeda
_et al._ (2008) All four plots assume $B_{sat}$ = 0.1 mT, no additional flux
concentration and an active resistor area of 3 mm2.

As is known, the $1/f$ contribution subsides at high frequencies, leaving
Johnson and shot noise as the remaining detectivity contributions. These are
dictated by the device resistance. With a RA product of nearly 400
k$\Omega-\mu$m2 The FFMTJ film approaches 1 pT/$\sqrt{Hz}$ at 1 kHz. The GMRML
film’s low resistance results in 700 fT/$\sqrt{Hz}$ detectivity around 100
kHz. The Ikeda film, with an RA = 10 $\Omega-\mu$m2 allows for the detectivity
to drop to 200 fT/$\sqrt{Hz}$ at 30 kHz.

Much work involving reaching fT detectivities involves some form of field or
signal modulation. Of these are microelectromechanical system-based (MEMS)
flux concentrators that serve to modulate the field at high frequencies where
Johnson noise is the only consideration.Edelstein  _et al._ (2006). Another
approach is to use specialized flux concentrators.Pannetier  _et al._ (2004).
Indeed, the utilization of other effects such as modulation by voltage
controlled magnetic anisotropy or spin transfer torques may also help to drop
the detectivity.

Interestingly, it should be noted that going to high frequencies can result in
an increased detectivity. Recent work by He et al. has shown that the
detectivity in flux concentrated MTJs can actually increase for large
frequencies, as the permeability of the flux concentrators decrease; limiting
the minimum detectivity in their sensor to 30 pT/$\sqrt{Hz}$.He  _et al._
(2018) Thus, the technique of sensor/field modulation may also be limited to a
particular frequency.

In conclusion we have performed a comparative study of the noise and
detectivity in three classes of sensors. We have found that on first
inspection, the GMRML sensor has the lowest detectivity, it also has the
largest size and power consumption. When the sensors are normalized to
$B_{sat}$ the differences are no longer evident. Extending the parameterized
comparisons to other materials, including the best demonstration Ikeda films,
shows that there is likely an ultimate performance limit for magnetoresistive
sensors in the 100s of fT range. However, normalizing the sensor’s noise
performance to a tunable parameter, such as the linear operating range
provides a much clearer and useful comparison. The hope is this work will
serve as a guide to MR sensor design, illustrating the present limits of the
technology.

We would like to acknowledge Cathy Nordman and Maria Torija for fruitful
discussions.

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

## References

  * Baibich  _et al._ (1988) M. N. Baibich, J. M. Broto, A. Fert, F. N. Van Dau, and F. Petroff, “Giant Magnetoresistance of (001)Fe/(001)Cr Magnetic Superlattices,” Physical Review Letters 61, 2472–2475 (1988).
  * Binasch  _et al._ (1989) G. Binasch, P. Grünberg, F. Saurenbach, and W. Zinn, “Enhanced magnetoresistance in layered magnetic structures with antiferromagnetic interlayer exchange,” Physical Review B 39, 4828–4830 (1989).
  * Dieny  _et al._ (1991) B. Dieny, V. Speriosu, S. Parkin, B. Gurney, D. Wilhoit, and D. Mauri, “Giant magnetoresistive in soft ferromagnetic multilayers,” Physical Review B 43, 1297–1300 (1991).
  * Parkin  _et al._ (1999) S. S. P. Parkin, K. P. Roche, M. G. Samant, P. M. Rice, R. B. Beyers, R. E. Scheuerlein, E. J. O’Sullivan, S. L. Brown, J. Bucchigano, D. W. Abraham, Y. Lu, M. Rooks, P. L. Trouilloud, R. A. Wanner, and W. J. Gallagher, “Exchange-biased magnetic tunnel junctions and application to nonvolatile magnetic random access memory (invited),” Journal of Applied Physics 85, 5828 (1999).
  * Slonczewski (2002) J. Slonczewski, “Currents and torques in metallic magnetic multilayers,” Journal of Magnetism and Magnetic Materials 247, 324–338 (2002).
  * Ralph and Stiles (2008) D. Ralph and M. Stiles, “Spin transfer torques,” Journal of Magnetism and Magnetic Materials 320, 1190–1216 (2008).
  * Sankey  _et al._ (2008) J. C. Sankey, Y.-T. Cui, J. Z. Sun, J. C. Slonczewski, R. A. Buhrman, and D. C. Ralph, “Measurement of the spin-transfer-torque vector in magnetic tunnel junctions,” Nature Physics 4, 67–71 (2008).
  * Wang, Anderson, and Daughton (1997) D. Wang, J. Anderson, and J. Daughton, “Thermally stable, low saturation field, low hysteresis, high GMR CoFe/Cu multilayers,” IEEE Transactions on Magnetics 33, 3520–3522 (1997).
  * Ikeda  _et al._ (2008) S. Ikeda, J. Hayakawa, Y. Ashizawa, Y. M. Lee, K. Miura, H. Hasegawa, M. Tsunoda, F. Matsukura, and H. Ohno, “Tunnel magnetoresistance of 604% at 300 K by suppression of Ta diffusion in CoFeB$/$MgO$/$CoFeB pseudo-spin-valves annealed at high temperature,” Applied Physics Letters 93, 082508 (2008).
  * Wang  _et al._ (2009) C. Wang, Y.-T. Cui, J. Z. Sun, J. A. Katine, R. A. Buhrman, and D. C. Ralph, “Sensitivity of spin-torque diodes for frequency-tunable resonant microwave detection,” Journal of Applied Physics 106, 053905 (2009).
  * Wolf  _et al._ (2001) S. A. Wolf, D. D. Awschalom, R. A. Buhrman, J. M. Daughton, S. von Molnár, M. L. Roukes, A. Y. Chtchelkanova, and D. M. Treger, “Spintronics: a spin-based electronics vision for the future.” Science (New York, N.Y.) 294, 1488–95 (2001).
  * Graham, Ferreira, and Freitas (2004) D. L. Graham, H. A. Ferreira, and P. P. Freitas, “Magnetoresistive-based biosensors and biochips,” Trends in Biotechnology 22, 455 – 462 (2004).
  * Caruso (1997) M. J. Caruso, “Applications of magnetoresistive sensors in navigation systems,” in  _SAE Technical Paper_ (SAE International, 1997).
  * Schewe and Schelter (1997) H. Schewe and W. Schelter, “Industrial applications of magnetoresistive sensors,” Sensors and Actuators A: Physical 59, 165 – 167 (1997), 1st European magnetic sensors and actuators conference.
  * Fujiwara  _et al._ (2018) K. Fujiwara, M. Oogane, A. Kanno, M. Imada, J. Jono, T. Terauchi, T. Okuno, Y. Aritomi, M. Morikawa, M. Tsuchida, N. Nakasato, and Y. Ando, “Magnetocardiography and magnetoencephalography measurements at room temperature using tunnel magneto-resistance sensors,” Applied Physics Express 11, 023001 (2018).
  * McFadyen, Fullerton, and Carey (2006) I. R. McFadyen, E. E. Fullerton, and M. J. Carey, “State-of-the-art magnetic hard disk drives,” MRS Bulletin 31, 379–383 (2006).
  * Tahoori  _et al._ (2018) M. Tahoori, S. M. Nair, R. Bishnoi, S. Senni, J. Mohdad, F. Mailly, L. Torres, P. Benoit, A. Gamatie, P. Nouet, F. Ouattara, G. Sassatelli, K. Jabeur, P. Vanhauwaert, A. Atitoaie, I. Firastrau, G. Di Pendina, and G. Prenat, “Using multifunctional standardized stack as universal spintronic technology for iot,” in  _2018 Design, Automation Test in Europe Conference Exhibition (DATE)_ (2018) pp. 931–936.
  * Liu  _et al._ (2019) X. Liu, K. H. Lam, K. Zhu, C. Zheng, X. Li, Y. Du, C. Liu, and P. W. T. Pong, “Overview of spintronic sensors with internet of things for smart living,” IEEE Transactions on Magnetics 55, 1–22 (2019).
  * Ouyang  _et al._ (2019) Y. Ouyang, Z. Wang, G. Zhao, J. Hu, S. Ji, J. He, and S. X. Wang, “Current sensors based on gmr effect for smart grid applications,” Sensors and Actuators A: Physical 294, 8 – 16 (2019).
  * Nowak  _et al._ (1998) E. R. Nowak, R. D. Merithew, M. B. Weissman, I. Bloom, and S. S. P. Parkin, “Noise properties of ferromagnetic tunnel junctions,” Journal of Applied Physics 84, 6195–6201 (1998), https://doi.org/10.1063/1.368936 .
  * Stutzke  _et al._ (2005) N. A. Stutzke, S. E. Russek, D. P. Pappas, and M. Tondra, “Low-frequency noise measurements on commercial magnetoresistive magnetic field sensors,” Journal of Applied Physics 97, 10Q107 (2005), https://doi.org/10.1063/1.1861375 .
  * Ingvarsson  _et al._ (2000) S. Ingvarsson, G. Xiao, S. S. P. Parkin, W. J. Gallagher, G. Grinstein, and R. H. Koch, “Low-frequency magnetic noise in micron-scale magnetic tunnel junctions,” Phys. Rev. Lett. 85, 3289–3292 (2000).
  * Klaassen, Xinzhi Xing, and van Peppen (2004) K. B. Klaassen, Xinzhi Xing, and J. C. L. van Peppen, “Signal and noise aspects of magnetic tunnel junction sensors for data storage,” IEEE Transactions on Magnetics 40, 195–202 (2004).
  * Zheng  _et al._ (2019) C. Zheng, K. Zhu, S. C. de Freitas, J.-Y. Chang, J. E. Davies, P. Eames, P. P. Freitas, O. Kazakova, C. Kim, C.-W. Leung, S.-H. Liou, A. Ognev, S. N. Piramanayagam, P. Ripka, A. Samardak, K.-H. Shin, S.-Y. Tong, M.-J. Tung, S. X. Wang, S. Xue, X. Yin, and P. W. T. Pong, “Magnetoresistive Sensor Development Roadmap (Non-Recording Applications),” IEEE Transactions on Magnetics 55, 1–30 (2019).
  * Egelhoff  _et al._ (2009) W. Egelhoff, P. Pong, J. Unguris, R. McMichael, E. Nowak, A. Edelstein, J. Burnette, and G. Fischer, “Critical challenges for picotesla magnetic-tunnel-junction sensors,” Sensors and Actuators A: Physical 155, 217 – 225 (2009).
  * Deak, Zhou, and Shen (2017) J. G. Deak, Z. Zhou, and W. Shen, “Tunneling magnetoresistance sensor with pt level 1/f magnetic noise,” AIP Advances 7, 056676 (2017), https://doi.org/10.1063/1.4978465 .
  * Zou  _et al._ (2001) W. Zou, H. Wadley, X. Zhou, R. Johnson, and D. Brownell, “Composition-morphology-property relations for giant magnetoresistance multilayers grown by rf diode sputtering,” MRS Proceedings 674, T1.5 (2001).
  * Jiang  _et al._ (2004) L. Jiang, E. R. Nowak, P. E. Scott, J. Johnson, J. M. Slaughter, J. J. Sun, and R. W. Dave, “Low-frequency magnetic and resistance noise in magnetic tunnel junctions,” Phys. Rev. B 69, 054407 (2004).
  * Edelstein  _et al._ (2006) A. S. Edelstein, G. A. Fischer, M. Pedersen, E. R. Nowak, S. F. Cheng, and C. A. Nordman, “Progress toward a thousandfold reduction in 1/f noise in magnetic sensors using an ac microelectromechanical system flux concentrator (invited),” Journal of Applied Physics 99, 08B317 (2006), https://doi.org/10.1063/1.2170067 .
  * Pannetier  _et al._ (2004) M. Pannetier, C. Fermon, G. Le Goff, J. Simola, and E. Kerr, “Femtotesla magnetic field measurement with magnetoresistive sensors,” Science 304, 1648–1650 (2004), https://science.sciencemag.org/content/304/5677/1648.full.pdf .
  * He  _et al._ (2018) G. He, Y. Zhang, L. Qian, G. Xiao, Q. Zhang, J. C. Santamarina, T. W. Patzek, and X. Zhang, “Picotesla magnetic tunneling junction sensors integrated with double staged magnetic flux concentrators,” Applied Physics Letters 113, 242401 (2018), https://doi.org/10.1063/1.5052355 .


# Prompt Learning for Domain Adaptation in Task-Oriented Dialogue

Makesh Narsimhan Sreedhar
University of Wisconsin - Madison
<EMAIL_ADDRESS>
&Christopher Parisien
NVIDIA
<EMAIL_ADDRESS>Work done as part of an internship with NVIDIA

###### Abstract

Conversation designers continue to face significant obstacles when creating
production-quality task-oriented dialogue systems. The complexity and cost
involved in schema development and data collection is often a major barrier
for such designers, limiting their ability to create natural, user-friendly
experiences. We frame the classification of user intent as the generation of a
canonical form, a lightweight semantic representation using natural language.
We show that canonical forms offer a promising alternative to traditional
methods for intent classification. By tuning soft prompts for a frozen large
language model, we show that canonical forms generalize very well to new,
unseen domains in a zero- or few-shot setting. The method is also sample-
efficient, reducing the complexity and effort of developing new task-oriented
dialogue domains.

##  1 Introduction

Task-oriented dialogue systems in Conversational AI are challenging for
developers to create. The current generation of dialogue frameworks requires
developers to define actions (intents) and parameters (slots) that the natural
language understanding (NLU) module accepts. This is then used to populate API
service calls that operate in the backend to fulfill the user request. Casting
natural language utterances from the user to a discrete set of intents and
slots is often not very intuitive. This in turn leads to a situation where
developers rely on hand-crafted rule-based grammars or a large annotated set
of training samples for machine learning models to implement a given design.
Any change to the design of the dialogue system would then require the
developers to revisit and modify the implementation which is very often a
time-consuming process. In this work, we aim to make dialogue system design
easier and more intuitive.

The tremendous success of pre-trained language models such as BERT (Devlin et
al., 2019) have made them the de facto standard for most intent classification
and slot-filling tasks. However, these models are not immune to the challenge
of adapting and extending existing models to new domains. One adaptation
approach that has exploded in popularity in recent times is the usage of
prompts with these language models. With a task description and few samples
showing the input-output pairs, these language models become extremely
effective at solving these tasks, especially at larger model sizes.

Manually specifying prompts suffers from sensitivity to phrasing; we get
widely varying results based on how we frame the prompt. Prompt tuning (Lester
et al., 2021) and p-tuning (Liu et al., 2021) have emerged as strong
alternatives to manual prompt designing and they help optimize task-specific
prompt tokens to get the best performance while keeping the language model
itself frozen. In this work, we explore the task of intent classification
using these large language models and p-tuning. Generative methods for
classification tasks have not been widely adopted because generation is
inherently difficult to control and utilize for further downstream tasks.
Using our experiments on the Schema Guided Dialogue (Rastogi et al., 2019)
dataset and the Virtual Assistant Benchmark (Liu et al., 2019), we show that
with p-tuning we can achieve promising zero-shot and few-shot generalization
capabilities to unseen domains.

In the task of intent classification, the intent labels provided as part of
the dataset are usually terse and rigid. Generative models generalize better
when intent labels are more descriptive but structured at the same. We borrow
some aspects and terminology from semantic parsing to cast the intent labels
to a more compositional format, known as canonical forms. In the traditional
sense, canonical forms are paraphrases of the user utterances to convert them
to a form that the semantic parser can operate on to output logical
representations. In our use case, we loosely use the term, canonical forms, to
refer to intent labels that are more descriptive than the discrete ones but
are not too verbose, e.g., "transfer_money" $\rightarrow$ "transfer money to
bank account". We manually frame these canonical forms and do not rely on any
grammar, simplifying the approach.

We observe that using such canonical forms as labels for the intent
classification task allows the model to generalize better to domains that are
adjacent, but not seen at train time (e.g., Flight Reservations $\rightarrow$
Bus Bookings). We also find that it is beneficial to do a two-stage P-tuning
for domain adaption, i.e., once we have a p-tuned large language model on a
wide set of domains, we can continue p-tuning this model on a small set of
labelled samples from the target domain to allow the model to generalize
better. We find that this few-shot approach works very well and this has
promising implications for developers for dialogue systems; with minimal
effort it would be feasible to adapt an existing model pre-trained on multiple
domains to a new domain. In summary, our contributions are:

  * •

We cast the problem of intent classification into a generative approach and
rewrite intent labels in a more descriptive format (canonical forms).

  * •

When using such canonical forms, generative approaches with Large Language
Models(LLMs) show promising results when compared with traditional methods for
intent classification.

  * •

Generative models generalize very well to unseen domains in zero-shot and few-
shot settings when compared with BERT-style approaches.

  * •

We demonstrate the sample efficiency of p-tuning LLMs where we can achieve
close to full dataset performance with a fraction of the data.

##  2 Method

In this section, we describe the creation of canonical forms and the prompt
tuning technique we adopt for intent classification in the task-oriented
dialogue setting.

###  2.1 Canonical Forms

Canonical forms are usually paraphrases of the user utterance to a
standardized form that can be utilized by downstream systems. These forms are
traditionally obtained by using a set of grammar rules written by experts. The
output of this process is a natural language sequence, but structured in a
form that makes it better suited for a semantic parser. Different semantic
parsers employ different canonical forms and thus transfer across datasets is
quite challenging.

Utterance | Canonical Form
---|---
what is the newest published article? | article that has the largest publication date
who has published the most articles? | person that is author of the most number of article

Table 1: Examples of canonical forms corresponding to user utterances from the
Overnight (Wang et al., 2015) semantic parsing dataset.

Our work uses canonical forms as a method of obtaining the intent of a user
utterance. Traditionally, intent labels tend to be terse, which makes it
difficult for models to generalize to unseen domains. The expressive and
compositional nature of language models can be exploited if the intent labels
are more verbose, allowing them to extrapolate the generated intents to
capture even novel domains. At the same time, if the intent labels tend to be
very long and riddled with descriptions, the language models become
susceptible to hallucinations. Our work proposes the use of canonical forms as
a way of establishing a balance between being terse and too verbose. We map
intent labels to short descriptive phrases, e.g., "check_balance"
$\rightarrow$ "check balance in bank account". Unlike traditional canonical
forms, we do not use any formal grammar to perform this mapping and the
phrases are manually specified. We believe that such an approach would reduce
the burden on developers and designers of conversational systems.

###  2.2 P-tuning

Large Language Models (LLMs) have exhibited remarkable generalization
capability when queried using prompts that contain examples of the task to be
performed. However, the performance of LLMs varies widely depending on how
such prompts are constructed. In order to overcome this issue of LLM
sensitivity to the format of the prompt, multiple studies have come up with
methods for automated prompt construction using discrete tokens (Lester et
al., 2021) as well as soft tokens (Liu et al., 2021).

In this work, we utilize the p-tuning approach that appends learnt soft tokens
into the prompt that is fed to the LLM. The soft tokens traditionally do not
have a mapping to words/subwords in the model vocabulary and are simply
vectors optimized using gradient descent. Following the setup proposed by Liu
et al. (2021), we use an LSTM model to learn and predict these soft tokens.
The parameters of the LLM are frozen and only the parameters of this LSTM
model are updated during p-tuning. We initialize the LSTM with random weights
at the beginning of the p-tuning process and then update it during the
training stage to output the optimal soft tokens. At the end of the training
phase, we store these soft tokens and append them with the prompt to the LLM
to get its prediction. The advantage of p-tuning is that we freeze the LM
weights and update only the weights of the LSTM (14M parameters). This results
in modifying only a very small fraction of the weights compared to traditional
finetuning where all of the weights are updated.

The LM of choice in our experiments are the Megatron-GPT (Narayanan et al.,
2021) models that are decoder-only transformers.

##  3 Experimental Setting

In this section, we describe the datasets used, the baselines we use for
comparison and the evaluation metrics.

###  3.1 Datasets

We consider two widely known datasets in the dialogue community, the Schema
Guided Dialogue (SGD) dataset (Rastogi et al., 2019) and the Virtual Assistant
dataset (Liu et al., 2019).

Schema Guided Dialogue \- This dataset covers 16 domains and has over 16k
annotated conversations. The domains span a variety of user actions, including
setting calendars and alarms, travel booking (car rentals, flights, buses and
trains), music, weather, movies, and more. The dataset also contains multi-
domain dialogues where the utterances switch between domains. For the purpose
of our experiments, we consider only the single-domain dialogues with 37
intents across all utterances.

Virtual Assistant Dataset \- This dataset covers 21 domains with 64 intents
across all utterances. As the name suggests, the domains relate to user
queries over a wide range of topics, including operating smart-home devices,
media consumption, weather and travel. It has over 25k annotated user
utterances that identify intents and slot values.

###  3.2 Prompt Template

The prompts that we use for intent classification have the following format

$\displaystyle<v_{1}..v_{n}>\quad{utterance}\quad intent:{canonical}$ (1)

where $<v_{1},v_{2},..,v_{n}>$ indicate the virtual tokens.

During the training stage of p-tuning, the model is shown the entire sequence,
but the loss is computed only on the answer which in this case is the
predicted canonical form. During inference, the context to the model includes
the sequence until the word "intent:" and the model completes the sequence
with its prediction for the intent. We use 100 virtual tokens with our prompt-
encoder being an LSTM model with 2 layers.

###  3.3 Evaluation Method

Intent Classification Evaluating generative models for a classification task
is not straightforward. This is further complicated by the fact that our model
generates a canonical form identifying the intent of a given user utterance.
We propose two methods to cast this generation problem to a classification
setting. The difficulty arises from the fact that generated sequences very
often differ from the exact gold truth sequence that the model sees as part of
training. We utilize two approaches based on associating the generated
canonical form to its closest label, i.e., a nearest neighbor search. Once the
canonical form label has been identified as the prediction, it becomes trivial
to compute the classification accuracy. Since we already have a one-to-one
mapping between canonical form labels and the discrete intent label, we can
easily measure the performance of our model.

  * •

Using Fasttext Embeddings (Bojanowski et al., 2016): We take the mean of all
the embedding vectors of the generated canonical form and consider the vector
obtained to be the representation of the whole sequence. We compute similar
vectors for all the canonical form labels and consider the canonical form
label that has the maximum cosine similarity with the generated one as the
model’s prediction.

  * •

Using Sentence Transformers (Reimers and Gurevych, 2019): We use the miniLM-QA
(Wang et al., 2020) transformer model that has been pretrained on multiple
datasets on the text entailment/semantic search task, i.e., given a query and
a set of keys (documents/labels), it ranks the keys in order of relevance. We
give as input to the model the generated canonical form (query) and the list
of canonical form labels (keys). The model then returns the closest canonical
form label to the generated canonical form which we consider as the
prediction.

###  3.4 Baselines

We consider the following baselines for the intent classification task.

  * •

BERT-based finetuned model (Intent Classification): We finetune BERT models on
the datasets described in section 3.1. While some of the Megatron-GPT models
we use are larger than the BERT model in terms of number of parameters, it
should be noted that the LM parameters are frozen during the training stage of
p-tuning and only the weights of the LSTM ( 14M parameters) are updated.

###  3.5 Evaluation Settings

We evaluate the performance of our model in two settings: in-domain and out-
of-domain.

####  3.5.1 In-Domain

This setting corresponds to the traditional dataset splits where the train and
test sets come from similar distributions. We p-tune the Megatron-GPT models
on the train set and evaluate them on the test set for intent classification.

####  3.5.2 Out-of-Domain

In this setting, we aim to explore the generalization capability of LLMs. We
hold out certain domains from the train set and use utterances from the held
out domains as our test. This helps us understand how well these LLMs can
generalize to unseen domains. The held out sets that we consider are:

  * •

Schema Guided Dialogue (SGD): We hold out utterances corresponding to bus
bookings and hotel reservations to form our test set. The train set includes
utterances from adjacent domains: flight booking and restaurant reservations.
This should be a relatively easy setting for the language model to generalize
to.

  * •

Virtual Assistant: To make things more challenging, we hold out utterances
corresponding to operating IOT devices and media consumption commands (e.g.,
commands that are variants of "play" - play movie, play audiobook). The train
set does not have utterances from similar domains and this setting is more
challenging for the model.

We consider the generalization capability of the model in two modes:

  * •

Zero-shot: P-tune the model on the train set and evaluate zero-shot on the
unseen domain test set.

  * •

Few-shot: After p-tuning on the train set, we do a second stage p-tuning on a
set of k samples from the target domain. Unless otherwise noted, k here is 5,
10, 50 or 100 samples.

The few-shot paradigm may be very useful for dialogue system developers in a
limited-resource setting. Developers can implement new domains using existing
language models and a small set of curated examples, without the burden and
expense of retraining or providing a large number of labelled samples.

##  4 Results

In this section, we review the quantitative performance of the models for
intent classification.

###  4.1 Intent Classification

We compute and list the accuracy of the baselines and our p-tuned GPT model in
identifying the intent given the user utterance.

####  4.1.1 In-domain

We find that both the p-tuned GPT model as well as the BERT baseline perform
very well on the standard in-domain split where both the train and test set
come from the same distribution (Table 2). The classification accuracy of
Megatron-GPT increases as we increase the model size. The trend of results
remains consistent for both the SGD and Assistant datasets.

Model | SGD | Assistant
---|---|---
BERT-Large | 0.88 | 0.91
Megatron-GPT - 345M | 0.87 | 0.88
Megatron-GPT - 1.3B | 0.91 | 0.92
Megatron-GPT - 5B | 0.95 | 0.94
Table 2: Classification Accuracy on test sets of the SGD and Assistant
datasets

####  4.1.2 Out-of-Domain

The out-of-domain setting is where the advantage of using a LLM becomes
apparent. It is not feasible to expect a finetuned BERT model to generalize to
an unseen domain not present in the train set. Such models continue to predict
that the intent belongs to one of the intent labels they see during training.
The p-tuned Megatron-GPT models, on the other hand, show impressive zero-shot
and few-shot generalization capabilities on the SGD dataset (Table 3). For
instance, having seen intents such as "buy flight roundtrip tickets" when
presented with utterances for Flight Reservations in training, we can expect
the model to reasonably generalize to utterances from Bus Reservations with
utterances like "Get me a return trip on the bus" with the model’s prediction
for the intent being "buy bus roundtrip tickets".

Mode | Bus Booking | Hotel Reservation
---|---|---
| 345M | 1.3B | 5B | 345M | 1.3B | 5B
Zero Shot | 0.755 | 0.762 | 0.787 | 0.379 | 0.448 | 0.467
FS - 10 samples | 0.907 | 0.789 | 0.942 | 0.793 | 0.720 | 0.939
FS - 50 samples | 0.953 | 0.965 | 0.975 | 0.957 | 0.968 | 0.970

Table 3: Zero-shot and Few Shot (FS) performance on the held out domains of
the SGD dataset. The columns indicate the size of the Megatron-GPT model.

In the Assistant dataset, the p-tuned models face the same issue as the BERT
models: they struggle to generalize to completely unseen domains and the
performance is close to random (Table 4). Unlike in SGD, the held-out domains
do not have sufficiently similar domains in training from which to generalize.
However, the few-shot setting holds promise as the performance of the models
improves with few samples. Since the held out domains have far more intents
compared to the held out domains from the SGD dataset, we employ stratified
sampling to ensure that the few-shot examples are representative of all
intents in the domain.

Mode | IOT devices | Media Consumption
---|---|---
| 345M | 1.3B | 5B | 345M | 1.3B | 5B
Zero Shot | 0.096 | 0.011 | 0.022 | 0.037 | 0.008 | 0.012
FS - 10 samples | 0.62 | 0.71 | 0.75 | 0.58 | 0.62 | 0.68
FS - 50 samples | 0.69 | 0.83 | 0.87 | 0.67 | 0.86 | 0.89

Table 4: Zero-shot and Few Shot (FS) performance on the held out domains of
the Assistant dataset. The columns indicate the size of the Megatron-GPT
model.

##  5 Discussion

The results on zero-shot and few-shot settings for unseen domains demonstrate
that p-tuning a LLM to have intents that are more verbose than discrete labels
can be very helpful.

In this section, we analyze the impact of the structure of canonical forms,
what helps the language model generalize, how sample efficient are these
language models and what all this means for a developer of chatbots and
dialogue systems.

###  5.1 How important is framing the right canonical form?

The phrasing of canonical forms has a significant impact on zero-shot cross
domain generalization. In our initial experiments, we observed that the
language models, especially the smaller ones, sometimes rely on spurious
correlations to predict the intent. For instance, if the intent
SearchFlightOneWay is mapped to the canonical form search tickets for flight
one way, the model correlates the word ticket in both the user utterance and
canonical form to identify the intent. When we use this model to predict the
intent of user utterances related to bus bookings in a zero-shot manner, the
model predicts that that the intent is related to a flight booking as most
utterances in the bus domain contain the word ticket.

Mode | Accuracy
---|---
| 345M | 1.3B | 5B
ZS - Original | 0.08 | 0.13 | 0.21
ZS - Modified | 0.755 | 0.762 | 0.787
Table 5: Zero shot (ZS) performance on utterances from Bus Bookings. Original
refers to having the canonical form for flight bookings as search tickets for
flight one way which led to incorrect generalizations. Modified refers to
having the improved canonical form for flight bookings as search for flights
one way.

Rephrasing the canonical form for the intent SearchFlightOneWay to search for
flights one way helps the model to avoid making the spurious correlation and
the performance in the zero-shot setting (Table 5) is significantly improved.

Mode | Accuracy
---|---
| 345M | 1.3B
ZS - Original | 0.08 | 0.13
FS 10 samples- Original | 0.76 | 0.72
FS 20 samples - Original | 0.84 | 0.87
Table 6: Zero shot (ZS) performance on utterances from Bus Bookings. Original
refers to having the canonical form for flight bookings as search tickets for
flight one way which led to incorrect generalizations. Adding a small number
of examples resolves the error.

However, the few-shot setting (Table 6) alleviates this problem of sensitivity
of the model to the canonical form structure. When we provide the model with a
few samples from the the target domain, it learns to associate that the
important words to distinguish between the domains are flight and bus and not
ticket.

###  5.2 What do good canonical forms looks like?

Based on our experiments, a set of good canonical forms has the following
properties:

  * •

Similarity in structure: Use similar verbs for similar actions/domains, e.g.,
book a flight, book bus tickets, search for hotels, search for restaurant
reservations.

  * •

Compositional: Using similar structures for canonical forms in similar domains
naturally lends to compositionality. This makes it easier for the model to
generalize in the zero-shot/few-shot setting while still allowing the
developers to easily map the generations to a supported service on the
backend.

  * •

Looks like natural language: Since LLMs are pretrained on very large corpora
of natural language, the benefit of pre-training is realized when the
canonical forms resemble natural language rather than complex semantic forms.
Making discrete intents look more like typical verb phrases brings out the
expressive nature of language models.

Future work will explore and refine methods to automate the creation of
canonical forms.

###  5.3 Do we need the entire training set for p-tuning?

We look for the fewest labelled samples for p-tuning needed to get an accuracy
close to accessing the entire train set. We randomly sample k samples per
intent ($k\in{5,10,20,30}$) to form the train set the model is p-tuned on, and
evaluate on the same test set as above. The train and test sets are from the
in-domain setting for both SGD (Table 7) and Assistant (Table 8) datasets.

#Samples/Intent | Train Size | Accuracy
---|---|---
|  | 345M | 1.3B | 5B
10 | 370 | 0.77 | 0.81 | 0.827
20 | 740 | 0.82 | 0.83 | 0.844
30 | 1110 | 0.84 | 0.85 | 0.87

Table 7: Accuracy on the SGD test set when using only k samples per intent.
The columns indicate the size of the Megatron-GPT model used.

#Samples/Intent | Train Size | Accuracy
---|---|---
|  | 345M | 1.3B | 5B
10 | 640 | 0.69 | 0.81 | 0.84
20 | 1280 | 0.74 | 0.84 | 0.91
30 | 1920 | 0.79 | 0.87 | 0.91

Table 8: Accuracy on the Assistant test set when using only k samples per
intent. The columns indicate the size of the Megatron-GPT model used.

####  5.3.1 Comparison with BERT

We observe that Megatron-GPT is more sample efficient than BERT-type models,
even when adjusting for the number of parameters. We use the 345M parameter
version of the Megatron-GPT for a fair comparison. We finetune BERT-Large and
p-tune the GPT model on the same training subset of the SGD dataset. Results
are shown in Table 9.

With a small number of samples (10 per intent), both Megatron-GPT and BERT-
Large have very similar performance. But with small increases in the number of
labelled samples per intent in the train set, we observe that the performance
of the GPT model improves faster than the BERT model.

#Samples/Intent | Accuracy
---|---
| 345M | BERT
10 | 0.77 | 0.75
20 | 0.82 | 0.767
30 | 0.84 | 0.773
Table 9: Accuracy on the SGD test set when using only k samples per intent.
MegatronGPT-345M is more sample efficient than BERT-Large.

###  5.4 What does this mean for dialogue system developers?

Task-oriented dialogue systems are challenging to create. Most common
frameworks cast utterances into discrete intents and slots, but it is often
not clear how to define these concepts for a given design. Such frameworks
also employ NLU models that often require the creation of either rule-based
grammars or a significantly large corpus of labelled samples. While ML-based
approaches have come a long way, distributional shifts in the way utterances
are structured can degrade performance. By leveraging LLMs, our approach
reduces the effort involved in framing intents and training classifiers.
Because of the flexibility in canonical form schemas and the sample efficiency
of p-tuning, we argue that development of new task-oriented dialogues becomes
simpler and faster. We envision a setting where a model publisher trains and
releases a general-purpose p-tuned language model covering a broad set of
cases. A conversation designer may then write a small set of example queries,
submit a brief p-tuning job, and deploy a new application with minimal cost.

##  6 Conclusion

We explore the use of Large Language Models and p-tuning for intent
classification in task-oriented dialogue systems. We show framing intent
labels into more verbose forms allows LMs to exploit the underlying structure
better and exhibit impressive zero-shot and few-shot generalization. We also
analyze how important the phrasing of the verbose forms are and how many
samples are needed to get good quantitative performance. We hope that this
work on using sample efficient LLMs serves to motivate further research in
making ToD systems simpler and quicker to develop.

##  7 Acknowledgements

The authors would like to thank Zhilin Wang, Virginia Adams, Sandeep
Subramanian, Vlad Getselevich, Prasoon Varshney, and Jonathan Cohen for many
useful discussions during the course of this work.

## References

  * Bojanowski et al. (2016) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016.  Enriching word vectors with subword information.  _arXiv preprint arXiv:1607.04606_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.
  * Liu et al. (2021) Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021.  Gpt understands, too.
  * Liu et al. (2019) Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2019.  Benchmarking natural language understanding services for building conversational agents.
  * Narayanan et al. (2021) Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. 2021.  Efficient large-scale language model training on gpu clusters using megatron-lm.
  * Rastogi et al. (2019) Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2019.  Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020.  Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.
  * Wang et al. (2015) Yushi Wang, Jonathan Berant, and Percy Liang. 2015.  Building a semantic parser overnight.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1332–1342, Beijing, China. Association for Computational Linguistics.


# Deep learning based landslide density estimation on SAR data for rapid
response

Vanessa Boehm
University of California Berkeley
United States
Wei Ji Leong
The Ohio State University
United States
Ragini Bal Mahesh
German Aerospace Center DLR
Germany
Ioannis Prapas
University of Valencia, Spain
National Observatory of Athens, Greece
Edoardo Nemni
United Nations Satellite Centre
(UNOSAT),Switzerland
Freddie Kalaitzis
University of Oxford
United Kingdom

Siddha Ganju
NVIDIA
United States
Raul Ramos-Pollan
Universidad de Antioquia
Colombia

###### Abstract

This work aims to produce landslide density estimates using Synthetic Aperture
Radar (SAR) satellite imageries to prioritise emergency resources for rapid
response. We use the United States Geological Survey (USGS) Landslide
Inventory data annotated by experts after Hurricane María in Puerto Rico on
Sept 20, 2017, and their subsequent susceptibility study which uses extensive
additional information such as precipitation, soil moisture, geological
terrain features, closeness to waterways and roads, etc. Since such data might
not be available during other events or regions, we aimed to produce a
landslide density map using only elevation and SAR data to be useful to
decision-makers in rapid response scenarios.

The USGS Landslide Inventory contains the coordinates of 71,431 landslide
heads (not their full extent) and was obtained by manual inspection of aerial
and satellite imagery. It is estimated that around 45% of the landslides are
smaller than a Sentinel-1 typical pixel which is 10m $\times$ 10m, although
many are long and thin, probably leaving traces across several pixels. Our
method obtains 0.814 AUC in predicting the correct density estimation class at
the chip level (128$\times$128 pixels, at Sentinel-1 resolution) using only
elevation data and up to three SAR acquisitions pre- and post-hurricane, thus
enabling rapid assessment after a disaster. The USGS Susceptibility Study
reports a 0.87 AUC, but it is measured at the landslide level and uses
additional information sources (such as proximity to fluvial channels, roads,
precipitation, etc.) which might not regularly be available in an rapid
response emergency scenario.

##  1 Introduction

According to the United Nations Office for Disaster Risk Reduction, landslides
have affected 4.8 million people and caused 18,414 deaths between
1998-2017111https://www.preventionweb.net/files/61119_credeconomiclosses.pdf.
Rising temperatures and climate change are projected to worsen this situation
[1, 2], and, given these predictions, there is a growing need for timely and
accurate landslide assessment methods that can inform decision-makers and
emergency responders. In the aftermath of a disaster event, optical satellite
imagery is commonly used to map the extent of the event. The fact that optical
data is often hindered by clouds and limited to daytime observations,
motivates the study of weather SAR as a monitoring technique.

The review in Tehrani et al. [3] provides a comprehensive view of machine
learning and deep learning (DL) methods applied to landslide detection,
including the usage of SAR data. Works on DL based landslide density or
susceptibility mapping are starting to emerge such as the recent work related
to [4] or [5]. However, to the best of our knowledge, very few studies are
carried out in challenging contexts such as this work, with weak labels, i.e.
landslide heads on small landslides.

### Application context

This work would enable the rapid generation of coarse landslide density maps
after a disaster event using only elevation and SAR data which is commonly
available around the globe. These maps can be used to prioritize emergency
resources and to validate landslide inventories reducing the need for experts
doing fieldwork. Moreover, our approach enables a scenario where landslides
are typically small (as compared to SAR acquisition resolution) and the
requirement for annotated datasets was relaxed to having weak labels, which
are more affordable to produce.

##  2 Data

### Landslide labels

We used the landslide inventory dataset produced by the USGS on slope failure
locations in Puerto Rico after Hurricane María [6], which is also used to
validate their landslide susceptibility map on the island [7]. The dataset
contains 71,431 landslides annotated by a group of experts who marked the head
of each landslide by inspecting optical imagery from different sources [7].
Figure 1 shows the overall density of landslides obtained from this dataset.

Additionally, the USGS produced a dataset of four selected regions in Puerto
Rico annotation the landslides contours [8] [9]. This accounts for 1,066
landslides, which we used only to get a better sense of the nature and extent
of the landslides. As shown in Figure 2, most landslides occupy an area
smaller than a Sentinel-1 typical pixel, but are thin and long, probably
leaving traces across several pixels. This makes this problem specially
challenging.

Figure 1:  Global landslide density, showing 1.28 km2 chips with 1 or more
landslides  Figure 2:  Left: example landslide heads annotations around Lago
Caonillas in the Utuado region. Right: size distribution of 1’066 landslides.

### Input data

We set forth to produce a discretized density estimation by using SAR
polarimetry and a digital elevation model (DEM). As detailed in table 1, we
used SAR acquisitions from Sentinel-1 pre and post Hurricane Maria as
downloaded from Microsoft Planetary Computing
service222https://planetarycomputer.microsoft.com/. For elevation data we used
the SRTM 1 Arc-second global [10] DEM and used the richdem Python package [11]
to generate derived elevation measures (slope, curvature and aspect).

Table 1: Sentinel-1 acquistions used. Hurricane Maria hit Puerto Rico on Sep 20, 2017, so we have three acquisitions pre-event and three post-event. All acquisitions are ascending which ensures comparable SAR geometry. date | satellite | id | orbit | path | frame | direction
---|---|---|---|---|---|---
20170823T223637 (pre) | S1A | 9D3C | 018057 | 135 | 55 | ascending
20170904T223637 (pre) | S1A | B225 | 018232 | 135 | 55 | ascending
20170916T223638 (pre) | S1A | AE4E | 018407 | 135 | 55 | ascending
20170922T223544 (pos) | S1B | D309 | 007511 | 135 | 52 | ascending
20171010T223638 (pos) | S1A | 99FA | 018757 | 135 | 55 | ascending
20171022T223638 (pos) | S1A | 26BB | 018932 | 135 | 55 | ascending

### Chips generation

Chips are numpy arrays of size 128$\times$128 with 17 channels corresponding
to 4 DEM channels (elevation, slope, curvature, aspect), 2 SAR channels (VV +
VH) for each of the 6 Sentinel-1 revisits, and 1 channel for the labels.
Pixels in the label channel have a value of 1 when there is a landslide head
annotated in that location, and zero otherwise. In total, we produced 5,517
chips of dimensions 128$\times$128 with a spatial resolution of 10m$\times$10m
covering 9,039 km2 which corresponds to the surface area of Puerto Rico.
Figure 3 shows an example chip.

Figure 3:  Example of one chip. Note how the backscattering intensity
increased after Hurricane María on Sept 20, 2017.

##  3 Methodology

### Model

Following the methodology in Hughes and Schulz [7] our end goal is to classify
each chip in one of the following _density classes_ : 0 for a chip with no
landslides, 1 for a chip containing between 1 and 25 landslides/km2 and 2 if
the chip contains more than 25 landslides/km2.

We propose a two-stage approach as illustrated in Figure 4. We trained a
convolutional DL model on SAR+DEM input to produce, for each chip, a mask of
the same dimensions (128x128) with pixel values $\in[0,1]$. This mask is the
_fused embeddings_ that the model will produce for a chip. The intent is that
embeddings on pixels with landslides will have a higher value than other
pixels. However, as opposed to a segmentation task, we devise a loss function,
described below, which we believe more adequate to our end task of density
estimation at a chip level. Second, for each chip embedding, we compute a
vector with 19 elements containing the percentiles 5%, 10%, … 95% of pixel
embeddings values. Thus, after inference through the DL model, each chip gets
described with a vector. Together with the associated class of each chip, we
feed this to a random forest classifier to get the final density class
prediction.

All activation maps from end to end are 128x128 in dimensions. Also, the
output layer is defined with a sigmoid activation so that the embeddings
values are $\in[0,1]$. Finally, notice that there are two independent
convolutional branches to deal with DEM and SAR channels as if they were
different modalities. Preliminary experiments showed that this architecture is
preferred for dealing with all channels together.

Figure 4:  Deep Learning model architecture and classification pipeline with
Random Forest

### Loss function

For the DL model, and in order to deal with the challenges of the landslide
head labels, we defined the following loss function at the batch level. Recall
that for a batch of $n$ chips as input the model will produce $n$ embeddings
of 128x128 pixels each. We denote with $e^{+}_{i}$ the set of values of pixels
of embedding $i$ that fall on pixels marked with a landslide head, and with
$e^{-}_{i}$ we denote the rest. We gather all pixel values in the batch on
landslides under $E^{+}=\bigcup\limits_{i=1}^{n}e^{+}_{i}$ and the rest under
$E^{-}=\bigcup\limits_{i=1}^{n}e^{-}_{i}$. Then, our loss for a batch is

$\mathcal{L}_{batch}=(1-\overline{E^{+}})+\overline{E^{-}}$ (1)

where $\overline{E^{+}}$ and $\overline{E^{-}}$ denote the average value of
all elements of $E^{+}$ and $E^{-}$, which value will always be $\in[0,1]$ as
per a sigmoid output. Intuitively, we want embedding pixels on landslide
labels to have in _average_ a high value, and the opposite for the rest of the
pixels. Observe that, usually, there are many more pixels without landslides
so $E^{-}$ will contain many more elements than $E^{+}$. For instance, in a
batch of 8 chips with 100 landslides in total, $E^{+}$ will contain 100 values
and $E^{-}$ will contain $128\times 128\times 8-100=130,972$ values

Since only the landslide heads are annotated, many landslide pixels will be
included in $E^{-}$, and not in $E^{+}$. However, since $E^{-}$ is much more
numerous, their value will get diluted and will not penalize the loss. This
loss will tend to generate many false positives at the pixel level, but in a
way that we expect to benefit the final density estimation task. See Figure 5
to get a sense of this.

### Experimental setup

Since we want to produce a prediction map for the full island of Puerto Rico,
we split all chips randomly into two sets of equal size. Then, we run the
training and prediction pipeline twice. First we train and run the model
selection with split 1 (40% train, 10% validation) and generate predictions
over split 2 (50% of the data). Then, we trained on split 2 (also subdivided
in 40/10) and generated predictions on split 1.

We devised a set of configurations to understand our pipeline behaviour with
respect to two aspects. First, we run configurations using only DEM input,
only SAR input and both fused, using only the relevant part of the
architecture in Figure 4. Second, we wanted to measure the effect of
additional revisits after hurricane Maria in the model results so we run
models using data for the first, second and third revisits after the event
corresponding to the last three lines (after Sep 20, 2017) in Table 1.

##  4 Results

Figure 5 shows the obtained three-class density map with three revisits using
DEM+SAR input data, together with the embeddings and landslides for two chips.
We observed that landslides fall mostly on light blue embeddings which was the
intention. Pixel-wise, there is a large false positive rate, many light blue
pixels with no landslides heads, but this is expected as the loss function 1
dilutes the penalty on these cases by design. This is aligned with the fact
that density estimation by the random forest is roughly done by taking the
average embedding value for each chip, a larger portion of light blue coverage
indicates a larger landslide density on that chip.

Table 2 details the results for each experiment configuration. Our method gets
0.782 AUC using the first revisit two days after the hurricane, and up to
0.814 when using three satellite revisits, one month after the event. It is
interesting to see that DEM-only models get an AUC of around 0.7, signalling
the correlation of the landslides with terrain features. Moreover, fusing SAR
and DEM inputs definitely boosts the model performance. We compare our results
with the USGS aggregated susceptibility index modified by soil moisture (SIAm)
[7] which, as mentioned, uses a weighted mixture of a variety of input data
and is measured on the same landslide inventory dataset.

Figure 5:  Resulting density map and example embeddings.  Table 2: Results summary. USGS SIAm is the aggregated susceptibility index pos event data | 1 revisit | 2 revisits | 3 revisits |
---|---|---|---|---
experiment | train | val | train | val | train | val |
DEM only | 0.705 | 0.698 | 0.699 | 0.687 | 0.701 | 0.692 |
SAR only | 0.710 | 0.708 | 0.715 | 0.710 | 0.742 | 0.740 |
DEM + SAR fused | 0.801 | 0.782 | 0.828 | 0.805 | 0.831 | 0.814 |
USGS SIAm |  |  |  |  |  |  | 0.87

##  5 Conclusions

This work shows that coarse density estimation using weak labels on small
landslides is feasible in a rapid response context, using only elevation and
SAR data. From here on, we believe there are several avenues towards having an
operational SAR based density estimation service. These include better
understanding the model’s response to data (Figure 3 seems to show that SAR is
responding to many things after the hurricane not only landslides), measuring
the transferability of models across different regions, validating the
interpretability for experts of the embeddings produced, and more.

##  6 Acknowledgements

This work was funded by the FDL/NASA 2022 research sprint. For their
invaluable insights we are indebt to Aaron Piña and Gerald Bawden (NASA SMD),
Eric Fielding, Erica Podest and Alex Handwerger (NASA JPL), John Stock, Jason
Stocker and Corina Cerovski-Darriau (USGS), Forrest Williams and Franz Mayer
(Alaska Satellite Facility), Ioannis Papoutsis (National Greek Observatory),
Ronny Haensch (DLR) and Brad Neuberg (Planet).

## References

  * Gariano and Guzzetti [2016] Stefano Luigi Gariano and Fausto Guzzetti.  Landslides in a changing climate.  _Earth-Science Reviews_ , 162:227–252, November 2016.  ISSN 00128252.  doi: 10.1016/j.earscirev.2016.08.011.  URL https://linkinghub.elsevier.com/retrieve/pii/S0012825216302458.
  * Huggel et al. [2012] Christian Huggel, Nikolay Khabarov, Oliver Korup, and Michael Obersteiner.  Physical impacts of climate change on landslide occurrence and related adaptation.  In John J. Clague and Douglas Stead, editors, _Landslides_ , pages 121–133. Cambridge University Press, 1 edition, August 2012.  ISBN 978-0-511-74036-7 978-1-107-00206-7.  doi: 10.1017/CBO9780511740367.012.  URL https://www.cambridge.org/core/product/identifier/9780511740367%23c00206-11-1/type/book_part.
  * Tehrani et al. [2022] Faraz S. Tehrani, Michele Calvello, Zhongqiang Liu, Limin Zhang, and Suzanne Lacasse.  Machine learning and landslide studies: recent advances and applications.  _Nat Hazards_ , jun 2022.  doi: 10.1007/s11069-022-05423-7.  URL https://doi.org/10.1007%2Fs11069-022-05423-7.
  * Ghasemian et al. [2022] Bahareh Ghasemian, Himan Shahabi, Ataollah Shirzadi, Nadhir Al-Ansari, Abolfazl Jaafari, Victoria R. Kress, Marten Geertsema, Somayeh Renoud, and Anuar Ahmad.  A robust deep-learning model for landslide susceptibility mapping: A case study of kurdistan province, iran.  _Sensors_ , 22(4), 2022.  ISSN 1424-8220.  doi: 10.3390/s22041573.  URL https://www.mdpi.com/1424-8220/22/4/1573.
  * Nava et al. [2022] Lorenzo Nava, Kushanav Bhuyan, Sansar Raj Meena, Oriol Monserrat, and Filippo Catani.  Rapid mapping of landslides on SAR data by attention u-net.  _Remote Sensing_ , 14(6):1449, mar 2022.  doi: 10.3390/rs14061449.  URL https://doi.org/10.3390%2Frs14061449.
  * Hughes et al. [2019] Kenneth Stephen Hughes, Desiree Bayouth Garcia, Gabriel Martinez Milian, William H Schulz, and Rex L Baum.  Map of slope-failure locations in puerto rico after hurricane maria, 2019\.  URL https://www.sciencebase.gov/catalog/item/5d4c8b26e4b01d82ce8dfeb0.
  * Hughes and Schulz [2020] K. Stephen Hughes and William Schulz.  Map depicting susceptibility to landslides triggered by intense rainfall, puerto rico, 2020.  URL https://doi.org/10.3133/ofr20201022.
  * Bessette-Kirton et al. [2019a] Erin K. Bessette-Kirton, Jeffrey A Coe, Corina Cerovski-Darriau, Matthew A. Kelly, and William H. Schulz.  Map data from landslides triggered by hurricane maria in four study areas of puerto rico, 2019a.  URL https://www.sciencebase.gov/catalog/item/5ca3c65fe4b0b8a7f6334309.
  * Bessette-Kirton et al. [2019b] Erin K Bessette-Kirton, Corina Cerovski-Darriau, William H Schulz, Jeffrey A Coe, Jason W Kean, Jonathan W Godt, Matthew A Thomas, and K Stephen Hughes.  Landslides triggered by hurricane maria: Assessment of an extreme event in puerto rico.  _GSA Today_ , 29(6):4–10, 2019b.
  * Earth Resources Observation And Science Center(2017) [EROS] Earth Resources Observation And Science (EROS) Center.  Shuttle radar topography mission (srtm) 1 arc-second global, 2017.  URL https://www.usgs.gov/centers/eros/science/usgs-eros-archive-digital-elevation-shuttle-radar-topography-mission-srtm-1-arc?qt-science_center_objects=0#qt-science_center_objects.
  * Barnes [2016] Richard Barnes.  _RichDEM: Terrain Analysis Software_ , 2016.  URL http://github.com/r-barnes/richdem.