2021

[1,2]Martin Tømterud

[1]Bodil Holst

[1]Department of Physics and Technology, University of Bergen, Allégaten 55,
5007 Bergen, Norway

2]Donostia International Physics Center (DIPC), Paseo Manuel de Lardizabal, 4,
20018 Donostia-San Sebastián, Spain

3]Center for Nanotechnology Innovation@NEST, Istituto Italiano di Tecnologia,
Piazza San Silvestro 12, 56127 Pisa, Italy

4]Department of Physics and Astronomy, Clemson University, Clemson, South
Carolina 29634, USA

# Temperature Dependent Bending Rigidity of Graphene

<EMAIL_ADDRESS>Simen K. Hellner<EMAIL_ADDRESS>Sabrina D. Eder
<EMAIL_ADDRESS>Stiven Forti<EMAIL_ADDRESS>Joseph R. Manson
<EMAIL_ADDRESS>Camilla Coletti<EMAIL_ADDRESS><EMAIL_ADDRESS>*  [  [  [

###### Abstract

The mechanical properties of two-dimensional (2D) materials are important for
a wide range of applications including composite materials, van der Waals
materials and flexible electronics. Several aspects are highly debated in the
literature: For example, the theoretically predicted bending rigidity,
$\kappa(T)$, of the most important 2D material, graphene, varies from 0.8 eV
to 10 eV for predictions at 0 K and there are predictions that it could
increase or decrease with temperature depending on the applied theory. Here we
present the first systematic experimental study of the temperature dependence
$\kappa(T)$ of graphene. Extracting values of $\kappa$ from the ZA-phonon
dispersion relation measured with helium atom scattering, we find the value of
$\kappa(T)$ to increase with sample temperature. A linear fit in the
temperature range (120 K-480 K) gives a bending rigidity of $\kappa(T)=(1.1\pm
0.1)[\textrm{eV}]+(0.0013\pm 0.0001)[\textrm{eV/K}]\times T$.

##  1 Introduction

The class of atomically thin material sheets and membranes, commonly referred
to as 2D materials, are foreseen to take center stage in the development of
van der Waals heterostructures and flexible electronics components. For these
applications, the materials’ mechanical properties are of immense importance.
In the last few years, helium atom scattering (HAS) has been established as a
technique uniquely suited for evaluating the bending rigidity of 2D materials
Holst2013 .

The bending rigidity, $\kappa$, sometimes referred to as the bending stiffness
or the flexural rigidity, is an expression of the flexibility of the material.
Classically, the bending rigidity of an isotropic plate is given by RN1 :

$\displaystyle\kappa(h)=\frac{Yh^{3}}{12(1-\sigma)},$ (1)

where $Y$ is Young’s modulus, $h$ is the plate thickness, and
$\ignorespaces\sigma$ is Poisson’s ratio. The unit of  $\ignorespaces\kappa$
is [Pa m3]=[J], but is usually expressed in [eV] for nanomaterials. For
hexagonal crystals, such as graphene, the elastic properties in the (0001)
plane can be treated as isotropic RN1 . Both Young’s modulus and Poisson’s
ratio for 2D materials can be measured relatively simply by using an atomic
force microscope and poking with a well-defined force and measuring the
response RN2 . However, in order to determine $\kappa$ using equation (1), one
needs to know the thickness $h$ of the material. This is often difficult to
determine for 2D materials RN3 ; RN4 ; RN5 . Furthermore, it is not evident
that an atomically thin layer behaves in a classical manner, and the use of
Eq. (1) for determining $\kappa$ is not well justified.

Many experiments have been performed to investigate the mechanical properties
of 2D materials, using various techniques of nanoengineering RN2 . The bending
rigidity has however proven difficult to measure.

The experimental predictions for the bending rigidity of monolayer graphene
ranges from 1.2 eV to 2.7 eV RN6 ; RN12 ; RN17 , with an outlier finding a
bending rigidity of $\mathcal{O}(10^{3})$ eV RN7 . The very large experimental
range is mainly due to challenges with out-of-plane corrugation and strain.
For bilayer graphene, the experimental prediction is 2.6 eV - 5.8 eV,
depending on the bending angle RN17 . Theoretical predictions give values from
0.8 eV to 35.5 eV for monolayer graphene RN8 ; RN9 ; RN10 ; RN11 ; RN13 ; RN14
; RN15 ; RN16 ; RN18 ; RN19  and 2.6 eV to 180 eV for bilayer graphene RN13 ;
RN15 ; RN16 ; RN18 ; RN19 .

A couple of years back Amorim and Guinea showed how the bending rigidity
$\kappa$ for an isotropic, free-standing thin membrane which is weakly bound
to a substrate is related to the phonon dispersion of the ZA mode via RN20 ,
see also RN26 ; 2Dsilica_Erratum :

$\displaystyle\omega^{2}_{ZA}(\Delta K)=\frac{\kappa}{\rho_{2D}}\Delta
K^{4}+{\omega_{0}}^{2},$ (2)

where $\omega$ is the phonon frequency, $\rho_{2D}$ is 2D material density,
$\Delta K$ is the parallel component of the change in the wave vector, and
$\omega_{0}$ is the binding energy between the membrane and the substrate.
Note that it is the free-standing 2D material value for $\kappa$ that is
measured, as long as the membrane is only weakly bound to the substrate.
Strictly speaking, for an infinite extent thin film with fixed or periodic
boundary conditions, Eq. (2) contains a quadratic term in $\Delta K$, implying
a linear term in the dispersion relation. However, this term is typically
small compared to the binding energy and is therefore usually ignored RN26 ;
2Dsilica_Erratum .

Al Taleb et al. applied this theory in 2015 to measure the bending rigidity of
free-standing monolayer graphene using phonon dispersion curves from monolayer
graphene weakly bound on a copper substrate obtained through helium scattering
RN12 . They obtained a value for $\kappa$ that shows good agreement with
theory. An often cited value in the literature that agrees well with theory,
$\kappa=1.2$ eV, was derived from Raman spectroscopy experiments performed on
bulk graphite by Nicklow et al. NicklowPRB . To our best knowledge, the work
by Al Taleb et al. remains the most precise measurement of the bending
rigidity of monolayer graphene up till now. In 2018 the bending rigidity of 2D
silica was obtained using the same method RN26 . The first measurements of the
temperature dependent bending rigidity of a 2D material were published in
2021. Eder et al. applied the method of Al Taleb et al. to AB stacked bilayer
graphene weakly bound on intercalated silicon carbide, obtaining the bending
rigidity by measuring the phonon dispersion curves Eder2021 . The change in
bending rigidity of a 2D material with material density has also been explored
using HAS Tomterud2022 .

In this paper we present the first measurements of the temperature dependent
bending rigidity of “the mother of all 2D materials”; graphene.

##  2 Experimental

The measurements were performed on a quasi free-standing monolayer graphene
sample prepared at the Istituto Italiano di Tecnologia (IIT) on a N-doped
($10^{18}/cm^{3}$) 6H-SiC(0001) crystal from Crystal GmbH. The sample was
prepared using an established procedure at the institute: first the substrate
was etched in a 50 $\%$ Ar and H2 mixture with a total pressure of 450 mbar
(fluxes 500 sccm (standard cubic centimetre per minute) each) at 1200°C for 5
min to regularize the step-terrace morphology. The sample was then exposed to
a 50$\%$ Ar and H2 mixture with a total pressure of 750 mbar (fluxes 500 sccm
each) at 800°C for 20 min. This leads to atomic hydrogen intercalating at the
hetero-interface between SiC(0001) and the carbon rich surface reconstruction,
thereby lifting the covalent interaction so that the carbon atoms previously
in the reconstruction form a monolayer of graphene on top of the now
H-terminated SiC(0001) RN35 ; RN36 ; RN37 . The graphene is weakly bound to
the substrate: it can be peeled off the substrate in a manner similar to
graphene on Cu RN47 .

Figure 1: Left panel: Diffraction curves showing the center diffraction peak
and two first order maxima at about $\pm 2.9$ Å-1. Two satellite peaks are
visible close to the zeroth order maximum. Three diffraction curves are
plotted, arbitrarily shifted upwards for visual clarity. The three curves are
obtained with different source pressure, $p_{0}$, as indicated bu labelling
above the curves. Right panel: Close-up of the zeroth order peak of the three
diffraction curves. A Gaussian fit, used to determine the FWHM of the peak, is
shown for the diffraction curve taken at the highest pressure (shown in red).

We can obtain measurements of the bending rigidity of the free-standing
monlayer using the phonon dispersion curve model discussed above because the
sample is weakly bound. The sample temperature can be varied easily and
uniformly by changing the temperature of the underlying SiC substrate.

After preparation, the sample was placed in a sealed package, filled with
ambient air from the laboratory (humidity 50$\%$, temperature 22∘C) and
shipped to the University of Bergen (UiB). The sealed package was stored in
the laboratory for several months before the sample was mounted in the argon
vented sample chamber of the molecular beam apparatus known as MAGIE at UiB,
which was then pumped down. The neutral helium beam used for sample
investigation was created by a supersonic (free-jet) expansion from a source
reservoir through a $10\pm 0.5~{}\mu m$ diameter nozzle RN48 . The central
part of the beam was selected by a skimmer, $410\pm 2~{}\mu m$ in diameter,
placed $11.6\pm 0.5$ mm in front of the nozzle. The sample was characterized
in vacuum, by helium diffraction scans and time of flight (TOF) spectra using
a pseudorandom chopper RN49 . The background pressure during all measurements
was around $1\times 10^{9}$ mbar or less. The incident beam spot size on the
sample was around 4 mm in diameter for all measurements. Directly after
mounting the sample and pumping it down, no diffraction pattern was observed.
A bake-out was performed for 1 h at a sample temperature $T_{s}=675$ K
followed by a gradual cooldown to the desired substrate temperature in 25 K
steps with 60 s wait time in between. This improved the signal strongly
yielding a high intensity specular peak, with a measured full width at half
maximum of $(0.039\pm 0.00162)$ Å-1 (the instrument resolution), see Fig. 1.
This bake-out procedure was repeated between the measurements. The diffraction
measurements were taken using cooled beam, $T_{n}=125$ K, at a sample
temperature of 400 K. The TOF measurements were taken using a beam of $p_{0}$
= 71 bar with beam energy $E_{b}=(28.9\pm 0.2)$ meV. The sample temperature
$T_{s}$ was (320, 370 and 400 $\pm$ 0.2) K. An angular range of 25°– 65°was
investigated for each temperature, and in total 698 TOF spectra were obtained.
For more details see supplementary information.

##  3 Results and Discussion

Examples of TOF spectra can be found in Fig 2. The arrow annotations identify
ZA peaks in the spectra.

Figure 2: Examples of time of flight spectra on momolayer graphene. The right
panel shows the intensity as a function of the helium atom flight time. In the
left panel, time of flight has been converted to energy exchange with the
surface. The incident angle for each experiment is indicated by annotations in
the right panel. Arrows identify ZA peaks in the spectra in the left panel.
There are no ZA excitations in the lowermost spectrum, which was taken close
to specular incidence (45°). The elastic peak is dominant in this spectrum.

Identified ZA mode excitations in the HAS spectra are plotted as a function of
their parallel momentum exchange in Fig. 3. The solid curves show fits of the
dispersion relation presented in Eq. (2) to the ZA modes. The three panels
show the different sample temperatures, with the measurement series at $T=320$
K also depicting a measurement taken with the sample azimuth rotated $8$°, to
check that the sample behaved isotropic in the plane as expected.

Figure 3: ZA mode excitations of SiC supported monolayer graphene measured
with inelastic HAS. The modes are fitted with the dispersion relation in Eq.
(2), shown as full coloured lines. The different panels show measurement data
for different sample temperatures, as indicated. In the lowermost panel,
$T=320$ K, two measurement series are shown, one of which rotated $8$°out of
plane to investigate the isotropy of the ZA mode.

The ZA mode behaviour is seen to be in good agreement with the model in Eq.
(2). With mass density $\rho_{2D}=7.6\times 10^{-7}$ kg/m2 we obtain
temperature dependent measurements of the bending rigidity presented in Fig.
4. As can be seen, the bending rigidity increases with temperature. From a
weighted, linear fit of the measurement points we obtain a relation for
$\kappa(T)$, given by

$\displaystyle\begin{split}\kappa(T)&=(1.1\pm 0.1)[\textrm{eV}]\\\
&+(0.0013\pm 0.0001)[\textrm{eV/K}]\times T.\end{split}$ (3)

The errors are one standard deviation of the weighted fit. The fit is
performed on a narrow temperature range, and it is not clear that a linear
relation between $\kappa$ and sample temperature is correct for a larger
temperature range.

Figure 4: An overview of different results, obtained through both experiments
and computations, of the bending rigidity of monolayer graphene. The present
experiments are show as blue circular points, with a linear fit of $\kappa$ as
a function of sample temperature shown as a blue, solid line. The linear
extrapolation to $T=0$ K of the present experiments is in agreement with
previous published work of monolayer graphene on copper foil RN12 , as well as
MD simulations at 300 K Ramirez2016 .

In Fig. 4, we have included the HAS measurement of monolayer graphene on
supported on copper foil RN12 . This measurement was obtained at a sample
temperature of 125 K and fits well, within errorbars, of the linear
extrapolation of the measurements series obtained for our SiC-supported
graphene. The measurement is however attributed significantly larger error
bars than ours, and should therefore not be taken as an argument for the
linear fit to hold in general. The linear extrapolation also coincides with MD
simulations at 300 K Ramirez2016 . Also shown in the Fig. 4 are theoretical
results obtained through various methods and models at zero temperature. These
were summarised in Table 1 of Ref. 16, including density functional theory
(DFT) RN14 ; Sanchez-Portal-PRB ; Kudin-PRB , tight-binding DFT (DFTB) RN16 ;
Munoz-Diamond , and empirical potential calculations Tersoff-PRB ; Tu-PRB ;
Arroyo-PRB ; Lu-APD . At finite temperature, theoretical results include
elastic membrane calculations Liu-APL , calculations of the bending rigidity
of carbon nanotubes Yi-Aip , and the bending rigidity of graphene from MD
simulations RN18 ; Costamagna-PRB . Furthermore, a comparative study between
calculations and electron microscopy by Han et al. in good agreement with our
measurements, is included Han-Nature . Note that we assume these measurements
were carried out at room temperature. We could not find an explicit mentioning
of this in the paper. At zero temperature there is a significant disagreement
between the theoretical predictions of the bending rigidity. Such a
disagreement is also present in the finite temperature theoretical results,
but to a smaller magnitude. The most significant disagreement at finite
temperature is the debate on whether the bending rigidity increases or
decreases with increasing temperature. This is the second set of measurements
we present where the bending rigidity is found to increase, which is also the
case for the majority of literature results. The starting point of Ref. 41,
which disagrees, is a liquid membrane theory. One interesting implication
could be that for a glassy 2D material, the bending rigidity will decrease
when heated. This is a topic for future investigation. Another observation
from the theoretical results is that they all underestimate the value we
measure experimentally. A plausible explanation is as follows. With HAS we are
measuring the phonon dispersion out to about $0.8$ Å-1. We observe that the ZA
mode peaks in our spectra follow the dispersion relation in Eq. (2) up until
this cutoff. When computing free-standing graphene phonon dispersions with
molecular dynamics one obtains the entire spectrum, and the bending rigidity
is often taken to be the fitting limit of the ZA dispersion in the limit
$k\longrightarrow 0$, i.e. close to the zone centre. This part of the
dispersion relation is hidden by the coupling energy in the substrate-graphene
system and not probable with HAS unless one obtains a truly free-standing
graphene system with $\omega_{0}=0$. It is therefore possible that important
physics are hidden close to the $\Gamma$-point that will decrease the measured
value of $\kappa$. Conversely, as already stated, our measurements show that
the ZA mode follow the dispersion relation at least until $0.8$ Å-1 \- which
signals that obtaining the bending rigidity by only fitting close to the
$\Gamma$-point misses out on important physics further out in the zone. A
“simple” way to test this hypothesis experimentally is obtaining graphene
samples on different substrates and measuring the effect of the binding energy
on the bending rigidity and extrapolating the free-standing value.

The experimental results for the binding energy between the graphene monolayer
and the SiC-substrate are presented in Fig. 5. We do not observe a significant
change with temperature in the binding energy within the investigated
temperature range.

Figure 5: The extracted value of the binding energy from fitting Eq. (2) to
the experimental data in Figure 3. We observe no significant change in the
binding energy with increasing temperature.

##  4 Conclusion

We have presented temperature resolved measurements of the ZA mode of SiC-
supported monolayer graphene. From the phonon dispersion we are able to
extract temperature dependent values of the bending rigidity, which we show
are increasing with temperature. Future work should include measurements over
a larger temperature range to investigate the linear behaviour we observe,
measurements of different 2D materials, and particularly glassy ones, to
investigate if they behave in a different manner when heated, and measurements
of graphene on different substrates to investigate the influence of the
binding energy on the values we are able to extract for the bending rigidity.

Supplementary information Supporting information, including sample
characterization, will be made available upon publication.

Acknowledgments

  * •

Funding: B. H. acknowledges funding from the Norwegian Research Council, Grant
No.

  * •

Conflict of interest: The authors declare no conflicts of interest.

  * •

Authors’ contributions: S. F. and C. C. made the sample. B. H. designed the
experiments. S. K. H and S. D. E. performed the experiments. M. T. performed
the data analysis. J. R. M. provided theoretical support. M. T. and B. H.
wrote the paper with support from all authors.

## References

  * (1) B. Holst and G. Bracco,  Probing surfaces with thermal he atoms: Scattering and microscopy with a soft touch.  In Surface Science Techniques, (Springer, Berlin, 2013) pp. 333 – 365.
  * (2) L. D. Landau, L. P. Pitaevskii, A. M. Kosevich, and E. M. Lifshitz.  Theory of Elasticity.  Butterworth.Heinemann, 2012.
  * (3) D. Akinwande, C. J. Brennan, J. Scott Bunch, P. Egberts, J. R. Felts, H. Gao, R. Huang, J. Kim, T. Li, Y. Li, K. M. Liechti, N. Lu, H. S. Park, E. J. Reed, P. Wang, B. I. Yakobson, T. Zhang, Y.-W. Zhang, Y. Zhou, and Y. Zhu.  A review on mechanics and mechanical properties of 2d materials - graphene and beyond.  Extreme Mechanics Letters, 13 42–77, 2017.
  * (4) L. Wang, Q. Zheng, J. Z. Liu, and Q.Jiang.  Size dependence of the thin-shell model for carbon nanotubes.  Phys . Rev. Lett., 95:105501, 2005.
  * (5) P. Pine, Y. E. Yaish, and J. Adler.  Vibrational analysis of thermal oscillations of single-walled carbon nanotubes under axial strain.  Phys. Rev. B, 89:115405, 2014.
  * (6) Y. Huang, J. Wu, and K. C. Hwang.  Thickness of graphene and single-wall carbon nanotubes.  Phys. Rev. B, 74:245413, 2006.
  * (7) J. Zhao, Q. Deng, T. H. Ly, G. H. Han, G. Sandeep, and M. H. Rümmeli.  Two-dimensional membrane as elastic shell with proof on the folds revealed by three-dimensional atmic mapping.  Nature Communications, 6:8935, 2015.
  * (8) A. Al Taleb, H. K. Yu, G. Anemone, D. Farias, and A. M. Wodtke.  Helium diffraction and acoustic phonons of graphene grown on copper foil.  Carbon, 95:731–737, 2015.
  * (9) E. Han, Y. Jaehyung, E. Annevelink, J. Son, D. A. Kang, K. Watanabe, T. Taniguchi, E. Ertekin, P. U. Huang, and A. M. van der Zande.  Ultrasoft slip-mediated bending in few-layer graphene.  Nature Materials, 19:305–309, 2020.
  * (10) M. K. Blees, A. W. Barnard, P. A. Rose, S. P. Roberts, K. L. McGill, P. Y. Huang, A. R. Ruyack, J. W. Kevek, B. Kobrin, D. A. Muller, and P. L. McEuen.  Graphene kirigami.  Nature, 524:204–207, 2015.
  * (11) Y. K. Shen and H. A. Wu.  Interlayer shear effect on multilayer graphene subjected to bending.  Appl. Phys. Lett., 100:101909, 2012.
  * (12) N. Lindahl, D. Midtvedt, J. Svensson, O. A. Nerushev, N. Lindvall, A. Isacsson, and E. E. B. Campbell.  Determination of the bending rigidity of graphene via electrostatic actuation of buckled membranes.  Nano Lett., 12:3526–2531, 2012.
  * (13) K. V. Zakharchenko, M. I. Katsnelson, and A. Fasolino.  Finite temperature lattice properties of grpahene beyond the quasiharmonic approximation.  Phys. Rev. Lett., 102:046808, 2019.
  * (14) E. Ertekin, D. C. Chrzan, and M. S. Daw.  Topological description of the stone-wales defect formation energy in carbon nanotubes and graphene.  Phys. Rev. B, 79:155521, 2009.
  * (15) Y. Guo, J. Qiu, and W. Guo.  Mechanical and electronic coupling in few-layer graphene and hbn wrinkles: a first-principles study.  Nanotechnology, 27:505702, 2016.
  * (16) Y. Wei, B. Wang, J. Wu, R. Yang, and M. I. Dunn.  Bending rigidity and gaussian bending stiffness of single-layered graphene.  Nano Lett., 13:26–30, 2013.
  * (17) X. Chen, C. Yi, and C. Ke.  Bending stiffness and interlayer shear modulus of few-layer graphene.  Appl. Phys. Lett., 106:101907, 2015.
  * (18) P. Koskinen and O. O. Kit.  Approximate modelling of spherical membranes.  Phys. Rev. B, 82:235420, 2010.
  * (19) K. V. Zakharchenko, J. H. Los, M. I. Katsnelson, and A. Fasolino.  Atomistic simulations of structural and thermodynamic properties of bilayer graphene.  Phys. Rev. B, 81:235439, 2010.
  * (20) D. B. Zhang, E. Akatyeva, and T. Dumitrica.  Bending ultrathin graphene at the margins of continuum mechanics.  Phys . Rev. Lett., 106:255503, 2011.
  * (21) B. Amorim and F. Guinea.  Flexural mode of graphene on a substrate.  Phys. Rev. B, 88:115418, 2013.
  * (22) C. Büchner, S. D. Eder, T. Nesse, P. Kuhness, P. Schlexer, G. Pacchioni, J. R. Manson, M. Heyde, B. Holst, and H.-J. Freund.  Bending rigidity of 2D silica, Phys. Rev. Lett. 120, 226101 (2018).
  * (23) C. Büchner, S. D. Eder, T. Nesse, P. Kuhness, P. Schlexer, G. Pacchioni, J. R. Manson, M. Heyde, B. Holst, and H.-J. Freund.  Erratum: Bending rigidity of 2D silica, Phys. Rev. Lett. 127, 029902 (2021).
  * (24) S. D. Eder, S. K. Hellner, S. Forti, J. M. Nordbotten, J. R. Manson, C. Colleti, and B. Holst.  Temperature-Dependent Bending Rigidity of AB-Stacked Bilayer Graphene  Phys . Rev. Lett., 127, 266102, 2021.
  * (25) R. Nicklow, N. Wakabayashi, and H. G. Smith, Lattice Dynamics of Pyrolytic Graphite,  Phys. Rev. B, 5:4951, 1972.
  * (26) M. Tømterud, S. D. Eder, C. Büchner, M. Heyde, H.-J. Freund, J. R. Manson, and B. Holst,  Variation of bending rigidity with material density: Bilayer silica with nanoscale holes,  Phys. Chem. Chem. Phys, 24, 17941, 2022.
  * (27) C. Riedl, C. Coletti, T. Iwasaki, A. A. Zakharov, and U. Starke.  Quasi-free-standing epitaxial graphene on sic obtained by hydrogen intercalation.  Phys. Rev. Lett., 103:246804, 2009.
  * (28) S. Forti, K. V. Emtsev, C. Coletti, A. A. Zakharov, C. Riedl, and U. Starke.  Large-area homogeneous quasifree standing epitaxial graphene on SiC(0001): Electronic and structural characterization.  Phys. Rev. B, 84:125449, 2011.
  * (29) S. Forti.  Growing Graphene on Semiconductors, pages 141–80.  Jenny Stanford Publishing, 2017.
  * (30) K. Liu, P. Yan, J. Li, C. He, T. Ouyang, C. Zhang, C. Tang, and J. Zhong.  Effect of hydrogen passivation on the decoupling of graphene on SiC(0001) substrate: First principles calculations,  Sci. Rep. 7, 8461, 2017.
  * (31) S. D. Eder, B. Samelin, G. Bracco, K. Ansperger, and B. Holst.  A free jet (supersonic), molecular beam source with automatized, 50 nm precision nozzle-skimmer positioning.  Review of Scientific Instruments, 84:093309, 2013.
  * (32) D. D. Koleske and S. J. Sibener.  Generation of pseudorandom sequences for use in cross-corelation modulation.  Rev. Sci. Inst., 63:3852, 1992.
  * (33) R. Ramírez, E. Chacón, and C. P. Herrero,  Anharmonic effects in the optical and acoustic bending modes of graphene  Phys . Rev. B, 93, 236519, 2016.
  * (34) D. Sánchez-Portal, E. Artacho, J. M. Soler, A Rubio, and P. Ordejón  Ab initio structural, elastic, and vibrational properties of carbon nanotubes  Phys . Rev. B, 59, 12678, 1999.
  * (35) K. N. Kudin, G. E. Scuseria, and B. I. Yakobson,  C2F, BN, and C nanoshell elasticity from ab initio computations  Phys . Rev. B, 64, 235406, 2001.
  * (36) E. Muñoz, A. K. Singh M. A. Ribas, E. S. Penev, and B. I. Yakobson,  The ultimate diamond slab: GraphAne versus graphEne,  Diamond Relat. Mater., 16, 368, 2010.
  * (37) J. Tersoff,  Energies of fullerenes  Phys . Rev. B, 46, 15546, 1992.
  * (38) Z.-C. Tu and Z.-C. Ou-Yang,  Single-walled and multiwalled carbon nanotubes viewed as elastic tubes with the effective Young’s moduli dependent on layer number  Phys . Rev. B, 65, 233407, 2002.
  * (39) M. Arroyo and T. Belytschko  Finite crystal elasticity of carbon nanotubes based on the exponential Cauchy-Born rule  Phys . Rev. B, 69, 115415, 2004.
  * (40) Q. Lu, M. Arroyo, and R. Huang,  Elastic bending modulus of monolayer graphene  J. Phys. D: Appl. Phys., 42, 102002, 2009.
  * (41) P. Liu and Y. W. Zhang,  Temperature-dependent bending rigidity of graphene  Appl. Phys. Lett., 94, 231912, 2009.
  * (42) L. Yi,  Temperature dependence bending rigidity of 2D membranes: Graphene as an example,  AIP Advances, 8, 075104, 2018.
  * (43) S. Costamagna, M. Neek-Amal, J. H. Los, and F. M. Peeters  Thermal rippling behavior of graphane,  Phys. Rev. B, 86, 041408, 2012.
  * (44) E. Han, J. Yu, E. Annevelink, J. Son, D. A. Kang, K. Watanabe, T. Taniguchi, E. Ertekin, P. Y. Huang, and A. M. van der Zande  Ultrasoft slip-mediated bending in few-layer graphene  Nat. Mater., 19, 305 - 309, 2020.


11institutetext: Texas A&M University, College Station TX 77843, USA
11email: {yangyong<EMAIL_ADDRESS>11email<EMAIL_ADDRESS>
22institutetext: Visa Research, Palo Alto CA 94306, USA
22email<EMAIL_ADDRESS>
# Practical Speech Re-use Prevention in Voice-driven Services

Yangyong Zhang 11 Maliheh Shirvanian 22 Sunpreet S. Arora 22 Jianwei Huang 11
Guofei Gu 11

###### Abstract

Voice-driven services (VDS) are being used in a variety of applications
ranging from smart home control to payments using digital assistants. The
input to such services is often captured via an open voice channel, e.g.,
using a microphone, in an unsupervised setting. One of the key operational
security requirements in such setting is the freshness of the input speech. We
present AEOLUS, a security overlay that proactively embeds a dynamic acoustic
nonce at the time of user interaction, and detects the presence of the
embedded nonce in the recorded speech to ensure freshness. We demonstrate that
acoustic nonce can (i) be reliably embedded and retrieved, and (ii) be non-
disruptive (and even imperceptible) to a VDS user. Optimal parameters
(acoustic nonce’s operating frequency, amplitude, and bitrate) are determined
for (i) and (ii) from a practical perspective. Experimental results show that
AEOLUS yields 0.5% FRR at 0% FAR for speech re-use prevention upto a distance
of 4 meters in three real-world environments with different background noise
levels. We also conduct a user study with 120 participants, which shows that
the acoustic nonce does not degrade overall user experience for 94.16% of
speech samples, on average, in these environments. AEOLUS can therefore be
used in practice to prevent speech re-use and ensure the freshness of speech
input.

##  1 Introduction

Voice-driven services (VDS) are widely deployed in commercial products to
enable personalized and convenient experiences for consumers. Examples include
digital assistants from Amazon, Apple and Google for performing tasks like
smart home control and online commerce in a hands-free manner, and automated
and human-assisted voice response systems for customer support. While this
increasing ubiquity can be primarily attributed to improved real world
performance of deep learning driven speech and speaker recognition, security
is an equally important consideration in operational settings. To secure VDS
in practice, one of the common security mechanisms used is “voice password”.
For example, applications such as Samsung Bixby, Wechat and others [15, 16,
14] prompt users to speak a password and perform two factor authentication by
checking (i) if the spoken passphrase is correct, and (ii) it was spoken by
the authorized user.

Figure 1: Difference between re-use and replay attacks. In both cases, an
adversary records a user’s interaction with a voice driven service (VDS) “A”,
over-the-air as shown in (a). In case of re-use (b), the recorded interaction
is used to attack the same type of VDS A, while in case of replay (c), the
interaction can be used to attack the same or different type of VDS A’.

Voice passwords, however, do not provide security against voice spoofing
techniques such as speech replays [31, 40, 37] and synthesis [4]. While there
are known practical limitations in conducting speech synthesis attacks (e.g.,
the attack proposed in [4] requires 24 hours of high quality training data),
it is easier to conduct replay attacks in practice. Broadly, there are two
categories of defense mechanisms that aim to prevent speech replays. The first
class of methods use machine learning to determine whether a speech sample is
produced by a human or replayed using a playback device [24, 49]. The second
category of techniques use external devices (e.g., a vibration sensor) to
check if the speech is produced by a human in real-time [30]. For machine
learning techniques, real world performance usually depends on training data
and its relevance to the test environment. A state-of-the-art method Void
[20], as an example, has significantly different equal error rates (EER), 0.3%
and 11.6%, on two speech datasets collected in two different environments.
Another practical limitation is unreliability of playback device signatures
used by machine learning techniques [49]. In addition, for methods that use
extra hardware, there are usability and cost implications for practical
deployment.

In this work, we aim to address a subset of replay attacks, called speech re-
use attacks. We define speech re-use attacks to be specific to a certain type
of VDS. As shown in Figure 1, let A and A’ be the VDS types for the capture
and attack steps, respectively. Replay attacks typically consider a variety of
VDS types for capture and attack steps (A = A’ and A $\neq$ A’), while re-use
attacks are a subset of replay attacks where the VDS types in consideration
for the capture and attack steps is similar (A = A’). The proposed security
overlay prevents re-use attacks on protected VDS. For example, if the security
overlay is integrated with each Alexa’s VDS, it can prevent speech captured
from user interaction with one Alexa’s VDS from being re-used on another
Alexa’s VDS. This is similar to the device specific random pattern used in
Face ID to counter digital and physical spoofs [6] (see Figure 2).

We present AEOLUS111Named after Aelous, the ”Keeper of the Winds” in Greek
mythology., a security overlay that proactively embeds a dynamic acoustic
nonce in the voice channel via the microphone at the time of user interaction,
and detects the presence of the embedded nonce in the speech recorded by the
speaker to ensure speech freshness. AEOLUS is designed as a software solution,
and can be integrated by a vendor with any closed-loop VDS that involves real-
time user interaction. It is useful for providing additional security in
critical applications such as user authentication and payments. AEOLUS does
not require any extra hardware and works with in-built speaker and microphone
in consumer products. AEOLUS is secure by design against attacks that remove
the embedded nonce for speech re-use. It uses Frequency-hopping spread
spectrum (FHSS) [27] [28] technique for dynamic acoustic nonce embedding.
Similar techniques have been previously used to watermark audio recordings for
copyright purposes [45] [21]. However, their application in offline audio
watermarking is fundamentally different from VDS operational setting where the
length and content of the input speech is unknown apriori. AEOLUS is designed
for real-time over-the-air use without prior knowledge of the input speech.

AEOLUS addresses two key practical challenges. The first challenge is to
reliably embed an acoustic nonce over-the-air and retrieve it successfully
from the recorded speech. Over-the-air acoustic nonce propagation is affected
by factors such as background noise and distance between microphone and
loudspeaker [25]. The latter, for instance, results in distorted and
attenuated speech signal which increases the likelihood of errors while
extracting the embedded acoustic nonce. The second challenge is to embed the
acoustic nonce such that it is imperceptible to a VDS user. Achieving
imperceptibility is non-trivial in light of the first challenge. This is
because an important consideration for reliability is embedding acoustic nonce
of certain strength which, in turn, makes the acoustic nonce perceptible.
Therefore, we model acoustic nonce generation as an optimization problem and
compute the set of optimal parameters, e.g., nonce’s operating frequency,
amplitude, bitrate in different environments using differential evolution.

Figure 2: (a) Face ID’s device-specific random IR pattern projection at the
time of user authentication for preventing re-use [6]. Analogous to (a), the
proposed security overlay (b) uses a random acoustic nonce at the time of user
interaction to prevent speech re-use in voice-driven systems.

We evaluate AEOLUS in three different environments and show that it works
reliably upto a range of 4 m. Additionally, we conduct a user study involving
120 subjects (approved by the institutional human study review board) to
evaluate the imperceptibility of the embedded nonce. The results of the study
show that majority of users find the embedded acoustic nonce to be either
imperceptible or non-disruptive.

In summary, the contributions of this work are as follows:

  * •

Design of a VDS security overlay called AEOLUS to ensure freshness of input
speech without any extra hardware dependency.

  * •

Modeling acoustic nonce propagation as an optimization problem to address i)
reliability, i.e. ensuring successful embedding and retrieval of dynamic
acoustic nonce without impacting VDS functionality, and (ii) imperceptibility,
i.e. to have minimal impact on VDS users’ experience.

  * •

Comprehensive real-world evaluation to show that AEOLUS can work effectively
(0.5% false reject rate (FRR) and 0% false accept rate (FAR) in detecting
speech re-use) up to a range of 4 meters in three different environments.

  * •

User study with 120 subjects to demonstrate that the embedded acoustic nonce
does not degrade user experience.

##  2 Background and Related Work

###  2.1 Example Use Cases

Voice-driven Payments. Voice-driven services are being increasingly used for
payments, especially in retail environments [17]. Amazon and Exxon recently
announced voice-driven payments at over 11,500 gas stations in the US [3]. A
recent user study [12] indicates that while most users are comfortable
conducting low-value purchases such as ordering a meal and shopping for
groceries using voice-driven payment services, the majority of them do not
have sufficient confidence in conducting high value purchases due to security
concerns.

Most voice-driven payment services authenticate users through speaker
verification. However, they often lack adequate protection mechanisms against
speech re-use attacks where an attacker records a user interaction and re-uses
it to attack the service either on-site or remotely. This is analogous to card
skimming [41] where an attacker re-uses a victim’s credit card information.
AEOLUS can be deployed to prevent speech re-use attacks in voice-driven
payments.

Workplace Automation. A variety of digital assistants are being used for
unsupervised or semi-supervised interactions in workplace environments [1]. In
addition to accurate speaker recognition (identification and/or verification),
it is important to have adequate protection mechanisms against speech re-use
to protect sensitive resources in such environments [18]. AEOLUS can be
incorporated in workplace digital assistants for this purpose.

###  2.2 Related Work

Speech Replay Detection Existing research related to speech replay detection
can be classified into two categories, software-based and hardware-based
methods. Software-based methods use machine learning to determine whether the
input speech is produced by a human or replayed using a recording device [24]
[20] [34]. Chen et al. [24] present a method that obtains 0% EER for distance
(between user and VDS input device) of only a few centimeters which limits its
applicability in practice. Ahmed et al. [20]’s method, on the other hand, is
reported to work well upto a distance of 2.6 meters. However, the EER varies
significantly in different environments with it being as low as 11.6%. One of
the best performing methods [34] reports EER of 6.7% on ASVSpoof 17 database.
A caveat, however, is that these methods aim to address general replay
attacks. In contrast, AEOLUS aims to prevent speech re-use and obtains 0% EER
upto a distance of 4 meters in three different environments.

The second category of approaches use additional hardware (e.g., a vibration
sensor on user’s neck) to check that speech is produced in real-time by a user
[30] [40]. Such approaches, in general, outperform machine learning-based
methods. For example, Feng et al. [30] report 97% successful detection rate,
on average, with no strict assumption on distance. However, there are two
major practical limitations using the aforementioned approaches for detecting
speech re-use, (i) requirement of extra hardware which has cost implications,
and (ii) inconvenience to users. AEOLUS only uses the built-in speaker and
microphone in smart devices yet achieves high speech re-use detection rate.

Audio Watermarking. Audio watermarking is widely used for multimedia copyright
protection [45, 42]. Both time domain [21] and frequency domain [26, 38]
watermarking techniques have been proposed in the literature. Typically, audio
watermark embedding and detection is self-synchronized [51]. Advanced
watermarking methods use spread-spectrum modulation [45] to prevent watermark
removal and provide imperceptibility [26, 35]. While useful, these techniques
cannot be directly used in VDS operational setting. They are designed for
fixed length offline audio files, e.g., music recordings in CDs [45], and do
not consider the impact of environmental factors, e.g., bitrate variability
and background noise, in over-the-air transmission. Such environmental factors
result in a dynamic and lossy acoustic environment in turn making watermark
embedding and retrieval significantly challenging. Further, they are not
designed for real-time speech and speaker recognition systems where input
speech is unknown apriori and can be of variable length. AEOLUS is designed to
take these pragmatic considerations into account. It is able to successfully
embed and retrieve acoustic nonce in the voice channel without degrading user
experience in practice.

##  3 Proposed Security Overlay

(a)

(b)

Figure 3: Use of the proposed security overlay to prevent speech re-use in
voice-driven services. (a) At the time of user interaction, an acoustic nonce
is generated and embedded in over-the-air channel, and its presence is
detected in the recorded speech to ensure speech freshness. (b) AEOLUS-enabled
VDS components with the associated data flows. Shown in blue and grey are the
components used to prevent speech re-use.

###  3.1 Threat Model

The following assumptions are made while designing AEOLUS to address the
speech re-use attack scenario. The hardware device (loudspeaker and
microphone) that is used by a user to interact with VDS is trusted and secure,
and an adversary cannot disable the speaker or the microphone. The proposed
overlay can be used by a VDS to generate, embed, and detect acoustic nonce. At
the time of user interaction, the method used to trigger a VDS, e.g., words or
phrases like “Alexa” and “Hey, Siri” are not protected but can initiate the
proposed overlay to prevent re-use of voice commands and confidential
passphrases. For example, to prevent re-use of a user’s voice password to
access recent account transactions when the user is interacting with a voice
banking VDS [11].

An adversary can use a commodity recording device to record prior user
interactions with “any” AEOLUS-enabled VDS, and re-use the recorded
interaction to attack any AEOLUS-enabled VDS. Attack scenarios where an
adversary can record or access clean user speech samples (i.e., recordings
without embedded acoustic nonce) to launch replay or speech synthesis attacks
are considered out of scope and are not addressed. The key underlying
assumptions are that (i) users typically interact with VDS using specific
commands (e.g., a user-defined passphrase [15]) and these commands can be
protected with the proposed overlay, and (ii) it is difficult to record clean
samples of these commands from interaction with unprotected devices or in
human conversations. However, this does not limit the use of AEOLUS in
conjunction with other defense mechanisms for replay or speech synthesis
attacks.

###  3.2 Core Components

To prevent speech re-use in VDS, AEOLUS uses three core components, nonce
generation, nonce embedding and nonce detection.

Nonce Generation. The nonce generation module is invoked upon initiation of a
new user interaction session (see Figure 3(b) ①). Akin to nonce-based
authentication [47, 22], the module generates a digital nonce (a random number
of a length $L$) that is valid for the current session. Nonce length $L$
should be sufficiently large to prevent use of duplicate nonces in different
sessions. Given limited bandwidth of over-the-air channel and estimated user
base for popular VDS (e.g., Alexa skill has 300,000 daily users [2]), the
minimum permitted value of $L$ is set to 32 bits. 32-bit long nonce is
sufficient to protect short speech samples (e.g., a voice passphrase) in a
particular session. Nonce repetition is permitted to protect longer speech
samples during the session. The nonce $\delta$ for session $S$ is stored in a
set $S_{set}$. $S_{set}$ is subsequently used by the nonce detection module to
determine if the embedded nonce is current.

Nonce Embedding. Post nonce generation, an encoding algorithm, Binary
Frequency Shift Keying (BFSK) [39, 46], is used to encode the generated
digital nonce as an acoustic nonce. Each bit is independently encoded as a
cosine wave $\omega_{i}$. Bit “0” is encoded as a 1kHz cosine wave, and bit
“1” as a 5 kHz cosine wave. All cosine waves in the set $\omega$ are
concatenated to yield the acoustic nonce $\delta$. The acoustic nonce $\delta$
is embedded in over-the-air channel using VDS playback device ( Figure 3(b)
②). An important parameter for nonce embedding is the time duration of the
embedded acoustic nonce relative to the time duration of user interaction.
Using a real-world dataset with 27,000 samples [32], we estimate that the
average user interaction is 3 sec. As an example, let the minimum duration of
embedded nonce be half of the average user interaction. In this case, the
acoustic nonce embedding would require a bitrate of 14 bits/sec, and the
maximum duration of each component cosine wave $w_{i}$ can be 42 ms. Note that
this estimate does not consider the idle time in user interaction.
Additionally, to prevent an adversary from obtaining clean speech sample by
removing the embedded nonce, the nonce embedding module leverages Frequency-
hopping Spread Spectrum (FHSS) [28, 27]. FHSS is a well-known technique that
provides high robustness against standard embedded signal removal attacks. For
high robustness, the module uses FHSS with a set of different frequencies and
periodically selects the operating frequency at random.

Nonce Detection. Once user interaction ends, the recorded audio is processed
using the nonce detection module (Figure 3(b) ③). The module decodes the
acoustic nonce using BFSK, and checks that (i) the nonce is completely
decoded, and (ii) the decoded nonce is current. The recorded speech is
subsequently processed using standard VDS modules (Figure 3(b) ④). If nonce
detection fails or if the decoded nonce has a high Bit Error Rate (BER), a
recourse action (e.g. a retry or reject) can be taken. Furthermore, if speech
is reused by an adversary and both the current nonce and the obsolete nonce
from re-used speech are detected, information about re-used speech such as
when and where it was recorded can be determined from $S_{set}$. Another
scenario can be signal interference between the current and obsolete nonce
resulting in inappropriate decoding result. In such case, pertinent recourse
action can be taken.

##  4 Practical Realization

###  4.1 Challenges

In this section, we discuss the two key challenges while implementing AEOLUS
in practice: reliability and imperceptibility.

Reliability. The first challenge is to reliably embed an acoustic nonce in
over-the-air channel, as well as to accurately detect nonce from the recorded
speech. Because over-the-air acoustic nonce transmission is impacted by
factors such as background noise and distance between microphone and playback
device, it is difficult to achieve this in operational settings. It is also
important to determine if any previously embedded acoustic nonce is present in
the recorded speech to ascertain speech re-use. The metric used to measure
reliability is the bit error rate (BER) between the embedded and detected
nonce. Recall that BFSK is used for acoustic nonce embedding and detection.
Under the assumption that only additive white Gaussian noise (AWGN) is present
in over-the-air channel, BER can be computed using the complimentary error
function $erfc()$ [36] as follows:

$BER=\frac{1}{2}erfc(\sqrt{E_{b}/N_{0}})$ (1)

Note that BER computation in Eqn. 1 involves the normalized per bit signal-to-
noise ratio ${E_{b}/N_{0}}$ [33] which depends on the frequency $f$ and
amplitude $\alpha$ of the carrier wave. However, other types of noises besides
AWGN are typically present in over-the-air channel. Hence, we conduct Room
Impulse Response (RIR) simulation experiments to study the impact of carrier
wave amplitude and frequency on BER.

Imperceptibility. The second challenge is to ensure that the embedded acoustic
nonce does not degrade VDS user experience. For this, the acoustic nonce
should be as imperceptible as possible to a VDS user. The nonce embedding and
generation modules presented earlier do not adequately address this challenge
because (i) they do not optimize any objective metric for imperceptibility,
and (ii) they do not account for dynamic and noisy environments where it is
difficult to achieve both reliability and imperceptibility simultaneously. For
measuring imperceptibility, Sound Pressure Level (SPL) is computed using the
following equation:

$SPL=2{\pi}^{2}{f}^{2}{\alpha}^{2}\rho{c}/{v}$ (2)

Here, $f$ represents the frequency and $\alpha$ is the amplitude of the
carrier wave. $\rho$ represents the density of the transmission channel (e.g.,
over-the-air channel), and $c$ denotes the speed of acoustic transmission.
Given that the average SPL for human speech is 60dBm [13], $f$ and $\alpha$
should ideally ensure that SPL is below this threshold for imperceptibility.
Like Equation 1, while this equation provides a theoretical basis to
understand SPL, we study how the aforementioned parameters impact SPL using
RIR simulation.

###  4.2 Key Parameters

There are two key parameters that impact reliability and imperceptibility: (i)
acoustic nonce generation and embedding parameters that include frequency and
amplitude of the carrier wave used for acoustic nonce generation and bitrate
used for acoustic nonce encoding, and (ii) environmental parameters that
include the distance between the microphone and user, the room size, and
background noise among others. To understand the impact of the these
parameters on reliability and imperceptibility, we implement AEOLUS as a
prototype and setup an RIR environment called MCRoomSim in MATLAB [48]. RIR is
a commonly used tool for estimating the performance of signal processing
applications in over-the-air channel [25, 19].

(a) Frequency-range=4-5kHz, hopping = 2, bitrate = 30 bits/sec

(b) Frequency-range=4-5kHz, hopping=2, amplitude = 1

(c) Freq-range=4-5kHz, hopping=2, bitrate=30 bits/sec, amplitude=1

Figure 4: Variation of bit error rate (%) with: (a) amplitude of carrier wave,
(b) bitrate used for encoding acoustic nonce, (c) background noise levels when
embedding the acoustic nonce. $E_{b}/N_{0}$ is calculated in decibels. The
distance between user (simulated using a speaker) and the microphone is set to
1 m.

####  4.2.1 Nonce Generation and Embedding Parameters

Amplitude. According to RIR simulation, a lower amplitude yields lower
$E_{b}/N_{0}$ ratio and consequently, higher BER (see Figure 4(a)). This is
consistent with Equation 1. Furthermore, as shown in Figure 5, SPL is
proportional to the amplitude of the carrier wave.

Frequency. BER and SPL are evaluated in the frequency range at which typical
speaker recognition systems operate (5 Hz - 8 kHz; sampling rate = 500 Hz).
The other parameters are fixed as follows: ${E_{b}/N_{0}}$ = 1, frequency
hopping = 2, amplitude (normalized) = 1, bitrate = 30 bits/sec, distance = 1
meter. It is observed that the carrier wave frequency neither impacts BER nor
SPL. The later is in contrast to existing research [44, 5] which shows that
the frequency of an acoustic wave affects the loudness as perceived by humans
in a non-linear manner. Hence, the frequency to SPL mapping from IoSR toolkit
[7] is used as reference.

Bitrate. Figure 4(b) shows the impact of acoustic nonce bitrate on BER. BER
increases significantly as bitrate increases beyond 20 bits/sec. One of the
causes is asynchronous signal overlapping in the time domain when two bits are
transmitted too close to each other.

####  4.2.2 Environmental Parameters

(a) $E_{b}/N_{0}$ = 1, carrier frequency range = 4-5kHz, hopping=2, bitrate =
30 bits/sec, distance = 1m

(b) $E_{b}/N_{0}$ = 1, carrier frequency range = 4-5kHz, hopping=2, bitrate =
30 bits/sec, Amplitude (normalized) = 1

Figure 5: Variation of sound pressure level (dBm) with (a) amplitude of
carrier wave, and (b) distance between user (simulated using a speaker) and
the microphone.

Frequency Shifting. Over-the-air acoustic nonce transmission induces
differences in the received frequencies and the transmitted frequencies called
frequency shifting. Frequency shifting can also occur because of hardware
limitations, e.g., a microphone with a 1.8 kHz frequency limit being used to
capture a 2 kHz wave. Since the received frequency is used to decode the
acoustic nonce, frequency shifting needs to be accounted for by AEOLUS. We
investigate frequency shifting in the simulation setting (operating frequency
range: 500 Hz-8 kHz, cardioid loudspeaker type, dipole microphone type) and
determine the frequency shifting range to be +/- 15 Hz.

Distance. Distance between a VDS user and VDS input impacts imperceptibility
of the acoustic nonce. Figure 5(b) indicates that this distance is inversely
proportional to SPL.

Background Noise. Figure 4(c) and Equation 1 suggest that the background noise
is proportional to BER. In an environment with high background noise, the
nonce generation and embedding parameters need to be configured appropriately
(e.g., decrease bitrate, increase amplitude) to limit the BER.

###  4.3 Mathematical Formulation

To address reliability and imperceptibility simultaneously, we model over-the-
air acoustic nonce transmission as an optimization problem. Let a user’s
recited speech be denoted by $x$, and the acoustic nonce embedded by AEOLUS be
$\delta$. Also, let the function that accounts for variations induced by over-
the-air channel, e.g., due to acoustic signal air absorption and distance, be
$h(\cdot)$. The speech recorded by the microphone is thus represented as
$h(x)$. When $\delta$ is embedded at the time of user interaction with the
VDS, $x$ mixes with $\delta$. As a result, the microphone records
$h(\textbf{x}*\delta)$, where $*$ denotes signal convolution between $x$ and
$\delta$.

Assume that AEOLUS detects nonce $\delta{{}^{\prime}}$ in $h(x*\delta)$. Let
us define an objective function $f(\cdot)$ denoting BER that takes the
parameter set $\mathcal{P}=\\{<f,\alpha,\tau>,...\\}$ as input, and a function
$g(\cdot)$ denoting SPL. The goal is to find the optimal parameter set
$\mathcal{P}$ that minimizes $f$ (for reliability) subject to the constraint
that the SPL difference between speech with and without embedded nonce is less
than or equal to a threshold $\theta$ (for imperceptibility):

$\arg\min_{\mathcal{P}}f(\mathcal{P})\\\ \text{ subject to
}g(h(\textbf{x}*\delta))-g(h(\textbf{x}))\leq\theta$ (3)

initialization:

1\. Create candidate population:$\mathcal{P}=\\{P<f,\alpha,\tau>,...\\}$

2\. Initialize: $\mathcal{P}_{output}=\\{\\}$, $i=0$, $MAX_{i}=100$,
$MAX_{output}=20$, $\beta=0.5$

2\. Acquire environmental parameters: $d$, $N_{0}$

3\. Set up Goals:$BER<10^{-9}$, $SPL^{{}^{\prime}}-SPL\leq\theta$

optimization:

while  _$i <=MAX_{i}$, $\mathcal{P}_{output}.Size<=MAX_{output}$ _ do

foreach  _individual $P_{i}(i=1,...)\in\mathcal{P}$_ do

if  _$P_{i}$ satisfies the Goals_ then

$\mathcal{P}_{output}\leftarrow P_{i}$

end if

Create candidate $C$ from parent $P_{i}$, $d$, $N_{0}$.

if  _the candidate is better than the parent_ then

$P_{feasible}\leftarrow$ Candidate $C$

Candidate $C$ replaces the parent

else

the candidate is discarded

end if

end foreach

Randomly select next $P$ in $\mathcal{P}$.

end while

output: $\mathcal{P}_{output}$ Candidate Creation: input : Parent $P_{par}$,
$d$, $N_{0}$, $\beta$

Randomly select individuals $P_{1}$ , $P_{2}$ from $\mathcal{P}$

Calculate candidate $C$ as $C=P_{par}+\beta(P_{1}-P_{2})$,

where $\beta$ is a scaling parameter decided by $d$ and $N_{0}$.

output : Candidate $C$

Algorithm 1 Computing Optimal Parameters

###  4.4 Computing Optimal Parameters

Initialization. The environmental parameters are first initialized (Figure
3(b) ⑤) using the VDS input. The background noise energy is calculated by
inspecting the ambient sound. Frequency shifting is calculated by transmitting
and decoding a nonce that covers the working frequency range (500 - 8 kHz,
sampled every 500 Hz). It is assumed that the distance between the user and
VDS input device is set by the VDS device vendor.

Optimization. Computing the gradients of the optimization parameters directly
using gradient-based methods is inefficient [23]. This is because the VDS’s
acoustic environment is dynamic (e.g., due to occlusions and varying
background noise levels). In addition, the formulated optimization is a multi-
dimensional problem with considerations of both reliability and
imperceptibility. Hence, we use differential evolution (DE) [29, 43] which is
a gradient-free method. DE generates diverse candidate solutions in each
iteration which avoids local minima and aids in convergence to a global
minima. It does not require apriori knowledge of the underlying acoustic
encoding method or acoustic environment. Algorithm 1 summarizes the use of DE
for obtaining optimal parameters. The initial set of candidate parameters
$\mathcal{P}$ includes feasible frequency $f$, amplitude $\alpha$, and bitrate
$\tau$, estimated using RIR simulation. Once AEOLUS is deployed, the
initialization module first estimates the environmental parameters, e.g.,
$N_{0}$ and $d$. Subsequently, the optimization module uses the pre-calculated
RIR parameters for efficient convergence to optimal parameters (see Section
5.1).

##  5 Experimental Evaluation

(a) Conference Room.

(b) Dining Hall

(c) Gas Station (Car).

(d) Setup in the Car.

Figure 6: Experimental setup to evaluate AEOLUS in different environments.

AEOLUS is implemented as a prototype in MATLAB and evaluated in three
different environments: (i) a conference room (approx. 5.5 m by 4 m) selected
as a reference for a semi-public indoor environment, (ii) a campus dining area
(approx. 25m by 50m) chosen to mimic a noisy indoor public area, and (iii)
inside a car parked at a gas station to simulate the use case of voice-driven
payments at gas stations [3]. For (i) and (ii), a FIFINE K669B microphone is
used to record speech, and a Logitech S120 Speaker System is used as playback
device to simulate users. The evaluation is performed at different distances
(0.5-4 m) between the microphone and playback device. For (iii), the
microphone is identical but the car’s in-built speakers are used instead of
the Logitech speakers. Also, the evaluation is performed at two different
distances 0.5 and 0.7 m.

AEOLUS is evaluated both on speech data captured from live participants in
ReMASC dataset [32] and synthesized speech from Amazon Polly. A total of ten
different speaker profiles are used (four of which are synthesized). For each
profile, three different speech samples are used. Microsoft Azure’s speech and
speaker recognition services [9] are used for speech-to-text and speaker
recognition. Text-independent speaker verification service [8] is used for
speaker verification experiments. Each speaker’s profile is first enrolled
using a speech sample longer than 20 seconds. Verification is performed using
the speaker’s speech samples with and without acoustic nonce, and yields one
of the following outputs: acceptance with “high”, “normal”, or “low”
confidence, or a rejection.

###  5.1 Performance

Table 1: Optimal parameters for achieving reliability and imperceptibility at a distance of 0.5 m between the microphone and user in different environments: (a) conference room, (b) dining hall, and (c) in a car parked at a gas station. Location |  SPL (dB-A) |  Frequency ($\pm$ 200 Hz) |  Amplitude (normalized) |  Bitrate (bits/sec)
---|---|---|---|---
Conference Room | 41 | 4000 Hz | 0.52 | 35
Dining Hall | 58 | 5200 Hz | 1 | 25
Gas Station (Car) | 47 | 4800 Hz | 0.61 | 33

Optimal Parameters. First, we estimate optimal parameters for achieving
reliability and imperceptibility simultaneously in a given environment using
Algorithm 1. The frequency step size is set to 200 Hz and $\theta$ is set as
0.1 to ensure that the generated nonce is below the average human speech SPL
of 60 db. It is observed that lower amplitudes and frequencies and higher
bitrates are comparatively optimal for acoustic nonce embedding in low noise
environments, e.g., conference room (see Table 1). Relatively higher
amplitudes and frequencies, and lower bitrates are more optimal in
environments with higher noise, such as the dining hall. This is because the
background noise in such environments typically vests as a low amplitude low
frequency noise.

Computational Overhead. Algorithm 1 iteratively optimizes the parameter set
$\mathcal{P}$ for the given environment. For conference room, a single
iteration is sufficient. However, for dining hall and gas station
environments, a few iterations ($<10$) are required and the computation takes
a few seconds. This does not cause a delay at the time of user interaction
because it overlaps with the time a user typically takes to set up a VDS. The
acoustic nonce embedding and decoding process occurs while user is interacting
with VDS. The computational overhead is negligible because the nonce size is
small, and the playback device is used only when the user is speaking.

Reliability Next, we measure the reliability of embedding and retrieving
acoustic nonce in the three environments. For this, we compute the average BER
over 5 independent trials at different distances between the VDS input device
(microphone) and user. Figure 7 shows that in all three environments, the
average BER upto a distance of 1.5 m is 0%. Note, however that BER increases
significantly beyond 1.5m in dining hall because the environment is relatively
noisy.

Table 2: Acoustic nonce recovery bit error rate (BER) and speech re-use detection performance (FAR and FRR) at different distances in the three environments.  Location | Dis. | BER/FAR/FRR | Location | Dis. | BER/FAR/FRR | Location | Dis. | BER/FAR/FRR
---|---|---|---|---|---|---|---|---
Conference Room | 0.5m | 0%/0%/0% |  Dining Hall | 0.5m | 0.2%/0%/0.5% |  Gas Station (Car) | 0.5m | 0%/0%/0%
| 1m | 0%/0%/0% |  | 1m | 1.2%/0%/ 1% |  | 0.7m | 0.5%0%0.5%
| 4m | 1.2%/0%/0.5% |  | 2m | 75%/0% /78% |  |  |

Speech Re-use Detection. In this experiment, we measure the efficacy of AEOLUS
in detecting speech re-use (see Table 2). The input speech is considered re-
used if (i) the current nonce is not decoded correctly, and (ii) if any trace
of an obsolete nonce is detected. FRR indicates the proportion of falsely
rejected samples because of incorrect nonce decoding or because the detected
nonce is obsolete. FAR indicates how many re-used speech samples with obsolete
nonces are falsely accepted. AEOLUS achieves 0.5% FRR at 0% FAR for speech re-
use prevention upto a distance of 4 meters in the three environments.

###  5.2 Robustness

Impact on Speaker and Speech Recognition. The goal of this experiment is to
assess the impact of acoustic nonce embedding and decoding on the performance
of a commercial speaker and speech recognition system (Microsoft Azure). The
results show that there is limited impact on the performance of the speaker
recognition system for the conference room and dining hall environments upto a
distance of 1.5 m (see Appendix Figure 8(a)). The embedded nonce marginally
increases the average word error rate (WER) of the speech recognition system
to 4.4%. The WER is close to the reported average WER of 5.1% for the same
system in the NIST 2000 Switchboard test set [52]. Manually reviewing the
results indicates that most errors are related to incorrect suffix “-ing”. A
few errors are due to small word omissions such as “an” and “a”. Relatively
more errors occur in the dining hall environment due to the background noise.

Table 3: Performance of Microsoft Azure speaker recognition system (similarity score 0 - 1) and speech recognition system (WER) after nonce removal using two different FHSS schemes. Without nonce removal, the respective average similarity score and WER of the two systems are 0.897 and 5.1% [52]. Lossy compression and equalization nonce removal techniques failed to remove the nonce and the speech samples were rejected by AEOLUS. Removal Technique |  4-5kHz, 2 Hop., Avg. of 30 samples |  1-8kHz, 10 Hop., Avg. of 30 samples
---|---|---
Speaker Verification (Score, 0-1) |  Speech Recognition (WER %) |  Speaker Verification (Score, 0-1) |  Speech Recognition (WER %)
Resampling | 0.233 | 29.73% | 0.170 | 91.83%
Amplitude Compression | 0.217 | 34.31% | 0.184 | 63.4%
Filtering | 0.623 | 17.12% | 0.319 | 62.21%
Additive Noise | 0.318 | 53.13% | 0.192 | 96.34%
Lossy Compression | Reject | Reject | Reject | Reject
Equalization | Reject | Reject | Reject | Reject

Acoustic Nonce Removal. An adversary may attempt to remove the embedded
acoustic nonce and obtain a “clean” speech sample to conduct re-use attacks.
To test the robustness of AEOLUS in this adversarial setting, we use 6 common
audio watermark removal techniques [51] (see Appendix 0.B for details). Each
technique is iteratively used until the acoustic nonce cannot be detected in a
speech sample. The removal attempts are repeated for all speech samples in the
dataset mentioned above. Table 3 shows that nonce removal techniques reduced
the average recognition performance of the commercial speaker recognition
system by 79.52%, and increased the average word error rate of the speech
recognition system by 33.57%.

###  5.3 Human Perception

####  5.3.1 Empirical Measurement.

In this experiment, we empirically measure the change in SPL due to the
embedded nonce. The distance between VDS output and the recording device is 1
m in dining hall and conference room, and 0.7 m in the car. The results (see
Appendix Figure 9) show that the average SPL difference in each environment is
less than 1.4%. Non-parametric Wilcoxon Signed Rank test [50] (at 95%
confidence level) is used to measure the statistical significance of the
difference in SPL before and after acoustic nonce embedding. The results
indicate that the differences are statistically insignificant.

Table 4: Perception of speech samples. Overall (%) is the proportion of participants that label speech samples as imperceptible, non-disruptive, or disruptive. Sample (%) is the proportion of speech samples (with nonce) that are determined to be imperceptible, non-disruptive, or disruptive by the majority of participants. Location |  Imperceptible (Overall/Sample%) |  Non-disruptive (Overall/Sample%) |  Disruptive, Nonce-caused (Overall/Sample%)
---|---|---|---
Conference Room | 34.87%/50% | 50%/50% | 1.31%/0%
Dining Hall | 32.64%/25% | 41.67%/75% | 7.64%/0%
Gas Station (Car) | 32.14%/25% | 42.86%/75% | 8.57%/0%

User Study. Next, we study the impact of AEOLUS on VDS usability. For this,
120 participants (with prior experience using a VDS) from various demographics
(age groups, gender shown in Appendix Table 6) are recruited on Amazon
Mechanical Turk. Each participant is presented with a survey that takes
approximately 10 minutes to complete. Post completion of the survey, a
participant is paid 4 USD as incentive. The survey questions are designed to
ensure bias is not introduced in the results (no leading questions).

Each participant is asked to enter their demographic information. The
participant is then instructed to play a test sample and adjust the playback
volume to a comfortable level. Following this, the participant is presented
with a survey with five speech samples and follow up questions. Included in
the five samples is one sample with only background noise (and no speech) to
check if the participant is attentive. The participants are asked to provide
speech transcriptions for all samples and the provided transcriptions are used
to check the participants’ attentiveness. Eleven participants failed to pass
this test and hence were excluded.

The test samples are uniformly selected at random from a dataset of 24 speech
recordings, 12 recordings each with and without embedded nonce from the three
environments. The average length of the speech samples is 6.4 seconds. To
ensure the user study is unbiased, all samples presented to a participant are
from the same speaker and pertain to the same voice command. After listening
to a speech sample, a participant is asked to rate the noise level of the
speech sample on a scale of 0 - 10 (10 being the highest noise level). The
participant is also instructed to report any noticeable noise source that
affects their experience, and to specify the characteristic of the noise that
resulted in this disruption (e.g., human conversation, train or car noise).
The answer is used to ascertain the source of noise that impacted the
usability (e.g., the embedded nonce).

Results. Table 4 reports the average human perception ratings. The acoustic
nonce does not degrade overall user experience for 94.16% of speech samples,
on average, in the three environments. Since this does not adequately measure
the participant’s perception of nonce embedding for each sample, per speech
sample perception is also reported. No speech sample with embedded nonce is
perceived to be disruptive by the majority of participants.

Table 5: Average perceived noise (on a scale of 0-10) in speech samples with and without acoustic nonce collected in the three environments. Location |  w/o Nonce |  w/ Nonce |  t-test, $\alpha=0.05$ | Statistically Significant
---|---|---|---|---
Conference Room | 3.8 | 4.8 | $t=4.2022,p=0.0001489$ | Yes
Dining Hall | 5.7 | 6.1 | $t=1.4595,p=0.1524$ | No
Gas Station (Car) | 5 | 5.6 | $t=3.0697,p=0.00389$ | Yes

Perceived Noise Level. Table 5 shows that the speech samples containing the
acoustic nonce are perceived to have a higher noise level. The dependent
samples paired t-test (at 95% confidence) indicates a statistically
significant difference (between samples with and without nonce) in the
conference room and car. However, this difference is statistically
insignificant in the dining hall due to higher background noise level compared
to the conference room and car. Despite perceiving higher noise level, the
majority of participants do not perceive any speech sample to be disruptive
(see Table 4).

##  6 Summary and Future Work

In this work, we present a security overlay called AEOLUS that can be
integrated with any voice-driven service to prevent speech re-use. AEOLUS
reliably embeds a dynamic acoustic nonce that is non-disruptive to a VDS user,
and detects the embedded nonce from the recorded user speech. AEOLUS can be
combined with other defense mechanisms to further enhance VDS security. As an
example, it can be used alongside a dynamic challenge-response method where in
each interaction, the user is expected to respond to a fresh challenge (e.g.,
repeat a phrase). The acoustic nonce can be embedded in each challenge, and
can be detected using AEOLUS. Experiments conducted in three different
environments show that AEOLUS can be used in operational VDS scenarios with
minimal overhead. In future, we plan to conduct in-depth investigation of
methods for generation and encoding of acoustic nonce, e.g., using variations
of music or other forms of sound along with different encoding schemes. We
also plan to conduct large-scale live subject testing to further evaluate
reliability and imperceptibility.

## References

  * [1] Alexa for Business.  https://aws.amazon.com/alexaforbusiness/.
  * [2] Alexa skill: 300,000 daily users.  https://vux.world/google-assistant-500m-active-users-popular-alexa-skill-300000-daily-users/.
  * [3] Amazon Alexa Pay in Gas Station.  https://bit.ly/37v25xN.
  * [4] Black Hat 2018: Voice Authentication is Broken.  https://bit.ly/3mxQIto.
  * [5] Equal-loudness contour.  https://bit.ly/2LF1ukH.
  * [6] Face ID Security.  https://apple.co/34mmO59.
  * [7] IoSR Matlab Toolbox: Equal-loudness Contour Mapping.  https://bit.ly/2Wtd1Wz.
  * [8] Microsoft Azure Speaker Recognition - Verification.  https://bit.ly/3h82Sbh.
  * [9] Microsoft Cognitive Services.  https://bit.ly/38haWCH.
  * [10] MP3 Wikipedia.  https://en.wikipedia.org/wiki/MP3.
  * [11] Paypal Voice Assistant Application with Pins.  https://bit.ly/3h8QkR6.
  * [12] Paysafe Study.  https://bit.ly/2WtgjJl.
  * [13] Personal Music Players & Hearing.  https://bit.ly/34tyWkN.
  * [14] Voice Password Applications in Google Play.  https://bit.ly/38dq1oG.
  * [15] Voice Password in Samsung Bixby.  https://bit.ly/34rlPAN.
  * [16] Voice Password in Wechat.  https://zd.net/3myAGQ2.
  * [17] Voice Payment Survey.  https://bit.ly/3p6k97z.
  * [18] Will voice assistant change workplace.  https://bit.ly/3mxRDKm.
  * [19] H. Abdullah, W. Garcia, C. Peeters, P. Traynor, K.R.B. Butler, and J. Wilson.  Practical hidden voice attacks against speech and speaker recognition systems.  arXiv preprint 1904.05734, 2019.
  * [20] Muhammad Ejaz Ahmed, Il-Youp Kwak, Jun Ho Huh, Iljoo Kim, Taekkyung Oh, and Hyoungshick Kim.  Void: A fast and light voice liveness detection system.  In 29th $\\{$USENIX$\\}$ Security Symposium ($\\{$USENIX$\\}$ Security 20), pages 2685–2702, 2020.
  * [21] P. Bassia, I. Pitas, and N. Nikolaidis.  Robust audio watermarking in the time domain.  IEEE Transactions on multimedia, 3(2):232–241, 2001.
  * [22] M. Bellare and P. Rogaway.  Provably secure session key distribution: the three party case.  In ACM symposium on Theory of computing, 1995.
  * [23] G. Chen, S. Chen, L. Fan, X. Du, Z. Zhao, F. Song, and Y. Liu.  Who is real bob? adversarial attacks on speaker recognition systems.  IEEE Symposium on Security and Privacy, 2021.
  * [24] S. Chen, K. Ren, S. Piao, C. Wang, Q. Wang, J. Weng, L. Su, and A. Mohaisen.  You can hear but you cannot steal: Defending against voice impersonation attacks on smartphones.  In International Conference on Distributed Computing Systems, 2017\.
  * [25] T. Chen, L. Shangguan, Z. Li, and K. Jamieson.  Metamorph: Injecting inaudible commands into over-the-air voice controlled systems.  Proceedings of NDSS, 2020.
  * [26] I. J. Cox, J. Kilian, T. Leighton, and T. Shamoon.  A secure, robust watermark for multimedia.  In International Workshop on Information Hiding, 1996.
  * [27] N. Cvejic.  Algorithms for audio watermarking and steganography.  2004\.
  * [28] N. Cvejic and T. Seppänen.  Spread spectrum audio watermarking using frequency hopping and attack characterization.  Signal processing, 84(1):207–213, 2004.
  * [29] S. Das and P. N. Suganthan.  Differential evolution: A survey of the state-of-the-art.  IEEE transactions on evolutionary computation, 15(1):4–31, 2010\.
  * [30] H. Feng, K. Fawaz, and K. G. Shin.  Continuous authentication for voice assistants.  In International Conference on Mobile Computing and Networking, 2017\.
  * [31] Y. Gong and C. Poellabauer.  Protecting voice controlled systems using sound source identification based on acoustic cues.  In International Conference on Computer Communication and Networks, 2018.
  * [32] Y. Gong, J. Yang, J. Huber, M. MacKnight, and C. Poellabauer.  Remasc: Realistic replay attack corpus for voice controlled systems.  arXiv preprint 1904.03365, 2019.
  * [33] D. H. Johnson.  Signal-to-noise ratio.  Scholarpedia, 1(12):2088, 2006.
  * [34] T. Kinnunen, M. Sahidullah, H. Delgado, M. Todisco, N. Evans, J. Yamagishi, and K. A. Lee.  The ASVspoof 2017 challenge: Assessing the limits of replay spoofing attack detection.  2017\.
  * [35] D. Kirovski and H. S. Malvar.  Spread-spectrum watermarking of audio signals.  IEEE transactions on signal processing, 51(4):1020–1033, 2003.
  * [36] P. Kumar, M. Jayakumar, and A.V. Vidyapeetham.  Comparison of bit error rate for propagation mechanisms of millimeter waves in a practical communication systems employing psk and fsk.  PIERS Proceedings, 2010.
  * [37] M. Leng, S. S. Arora, and K. Wagner.  Replay spoofing detection for automatic speaker verification system, 2020\.  US Patent App. 16/535,836.
  * [38] C. Neubauer and J. Herre.  Digital watermarking and its influence on audio quality.  In Audio Engineering Society Convention 105. Audio Engineering Society, 1998.
  * [39] J. Park, K. Park, and J. R. Yoon.  Underwater acoustic communication channel simulator for flat fading.  Japanese Journal of Applied Physics, 49(7S), 2010.
  * [40] S. Pradhan, W. Sun, G. Baig, and L. Qiu.  Combating replay attacks against voice assistants.  ACM Interactive, Mobile, Wearable and Ubiquitous Technologies, 3(3), 2019.
  * [41] N. Scaife, C. Peeters, and P. Traynor.  Fear the reaper: Characterization and fast detection of card skimmers.  In 27th USENIX Security Symposium, 2018.
  * [42] J. W. Seok and J. W. Hong.  Audio watermarking for copyright protection of digital audio data.  Electronics Letters, 37(1):60–61, 2001.
  * [43] R. Storn and K. Price.  Differential evolution–a simple and efficient heuristic for global optimization over continuous spaces.  Journal of global optimization, 11(4):341–359, 1997.
  * [44] Y. Suzuki and H. Takeshima.  Equal-loudness-level contours for pure tones.  The Journal of the Acoustical Society of America, 116(2):918–933, 2004.
  * [45] M. D. Swanson, B. Zhu, A. H. Tewfik, and L. Boney.  Robust audio watermarking using perceptual masking.  Signal processing, 66(3):337–355, 1998.
  * [46] B. Truax.  Acoustic communication.  Greenwood Publishing Group, 2001.
  * [47] J. L. Tsai.  Efficient nonce-based authentication scheme for session initiation protocol.  IJ Network Security, 9(1):12–16, 2009.
  * [48] A. Wabnitz, N. Epain, C. Jin, and A. Van Schaik.  Room acoustics simulation for multichannel microphone arrays.  In International Symposium on Room Acoustics, 2010.
  * [49] S. Wang, J. Cao, X. He, K. Sun, and Q. Li.  When the differences in frequency domain are compensated: Understanding and defeating modulated replay attacks on automatic speech recognition.  In ACM SIGSAC Conference on Computer and Communications Security, 2020.
  * [50] F. Wilcoxon, S. K. Katti, and R. A. Wilcox.  Critical values and probability levels for the wilcoxon rank sum test and the wilcoxon signed rank test.  Selected tables in mathematical statistics, 1:171–259, 1970.
  * [51] S. Wu, J. Huang, D. Huang, and Y. Q. Shi.  Efficiently self-synchronized audio watermarking for assured audio data transmission.  IEEE Transactions on Broadcasting, 51(1):69–76, 2005.
  * [52] W. Xiong, L. Wu, F. Alleva, J. Droppo, X. Huang, and A. Stolcke.  The Microsoft 2017 conversational speech recognition system.  In IEEE International Conference on Acoustics, Speech and Signal Processing, 2018.

##  Appendix 0.A Appendix: Distance between adversary and VDS speaker.

To launch a speech re-use attack, an adversary first records a user’s
interaction with VDS. In this experiment, we measure how effectively the
acoustic nonce is preserved with varying distance between the VDS speaker and
adversary’s recording device (microphone) at the time of capture. Figure 8(b)
shows that this distance significantly impacts BER. The computed BER is 0%
upto a distance of 4 m when the signal to noise ratio is low. If the captured
speech is being re-used, assuming a lossless audio input to VDS, the entire
nonce can be recovered and used for determining the particular user
interaction session the speech was previously used in.

Figure 7: Variation of bit error rate (%) with different distances between
microphone and user (simulated using a playback device) in different
environments: (a) conference room, (b) dining hall, and (c) in a car parked at
a gas station. Given a fixed size car in (c), results can only be computed
upto a distance of 0.7 m.

(a)

(b)

Figure 8: (a) Impact of the proposed security overlay on prediction confidence
of Microsoft Azure Speaker Recognition system in three different environments.
(b) Impact of distance between the VDS speaker and an adversary’s recording
device at the time of capture on speech re-use.  Figure 9: Loudness of speech
with and without embedded nonce (average value of 5 measurements).

##  Appendix 0.B Appendix: Nonce Removal Techniques

  * •

Resampling. Samples the audio at a different frequency (e.g. 44.1 KHz) to
remove sampling-dependent watermark.

  * •

Amplitude Compression. Alters the amplitude of recorded audio to bypass
amplitude-related watermark detection.

  * •

Filtering. Uses a high pass or low pass filter to remove a specific range of
frequency in the audio. For example, separate audible signals from inaudible
frequencies (e.g. ultrasound).

  * •

Lossy Compression. Leverages data encoding methods that uses inexact
approximations and partial data discarding to compress the audio. For example,
MP3 [10] or MPEG-1 Audio Layer III reduce audio file size by taking advantage
of a perceptual limitation of human hearing.

  * •

Equalization. Modifies the frequency response of an audio system using linear
filters. Frequency response is the measure of the audio output in comparison
to the input as a function of frequency. For example, a 5kHz wave input may
have a 4kHz output response.

  * •

Additive Noise. Adds noise to decrease SNR (signal-to-noise ratio) so that
watermark cannot be extracted successfully.

##  Appendix 0.C Appendix: User Study Demographics

The demographics for the 120 participants in the user study are shown in
Figure 6. There are more female participants than male participants, and 56.7%
of the participants are between age 25 to 35.

Table 6: User study participant demographic information. Category | Gender Groups | Age Groups
---|---|---
Male | Female | Others | 18-25, | 25-35, | 35-45, | 45-55
Number | 65 | 53 | 2 | 42 | 68 | 41 | 11