# Growth of a viscoplastic blister underneath an elastic sheet

Torstein Sæter1 Olivier Galland2 Blandine Feneuil1,3 Andreas Carlson1
<EMAIL_ADDRESS>1 Department of Mathematics, Mechanics Division,
University of Oslo, Oslo 0316, Norway  2 Physics of Geological Processes, The
Njord Centre, Department of Geosciences, University of Oslo, Blindern, 0316,
Oslo, Norway  3 Current address: SINTEF Industry, Petroleum department,
Trondheim, Norway.

###### Abstract

Inspired by the formation of geological structures as earth’s crust deforms by
magmatic intrusions, we investigate the elastohydrodynamic growth of a
viscoplastic blister under an elastic sheet. By combining experiments, scaling
analysis and numerical simulations we reveal a new regime for the growth of
the blister’s height $\sim t^{5/9}$ and radius $\sim t^{2/9}$. A plug like
flow inside the blister dictates its dynamics, whereas the blister takes a
quasi-static self-similar shape given by a balance in the pressure gradient
induced by bending of the elastic sheet and the fluid’s yield stress.

The intrusion of a liquid underneath an elastic sheet resting on a non-
deformable solid has been deployed as a generic model to understand the
formation of geological structures such as magmatic sills and laccoliths [1,
2, 3], and ice sheet relaxation [4]. In models of e.g., sill and laccolith
emplacement, the intruding magma is commonly assumed to be Newtonian, and the
solid is assumed to deform by elastic bending and tensile fracturing. However,
there is growing evidence that some magmas exhibit highly non-Newtonian
behaviour [5, 6, 7], such that the magma can transport as plug flow [8].
Conversely, rock layers along which magma is emplaced can exhibit complex
viscoplastic rheological response to magma flow [9, 10]. To which extent these
non-linear phenomena affect the dynamics of these geophysical systems is
unclear.

In this context, the dual nature of a viscoplastic fluid, i.e. when the
applied stress is above the yield stress $\tau_{0}$ it flows like a liquid and
below $\tau_{0}$ it is a solid, is particularly interesting to capture the
consequences of the fluid-and-solid nature of the flowing matter. We describe
in here the spatiotemporal growth of a viscoplastic blister underneath an
elastic sheet.

When an elastic sheet is resting on a solid pre-wetted by a fluid film, it has
been shown that the height and radius of the formed blister is determined by
local effects at the intrusion tip, which couple to the interior quasi-static
solution of the shape of the sheet [11]. Other effects have also been
demonstrated to affect the blister’s dynamics, e.g. vapour at the intrusion
tip [12], adhesion to the supporting solid [13, 14, 15], gravitational flow
[16], the height of the pre-coated film layer [17] as well as stochastic
effects [18]. The influence of a deformable elastic sheet on the resulting
flow also has been shown in other settings such as the suppression of viscous
fingering in a Hele-Shaw geometry [19] and the effect of the porous-base [4,
20] .

Fluid models that account for viscoplasticity have been used to provide a more
realistic representation of mud, glacier and magma flows on the Earth’s
surface [21, 5, 22]. Recently, viscoplastic fluids have also attracted much
interest in interfacial flows, demonstrated to affect coating, drop spreading
and coalescence [23, 24, 25].

Figure 1:  a) Schematic drawing of the studied system. An elastic sheet, of
Young’s modulus $E=0.126\;$ MPa, thickness $d=10\;$mm, and Poisson’s ratio
$\nu=0.5$, rests on top of pre-wetted layer of yield stress fluid Carbopol, of
thickness $h_{0}$. Solutions of the same Carbopol as that of pre-wetted layer
is injected at constant $Q$ beneath the elastic sheet through a small tube of
radius $r_{t}=2\;$mm. The resulting blister is measured by tracking the laser-
line and extracting the height profile, $h(r,t)$. Experimental profiles
$h(r,t)$ of the sheet at $t=[20,340,660]$ s when injecting a fluid with b) low
($\tau_{0}=2.6\;Pa$) and c) high ($\tau_{0}=31.8\;Pa$) yield stress.

We combine here the effects of having an elastic interface that can bend and
look at the formation and growth of a blister as we inject a viscoplastic
fluid. Intuitively, we might suspect that the large stress at the peeling
front, as predicted from the Newtonian case [11], will locally fluidize the
material, but how is the peeling front dynamics affected by the yield stress
threshold? To answer this, we make experiments adapting the design from
previous Newtonian elastohydrodynamics studies [11, 26]. Fig. 1a shows a
schematic representation of the experimental system, where we use Carbopol as
a model yield-stress fluid as it has well characterised rheological properties
and small thixotropic effects [27]. We prepare four different Carbopol
solutions, giving a range in yield stresses $\tau_{0}\cap[2.6-47.2]$ Pa when
fitting the Herschel-Bulkley model to the rheological measurements, see
Supplemental Material (SM) for details.

A solid plate is coated by a thin layer of Carbopol with height $h_{0}$
$\approx[0.15-0.35]$ mm. The elastic sheet initially rests upon the Carbopol
layer, before injecting Carbopol with a controlled flux
$Q\cap[1.7,3.3,6.7]\cdot 10^{-7}m^{3}/s$ through the tube at the center of the
plate. The sheet is made of a silicone based elastomer (Zhermack, Elite
Double) with a Young’s modulus $E=0.126$ MPa. A laser line is used to
visualise the sheet’s deflections $h(r,t)$ as the blister grows in time and
space by using side view imaging.

The spatiotemporal growth of the blister for two Carbopol solutions,
$\tau_{0}=2.6$ Pa and $\tau_{0}=31.8$ Pa, are shown in Fig. 1b-c,
respectively. There are some noteworthy features to highlight in Fig. 1b. A
distinct effect of $\tau_{0}$ on the growth of the blister’s radius $R(t)$ and
its height at the center $h(0,t)$ is observed. The ratio between the
deflection and the radius remains small throughout the experiments
($h(0,t)/R(t)\leq 0.1$), and the flow is highly viscous with a small Reynolds
number. Our measurements of the blister profiles show a high degree of axis-
symmetry around the tube where Carbopol is injected in the experiments.

To give a theoretical description of the elastohydrodynamics of the
viscoplastic intrusion observed in Fig. 1b, we use the small aspect ratio and
the viscous flow allowing us adopt the axisymmetric lubrication theory. There
is a minor effect from gravity (illustrated in the SM), which we have
neglected here for clarity, an assumption strictly valid for lengths below the
elastogravity length $L_{eg}=\big{(}\frac{B}{\rho g}\big{)}^{1/4}\approx
0.035$ m, with $\rho$ the liquid density and $g$ gravitational acceleration.
The bending modulus of the sheet is fixed
$B=\frac{Ed^{3}}{12(1-\nu^{2})}=0.014$ N$\cdot$m, with $d$ the sheet thickness
and $\nu=0.5$ the Poisson’s ratio. We use the fact that there are only small
deformations of the sheet and we can then neglect in-plane tension [28]. Thus
the bending pressure is described as $p(r,t)=B\nabla^{4}h(r,t)$ for small
slopes. To capture the principal viscoplastic effect we use the Bingham model,
which can be combined with the integrated lubrication equations to derive the
expression for the vertical yielding line $Y(r,t)$ of the fluid, where the
shear stress $\tau_{rz}$ equals the yield stress $\tau_{0}$ [21, 5, 29],

$Y(r,t)=max\bigg{(}0,\frac{h(r,t)}{2}-\frac{\tau_{0}}{\mid\frac{\partial
p(r,t)}{\partial r}\mid}\bigg{)}.$ (1)

We expect the shear profile $\tau_{rz}$ to be symmetric, and will therefore
have another corresponding yielding position along $z(r,t)=h(r,t)-Y(r,t)$.
Between the two yielding lines we have a plug flow, which flows with a
constant speed and zero shear rate [30].

We follow the standard procedure to derive the thin film equation, where an
additional term including $Y(r,t)$ appear from the viscoplastic Bingham model
[31, 21, 5, 29]:

$\begin{split}\frac{\partial h(r,t)}{\partial t}=&\frac{\partial}{6\mu
r\partial r}\bigg{(}rY(r,t)^{2}(3h(r,t)-2Y(r,t))\frac{\partial
p(r,t)}{\partial r}\bigg{)}\\\ +&w(r).\end{split}$ (2)

We have here introduced $w(r)=\frac{2Q}{\pi
r_{t}^{2}}\bigg{(}1-\big{(}\frac{r}{r_{t}}\big{)}^{2}\bigg{)}$ for $r\leq
r_{t}$, which is the Poisseuille flow through the tube with radius $r_{t}$
allowing us to prescribe the influx $Q$. Note that a no-slip condition has
been applied at both surfaces and that $\mu$ is the effective fluid viscosity.
Equation (2) has two particularly interesting limits. If $Y(r,t)=h(r,t)$
Newtonian flow is recovered and predicts the blister to grow with a height
$\sim t^{8/22}$ [11]. As $Y(r,t)\rightarrow 0$ we expect a plug flow to form
and the flow to be dominated by the effects from $\tau_{0}$.

Figure 2:  a) Plots of experimental results for the blister height
$h(0,t)-h_{0}$ for different yield stress $\tau_{0}$ and flux $Q$. b) Results
from a) scaled by the length $L=\big{(}{B}/{\tau_{0}}\big{)}^{1/3}$ and the
time $T={B}/{Q\tau_{0}}$ while comparing with the numerical simulations of the
non-dimensional version of equation (2). The dashed-dotted line shows the
derived scaling law of equation (3), where we have determined $\alpha$ = 0.25.
Figure 3:  a) Plot of laboratory blister profiles at different times with
$\tau_{0}=47.2\;Pa$ and $Q=3.3\cdot 10^{-7}\;m^{3}/s$. b) Plots of numerical
blister profiles for two times, computed from non-dimensional version of
equation (2) with $\Pi=1000$. Dashed lines for each profile represent the non-
dimensional yield limits $Y(r,t)/L$ and $(h(r,t)-Y(r,t))/L$; plug flow occurs
in-between the yield lines. In c) we scale our experimental and numerical
height profiles from a) and b) (11 profiles between the times $t/T=0.12-0.3$)
by using the scaling laws in equation (3) and equation (4). The profiles fall
onto a self-similar shape, described by the solution of the non-dimensional
form of equation (1) when $Y/L\to 0$, where we have
$L=\big{(}{B}/{\tau_{0}}\big{)}^{1/3}$, $T={B}/{Q\tau_{0}}$,
$h_{max}=\frac{h(0,t)T^{5/9}}{Lt^{5/9}}$ and
$r_{max}=\frac{R(t)T^{2/9}}{Lt^{2/9}}$.

Before moving on to the experiments and the numerical simulations of equation
(2), we derive a scaling law for the spatiotemporal blister growth. If we
scale the height $h(r,t)$ with $h(0,t)$ and the radial direction $r$ with
$R(t)$, while assuming unidirectional flow from $r=0$ and outwards, in the
limit $Y(r,t)\rightarrow 0$, equation (1) gives $\frac{h(r,t)\partial
p(r,t)}{2\partial r}\sim\frac{Bh^{2}(0,t)}{R(t)^{5}}\sim(h(0,t)/2\tau_{0})$.
We impose mass conservation, where the volume $V=\int_{0}^{R(t)}2\pi
h(r,t)rdr\sim h(0,t)R(t)^{2}\sim Qt$, implicit in equation (2), which gives
the scaling laws for the blister height $h(0,t)$ and radius $R(t)$:

$h(0,t)\sim\bigg{(}\frac{4\tau_{0}^{2}Q^{5}}{B^{2}}\bigg{)}^{1/9}t^{5/9}\sim
LT^{-5/9}t^{5/9}$ (3)
$R(t)\sim\bigg{(}\frac{Q^{2}B}{2\tau_{0}}\bigg{)}^{1/9}t^{2/9}\sim
LT^{-2/9}t^{2/9}$ (4)

A length $L=(\frac{B}{\tau_{0}})^{1/3}$ and time $T=\frac{B}{Q\tau_{0}}$
appear in equation (3) and equation (4), which characterise our system.

We make equation (1) and equation (2) non-dimensional by scaling $h(r,t)$ and
$r$ with $L$ and time $t$ with $T$, giving only one non-dimensional number
appearing in front of the first term on the right-hand-side of equation (2)
i.e. $\Pi=\frac{B}{6\mu Q}$, which gives the ratio between elastic and viscous
forces. We assume the effective viscosity in our experiments to be set by the
shear rate in the regions where the fluid is yielded, because the flows here
are determining our viscous timescale. We estimate the shear rate by assuming
a parabolic velocity profile for $z>h(r,t)-Y(r,t)$ and $z<Y(r,t)$, and obtain
$\dot{\gamma}_{max}\approx\frac{\partial u}{\partial
z}|_{z=0,z=h}\sim\frac{Q}{R(t)Y(r,t)^{2}}$. Setting $Q\approx
10^{-7}\;m^{3}$/s, $R(t)\approx 10^{-1}\;$ m, and $Y(r,t)\approx 10^{-4}$ m
gives $\dot{\gamma}_{max}\approx 100\;s^{-1}$. This is a conservative estimate
for the shear rate and we therefore assume $\dot{\gamma}\approx[10-100]s^{-1}$
in the experiments. From the rheological viscosity curves (see SM), we
estimate the effective viscosity $\mu\approx[0.1-10]$ Pa$\cdot$s, which
translates to $\Pi\approx[10^{2}-10^{4}]$.

We use the blister’s maximal height $h(0,t)$ as a metric to characterise the
dynamics to test the predictions from equation (3) and equation (4). The
bending stiffness $B$ is fixed, while we vary the yield stress $\tau_{0}$ and
the flux $Q$ in the experiments shown in Fig. 2a. It is clear that $h(0,t)$ is
affected by both parameters by steepening of the blister profile. With these
experiments at hand, the experimental data can be rescaled by $h(0,t)/L$ and
$t/T$, as shown in Fig. 2b, and collapse onto a single curve following the
proposed scaling law of equation (3). At late times in the experiments,
gravity is starting to affect the dynamics and we enter a transitional regime
where bending and gravitational forces become comparable.

The experiments are complemented by numerical solutions of equation (2), which
is discretised by linear finite elements and the solution method is described
in [17]. We use an adaptive time-stepping routine, with a time step limit of
$\Delta t/T=9\cdot 10^{-6}$, and a discretisation in space $\Delta
r/L\in[0.0005-0.005]$. We have defined
$Y(r,t)=max\bigg{(}\epsilon,h(r,t)/2-\frac{1}{\mid\frac{\partial}{\partial
r}\nabla^{4}h(r,t)\mid}\bigg{)}$, with a regularisation parameter
$\epsilon=10^{-6}$ similar to [24], and we tested that the results are
unaffected by this choice of $\epsilon$. The simulations are started with a
pre-wetted layer $h_{0}/L=0.0069$, and with $Y(r,t)=0$, which is gradually
introduced until $t/T=10^{-4}$. The results from the numerical simulations are
shown with the experiments in Fig. 2b, they are fairly insensitive to the non-
dimensional number $\Pi$, and recover the scaling law in equation (3). The
small time shift separating the experiments and the simulations (a factor of
$2.5$ in time) may stem from the minimal rheological model we use to describe
Carbopol [27], wall slip or the simplified influx condition. Regardless, the
simulations overall represent well the elastohydrodynamic growth of the
viscoplastic blister.

Next, we want to understand the interface dynamics of the blister and if it
adopts a self-similar shape. Fig. 3a shows the blister shape in time from the
experiments. To better understand the viscoplastic flow in the blister, we
plot in Fig. 3b the non-dimensional height profile and the yielded lines for
two time instances from a numerical simulation. It is clear that a major part
of the area is encompassed between the yielding lines (dashed lines) in Fig.
3b with a significant plug flow, whereas a flow with a viscous gradient is
limited to thin areas near the two bounding solid surfaces. This further
supports our argument allowing us to derive equation (3) and equation (4). To
go one step further, we now rescale the height profiles from the experiments
and the numerical simulations data (eleven profiles are shown in-between the
two time-points in Fig. 3b, using the predictions from equation (3) and
equation (4), where we normalise each profile with:
$h_{max}=\frac{h(0,t)T^{5/9}}{Lt^{5/9}}$ and radial value,
$r_{max}=\frac{R(t)T^{2/9}}{Lt^{2/9}}$, see SM. The experimental and numerical
data fall onto a profile for a universal blister shape, see Fig. 3c.

The self-similar blister shape can be understood if we look back to equation
(1) in the limit $Y(r,t)\rightarrow 0$ that reduces to a non-linear ordinary
differential equation (ODE): $h(r)/d\nabla^{4}h(r)/dr=2\tau_{0}sgn(h_{0}-h)$
[24]. By scaling $h(r)$ and $r$ with $L$, equation (1) becomes parameter free.
To determine the blister shape we use a numerical profile from our simulation,
$\Pi=1000$ and $t/T=0.75$, as an initial guess. We need five boundary
conditions to define our problem. We choose three conditions at the
centerline, where we pin the height to the maximum height of the simulation
profile, $h_{max}$, ${h(0)}=h_{max}$, and set two symmetry conditions,
$\frac{\partial{h}(0)}{\partial r}=0$ and $\frac{\partial^{3}{h}(0)}{\partial
r^{3}}=0$. We supplement with two boundary conditions at $\tilde{R}=1.3\times
R$, where we set ${h}(\tilde{R})=h_{0}$ and
$\frac{\partial^{2}{h}(\tilde{R})}{\partial r^{2}}=0$. The solver Solve_bvp
retrieved from the scipy package in Python is used to solve the ODE. In Fig.
3c, the solution for the height profile is shown together with the experiments
and the numerical simulations of equation (2), which gives a perfect agreement
for the predicted shapes.

Intrusion of flowing matter with complex rheological properties underneath an
elastic layer or inside an elastic matrix appear in a myriad of geological
processes. Both non-linear magma plug flow [8] and viscoplastic deformation of
magma intrusion’s host rocks [10] have been documented to play a key role in
magma emplacement in the Earth’s crust. Our study quantifies the dynamics of a
viscoplastic blister growth underneath an elastic sheet, and evidences a
viscoplastic regime for the growth of its height and radial expansion. The
blister is shown to adopt a self-similar quasi-static shape in time that is
set by the balance between the pressure gradient induced by the bending of the
elastic sheet and the yield stress of the intruding fluid. These results
suggest that the growth of geological structures such as sills and laccoliths
exhibit distinct dynamics than those predicted from a Newtonian model if the
intruding fluid can support a critical stress before starting to flow. Our
work gives a first look at the elastohydrodynamics of a growing viscoplastic
blister, but highlight that there are many more questions in these
configurations that remain to be understood such as additional non-Newtonian
effects in the bulk and the elastic sheet [32] and to combine the yield-stress
intrusion model with an elastic fracture toughness at the tip.

The authors thank for the funding through the EarthFlow initiative at the
University of Oslo.

## References

  * Pollard and Johnson [1973] D. D. Pollard and A. M. Johnson, Mechanics of growth of some laccolithic intrusions in the Henry mountains, Utah, ii: Bending and failure of overburden layers and sill formation, Tectonophysics 18, 311 (1973).
  * Michaut [2011] C. Michaut, Dynamics of magmatic intrusions in the upper crust: Theory and applications to laccoliths on earth and the moon, Journal of Geophysical Research: Solid Earth 116 (2011).
  * Lister and Kerr [1991] J. R. Lister and R. C. Kerr, Fluid-mechanical models of crack propagation and their application to magma transport in dykes, Journal of Geophysical Research: Solid Earth 96, 10049 (1991).
  * Lai  _et al._ [2021] C.-Y. Lai, L. A. Stevens, D. L. Chase, T. T. Creyts, M. D. Behn, S. B. Das, and H. A. Stone, Hydraulic transmissivity inferred from ice-sheet relaxation following Greenland supraglacial lake drainages, Nature communications 12, 1 (2021).
  * Balmforth  _et al._ [2000] N. Balmforth, A. Burbidge, R. Craster, J. Salzig, and A. Shen, Visco-plastic models of isothermal lava domes, Journal of Fluid Mechanics 403 (2000).
  * Caricchi  _et al._ [2007] L. Caricchi, L. Burlini, P. Ulmer, T. Gerya, M. Vassalli, and P. Papale, Non-newtonian rheology of crystal-bearing magmas and implications for magma ascent dynamics, Earth and Planetary Science Letters 264, 402 (2007).
  * Cordonnier  _et al._ [2012] B. Cordonnier, L. Caricchi, M. Pistone, J. Castro, K.-U. Hess, S. Gottschaller, M. Manga, D. Dingwell, and L. Burlini, The viscous-brittle transition of crystal-bearing silicic melt: Direct observation of magma rupture and healing, Geology 40, 611 (2012).
  * Morgan  _et al._ [2008] S. S. Morgan, A. Stanik, E. Horsman, B. Tikoff, M. de Saint Blanquat, and G. Habert, Emplacement of multiple magma sheets and wall rock deformation: Trachyte mesa intrusion, henry mountains, utah, Journal of Structural Geology 30, 491 (2008).
  * Scheibert  _et al._ [2017] J. Scheibert, O. Galland, and A. Hafver, Inelastic deformation during sill and laccolith emplacement: Insights from an analytic elastoplastic model, Journal of Geophysical Research: Solid Earth 122, 923 (2017).
  * Galland  _et al._ [2019] O. Galland, J. B. Spacapan, O. Rabbel, K. Mair, F. G. Soto, T. Eiken, M. Schiuma, and H. A. Leanza, Structure, emplacement mechanism and magma-flow significance of igneous fingers – implications for sill emplacement in sedimentary basins, Journal of Structural Geology 124, 120 (2019).
  * Lister  _et al._ [2013] J. R. Lister, G. G. Peng, and J. A. Neufeld, Viscous control of peeling an elastic sheet by bending and pulling, Phys. Rev. Lett. 111, 154501 (2013).
  * Wang and Detournay [2018] Z.-Q. Wang and E. Detournay, The tip region of a near-surface hydraulic fracture, Journal of Applied Mechanics 85, 041010 (2018).
  * Ball and Neufeld [2018] T. V. Ball and J. A. Neufeld, Static and dynamic fluid-driven fracturing of adhered elastica, Phys. Rev. Fluids 3, 074101 (2018).
  * Lister  _et al._ [2019] J. R. Lister, D. J. Skinner, and T. M. J. Large, Viscous control of shallow elastic fracture: peeling without precursors, Journal of Fluid Mechanics 868, 119–140 (2019).
  * Hosoi and Mahadevan [2004] A. Hosoi and L. Mahadevan, Peeling, healing, and bursting in a lubricated elastic sheet, Physical review letters 93, 137802 (2004).
  * Hewitt  _et al._ [2015] I. J. Hewitt, N. J. Balmforth, and J. R. De Bruyn, Elastic-plated gravity currents, European Journal of Applied Mathematics 26, 1–31 (2015).
  * Pedersen  _et al._ [2019] C. Pedersen, J. F. Niven, T. Salez, K. Dalnoki-Veress, and A. Carlson, Asymptotic regimes in elastohydrodynamic and stochastic leveling on a viscous film, Phys. Rev. Fluids 4, 124003 (2019).
  * Carlson [2018] A. Carlson, Fluctuation assisted spreading of a fluid filled elastic blister, Journal of Fluid Mechanics 846, 1076 (2018).
  * Pihler-Puzović  _et al._ [2018] D. Pihler-Puzović, G. G. Peng, J. R. Lister, M. Heil, and A. Juel, Viscous fingering in a radial elastic-walled hele-shaw cell, Journal of Fluid Mechanics 849, 163–191 (2018).
  * Chase  _et al._ [2021] D. L. Chase, C.-Y. Lai, and H. A. Stone, Relaxation of a fluid-filled blister on a porous substrate, Phys. Rev. Fluids 6, 084101 (2021).
  * Liu and Mei [1989] K. F. Liu and C. C. Mei, Slow spreading of a sheet of Bingham fluid on an inclined plane, Journal of Fluid Mechanics 207, 505–529 (1989).
  * Petford [2003] N. Petford, Rheology of granitic magmas during ascent and emplacement, Annual Review of Earth and Planetary Sciences 31, 399 (2003).
  * Smit  _et al._ [2019] W. J. Smit, C. Kusina, J.-F. Joanny, and A. Colin, Stress field inside the bath determines dip coating with yield-stress fluids in cylindrical geometry, Phys. Rev. Lett. 123, 148002 (2019).
  * Jalaal  _et al._ [2021] M. Jalaal, B. Stoeber, and N. J. Balmforth, Spreading of viscoplastic droplets, Journal of Fluid Mechanics 914, A21 (2021).
  * Sanjay  _et al._ [2021] V. Sanjay, D. Lohse, and M. Jalaal, Bursting bubble in a viscoplastic medium, Journal of Fluid Mechanics 922, A2 (2021).
  * Berhanu  _et al._ [2019] M. Berhanu, A. Guérin, S. Courrech du Pont, F. Raoult, R. Perrier, and C. Michaut, Uplift of an elastic membrane by a viscous flow, Physical Review E 99 (2019).
  * Coussot [2014] P. Coussot, Yield stress fluid flows: A review of experimental data, Journal of Non-Newtonian Fluid Mechanics 211, 31 (2014).
  * Audoly and Pomeau [2010] B. Audoly and Y. Pomeau, Elasticity and geometry: From hair curls to the non-linear response of shells, Oxford University Press (2010).
  * Balmforth  _et al._ [2006] N. J. Balmforth, R. V. Craster, A. C. Rust, and R. Sassi, Viscoplastic flow over an inclined surface, Journal of Non-Newtonian Fluid Mechanics 139, 103 (2006).
  * Balmforth and Craster [1999] N. J. Balmforth and R. V. Craster, A consistent thin-layer theory for Bingham plastics, Journal of Non-Newtonian Fluid Mechanics 84, 65 (1999).
  * Oron  _et al._ [1997] A. Oron, S. H. Davis, and S. G. Bankoff, Long-scale evolution of thin liquid films, Reviews of modern physics 69, 931 (1997).
  * Ball and Balmforth [2021] T. V. Ball and N. J. Balmforth, Viscoplastic plates, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences 477, 20210509 (2021).


# Frequency-Guided Word Substitutions for Detecting
Textual Adversarial Examples

Maximilian Mozes1 Pontus Stenetorp1 Bennett Kleinberg2,1 Lewis D. Griffin1
1University College London
2Tilburg University
{m.mozes, p.stenetorp<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Recent efforts have shown that neural text processing models are vulnerable to
adversarial examples, but the nature of these examples is poorly understood.
In this work, we show that adversarial attacks against CNN, LSTM and
Transformer-based classification models perform word substitutions that are
identifiable through frequency differences between replaced words and their
corresponding substitutions. Based on these findings, we propose frequency-
guided word substitutions (FGWS), a simple algorithm exploiting the frequency
properties of adversarial word substitutions for the detection of adversarial
examples. FGWS achieves strong performance by accurately detecting adversarial
examples on the SST-2 and IMDb sentiment datasets, with $F_{1}$ detection
scores of up to 91.4% against RoBERTa-based classification models. We compare
our approach against a recently proposed perturbation discrimination framework
and show that we outperform it by up to 13.0% $F_{1}$.

##  1 Introduction

Artificial neural networks are vulnerable to adversarial examples—carefully
crafted perturbations of input data that lead a learning model into making
false predictions Szegedy et al. (2014).

Attack | Original or perturbed sequence
---|---
None | A clever blend of fact and fiction
Genetic | A $\overset{\raise 0.0pt\hbox{\hbox{\pagecolor{blue!15}1.39}}}{\textbf{brainy}}\\!\\!\overset{\raise 3.00003pt\hbox{$\,\dashleftarrow$}}{{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}I}}\\!\\!\overset{\hbox{\pagecolor{blue!40}5.55}}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{clever}}}\text{]}}$ blend of fact and fiction
PWWS | A $\overset{\raise 0.0pt\hbox{\hbox{\pagecolor{blue!17}1.61}}}{\textbf{cunning}}\\!\\!\overset{\raise 3.00003pt\hbox{$\dashleftarrow$\,\,}}{{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}I}}\\!\\!\overset{\hbox{\pagecolor{blue!40}5.55}}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{clever}}}\text{]}}$ $\overset{\raise 0.0pt\hbox{\hbox{\pagecolor{blue!10}0.00}}}{\textbf{blending}}\\!\\!\overset{\raise 3.00003pt\hbox{$\dashleftarrow$\,\,}}{{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}I}}\\!\\!\overset{\hbox{\pagecolor{blue!30}3.81}}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{blend}}}\text{]}}$ of
| fact and $\overset{\raise
0.0pt\hbox{\hbox{\pagecolor{blue!10}0.00}}}{\textbf{fabrication}}\\!\\!\overset{\raise
3.00003pt\hbox{$\dashleftarrow$\,\,}}{{\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color@gray@fill{1}I}}\\!\\!\overset{\hbox{\pagecolor{blue!35}4.39}}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{fiction}}}\text{]}}$

Figure 1: Corpus $\log_{e}$ frequencies of the replaced words (bold, italic,
red) and their corresponding adversarial substitutions (bold, black) using the
Genetic Alzantot et al. (2018) and PWWS Ren et al. (2019) attacks on SST-2
Socher et al. (2013).

While initially discovered for computer vision tasks, natural language
processing (NLP) models have also been shown to be oversensitive to
adversarial input perturbations for a variety of tasks Papernot et al. (2016);
Jia and Liang (2017); Belinkov and Bisk (2018); Glockner et al. (2018); Iyyer
et al. (2018). Here we focus on highly successful synonym substitution attacks
Alzantot et al. (2018); Ren et al. (2019); Zang et al. (2020), in which
individual words are replaced with semantically similar ones. Existing defense
methods against these attacks mainly focus on adversarial training Jia and
Liang (2017); Ebrahimi et al. (2018); Ribeiro et al. (2018); Ren et al.
(2019); Jin et al. (2019) and hence typically require a priori attack
knowledge and models to be retrained from scratch to increase their
robustness. Recent work by Zhou et al. (2019) instead proposes DISP (learning
to discriminate perturbations), a perturbation discrimination framework that
exploits pre-trained contextualized word representations to detect and correct
word-level adversarial substitutions without having to retrain the attacked
model. In this paper, we show that we can achieve an improved performance for
the detection and correction of adversarial examples based on the finding that
various word-level adversarial attacks have a tendency to replace input words
with less frequent ones.111This frequency difference is expected for attacks
that explicitly conduct symbol substitutions resulting in out-of-vocabulary
(OOV) terms Gao et al. (2018). We therefore study attacks that do not
explicitly enforce a mapping to words that have lower frequencies. Figure 1
illustrates this tendency for two state-of-the-art attacks. We provide
statistical evidence to support this observation and propose a rule-based and
model-agnostic algorithm, frequency-guided word substitutions (FGWS), to
detect adversarial sequences and recover model performances for perturbed test
set sequences. FGWS effectively detects adversarial perturbations, achieving
$F_{1}$ scores of up to 91.4% against RoBERTa-based models Liu et al. (2019)
on the IMDb sentiment dataset Maas et al. (2011). Furthermore, our results
show that FGWS outperforms DISP by up to 13.0% $F_{1}$ when differentiating
between unperturbed and perturbed sequences, despite representing a
conceptually simpler approach to this task.

##  2 Generating adversarial examples

In our experiments, we investigate two baseline attacks introduced by Ren et
al. (2019) as well as two state-of-the-art attacks.

Random. Our first baseline attack is a simple word substitution model that
randomly selects words in an input sequence and replaces them with synonyms
randomly sampled from a set of synonyms related to the specific word. We
follow Ren et al. (2019) by using WordNet Fellbaum (1998) to identify synonym
substitutions for each selected word.

Prioritized. Our second baseline builds upon Random by selecting the
replacement word from the synonym set that maximizes the change in prediction
confidence for the true label of an input.

Genetic. We additionally analyze an attack suggested by Alzantot et al.
(2018), consisting of a population-based black-box mechanism based on genetic
search that iteratively performs individual word-level perturbations to an
input sequence to cause a misclassification.

PWWS. Lastly, we analyze the probability weighted word saliency (PWWS)
algorithm Ren et al. (2019). For each word in an input sequence, PWWS selects
a set of synonym replacements from WordNet and chooses the synonym yielding
the highest difference in prediction confidence for the true class label after
replacement. The algorithm furthermore computes the word saliency Li et al.
(2016a, b) for each input word and ranks word replacements based on these two
indicators.

#### Datasets and models.

We perform experiments on two binary sentiment classification datasets, the
Stanford Sentiment Treebank (SST-2, Socher et al., 2013) and the IMDb reviews
dataset Maas et al. (2011), both of which are widely used in related work
focusing on adversarial examples in NLP Jia et al. (2019); Ren et al. (2019);
Zhou et al. (2019). Dataset details can be found in Appendix A. Adhering to
Zhou et al. (2019), we attack a pre-trained model based on the Transformer
architecture Vaswani et al. (2017). Zhou et al. (2019) use BERT Devlin et al.
(2019) in their experiments, but we found that RoBERTa Liu et al. (2019)
represents a stronger model for the specified tasks.

We additionally experiment with both a CNN Kim (2014) and an LSTM Hochreiter
and Schmidhuber (1997) text classification model, both of which have been
employed in existing work studying textual adversarial attacks Alzantot et al.
(2018); Lei et al. (2019); Jia et al. (2019); Tsai et al. (2019); Ren et al.
(2019).

The fine-tuned RoBERTa model achieves 93.4% and 94.9% accuracy on the IMDb and
SST-2 test sets, which is comparable to existing work Beltagy et al. (2020);
Liu et al. (2019). On the IMDb test set, the CNN achieves an accuracy of 86.0%
and the LSTM achieves 83.1%. These performances are close to existing work
using comparable settings Zhang et al. (2019); Ren et al. (2019). On the SST-2
test set, the CNN achieves 84.0% and the LSTM 85.2% accuracy, which are also
close to comparable experiments Huang et al. (2019).

Following Ren et al. (2019), we apply all four attacks to a random subset of
2,000 sequences from the IMDb test set as well as the entire test set of SST-2
(1,821 samples). Implementation details for the models and attacks can be
found in Appendix B. We report the after-attack accuracies222The after-attack
accuracy represents the model accuracy on the test set after perturbing all
correctly classified inputs. A lower after-attack accuracy indicates a
stronger attack. for the RoBERTa model in Table 2 and for the CNN/LSTM models
in Table 3 (column Adv.). We observe that all four attacks cause notable
decreases in model accuracy on the test sets, and that Genetic and PWWS are
more successful than the baseline attacks in most comparisons.

##  3 Analyzing frequencies of adversarial word substitutions

Next, we conduct an analysis of the word frequencies of individual words
replaced by the attacks and their substitutions. We compute the $\log_{e}$
training set frequencies $\phi(x)$ of all words $x$ that have been replaced by
the respective attacks and all of their corresponding substitutions. Then, we
conduct Bayesian hypothesis testing Rouder et al. (2009) to statistically
compare the two samples. This is achieved by computing the Bayes factor
$\mathrm{BF}_{10}$, representing the degree to which the data favor the
alternative hypothesis over the null hypothesis. Here, the alternative
hypothesis $\mathcal{H}_{1}$ states that the frequencies of replaced words
differ from the frequencies of the adversarial substitutions. The null
hypothesis $\mathcal{H}_{0}$ states that there is no such difference. The
higher $\mathrm{BF}_{10}$, the stronger the evidence in favor of the
alternative hypothesis $\mathcal{H}_{1}$.333A Bayes factor
$\mathrm{BF}_{10}>100$ can be interpreted as “extreme” evidence for
$\mathcal{H}_{1}$ Wagenmakers et al. (2011). We additionally calculate Cohen’s
$d$ effect sizes for all mean frequency comparisons.444Cohen’s $d$ indicates
the magnitude of the frequency differences of the two samples—larger effect
sizes suggest a higher magnitude of the frequency difference. A value of
$d=0.8$ can be interpreted as a large effect, $d=0.5$ is considered a moderate
effect Cohen (1988).

Dataset | Attack | Replaced | Subst. | non-OOV
---|---|---|---|---
$\mu_{\phi}$ | $\sigma_{\phi}$ | $\mu_{\phi}$ | $\sigma_{\phi}$ | $d$ | $\mu_{\phi}$ | $\sigma_{\phi}$ | $d$
IMDb | Random | 7.6 | 2.5 | 3.4 | 2.8 | 1.6 | 4.4 | 2.4 | 1.3
Prioritized | 7.6 | 2.5 | 3.6 | 2.8 | 1.5 | 4.4 | 2.4 | 1.3
Genetic | 6.5 | 2.0 | 3.7 | 2.3 | 1.3 | 4.0 | 2.2 | 1.2
PWWS | 6.9 | 2.3 | 4.4 | 2.5 | 1.0 | 5.0 | 2.1 | 0.9
SST-2 | Random | 5.4 | 2.6 | 2.1 | 2.4 | 1.4 | 4.0 | 1.8 | 0.6
Prioritized | 5.4 | 2.6 | 2.1 | 2.4 | 1.3 | 4.0 | 1.8 | 0.6
Genetic | 4.4 | 1.9 | 1.9 | 2.2 | 1.2 | 3.6 | 1.6 | 0.4
PWWS | 4.8 | 2.1 | 2.9 | 2.2 | 0.9 | 4.0 | 1.5 | 0.4

Table 1: Mean $\log_{e}$ frequencies of replaced words and their
substitutions. Values in bold denote largest effect sizes per dataset.

Table 1 shows the $\log_{e}$ frequencies (mean $\mu_{\phi}$ and standard
deviation $\sigma_{\phi}$) and Cohen’s $d$ for the specified samples generated
by the attacks against the RoBERTa model (the results for the CNN and LSTM
models can be found in Appendix C). We report the mean frequencies of all
adversarial substitutions (Subst.) and only those that occur in the training
set (non-OOV), to demonstrate that the frequency differences are not solely
caused by OOV substitutions. Across datasets and attacks, the substitutions
are consistently less frequent than the words selected for replacement. We
observe large Cohen’s $d$ effect sizes for the majority of comparisons,
statistically supporting the observation of mean frequency differences between
replaced words and their corresponding substitutions. We furthermore observe
that $\mathrm{BF}_{10}>10^{55}$ holds for all comparisons—both when
considering all and only non-OOV substitutions (the $\mathrm{BF}_{10}$ scores
can be found in Appendix D). This provides strong empirical evidence that
$\mathcal{H}_{1}$ is more likely to be supported by the measured word
frequencies (see Appendix E for additional illustrations).

Dataset | Attack | Adv. | Restored acc. |  TPR (FPR) | $F_{1}$
---|---|---|---|---|---
DISP | FGWS | DISP | FGWS | DISP | FGWS
IMDb | Random | 87.3 | 89.2 | 91.0 | 63.6 (9.4) |  83.5 (9.3) | 73.6 | 86.6
Prioritized | 41.5 | 81.0 | 85.9 | 87.8 (9.4) |  92.0 (9.3) | 89.0 | 91.4
Genetic | 47.7 | 74.1 | 80.6 | 70.4 (9.4) |  81.5 (9.3) | 78.3 | 85.4
PWWS | 41.0 | 68.7 | 75.4 | 66.2 (9.4) |  76.4 (9.3) | 75.4 | 82.3
SST-2 | Random | 87.2 | 86.6 | 90.0 |  66.2 (11.9) | 61.3 (11.4) | 74.4 | 71.0
Prioritized | 68.9 | 80.8 | 84.8 | 69.1 (11.9) |  74.7 (11.4) | 76.3 | 80.3
Genetic | 40.8 | 60.1 | 61.7 |  57.2 (11.9) | 57.0 (11.4) | 67.7 | 67.7
PWWS | 57.4 | 71.0 | 78.2 | 59.6 (11.9) |  65.6 (11.4) | 69.6 | 74.2

Table 2: Adversarial example detection performances for DISP and FGWS when
evaluated on attacks against RoBERTa. Adv. shows the model’s classification
accuracy on the perturbed sequences. Restored acc. denotes model accuracy on
the adversarial sequences after transformation. Values in bold represent best
scores per metric, dataset and attack.

Model/ Dataset | Attack | Adv. | Restored acc. | $F_{1}$
---|---|---|---|---
NWS | FGWS | NWS | FGWS
CNN/ IMDb | Random | 73.0 | 79.5 | 84.7 | 75.2 | 83.5
Prioritized | 14.0 | 41.6 | 78.9 | 71.5 | 89.3
Genetic | 10.7 | 21.3 | 68.5 | 37.9 | 83.5
PWWS | 10.2 | 27.4 | 70.2 | 45.4 | 83.9
LSTM/ IMDb | Random | 64.7 | 75.7 | 80.9 | 80.5 | 83.9
Prioritized | 3.2 | 32.0 | 71.6 | 62.4 | 86.6
Genetic | 1.2 | 10.9 | 54.9 | 34.3 | 78.0
PWWS | 1.6 | 17.3 | 57.1 | 41.7 | 77.4
CNN/ SST-2 | Random | 71.8 | 77.1 | 78.4 | 71.4 | 69.2
Prioritized | 50.3 | 60.1 | 69.3 | 54.8 | 67.8
Genetic | 19.6 | 34.9 | 48.8 | 49.9 | 60.3
PWWS | 28.1 | 47.4 | 58.1 | 55.1 | 63.9
LSTM/ SST-2 | Random | 73.4 | 79.3 | 80.5 | 69.2 | 62.2
Prioritized | 48.5 | 59.9 | 74.0 | 54.9 | 67.3
Genetic | 21.3 | 37.6 | 61.1 | 51.2 | 62.8
PWWS | 28.6 | 49.7 | 67.2 | 55.9 | 63.4

Table 3: Performance results of NWS and FGWS on attacks against the CNN and
LSTM models. Values in bold indicate best performances per model-dataset-
attack combination and metric.

##  4 Frequency-guided word substitutions

Based on the observation of consistent frequency differences between replaced
words and adversarial substitutions, we argue that the effects of such
substitutions can be mitigated through simple frequency-based transformations.
To do this, we propose frequency-guided word substitutions (FGWS), a detection
method that estimates whether a given input sequence is an adversarial
example.555Code is available at https://github.com/maximilianmozes/fgws. We
denote a classification model by a function $f(X)$ that maps a sequence $X$ to
a $C$-dimensional vector representing the probabilities for predicting each of
the $C$ possible classes. We represent a sequence as
$X=\\{x_{1},\dots,x_{n}\\}$, where $x_{i}$ denotes the $i$-th word in the
sequence. We furthermore introduce the notation
$f^{\ast}(X)\in\\{1,\dots,C\\}$ representing the class label predicted by $f$
given input $X$. FGWS transforms a given sequence $X$ into a sequence
$X^{\prime}$ by replacing infrequent words with more frequent, semantically
similar substitutions. We initially define the subset $X_{E}:=\\{x\in
X\,|\,\phi(x)<\delta\\}$ of words that are eligible for substitution, where
$\delta\in\mathbb{R}_{>0}$ is a frequency threshold. FGWS then generates a
sequence $X^{\prime}$ from $X$ by replacing all eligible words with words that
are semantically similar, but have higher occurrence frequencies in the
model’s training corpus. For each eligible word $x\in X_{E}$ we consider the
set of replacement candidates $\mathcal{S}(x)$ and find a replacement
$x^{\prime}$ by selecting
$x^{\prime}={\mathrm{argmax}}_{w\in\mathcal{S}(x)}\,\,\phi(w)$. We then
generate $X^{\prime}$ by replacing each eligible word $x$ with $x^{\prime}$ if
$\phi(x^{\prime})>\phi(x)$. Given the prediction label $y=f^{\ast}(X)$ for $X$
and a threshold $\gamma\in[0,1]$, the sequence $X$ is considered adversarial
if $f(X)_{y}-f(X^{\prime})_{y}>\gamma$, i.e., if the difference in prediction
confidence on class $y$ before and after transformation exceeds the threshold
$\gamma$. The threshold allows control of the rate of false positives (i.e.,
unperturbed sequences that are erroneously identified as adversarial) flagged
by our method.

###  4.1 Comparisons

####  DISP.

We compare FGWS to the DISP framework Zhou et al. (2019), which is, to the
best of our knowledge, the best existing approach for the detection of word-
level adversarial examples. DISP uses two independent BERT-based components, a
perturbation discriminator and an embedding estimator for token recovery, to
identify perturbed tokens and to reconstruct the replaced ones.

####  NWS.

For the CNN and LSTM models, we compare FGWS with the naive word substitutions
(NWS) baseline. For a given input sequence, NWS selects all OOV words in that
sequence and replaces each with a random choice from a set of semantically
related words. We restrict NWS to allow only substitutions for which the
replacement word occurs in the model’s training vocabulary. NWS can be
interpreted as a variant of FGWS that is not explicitly guided by word
frequencies.

###  4.2 Experiments

We apply both methods to the adversarial examples crafted by the four attacks
on the subsets of both the IMDb and SST-2 datasets as described in Section 2.
To account for an imbalance between unperturbed and perturbed sequences, we
repeatedly bootstrap a balanced set of unperturbed sequences for each set of
perturbed sequences for 10,000 times and compute the average detection scores.
For FGWS, we tune the frequency threshold $\delta$ for each model-dataset
combination on the validation set. To do this, we utilize the Prioritized
attack to craft adversarial examples from all sequences of the validation
set666We assume both baseline attacks as given to the defender, and prefer
Prioritized over Random due to increased effectiveness and hence a larger
sample size for parameter tuning. and compare FGWS detection performances with
different values for $\delta$. Specifically, we set $\delta$ equal to the
$\log_{e}$ frequency representing the $q^{\text{th}}$ percentile of all
$\log_{e}$ frequencies observed by the words eligible for replacement in the
training set, and experiment with $q\in\\{0,10,\dots,100\\}$. We select
$\gamma$ so that not more than $10\%$ of the unperturbed sequences in the
validation set are labeled as adversarial.777We provide additional results
with varying false positive thresholds in Appendix F. For FGWS, we define the
set of replacement candidates for each word $x\in X_{E}$ as the union of the
word’s $K$ nearest neighbors in a pre-trained GloVe Pennington et al. (2014)
word embedding space and its synonyms in WordNet. We set $K$ equal to the
average number of WordNet synonyms for each word in the validation set
(yielding $K=6$ for IMDb and $K=8$ for SST-2).

Unperturbed | a smart sweet and playful romantic comedy |  positive (99.9%)
---|---|---
(A) | PWWS | a $\\!\overset{0.00}{\textbf{impertinent}}$ $\overset{5.69}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{smart}}}\text{]}}\\!$ $\\!\overset{1.79}{\textbf{odoriferous}}$ $\overset{5.77}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{sweet}}}\text{]}}\\!$ and playful romantic comedy |  negative (56.3%)
(D) | DISP |  $\\!\overset{10.22}{\textbf{the}}$ $\overset{9.99}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{a}}}\text{]}}\\!$ $\\!\overset{6.83}{\textbf{little}}$ $\overset{0.00}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{impertinent}}}\text{]}}\\!$ odoriferous and playful romantic comedy |  positive (79.3%)
(D) | FGWS | a $\\!\overset{5.69}{\textbf{smart}}$ $\overset{0.00}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{impertinent}}}\text{]}}\\!$ $\\!\overset{5.77}{\textbf{sweet}}$ $\overset{1.79}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{odoriferous}}}\text{]}}\\!$ and playful romantic comedy |  positive (99.9%)

Figure 2: The detection methods applied to an adversarial example from the
PWWS attack against RoBERTa on SST-2. The words highlighted in bold, italic
and red were selected for replacement by the attack (A) and the detection
methods (D), the ones in bold and black denote the substitutions. The values
above the words denote their $\log_{e}$ frequencies.

###  4.3 Results

We report the results comparing FGWS to DISP on attacks against RoBERTa in
Table 2. Here, the true positive rate (TPR) represents the percentage of
successful adversarial examples that were correctly identified as such, and
the false positive rate (FPR) denotes the percentage of unperturbed sequences
that were identified as adversarial. The column Adv. gives the classification
accuracy on the perturbed sequences, and Restored acc. the model’s accuracy on
the adversarial sequences after transformation. We observe that FGWS best
restores the model’s classification accuracy across all comparisons, showing
it to be effective in mitigating the effects of the individual attacks.
Furthermore, FGWS outperforms DISP in terms of true positive rates and $F_{1}$
across the majority of experiments. These results show that, although
contextualized word representations (DISP) serve as a competitive method to
detect adversarial examples, relying solely on frequency-guided substitutions
(FGWS) shows to be more effective. Figure 2 provides an example adversarial
sequence generated with the PWWS attack and the two corresponding transformed
sequences using DISP and FGWS (see Appendix G for additional examples).

The results of NWS and FGWS against the CNN and LSTM models are shown in Table
3. We observe that FGWS outperforms NWS across all comparisons in terms of
restored model accuracy and in the majority of comparisons in terms of
$F_{1}$. Moreover, the direct comparison between NWS and FGWS again underlines
the importance of utilizing word frequencies as guidance for the word
substitutions: while NWS is not guided by word frequency characteristics to
perform the word replacements, we observe that FGWS outperforms NWS by a large
margin in most comparisons, demonstrating the effectiveness of mapping
infrequent words to their most frequent semantically similar counterparts to
detect adversarial examples.

###  4.4 FGWS on unperturbed data

We furthermore investigate the effect of FGWS on model performance on
unperturbed sequences after transformation. To do this, we transform the
sampled test sets using FGWS and evaluate classification accuracies after
sequence transformation. The differences in accuracy for the CNN, LSTM and
RoBERTa models before and after transformation are $0.0$%, $+1.0$% and $-0.2$%
for IMDb and $-1.8$%, $-2.9$% and $-1.8$% for SST-2. This indicates that FGWS
applied to unperturbed data has only small effects on classification accuracy,
and in some cases even slightly increases prediction accuracy.

##  5 Limitations

It is worth mentioning that compared to FGWS, DISP represents a more general
perturbation discrimination approach since it is trained to detect both
character- and word-level adversarial perturbations, whereas FGWS solely
focuses on word-level attacks.

Furthermore, it remains open whether FGWS would be effective against attacks
for which the frequency difference is less evident. To investigate this, we
conducted preliminary experiments by restricting the investigated attacks to
only allow equifrequent substitutions. However, we observed that introducing
this constraint has a substantial effect on attack performance, since the
attacks are supplied with fewer candidate replacements. We will further
investigate this in future work.

##  6 Conclusion

We have shown that the word frequency characteristics of adversarial word
substitutions can be leveraged effectively to detect adversarial sequences for
neural text classification. Our proposed approach outperforms existing
detection methods despite representing a conceptually simpler approach to this
task.

## Acknowledgements

This research was supported by the Dawes Centre for Future Crime at University
College London.

## References

  * Alzantot et al. (2018) Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018.  Generating natural language adversarial examples.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2890–2896, Brussels, Belgium. Association for Computational Linguistics.
  * Belinkov and Bisk (2018) Yonatan Belinkov and Yonatan Bisk. 2018.  Synthetic and natural noise both break neural machine translation.  In _Proceedings of the International Conference on Learning Representations (ICLR)_.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_.
  * Chelba et al. (2013) Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2013.  One billion word benchmark for measuring progress in statistical language modeling.  Technical report, Google.
  * Cohen (1988) Jacob Cohen. 1988.  _Statistical power analysis for the behavioral sciences_.  Academic press.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ebrahimi et al. (2018) Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018.  HotFlip: White-box adversarial examples for text classification.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 31–36, Melbourne, Australia. Association for Computational Linguistics.
  * Fellbaum (1998) Christiane Fellbaum, editor. 1998.  _WordNet: an electronic lexical database_.  MIT Press.
  * Gao et al. (2018) J. Gao, J. Lanchantin, M. L. Soffa, and Y. Qi. 2018.  Black-box generation of adversarial text sequences to evade deep learning classifiers.  In _2018 IEEE Security and Privacy Workshops (SPW)_ , pages 50–56.
  * Glockner et al. (2018) Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018.  Breaking NLI systems with sentences that require simple lexical inferences.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 650–655, Melbourne, Australia. Association for Computational Linguistics.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural Computation_ , 9(8):1735–1780.
  * Huang et al. (2019) Po-Sen Huang, Robert Stanforth, Johannes Welbl, Chris Dyer, Dani Yogatama, Sven Gowal, Krishnamurthy Dvijotham, and Pushmeet Kohli. 2019.  Achieving verified robustness to symbol substitutions via interval bound propagation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4081–4091, Hong Kong, China. Association for Computational Linguistics.
  * Iyyer et al. (2018) Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018.  Adversarial example generation with syntactically controlled paraphrase networks.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1875–1885, New Orleans, Louisiana. Association for Computational Linguistics.
  * Jia and Liang (2017) Robin Jia and Percy Liang. 2017.  Adversarial examples for evaluating reading comprehension systems.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2021–2031, Copenhagen, Denmark. Association for Computational Linguistics.
  * Jia et al. (2019) Robin Jia, Aditi Raghunathan, Kerem Göksel, and Percy Liang. 2019.  Certified robustness to adversarial word substitutions.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4120–4133, Hong Kong, China. Association for Computational Linguistics.
  * Jin et al. (2019) Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2019.  Is bert really robust? a strong baseline for natural language attack on text classification and entailment.  _arXiv preprint arXiv:1907.11932_.
  * Kim (2014) Yoon Kim. 2014.  Convolutional neural networks for sentence classification.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1746–1751, Doha, Qatar. Association for Computational Linguistics.
  * Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  In _Proceedings of the 3rd International Conference on Learning Representations (ICLR)_.
  * Lei et al. (2019) Qi Lei, Lingfei Wu, Pin-Yu Chen, Alexandros G Dimakis, Inderjit S Dhillon, and Michael Witbrock. 2019.  Discrete adversarial attacks and submodular optimization with applications to text classification.  In _SysML 2019_.
  * Li et al. (2016a) Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016a.  Visualizing and understanding neural models in NLP.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 681–691, San Diego, California. Association for Computational Linguistics.
  * Li et al. (2016b) Jiwei Li, Will Monroe, and Dan Jurafsky. 2016b.  Understanding neural networks through representation erasure.  _arXiv preprint arXiv:1612.08220_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
  * Papernot et al. (2016) Nicolas Papernot, Patrick Drew McDaniel, Ananthram Swami, and Richard Harang. 2016\.  Crafting adversarial input sequences for recurrent neural networks.  In _MILCOM 2016 - 2016 IEEE Military Communications Conference_ , Proceedings - IEEE Military Communications Conference MILCOM, pages 49–54, United States. Institute of Electrical and Electronics Engineers Inc.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI Blog_ , 1(8).
  * Ren et al. (2019) Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019.  Generating natural language adversarial examples through probability weighted word saliency.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1085–1097, Florence, Italy. Association for Computational Linguistics.
  * Ribeiro et al. (2018) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018.  Semantically equivalent adversarial rules for debugging NLP models.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 856–865, Melbourne, Australia. Association for Computational Linguistics.
  * Rouder et al. (2009) Jeffrey N Rouder, Paul L Speckman, Dongchu Sun, Richard D Morey, and Geoffrey Iverson. 2009.  Bayesian t tests for accepting and rejecting the null hypothesis.  _Psychonomic bulletin & review_, 16(2):225–237.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: A simple way to prevent neural networks from overfitting.  _Journal of Machine Learning Research_ , 15:1929–1958.
  * Szegedy et al. (2014) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014.  Intriguing properties of neural networks.  In _International Conference on Learning Representations_.
  * Tsai et al. (2019) Yi-Ting Tsai, Min-Chu Yang, and Han-Yu Chen. 2019.  Adversarial attack on sentiment classification.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 233–240, Florence, Italy. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_ , pages 5998–6008. Curran Associates, Inc.
  * Wagenmakers et al. (2011) Eric-Jan Wagenmakers, Ruud Wetzels, Denny Borsboom, and Han LJ Van Der Maas. 2011\.  Why psychologists must change the way they analyze their data: the case of psi: comment on bem (2011). _Journal of Personality and Social Psychology_ , 100(3):426 – 432\.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _ArXiv_ , abs/1910.03771.
  * Zang et al. (2020) Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. 2020.  Word-level textual adversarial attacking as combinatorial optimization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6066–6080, Online. Association for Computational Linguistics.
  * Zhang et al. (2019) Huangzhao Zhang, Hao Zhou, Ning Miao, and Lei Li. 2019.  Generating fluent adversarial examples for natural languages.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5564–5569, Florence, Italy. Association for Computational Linguistics.
  * Zhou et al. (2019) Yichao Zhou, Jyun-Yu Jiang, Kai-Wei Chang, and Wei Wang. 2019.  Learning to discriminate perturbations for blocking adversarial attacks in text classification.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4903–4912, Hong Kong, China. Association for Computational Linguistics.

Dataset | Model | Attack | Replaced | Subst. | non-OOV
---|---|---|---|---|---
$\mu_{\phi}$ | $\sigma_{\phi}$ | $\mu_{\phi}$ | $\sigma_{\phi}$ | $d$ | $\mu_{\phi}$ | $\sigma_{\phi}$ | $d$
IMDb | CNN | Random | 7.6 | 2.5 | 3.5 | 2.8 | 1.6 | 4.4 | 2.4 | 1.3
Prioritized | 7.6 | 2.5 | 3.3 | 2.7 | 1.6 | 3.9 | 2.5 | 1.5
Genetic | 6.3 | 2.0 | 3.5 | 2.2 | 1.3 | 3.7 | 2.1 | 1.3
PWWS | 6.7 | 2.3 | 4.0 | 2.4 | 1.1 | 4.5 | 2.1 | 1.0
LSTM | Random | 7.6 | 2.5 | 3.5 | 2.8 | 1.6 | 4.4 | 2.4 | 1.3
Prioritized | 7.6 | 2.5 | 2.8 | 2.3 | 2.0 | 3.2 | 2.2 | 1.8
Genetic | 6.2 | 2.0 | 3.1 | 1.9 | 1.6 | 3.3 | 1.8 | 1.5
PWWS | 6.4 | 2.2 | 3.5 | 2.1 | 1.4 | 3.7 | 1.9 | 1.3
SST-2 | CNN | Random | 5.4 | 2.5 | 2.0 | 2.3 | 1.4 | 3.8 | 1.7 | 0.7
Prioritized | 5.4 | 2.5 | 2.4 | 2.1 | 1.3 | 3.5 | 1.7 | 0.8
Genetic | 4.3 | 1.8 | 2.2 | 1.9 | 1.2 | 3.2 | 1.4 | 0.6
PWWS | 4.8 | 2.1 | 2.8 | 2.1 | 1.0 | 3.8 | 1.5 | 0.6
LSTM | Random | 5.4 | 2.6 | 2.0 | 2.3 | 1.4 | 3.8 | 1.7 | 0.7
Prioritized | 5.4 | 2.5 | 2.3 | 2.1 | 1.3 | 3.4 | 1.6 | 0.9
Genetic | 4.3 | 1.7 | 2.0 | 1.9 | 1.3 | 3.1 | 1.4 | 0.8
PWWS | 4.8 | 2.0 | 2.7 | 2.1 | 1.0 | 3.7 | 1.4 | 0.6

Table 4: Mean $\log_{e}$ frequencies of replaced words and their corresponding
substitutions by attack, model and dataset. The shown values are the mean
$\mu_{\phi}$ and standard deviation $\sigma_{\phi}$ of the $\log_{e}$
frequencies corresponding to each setting, and additionally the Cohen’s $d$
effect sizes for the substitutions. Values in bold denote largest effect sizes
per dataset and model.

##  Appendix A Dataset statistics

The SST-2 dataset comes with a pre-defined split of 67,349 samples for
training, 872 for validation and 1,821 for testing. The IMDb dataset consists
of 50,000 positive and negative movie reviews with a pre-defined split of
25,000 training and 25,000 test samples. Since this dataset does not have a
pre-defined validation set, we hold out 1,000 randomly selected training set
samples for validation. We select a validation set of roughly the same size as
for SST-2 for fair comparisons when tuning parameters for adversarial example
detection. To the best of our knowledge, the compared work Alzantot et al.
(2018); Ren et al. (2019) does not validate model performance on held-out
training data.

Dataset | Model | Attack | Subst. | non-OOV
---|---|---|---|---
IMDb | CNN | Random | $>10^{10594}$ | $>10^{7004}$
Prioritized | $>10^{6549}$ | $>10^{5009}$
Genetic | $>10^{2581}$ | $>10^{2318}$
PWWS | $>10^{2182}$ | $>10^{1673}$
LSTM | Random | $>10^{9643}$ | $>10^{6338}$
Prioritized | $>10^{5949}$ | $>10^{4967}$
Genetic | $>10^{2550}$ | $>10^{2369}$
PWWS | $>10^{1666}$ | $>10^{1442}$
RoBERTa | Random | $>10^{12138}$ | $>10^{7948}$
Prioritized | $>10^{9014}$ | $>10^{6043}$
Genetic | $>10^{4215}$ | $>10^{3672}$
PWWS | $>10^{5182}$ | $>10^{3656}$
SST-2 | CNN | Random | $>10^{754}$ | $>10^{138}$
Prioritized | $>10^{573}$ | $>10^{222}$
Genetic | $>10^{388}$ | $>10^{104}$
PWWS | $>10^{397}$ | $>10^{131}$
LSTM | Random | $>10^{800}$ | $>10^{153}$
Prioritized | $>10^{648}$ | $>10^{264}$
Genetic | $>10^{522}$ | $>10^{148}$
PWWS | $>10^{456}$ | $>10^{144}$
RoBERTa | Random | $>10^{867}$ | $>10^{149}$
Prioritized | $>10^{779}$ | $>10^{130}$
Genetic | $>10^{584}$ | $>10^{55}$
PWWS | $>10^{600}$ | $>10^{125}$

Table 5: Bayes factors ($\mathrm{BF}_{10}$) for the Bayesian hypothesis tests.

##  Appendix B Model and attack details

###  B.1 RoBERTa

We utilize a pre-trained RoBERTa (base) model Liu et al. (2019) provided by
the Hugging Face Transformers library Wolf et al. (2019). We use maximum input
sequence lengths of 256 and 128 after byte-pair encoding Sennrich et al.
(2016) for the IMDb and SST-2 datasets, respectively. The RoBERTa model
consists of 125 million
parameters.888https://github.com/pytorch/fairseq/tree/master/examples/roberta
The model was trained for $10$ epochs with batch size $32$ (SST-2) and $16$
(IMDb) and a learning rate of $1\cdot 10^{-5}$. We evaluated model performance
after each epoch on the validation set and selected the best-performing
checkpoints for testing.

###  B.2 CNN/LSTM

The CNN architecture consists of $3$ convolutional layers with kernel sizes
$2$, $3$ and $4$ and $100$ feature maps for each convolutional layer. The LSTM
operates on a hidden state size of $128$. Following Alzantot et al. (2018), we
initialize the LSTM with pre-trained GloVe Pennington et al. (2014) word
embeddings, and do the same for the CNN.

Both the LSTM and the CNN use Dropout Srivastava et al. (2014) during training
with a rate of $0.1$ before applying the output layer. We trained both models
for $20$ epochs using the Adam optimizer (Kingma and Ba, 2014). We evaluated
model performance after each epoch on the validation set and selected the
best-performing checkpoints for testing. The CNN and LSTM models were trained
with batch size $100$ and a learning rate of $1\cdot 10^{-3}$.

###  B.3 PWWS

Our implementation of PWWS is based on the code as provided by Ren et al.
(2019) on GitHub.999https://github.com/JHL-HUST/PWWS

###  B.4 Genetic

Note that we utilize a different language model for the Perturb subroutine as
compared to the original implementation by Alzantot et al. (2018). While
Alzantot et al. (2018) employ the Google $1$ billion words language model
Chelba et al. (2013), we instead utilize the recently proposed GPT-2 language
model Radford et al. (2019) and compute the sequences’ perplexity scores using
the exponentialized language modelling loss (we employ the pre-trained
GPT2LMHeadModel language model from Wolf et al. (2019)). We compute the
perplexity scores for each perturbed sequence only around the respective
replacement words by only considering a subsequence ranging from five words
before to five words after an inserted replacement. The motivation for using a
different language model as compared to the original implementation is due to
computational efficiency, since we observed a notable decrease in attack
runtime with our modification. This does not have an impact on attack
performance, since our implementation of the Genetic has an attack success
rate of 98.6% against the LSTM on IMDb, whereas Alzantot et al. (2018) report
an attack success rate of 97%.

For attacks against SST-2, we furthermore increase the $\delta$ threshold for
the maximum distance between replaced words and substitutions to $\delta=1.0$,
since we observed poor attack performances with $\delta=0.5$ (which was used
by Alzantot et al. (2018) and in our experiments on IMDb). All other
parameters of the attack (e.g., the number of generations and population size)
are directly adapted from Alzantot et al. (2018).

We restrict the words eligible for replacement by the Genetic attack to non-
stopwords, in accordance to Alzantot et al. (2018). Since the attack computes
nearest neighbors for a selected word from a pre-trained embedding space, we
furthermore can only select words for which there exists an embedding
representation in this pre-trained space. On the SST-2 test set, we found
three input sequences consisting of only one word which we excluded from our
evaluation, since the used GPT-2 language model implementation requires an
input sequence consisting of more than one word.

(a) Random on SST-2 (b) Random on IMDb

(c) Prioritized on SST-2 (d) Prioritized on IMDb

(e) Genetic on SST-2 (f) Genetic on IMDb

(g) PWWS on SST-2 (h) PWWS on IMDb

Figure 3: Histograms showing the frequency distribution of words replaced by
the attacks and their corresponding substitutions against the RoBERTa model.
The $x$-axis represents the words’ $\log_{e}$ frequency with respect to the
model’s training corpus, the $y$-axis denotes their respective frequencies
among the perturbed test set sequences.

###  B.5 Random, Prioritized, PWWS, Genetic

For the Genetic attack, we follow Alzantot et al. (2018) by limiting the
maximum amount of word replacements to 20% of the input sequence length. We
apply the same threshold to the Random and Prioritized attacks, but not to
PWWS since we observed low replacement rates despite the attack’s
effectiveness. This is in agreement to the results reported in Ren et al.
(2019).

##  Appendix C Frequency differences for CNN and LSTM models

The $\log_{e}$ frequencies for the four attacks against the CNN and LSTM
models can be found in Table 4. In accordance to the experiments with RoBERTa
(see Section 3 in the paper), we observe large Cohen’s $d$ effect sizes for
the majority of the comparisons, which shows that the statistical frequency
differences between replaced words and their substitutions are present for
adversarial attacks against these two models as well.

##  Appendix D Bayes factors

The Bayes factors for the mean frequency comparisons between replaced words
and their adversarial substitutions can be found in Table 5. We observe high
values for $\mathrm{BF}_{10}$ across all comparisons, providing strong
evidence for the hypothesis that the $\log_{e}$ frequency means between
replaced words and their substitutions are different.

(a) IMDb (b) SST-2

Figure 4: The trade-off between true and false positive rates on the test sets
with all four attacks against RoBERTa on (a) IMDb and (b) SST-2. The true
positive rates ($y$-axis) are computed when $\gamma$ is set to allow for
different quasi-fixed amounts of false positives ($x$-axis).

##  Appendix E Visualizations of frequency differences

Figure 3 illustrates the frequency differences for attacks against the RoBERTa
model using histograms. We observe that for the majority of the attacks, OOV
substitutions occur most often among the perturbed sequences.

##  Appendix F Varying false positive thresholds

The rate of false positives predicted by a detection system is crucial for its
practicability, and a limited amount of false positives is hence highly
desirable. Figure 4 illustrates the true positive rates predicted by FGWS for
all attacks against RoBERTa with different quasi-fixed false positive
thresholds (as in Section 4.2 of the paper, $\delta$ was tuned on the
validation set for each value of $\gamma$ corresponding to the specific false
positive threshold). As expected, we observe a trade-off between true and
false positive rates for varying values of $\gamma$, such that lower false
positive rates imply lower true positive rates. However, even for false
positive rates of 1% and 5%, we observe that FGWS is able to detect between
33.6% and 90.0% of adversarial examples on IMDb and between 31.7% and 67.2% on
SST-2. This indicates that FGWS has the potential to detect a useful fraction
of adversarial examples without creating an excessive burden of false
positives.

##  Appendix G Additional FGWS examples

Additional examples of FGWS can be found in Table 6 (true positives), Table 7
(false positives), Table 8 (true negatives) and Table 9 (false negatives).

Model:RoBERTa on SST-2

Unperturbed |  first good then bothersome |  negative (74.5%)
---|---|---
Genetic |  first good then $\\!\overset{0.00}{\textbf{galling}}$ $\overset{0.00}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{bothersome}}}\text{]}}\\!$ |  positive (88.7%)
DISP |  first good $\\!\overset{8.96}{\textbf{that}}$ $\overset{5.31}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{then}}}\text{]}}\\!$ galling |  positive (84.8%)
FGWS |  first good then $\\!\overset{4.32}{\textbf{annoying}}$ $\overset{0.00}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{galling}}}\text{]}}\\!$ |  negative (91.3%)

Model:RoBERTa on IMDb

Unperturbed |  i am a huge rupert everett fan . i adore kathy bates so when i saw it available i decided to check it out . the synopsis didn t really tell you much . in parts it was silly touching and in others some parts were down right hysterical . any person that is a huge fan of a personality of any type will find some small identifying traits with the main character . of course there are many they won t but that is the point if you like any of the actors give it a watch but don t look for any thing too dramatic it s good fun . i might also mention you can see how darn tall rupert is . i mean i knew he was 6 4 but he seems even more in this film . he even seemed to stoop a bit due to the other characters height in this . he is tall i mean tall and for you rupert fans there is a bare chest scene … wonderful |  positive (99.2%)
---|---|---
PWWS |  i am a huge rupert everett fan . i adore kathy bates so when i saw it available i decided to $\\!\overset{6.54}{\textbf{stop}}$ $\overset{6.19}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{check}}}\text{]}}\\!$ it out . the synopsis didn t really tell you much . in parts it was silly touching and in others some parts were down right hysterical . any person that is a huge fan of a personality of any type will find some small identifying traits with the main character . of course there are many they won t but that is the point if you like any of the actors give it a watch but don t look for any thing too dramatic it s $\\!\overset{0.00}{\textbf{undecomposed}}$ $\overset{9.22}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{good}}}\text{]}}\\!$ fun . i might also mention you can see how darn tall rupert is . i mean i knew he was 6 4 but he seems even more in this film . he even seemed to stoop a bit $\\!\overset{0.00}{\textbf{imputable}}$ $\overset{6.31}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{due}}}\text{]}}\\!$ to the other characters height in this . he is tall i mean tall and for you rupert fans there is a bare chest scene … $\\!\overset{4.45}{\textbf{tremendous}}$ $\overset{7.08}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{wonderful}}}\text{]}}\\!$ |  negative (60.1%)
DISP |  i am a huge rupert everett fan . i adore kathy bates so when i saw it available i decided to $\\!\overset{9.27}{\textbf{out}}$ $\overset{6.54}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{stop}}}\text{]}}\\!$ it out . the synopsis didn t really tell you much . in parts it was silly touching and in others some parts were down right hysterical . any person that is a huge fan of a personality of any type will find some small identifying traits with the main character . of course there are many they won $\\!\overset{0.00}{\textbf{,}}$ $\overset{9.97}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{t}}}\text{]}}\\!$ but that is the point if you like any of the actors give it a watch but don t look for any thing too dramatic it $\\!\overset{0.00}{\textbf{'s}}$ $\overset{10.54}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{s}}}\text{]}}\\!$ $\\!\overset{9.50}{\textbf{so}}$ $\overset{0.00}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{undecomposed}}}\text{]}}\\!$ fun . i $\\!\overset{9.20}{\textbf{can}}$ $\overset{7.44}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{might}}}\text{]}}\\!$ also mention you can see how darn tall rupert is . i mean i knew he was 6 4 but he seems even more in this film . he even seemed to stoop a bit imputable to the other characters height in this . he is tall i mean tall and for you rupert fans there is a bare chest scene … $\\!\overset{12.05}{\textbf{.}}$ $\overset{4.45}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{tremendous}}}\text{]}}\\!$ |  positive (92.0%)
FGWS |  i am a huge rupert everett fan . i adore kathy bates so when i saw it available i decided to stop it out . the synopsis didn t really tell you much . in parts it was silly touching and in others some parts were down right hysterical . any person that is a huge fan of a personality of any type will find some small $\\!\overset{7.26}{\textbf{place}}$ $\overset{2.48}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{identifying}}}\text{]}}\\!$ traits with the main character . of course there are many they won t but that is the point if you like any of the actors give it a watch but don t look for any thing too dramatic it s $\\!\overset{9.22}{\textbf{good}}$ $\overset{0.00}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{undecomposed}}}\text{]}}\\!$ fun . i might also mention you can see how darn tall rupert is . i mean i knew he was 6 4 but he seems even more in this film . he even seemed to $\\!\overset{6.22}{\textbf{sit}}$ $\overset{2.40}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{stoop}}}\text{]}}\\!$ a bit $\\!\overset{6.31}{\textbf{due}}$ $\overset{0.00}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{imputable}}}\text{]}}\\!$ to the other characters height in this . he is tall i mean tall and for you rupert fans there is a bare chest scene … tremendous |  positive (88.9%)

Table 6: Illustration of true positives generated with FGWS against RoBERTa on
SST-2 (top) and IMDb (bottom). The substitutions caused the model to change
the predicted label back to its ground-truth for the given adversarial
examples.

Model: RoBERTa on SST-2

Unperturbed |  imagine if you will a tony hawk skating video interspliced with footage from behind enemy lines and set to jersey shore techno |  negative (83.6%)
---|---|---
DISP |  imagine if you $\\!\overset{5.97}{\textbf{get}}$ $\overset{6.84}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{will}}}\text{]}}\\!$ a tony hawk skating video $\\!\overset{0.00}{\textbf{,}}$ $\overset{0.00}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{interspliced}}}\text{]}}\\!$ with footage from behind enemy lines and set to jersey shore techno |  negative (87.1%)
FGWS |  imagine if you will a $\\!\overset{3.76}{\textbf{kevin}}$ $\overset{1.10}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{tony}}}\text{]}}\\!$ $\\!\overset{3.93}{\textbf{pitch}}$ $\overset{2.48}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{hawk}}}\text{]}}\\!$ skating video interspliced with footage from behind enemy lines and set to $\\!\overset{6.55}{\textbf{new}}$ $\overset{2.08}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{jersey}}}\text{]}}\\!$ $\\!\overset{3.93}{\textbf{sea}}$ $\overset{2.08}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{shore}}}\text{]}}\\!$ $\\!\overset{5.69}{\textbf{music}}$ $\overset{1.61}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{techno}}}\text{]}}\\!$ |  positive (65.7%)

Model: RoBERTa on IMDb

Unperturbed |  admittedly alex has become a little podgey but they are still for me the greatest rock trio ever . i wholeheartedly recommend this dvd to any fan . i was very disappointed that they canceled their planned recent munich gig logistics and regret not making an effort to see them elsewhere . the dvd is a small consolation the greatest incentive to acquire a proper dvd playback setup . naive perhaps but i still don t understand the significance of the tumble driers on stage i would be grateful for any clarification . cheers iain . |  positive (99.4%)
---|---|---
DISP |  admittedly alex has become a little podgey but they are still for me the greatest rock trio ever . i wholeheartedly recommend this dvd to any fan . i was very disappointed that they canceled their planned recent munich gig logistics and regret not making an effort to see them elsewhere . the dvd is a small consolation the greatest incentive to acquire a proper dvd playback setup . naive perhaps but i still don t understand the significance of the $\\!\overset{9.77}{\textbf{one}}$ $\overset{1.95}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{tumble}}}\text{]}}\\!$ driers on stage i would be grateful for any clarification . cheers $\\!\overset{10.73}{\textbf{that}}$ $\overset{0.00}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{iain}}}\text{]}}\\!$ . |  positive (99.3%)
FGWS |  admittedly alex has become a little podgey but they are still for me the greatest rock trio ever . i $\\!\overset{4.55}{\textbf{disagree}}$ $\overset{2.30}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{wholeheartedly}}}\text{]}}\\!$ recommend this dvd to any fan . i was very disappointed that they canceled their planned recent $\\!\overset{5.03}{\textbf{germany}}$ $\overset{2.08}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{munich}}}\text{]}}\\!$ gig $\\!\overset{3.40}{\textbf{transport}}$ $\overset{0.69}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{logistics}}}\text{]}}\\!$ and regret not making an effort to see them elsewhere . the dvd is a small $\\!\overset{5.69}{\textbf{win}}$ $\overset{2.08}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{consolation}}}\text{]}}\\!$ the greatest $\\!\overset{5.53}{\textbf{opportunity}}$ $\overset{1.61}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{incentive}}}\text{]}}\\!$ to acquire a proper dvd $\\!\overset{6.21}{\textbf{editing}}$ $\overset{2.08}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{playback}}}\text{]}}\\!$ setup . naive perhaps but i still don t understand the significance of the $\\!\overset{6.19}{\textbf{fall}}$ $\overset{1.95}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{tumble}}}\text{]}}\\!$ $\\!\overset{1.61}{\textbf{dryer}}$ $\overset{0.00}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{driers}}}\text{]}}\\!$ on stage i would be grateful for any $\\!\overset{5.11}{\textbf{explanation}}$ $\overset{1.10}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{clarification}}}\text{]}}\\!$ . cheers iain . |  negative (50.1%)

Table 7: Illustration of false positives generated with FGWS against RoBERTa
on SST-2 (top) and IMDb (bottom). The substitutions caused the model to change
the predicted label for the given unperturbed sequences.

Model: RoBERTa on SST-2

Unperturbed |  it s a hoot and a half and a great way for the american people to see what a candidate is like when he s not giving the same 15 cent stump speech |  positive (100.0%)
---|---|---
DISP |  it $\\!\overset{0.00}{\textbf{'s}}$ $\overset{9.09}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{s}}}\text{]}}\\!$ a hoot and a half and a great way for the american people to see what a candidate is like when he $\\!\overset{0.00}{\textbf{'s}}$ $\overset{9.09}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{s}}}\text{]}}\\!$ not giving the same 15 $\\!\overset{6.01}{\textbf{minutes}}$ $\overset{0.00}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{cent}}}\text{]}}\\!$ $\\!\overset{10.22}{\textbf{the}}$ $\overset{0.00}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{stump}}}\text{]}}\\!$ speech |  positive (100.0%)
FGWS |  it s a hoot and a half and a great way for the american people to see what a $\\!\overset{3.71}{\textbf{nomination}}$ $\overset{1.95}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{candidate}}}\text{]}}\\!$ is like when he s not giving the same 15 cent $\\!\overset{2.48}{\textbf{stamp}}$ $\overset{0.00}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{stump}}}\text{]}}\\!$ $\\!\overset{4.45}{\textbf{words}}$ $\overset{0.00}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{speech}}}\text{]}}\\!$ |  positive (100.0%)

Model: RoBERTa on IMDb

Unperturbed |  it was awful plain and simple . what was their message where was the movie going with this it has all the ingredients of a sub b grade movie . from plotless storyline the bad acting to the cheesey slow mo cinematography . i d sooner watch a movie i ve already seen like goodfellas a bronx tale even grease . there are no likeable characters . in the end you just want everyone to die already . save 2 hours of your life and skip this one . |  negative (99.9%)
---|---|---
DISP |  it was awful plain and simple . what was their message where was the movie going with this it has all the ingredients of a sub b grade movie . from plotless storyline the bad acting to the cheesey slow mo cinematography . i $\\!\overset{8.94}{\textbf{would}}$ $\overset{7.56}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{d}}}\text{]}}\\!$ sooner watch a movie i $\\!\overset{9.79}{\textbf{have}}$ $\overset{8.17}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{ve}}}\text{]}}\\!$ already seen like goodfellas a bronx tale $\\!\overset{10.97}{\textbf{in}}$ $\overset{8.97}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{even}}}\text{]}}\\!$ grease . there are no likeable characters . in the end you just want everyone to die already . save 2 hours of your life and skip this one . |  negative (99.9%)
FGWS |  it was awful plain and simple . what was their message where was the movie going with this it has all the ingredients of a sub b grade movie . from $\\!\overset{4.28}{\textbf{unwatchable}}$ $\overset{1.39}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{plotless}}}\text{]}}\\!$ storyline the bad acting to the $\\!\overset{6.10}{\textbf{cheesy}}$ $\overset{1.95}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{cheesey}}}\text{]}}\\!$ slow mo cinematography . i d sooner watch a movie i ve already seen like goodfellas a bronx tale even grease . there are no likeable characters . in the end you just want everyone to die already . save 2 hours of your life and skip this one . |  negative (99.9%)

Table 8: Illustration of true negatives generated with FGWS against RoBERTa on
SST-2 (top) and IMDb (bottom). The substitutions did not cause the model to
change the predicted label for the given unperturbed sequences.

Model: RoBERTa on SST-2

Unperturbed |  the spark of special anime magic here is unmistakable and hard to resist |  positive (100.0%)
---|---|---
PWWS |  the spark of special anime $\\!\overset{2.83}{\textbf{deception}}$ $\overset{4.52}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{magic}}}\text{]}}\\!$ here is unmistakable and $\\!\overset{2.77}{\textbf{laborious}}$ $\overset{6.15}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{hard}}}\text{]}}\\!$ to $\\!\overset{4.58}{\textbf{hold}}$ $\overset{3.91}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{resist}}}\text{]}}\\!$ |  negative (84.4%)
DISP |  the spark of special anime deception here is unmistakable and $\\!\overset{4.88}{\textbf{able}}$ $\overset{2.77}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{laborious}}}\text{]}}\\!$ to hold |  positive (99.9%)
FGWS |  the spark of special anime deception here is $\\!\overset{4.52}{\textbf{subtle}}$ $\overset{2.48}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{unmistakable}}}\text{]}}\\!$ and laborious to hold |  negative (97.8%)

Model: RoBERTa on IMDb

Unperturbed |  graduation day is a result of the success of friday the 13th . both of those films are about creative bloody murders rather than suspense . if you enjoy that type of film i d recommend graduation day . if not i wouldn t. there s nothing new here just the same old killings . even though i ve given the film a 4 out of 10 i will say that it s not a repulsive film . it is watchable if your curious about it just not creative . |  negative (71.3%)
---|---|---
Genetic |  graduation day is a result of the success of friday the 13th . both of those films are about creative bloody murders rather than suspense . if you enjoy that type of film i d recommend graduation day . if not i wouldn t. there s nothing new here just the same $\\!\overset{5.06}{\textbf{ancient}}$ $\overset{8.03}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{old}}}\text{]}}\\!$ killings . even though i ve given the film a 4 out of 10 i will say that it s not a repulsive film . it is watchable if your curious about it just not creative . |  positive (53.5%)
DISP |  graduation day is a result of the success of friday the 13th . both of those films are about creative bloody murders rather than suspense . if you enjoy that type of film i $\\!\overset{8.94}{\textbf{would}}$ $\overset{7.56}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{d}}}\text{]}}\\!$ recommend graduation day . if not i $\\!\overset{8.64}{\textbf{do}}$ $\overset{6.48}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{wouldn}}}\text{]}}\\!$ t. there $\\!\overset{11.14}{\textbf{is}}$ $\overset{10.54}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{s}}}\text{]}}\\!$ nothing new here just the same ancient killings . even though i $\\!\overset{9.79}{\textbf{have}}$ $\overset{8.17}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{ve}}}\text{]}}\\!$ given the film a 4 out of 10 i will say that it $\\!\overset{0.00}{\textbf{'s}}$ $\overset{10.54}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{s}}}\text{]}}\\!$ not a $\\!\overset{9.22}{\textbf{good}}$ $\overset{3.69}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{repulsive}}}\text{]}}\\!$ film . it is watchable if your curious about it $\\!\overset{11.14}{\textbf{is}}$ $\overset{9.32}{\text{[}{\color[rgb]{0.86328125,0.1953125,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.1953125,0.18359375}\textit{{just}}}\text{]}}\\!$ not creative . |  negative (99.5%)
FGWS |  graduation day is a result of the success of friday the 13th . both of those films are about creative bloody murders rather than suspense . if you enjoy that type of film i d recommend graduation day . if not i wouldn t. there s nothing new here just the same ancient killings . even though i ve given the film a 4 out of 10 i will say that it s not a repulsive film . it is watchable if your curious about it just not creative . |  positive (53.5%)

Table 9: Illustration of false negatives generated with FGWS against RoBERTa
on SST-2 (top) and IMDb (bottom). The substitutions did not cause the model to
change the predicted label back to its ground-truth for the given adversarial
examples.