# Mischief: A Simple Black-Box Attack Against Transformer Architectures

Adrian de Wynter
Amazon Alexa
<EMAIL_ADDRESS>

###### Abstract

We introduce Mischief, a simple and lightweight method to produce a class of
human-readable, realistic adversarial examples for language models. We perform
exhaustive experimentations of our algorithm on four transformer-based
architectures, across a variety of downstream tasks, as well as under varying
concentrations of said examples. Our findings show that the presence of
Mischief-generated adversarial samples in the test set significantly degrades
(by up to 20%) the performance of these models with respect to their reported
baselines. Nonetheless, we also demonstrate that, by including similar
examples in the training set, it is possible to restore the baseline scores on
the adversarial test set. Moreover, for certain tasks, the models trained with
Mischief set show a modest increase on performance with respect to their
original, non-adversarial baseline.

##  1 Introduction

An adversarial attack on deep learning systems, as introduced by ?) and ?),
consists on any input that may be designed explicitly to cause poor
performance on a model. They are traditionally split in two major categories:
white-box and black-box. In the former, the adversarial inputs are found–or
rather, learned–through a perturbation of the gradient. The latter, on the
other hand, assumes that there is no access to the model’s gradient, and thus
said adversarial examples are often found by trial-and-error.

In computer vision, such attacks typically involve injecting learned noise
into small areas of the input image. This noise is unnoticeable to a human
user, but just about complex enough to cause the network to fail to perform as
expected. In contrast, for text-based systems, this noticeability-versus-
failure tradeoff is not as clear. Since machine learning-based language models
embed an input string into a vector space for further processing in other
tasks, from a feasibility point of view it would be more realistic to
determine which changes in the string, and not the vector, are more likely to
negatively affect the model.

In an attempt to make our work more readily applicable to existent systems, we
concentrate ourselves solely on such black-box attacks. Moreover, we focus
mostly on architectures that leverage the transformer layer from ?), such as
BERT [Devlin et al. (2018], as their high performance in multiple language
modeling tasks makes them ubiquitous in both research and production
pipelines.

In this work we present Mischief, a procedure that generates a class of such
adversarial examples. In order to remain within our constraints, Mischief
leverages a well-known phenomenon from psycholinguistics first described by
?). We characterize the impact of our algorithm on the performance of four
selected transformer-based architectures, by carrying out exhaustive
experiments across a variety of tasks and concentrations of adversarial
examples.

Our experimentation shows that the presence of Mischief-generated examples is
able to significantly downgrade the performance of the language models
evaluated. However, we also demonstrate that, at least for the architectures
evaluated, including Mischief-generated examples into the training process
allows the models to regain, and sometimes increase, their baseline
performance in a variety of downstream tasks.

##  2 Related Work

Adversarial attacks in the context of learning theory were perhaps first
described by ?). However, such examples _per se_ predate machine learning
(e.g., techniques to circumvent spam filters) by a wide margin [Ollman (2007].
On the other hand, the study of adversarial attacks on deep neural networks,
albeit relatively recent, fields a large number of important contributions in
addition to the ones mentioned on the introduction, and it is hard to name
them all. However, an excellent introduction on this topic, along with
historical notes, can be found in ?). In the context of Natural Language
Understanding (NLU), the work by ?) was arguably the first where such notions
were formally applied to the intersection of language modeling and deep
learning. Moreover, the well-known research by ?), ?), and ?) showed that the
large majority of existing language models are extremely vulnerable to both
black-box and white-box attacks. Indeed, the Mischief algorithm is similar to
that one of ?) with variations on noise levels, and applied over a wider range
of natural language tasks.

Nonetheless, the large majority of the procedures presented in such papers
were often considered to be unrealistic [Zhang et al. (2019], and it wasn’t
until ?) and ?) where more practical attack and defense mechanisms were
introduced. This paper is more closely aligned to theirs, although it differs
in key aspects regarding contributions and methodology. Regardless, our work
is meant to add to this body of research, with a specific focus on black box-
based sentence level attacks for transformer architectures. The interested
reader can find a more comprehensive compendium of the history of adversarial
attacks for NLU in the survey by ?).

We elaborate on a few studies of the research first reported by Graham
Rawlinson [Rawlinson (2007] in Section 3. In addition to these papers, it is
important to point out that there are quite a few papers around this
phenomenom. For example, the work by ?) and ?) expanded upon said research by
exploring other types of permutations; while ?) and ?) attempted to explain
this phenomenon from a statistical perspective, and an analysis of some of the
leading theories around positional encoding can be found in the articles by ?)
and ?). Finally, a compilation of the works around this effect can be found in
?).

##  3 Mischief

###  3.1 Background

Graham Rawlinson described in his doctoral thesis [Rawlinson (1976] a
phenomenon where permuting the middle characters in a word, but leaving the
first and last intact, had little to no effect on the ability of a human
reader to understand it. It was shown in a few other studies that said
permutation does tend to slow down readability [Rayner et al. (2006], and that
the type of permutation (i.e., the position of the permuted substring) is
relevant to the comprehension of the text [Schoonbaert and Grainger (2004], as
well as any context [Pelli et al. (2003] added.

It could be argued that the act of shuffling the characters in an input word
will have a naturally detrimental effect on any language model. Most models
rely on a tokenizer and a vocabulary to parse the input string; thus, the
presence of an adversarial example as an input to a pretrained model implies
that the input will very likely be mapped to a low-information, or even
incorrect, vocabulary element. On the other hand, the attention mechanism that
lies at the heart of the transformer architecture does not have a concept of
word order [Vaswani et al. (2017], and relies on statistical methods to learn
syntax [Peters et al. (2018]. It has also been shown to prefer, in some
architectures, certain specific tokens and other coreferent objects [Clark et
al. (2019, Kovaleva et al. (2019]. This suggests that, although models relying
on these artifacts may be resilient to slight changes in the input, the right
concentration of permuted words may lead to a degradation in performance–all
while remaining understandable by a human reader.

Note that we do not alter the order of the words in the sentence, as that may
risk losing a significant amount of semantic and lexical information, and thus
it will no longer be considered a practical adversarial attack.

###  3.2 The Mischief Algorithm

We define the Generalized Rawlinson adversarial example (GRA) as a permutation
$\sigma$ on a word $w=w_{1},w_{2},\dots,w_{n}$, where $n>3$, such that
GRA$(w)=w_{1},\sigma(w_{2}),\dots,\sigma(w_{n-1}),w_{n}$. The algorithm that
we use to generate such examples, which we call Mischief, is a function that
acts on a text corpus and takes in two parameters $p,r\in[0,1]$. Here, $p$
denotes the proportion of the dataset to perform “Mischief” on, and $r$ is the
probability of a word $w$ in a given line to be randomized; that is, to
perform GRA$(w)$. An implementation of Mischief can be seen in Algorithm 1.

Algorithm 1 Simplified Mischief: The $+$ operator denotes string
concatenation.

Input: dataset $D$, proportion $p$, probability $r$

for sentence $s\in D$ do

Draw probability $\pi_{s}\sim P$

if $\pi_{s}\leq p$ then

for word $w\in s$ do

Draw probability $\pi_{r}\sim P$

if $\pi_{r}\leq r\land|w|>3$ then

$w^{\prime}=w_{1}+\text{GRA}(w_{2:-2})+w_{-1}$

Replace $w\in s$ with $w^{\prime}$

end if

end for

end if

end for

##  4 Experimentation

We evaluate our approach with four transformer-based models: BERT (large,
cased) [Devlin et al. (2018], RoBERTa (large) [Liu et al. (2019], XLM
(2048-en) [Lample and Conneau (2019], and XLNet (large) [Yang et al. (2019].
All of them were selected due to their high performances on the Generalized
Language Understanding Evaluation
(GLUE)111https://gluebenchmark.com/leaderboard benchmarks, which is a set of
ten distinct NLU tasks designed to showcase the candidate’s ability to model
and generalize language [Wang et al. (2018].

For every model and task, we apply four different concentrations
$r=\\{25\%,50\%,75\%,100\%\\}$ of Mischief on the training set; additionally,
for each of these concentrations we test different combinations of Mischief-no
Mischief on the test and training set, totaling $640$ distinct experiments.
Due to the large complexity of the task, we maintain $p=1$ across all
experiments. A summary of our findings and general experimental set up is
described in Table 1.

Training Test |  Mischief |  No Mischief
---|---|---
No Mischief |  An adversarial attack: significant performance degradation. |  Baseline from the GLUE benchmarks.
Mischief |  Proposed defense: minimal performance degradation. |  Minimal degradation, or increased performance.
Table 1: Summary of our findings for all experiments, models, and variations.

###  4.1 Methodology

In order to obtain the performance of a model on a test set, an experimenter
must first upload the raw predictions to the GLUE website. Due to the number
of experiments we performed, along with our need to modify the test sets, we
opted out from evaluating every result on the website. Instead, we treated the
provided validation set as a test set, and generated a small validation set
from a $90\%-10\%$ split of the training set.

It is well-known that the language models we tested are highly sensitive to
initialization. Given that in most tasks we observed significant variation of
results accross multiple experiments, we report the average result for ten
random seeds, bringing up the total number of experiments to $6,\\!400$.
However, as done in ?), we opt to not report the CoLA or WNLI benchmarks, as
their small training set size made them remarkably sensitive to variations in
the experimentation, and their inclusion could bias our summary results for
the following sections.

###  4.2 Effects of Mischief in the Test Set

Our first set of experiments, corresponding to the first column of Table 1,
involved exploring the effects of a Mischief-generated adversarial test set,
as well as a simple defense schema. To simulate an adversarial attack, on our
first setup we fine-tuned the models on each GLUE task as described on their
original papers, and evaluated them on test sets with Mischief. Then, we
simulated a simple defense by applying Mischief to the training sets, and
subsequently fine-tuning and evaluating the models. The results can be seen in
Figure 1.

Figure 1: Resulting scores under an adversarial test set, and for two
situations: the adversarial setting (no Mischief on the training set, in red),
and our proposed defense (with Mischief on the training set, in blue). The
results are averaged out across all values of $r$, and measured in terms of
F$1$, Pearson correlation ($p$ in the plot), Spearman-$\rho$ ($s$ in the
plot), and accuracy (not indicated). Note the high variance in RTE, which we
attribute to the small size of the dataset.

We found that models trained without Mischief are vulnerable to adversarial
attacks, with performance drops averaging almost 20% on the case where
$r=25\%$. However, such degradation can be easily recovered by training with
Mischief, as well as performing minor hyperparameter tuning to compensate for
the variations in the new training set. A plot of the mean task degradation
observed by varying $r$ can be seen in Figure 2. We conjecture that the “dip”
at $r=25\%$ can be explained by the fact that the concentration of GRA
examples is enough to degrade the performance of the tested models, but not
sufficient to allow for learning.

Figure 2: Average (per-task) performance degradation, for varying proportions
of $r$. On average, applying Mischief to the training set is an effective
defense against this class of adversarial attacks. Note also how the
architectures have a consistent ordering in their performance degradation,
regardless of $r$.

###  4.3 Effects of Mischief in the Training Set

Our second set of experiments involve evaluating the performance on the
unmodified test dataset, after training the models with Mischief. The results
for every proportion $r$ can be seen in Figure 3. We observed increased
performances on various tasks. However, it does appear that the size of the
dataset, as well as the objective of the task, have an important influence as
to whether Mischief-trained models can have such performance increase.

This is to be expected, as certain tasks do rely more heavily on “masking”
certain tokens. For example, MRPC (the Microsoft Research Paraphrase Corpus,
by ?)), is, as it name indicates, a classification task where the model must
determine whether two sentences $p,q$ are paraphrases of one another. A
paraphrase of $p$ would normally retain most of the semantic content, while
altering the lexical relations as much as possible, in which Mischief clearly
allows for a more fine-grained data expansion at the tokenizer level. It is
also important to point out that MRPC is a relatively large dataset, with
$5801$ sentence pairs.

On the other hand, some other tasks would actually be harmed by the
unintentional “masking” induced by Mischief. As an example, RTE (Recognizing
Textual Entailment) is a dataset merged by ?) from the corpora by ?), ?), ?),
and ?). Its objective is to determine whether a pair of sentences $p,q$ have
the relation $p\implies q$. It could be argued that such a task cannot benefit
from Mischief, as it would lose critical lexical information and simply
obfuscate the dataset further. However, MNLI (the Multi-Genre NLI corpus, by
?)) also involves textual entailment, but it is significantly larger than RTE:
the latter is the smallest dataset presented in this paper, with $2769$
examples total, while the former is nearly $150$ times larger, topping about
$433\\!,000$ sentence pairs.

Figure 3: Resulting performance change, across all tasks, for every $r$. In
blue we report the best performance of a model trained with Mischief, and
evaluated on its original test set. In red, we report the baseline. In
general, large corpora consistently benefit from Mischief-based training.

###  4.4 Discussion

Mischief as an adversarial attack is remarkably effective, although its
ability to degrade the performance of a language model is, fortunately, easily
lost if the model has been exposed to other GRA samples before. We hypothesize
that this is, as mentioned in Section 3.1, due to the way these models
construct their vocabulary. The models tested employ a byte pair encoding
(BPE) preprocessing step [Gage (1994], which segments subwords iteratively and
stores them in the vocabulary based on their frequency. It follows that any
model trained on Mischief-generated samples will become more robust to the
perturbations induced by this algorithm. Moreover, the models tested have
large parameter sizes, which translates into a much stronger ability to
memorize, and thus be resilient to, new input examples.

This can also help partially explain the results observed in Section 4.3: let
$w_{i},w_{j}$ be two words occurring in different parts of the dataset, and
where $w_{i}=w_{j}$, and $|w_{i}|=|w_{j}|:=n$. For $n\geq 3$, and assuming a
uniform distribution, the probability that these two words are transformed the
same way is

$\text{Pr}[\text{GRA}(w_{i})=\text{GRA}(w_{j})]=\left(\frac{1}{(n-2)!}\right)^{2}$
(1)

which in turn means that Mischief is effectively a data augmentation
technique–the average English word length is $n=5.1$
characters.222https://www.wolframalpha.com/input/?i=average+english+word+length
Although this number would naturally vary with the corpus being utilized,
ultimately the models tested will be exposed to a wider variety of slight
perturbations on the inputs, and in turn allows it to focus better on the
linguistic relations between the tokens.

However, Equation 1 does not account for the fact that some tasks do not
benefit from Mischief. For example, the QQP (Quora Question Pairs) dataset
attempts to relate a question-answer pair
semantically333https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-
Question-Pairs, and Mischief-based models consistently underperformed in spite
of the fact that this corpus has nearly $400,\\!000$ lines. Given the scores
in STS-B [Cer et al. (2017], and SST-2 [Socher et al. (2013], it appears that,
generally speaking, tasks where semantic similarity is the primary measurement
are more likely to be impacted negatively. There were some exceptions to the
rule, however, as some models did outperform their baseline, for example, BERT
in STS-B for $r=100\%$ and XLNet in SST-2 for $r=75\%$, and $r=100\%$.

##  5 Conclusion

We presented Mischief, a simple algorithm that allows us to construct a class
of human-readable adversarial examples; and showed that the injection of such
examples in the dataset is capable of significantly degrading the performance
of transformer-based models. Such models can be made resistant to Mischief-
based attacks by simply training with similar examples, and without relying on
other components (e.g., a spell-checker).

However, Mischief has also value as a data augmentation technique, as we saw
that certain NLU tasks benefit from the inclusion of such examples. It is
important to point out that, in general, adversarial attacks are architecture-
independent [Szegedy et al. (2014]. Although we attempted to provide an in-
depth analysis of select transformer-based architectures, it remains an open
problem as to whether the results of this paper are applicable to other
families of models. We conjecture that, as long as their tokenizer operates in
a similar fashion to the WordPiece tokenizer from ?), and their parameter size
is large enough, the effects from this study extend to them. In the case of
smaller-capacity models or other word-segmentation techniques where out-of-
vocabulary words are frequently mapped to the same token, the outcome of a
Mischief-based attack can only be more detrimental.

Finally, one area we did not pursue in this paper is synonym injection. We
argue that synonym injection is arguably far more impactful in terms of
supplying strong adversarial examples, and a Mischief-based approach to
training with such examples may also increase performance in the tasks where
Mischief did not show an improvement. However, given how sensitive is the
meaning of a word–let alone their synonym–to context, such process cannot be
done in an automated fashion, and without expert knowledge being invested.

## Acknowledgments

The author is grateful to B. d’Iverno, Y. Ibrahim, V. Khare, A. Mottini, and
Q. Wang for their helpful comments and suggestions throughout this project,
and to the anonymous reviewers whose comments greatly improved this work.

## References

  * [Bar-Haim et al. (2006]  Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, and Danilo Giampiccolo.  2006\.  The second pascal recognising textual entailment challenge.  Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, 01.
  * [Belinkov and Bisk (2018]  Yonatan Belinkov and Yonatan Bisk.  2018\.  Synthetic and natural noise both break neural machine translation.  In International Conference on Learning Representations.
  * [Bentivogli et al. (2009]  Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo.  2009\.  The fifth pascal recognizing textual entailment challenge.  In TAC.
  * [Biggio and Roli (2018]  Battista Biggio and Fabio Roli.  2018\.  Wild patterns: Ten years after the rise of adversarial machine learning.  In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, page 2154–2156, New York, NY, USA. Association for Computing Machinery.
  * [Cer et al. (2017]  Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia.  2017\.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada, August. Association for Computational Linguistics.
  * [Clark et al. (2019]  Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning.  2019\.  What does BERT look at? an analysis of BERT’s attention.  In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy, August. Association for Computational Linguistics.
  * [Dagan et al. (2005]  Ido Dagan, Oren Glickman, and Bernardo Magnini.  2005\.  The pascal recognising textual entailment challenge.  In Proceedings of the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment, MLCW’05, page 177–190, Berlin, Heidelberg. Springer-Verlag.
  * [Davis and Bowers (2006]  Colin J. Davis and Jeffrey S. Bowers.  2006\.  Contrasting five different theories of letter position coding: evidence from orthographic similarity effects.  Journal of experimental psychology. Human perception and performance, 32 3:535–57.
  * [Davis (2003]  Matt Davis.  2003\.  Cmabridge.  https://www.mrc-cbu.cam.ac.uk/people/matt.davis/cmabridge/.  Accessed January 30, 2020.
  * [Devlin et al. (2018]  Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  2018\.  BERT: Pre-training of deep bidirectional transformers for language understanding.  CoRR, abs/1810.04805.
  * [Dolan and Brockett (2005]  Bill Dolan and Chris Brockett.  2005\.  Automatically constructing a corpus of sentential paraphrases.  In Third International Workshop on Paraphrasing (IWP2005), January.
  * [Ebrahimi et al. (2018a]  Javid Ebrahimi, Daniel Lowd, and Dejing Dou.  2018a.  On adversarial examples for character-level neural machine translation.  In Proceedings of the 27th International Conference on Computational Linguistics, pages 653–663, Santa Fe, New Mexico, USA, August. Association for Computational Linguistics.
  * [Ebrahimi et al. (2018b]  Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou.  2018b.  HotFlip: White-box adversarial examples for text classification.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Melbourne, Australia, July.
  * [Gage (1994]  Philip Gage.  1994\.  A new algorithm for data compression.  The C Users Journal., february.
  * [Giampiccolo et al. (2007]  Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan.  2007\.  The third PASCAL recognizing textual entailment challenge.  In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1–9, Prague, June. Association for Computational Linguistics.
  * [Gomez et al. (2008]  Pablo Gomez, Roger Ratcliff, and Manuel Perea.  2008\.  The overlap model: A model of letter position coding.  Psychological review, 115(3):577.
  * [Goodfellow et al. (2015]  Ian Goodfellow, Jonathon Shlens, and Christian Szegedy.  2015\.  Explaining and harnessing adversarial examples.  In International Conference on Learning Representations.
  * [Jia and Liang (2017]  Robin Jia and Percy Liang.  2017\.  Adversarial examples for evaluating reading comprehension systems.  In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.
  * [Kearns and Li (1993]  Michael. Kearns and Ming. Li.  1993\.  Learning in the presence of malicious errors.  SIAM Journal on Computing, 22(4):807–837.
  * [Kovaleva et al. (2019]  Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky.  2019\.  Revealing the dark secrets of BERT.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4365–4374, Hong Kong, China, November. Association for Computational Linguistics.
  * [Lample and Conneau (2019]  Guillaume Lample and Alexis Conneau.  2019\.  Cross-lingual language model pretraining.  CoRR, abs/1901.07291.
  * [Liu et al. (2019]  Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  2019\.  Roberta: A robustly optimized BERT pretraining approach.  CoRR, abs/1907.11692.
  * [Minervini and Riedel (2018]  Pasquale Minervini and Sebastian Riedel.  2018\.  Adversarially regularising neural NLI models to integrate logical background knowledge.  In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 65–74, Brussels, Belgium, October. Association for Computational Linguistics.
  * [Norris (2006]  Dennis Norris.  2006\.  The bayesian reader: Explaining word recognition as an optimal bayesian decision process.  Psychological review, 113(2):327.
  * [Ollman (2007]  Gunter Ollman.  2007\.  The phishing guide.  IBM Internet Security Systems, page 3.
  * [Pelli et al. (2003]  Denis G. Pelli, Bart Farell, and Deborah C. Moore.  2003\.  The remarkable inefficiency of word recognition.  Nature, 423:752–756.
  * [Perea and Lupker (2003]  Manuel Perea and Stephen J. Lupker.  2003\.  Does jugde activate COURT? transposed-letter similarity effects in masked associative priming.  Memory & Cognition, 31(6):829–841, Sep.
  * [Perea and Lupker (2004]  M. Perea and S. J. Lupker.  2004\.  Can caniso activate casino? transposed-letter similarity effects with nonadjacent letter positions.  Journal of Memory and Language, 51(2):231–246.
  * [Peters et al. (2018]  Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih.  2018\.  Dissecting contextual word embeddings: Architecture and representation.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1499–1509, Brussels, Belgium, October-November. Association for Computational Linguistics.
  * [Pruthi et al. (2019]  Danish Pruthi, Bhuwan Dhingra, and Zachary C. Lipton.  2019\.  Combating adversarial misspellings with robust word recognition.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5582–5591, Florence, Italy, July. Association for Computational Linguistics.
  * [Rawlinson (1976]  Graham Rawlinson.  1976\.  The significance of letter position in word recognition.  Unpublished PhD thesis.
  * [Rawlinson (2007]  Graham Rawlinson.  2007\.  The significance of letter position in word recognition.  Aerospace and Electronic Systems Magazine, IEEE, 22:26–27, 02.
  * [Rayner et al. (2006]  Keith Rayner, Sarah J. White, Rebecca L. Johnson, and Simon P. Liversedge.  2006\.  Raeding wrods with jubmled lettres: there is a cost.  Psychological Science, 17(3):192–193.
  * [Schoonbaert and Grainger (2004]  Sofie Schoonbaert and Jonathan Grainger.  2004\.  Letter position coding in printed word perception: Effects of repeated and transposed letters.  Language and Cognitive Processes, 19(3):333–367.
  * [Schuster and Nakajima (2012]  M. Schuster and K. Nakajima.  2012\.  Japanese and korean voice search.  In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149–5152, Kyoto, Japan.
  * [Socher et al. (2013]  Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts.  2013\.  Recursive deep models for semantic compositionality over a sentiment treebank.  In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642, Seattle, Washington, USA, October. Association for Computational Linguistics.
  * [Szegedy et al. (2014]  Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  2014\.  Intriguing properties of neural networks.  In International Conference on Learning Representations.
  * [Vaswani et al. (2017]  Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  2017\.  Attention is all you need.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc.
  * [Wang et al. (2018]  Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  2018\.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium, November. Association for Computational Linguistics.
  * [Whitney (2008]  Carol Whitney.  2008\.  Comparison of the seriol and solar theories of letter-position encoding.  Brain and Language, 107(2):170–178.
  * [Williams et al. (2018]  Adina Williams, Nikita Nangia, and Samuel Bowman.  2018\.  A broad-coverage challenge corpus for sentence understanding through inference.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana, June. Association for Computational Linguistics.
  * [Yang et al. (2019]  Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le.  2019\.  Xlnet: Generalized autoregressive pretraining for language understanding.  CoRR, abs/1906.08237.
  * [Zhang et al. (2019]  Wei Emma Zhang, Quan Z. Sheng, and Ahoud Abdulrahmn F. Alhazmi.  2019\.  Generating textual adversarial examples for deep learning models: A survey.  CoRR, abs/1901.06796.