# Efficient privacy-preserving inference
for convolutional neural networks

Han Xuanyuan, Francisco Vargas & Stephen Cummins
Department of Computer Science and Technology
University of Cambridge
<EMAIL_ADDRESS>

###### Abstract

The processing of sensitive user data using deep learning models is an area
that has gained recent traction. Existing work has leveraged homomorphic
encryption (HE) schemes to enable computation on encrypted data. An early work
was CryptoNets, which takes 250 seconds for one MNIST inference. The main
limitation of such approaches is that of the expensive FFT-like operations
required to perform operations on HE-encrypted ciphertext. Others have
proposed the use of model pruning and efficient data representations to reduce
the number of HE operations required. We focus on improving upon existing work
by proposing changes to the representations of intermediate tensors during CNN
inference. We construct and evaluate private CNNs on the MNIST and CIFAR-10
datasets, and achieve over a two-fold reduction in the number of operations
used for inferences of the CryptoNets architecture.

##  1 Introduction

Machine Learning as a Service (MLaaS) is a framework in which cloud services
apply machine learning algorithms on user-supplied data to produce an
inference result which is then returned to the user. However, the data has to
be decrypted before inference, which allows a server-side adversary to have
access to the user’s information. Homomorphic encryption (HE), can be applied
to enable inference to be performed on encrypted data, enabling the result to
be delivered to the user without risk of the server accessing the original
data or the inference result. CryptoNets (Gilad-Bachrach et al., 2016) was the
first application of HE to secure neural network inference, and leveraged the
YASHE’ scheme to perform MNIST classifications. Their approach requires a
notably high number of homomorphic operations (HOPs), with a single MNIST
inference requiring $\mathord{\sim}290000$ homomorphic multiplications and
$\mathord{\sim}250$ seconds of inference latency. Later works utilised
ciphertext rotations as opposed to the SIMD packing scheme, enabling
convolutional and fully connected layers to be computed using much fewer HOPs
(Juvekar et al., 2018; Mishra et al., 2020). This has been shown to reduce the
inference latency of MNIST models by more than an order of magnitude, bringing
confidence that private inference can be practical. LoLa (Brutzkus et al.,
2019) proposed novel representations for intermediate tensors and their MNIST
model requires only 2.2 seconds for one inference.

In this work, we introduce a framework for secure inference on secure CNNs,
designed to reduce the number of HOPs required per inference whilst preserving
prediction accuracy. We integrate the convolution-packing method from LoLa
with the fast matrix-vector product method introduced by Halevi and Shoup
(Halevi & Shoup, 2019) and utilised by Juvekar et al. (2018) in their multi-
party computation framework. We show that utilising the Halevi-Shoup method
allows the use of rotations and ciphertext packing to scale better compared
with the representations in LoLa, when applied to larger convolutional layers.
We perform a more detailed investigation on the scalability of the methods
used in LoLa to larger models and show that they are significantly
outperformed by our proposed method. In addition, we compare our framework
against LoLa by constructing models for MNIST and CIFAR-10. With the same
layer parameters as LoLa, we are able to obtain over a two-fold reduction in
the number of HOPs per inference. Our CIFAR-10 model achieves similar accuracy
to that of LoLa’s but uses far fewer operations.

##  2 Prerequisites

###  2.1 Homomorphic operations

Several recent HE schemes such as BFV (Brakerski & Vaikuntanathan, 2011) and
CKKS (Cheon et al., 2017) are based on the RLWE problem and support SIMD
ciphertext operations. On a high level, such schemes establish a mapping
between real vectors and a plaintext space. The plaintext space is usually the
polynomial ring $\mathcal{R}={\mathbb{Z}[X]}/(X^{N}+1)$. In particular, this
is a _cyclotomic polynomial ring_ $\mathcal{R}={\mathbb{Z}[X]}/(\Phi_{M}(X))$
where $\Phi_{M}(X)$ is the $M$-th cyclotomic polynomial and $M=2N$ is a power
of two. The _decoding_ operation maps an element in $\mathcal{R}$ to a vector
that is either real or complex, depending on the scheme used. The _encoding_
operation performs the reverse. Plaintext polynomials are encrypted into
ciphertext polynomials using a _public key_. The operations of addition and
multiplication can be performed over ciphertexts using an _evaluation key_.
Since each ciphertext corresponds to a vector of real (or complex) values, a
single homomorphic operation between two ciphertexts constitutes an element-
wise operation between two vectors. In addition, such schemes support
rotations of the slots within a ciphertext, with the use of Galois
automorphisms.

###  2.2 Fast encrypted convolution

The first convolutional layer in a CNN can be represented using convolution-
packing (Brutzkus et al., 2019). The convolution of an input image $f$ with a
filter $g$ of width $w$, height $h$ and depth $d$ can be formulated as

$\displaystyle
f*g=\sum_{x=0}^{w-1}\sum_{y=0}^{h-1}\sum_{z=0}^{d-1}g[x,y,z]\cdot\mathbf{F}^{(x,y,z)},$
(1)

where $\mathbf{F}^{(x,y,z)}$ is a matrix such that
$\mathbf{F}^{(x,y,z)}_{ij}=f[i+x,j+y,z]$. For an input image
$I\in\mathbb{R}^{c_{\text{in}}\times d_{\text{in}}\times d_{\text{in}}}$
feature maps and kernel of window size $k\times k$, the input image is
represented as $k^{2}\cdot c_{\text{in}}$ vectors
$\textbf{v}_{1},\ldots,\textbf{v}_{k^{2}\cdot c_{\text{in}}}$, where
$\textbf{v}_{i}$ contains all elements convolved with the $i$-th value in the
filter. Denote corresponding ciphertexts as
$\text{ct}_{1},\ldots,\text{ct}_{k^{2}\cdot c_{\text{in}}}$. The process of
producing the $j$-th output feature map is now reduced to ciphertext-plaintext
multiplication of each $\text{ct}_{i}$ with the $i$-th value in the $j$-th
filter. In total, the process requires $k^{2}\cdot c_{\text{in}}$ ciphertext-
scalar multiplications per output feature map, leading to a total of
$k^{2}\cdot c_{\text{in}}\cdot c_{\text{out}}$ multiplications.

##  3 Method

In this section, we present our method for achieving privacy-preserving CNN
inference with low numbers of HOPs. In summary, we adopt the fast convolution
method from LoLa but compute the intermediate convolutional layers in a
network using an efficient matrix-vector product method proposed by Halevi &
Shoup (2019). This enables large convolutions to be performed in far fewer
ciphertext rotations than LoLa’s approach of computing a rotate-and-sum
procedure for each row in the weight matrix. In section 3.1 we explain the
approach we use. In section 3.2, we perform an analysis to show the
improvements made by our modifications compared with the approach from LoLa.
In section 4, we apply our approach to models for the MNIST and CIFAR-10
datasets.

###  3.1 Performing intermediate convolutions

Consider the convolution of an image $\mathbf{I}$ with a filter $f$. For
simplicity, assume that both the image and filter are square, the vertical and
horizontal strides of the filter are equal. Let
$\mathbf{I}\in\mathbb{R}^{d_{\text{in}}\times d_{\text{in}}\times
c_{\text{in}}}$, and $f\in\mathbb{R}^{k\times k\times c_{\text{in}}}$. It is
well known that this convolution can be flattened and represented as a matrix-
vector product
$\mathbf{A}\cdot\mathbf{w}\in\mathbb{R}^{d_{\text{out}}^{2}\cdot
c_{\text{out}}}$ where $\mathbf{A}\in\mathbb{R}^{d_{\text{out}}^{2}\cdot
c_{\text{out}}\times d_{\text{in}}^{2}\cdot c_{\text{in}}}$ and $\mathbf{w}$
is the flattened input.

Halevi & Shoup (2019) introduced an efficient method of computing the
encrypted matrix-vector product $\mathbf{A}\cdot\mathbf{v}$ for square
$\mathbf{A}$. GAZELLE (Juvekar et al., 2018) extended the approach to support
rectangular $\mathbf{A}\in\mathbb{R}^{m\times n}$. The method works by
decomposing $\mathbf{A}$ into its $m$ diagonals, denoted
$\left\\{d_{1},d_{2},\ldots,d_{m}\right\\}$ such that
$d_{i}=\left[\mathbf{A}_{i,0},\mathbf{A}_{i+1,1},\ldots,\mathbf{A}_{i+n-1,n-1}\right]$.
Note that all row positions are in modulo $m$. Then each $d_{i}$ is _rotated_
$i$ positions to align values belonging to the same row of A into the same
column(s), and finally each rotated diagonal is multiplied with with
corresponding rotations of v. The ciphertexts are summed, and the last stage
is to apply a rotate-and-sum procedure to the resulting ciphertext. Overall,
this procedure requires $O(m)$ multiplications and $O(m+\log_{2}n)$ rotations.
We show that this can be applied to the convolutions in LOLA, but under
certain constraints. Neither the original method (Halevi & Shoup, 2019) nor
GAZELLE (Juvekar et al., 2018) discuss the constraints the ciphertext slot
count $N$ imposes in this context. In Proposition 1 we show that certain
constraints must be applied on the sizes of the input and output of the
convolution. In particular, we must have $N\geq d_{\text{in}}^{2}\cdot
c_{\text{in}}+d_{\text{out}}^{2}\cdot c_{\text{out}}-1$, unless
$d_{\text{out}}$ and $c_{\text{out}}$ are both powers of 2.

######  Proposition 1.

Let $N$ denote the ciphertext slot count, and $n=d_{\text{in}}^{2}\cdot
c_{\text{in}}$ be the size of the convolution input, and
$m=d_{\text{out}}^{2}\cdot c_{\text{out}}$ be the size of the output. The
basic Halevi-Shoup method, which takes
$m-1+\lceil\log_{2}{\left(\frac{m+n-1}{m}\right)}\rceil$ rotations, requires
the condition that $N\geq m+n-1$. If this does not hold, but it is the case
that $m=2^{l},\;0\leq l\leq\log_{2}N$, then $m-1+\log_{2}\frac{N}{m}$
rotations are required.

###### Proof.

Applying the Halevi-Shoup technique requires $m$ ciphertext diagonals
$\mathbf{d}^{(1)},\ldots,\mathbf{d}^{(m)}$ of length $n$ to be extracted,
rotated and summed. Note that $\mathbf{d}_{j}^{(i)}=\mathbf{A}_{i+j,j}$. Now,
if $N\geq m+n-1$ then $N$ is sufficiently large for all rotations to be
performed without wrapping around the ciphertext. If $N<m+n-1$, then wrap-
around will occur for at least one of the diagonal ciphertexts during
rotation. For any slot $\mathbf{d}^{(i)}_{j}$, the rotation by $i$ positions
will shift the slot to position $k=i+j\ (\text{mod}\ N)$. The slot
$\mathbf{d}^{(i)}_{j}$ is from the row of $\mathbf{A}$ with index $r=i+j\
(\text{mod}\ m)$, and so must be shifted into an index that is equivalent to
$r$ in modulo $m$, in order for the rotate-and-sum algorithm to be used. If
$N$ is an integer multiple of $m$, then we see $k\equiv r\ (\text{mod}\ m)$
indeed holds. Otherwise, wrap-around will cause the diagonals to be misaligned
when summed together. Since $N$ is a power of 2, the requirement that $m$
divides $N$ is satisfied whenever $d_{\text{out}}$ and $c_{\text{out}}$ are
also powers of 2 such that
$2\log_{2}{d_{\text{out}}}+\log_{2}{c_{\text{out}}}\leq\log_{2}N$. ∎

Based on Proposition 1, we utilise a procedure to ensure that the method can
be applied for all choices of
$(d_{\text{in}},d_{\text{out}},c_{\text{in}},c_{\text{out}})$ where
$d_{\text{out}}^{2}\cdot c_{\text{out}}\leq N$: if the condition that $N\geq
m+n-1$ does not hold, then we ‘round’ $m$ to the closest power of $2$ not less
than itself, and add corresponding rows filled with 0’s to the weight matrix.
This procedure is illustrated below for the simple case where $N=4,m=3$ and
$n=4$. Notice that the padding allows the diagonals to align correctly after
rotation.

Figure 1: An illustration of our version of the Halevi-Shoup method that
utilises a padding method.

###  3.2 Comparison with LoLa

LoLa proposes a _dense-vector row-major_ matrix-vector product method (which
we call LoLa-dense) that maps the input vector in the _dense_ representation
to an output vector in the _sparse_ representation. A vector $\mathbf{v}$ is
dense when represented as a single ciphertext where the first $n$ slots
correspond to the values in $\mathbf{v}$, and is sparse when represented as
$n$ ciphertexts, where the $i$-th ciphertext contains the $i$-th element of
$\mathbf{v}$ in all its slots. Let $\mathbf{A}\in\mathbb{R}^{m\times n}$. The
dense-vector row-major method computes
$\mathbf{A}\mathbf{v}_{1}=\mathbf{v}_{2}$ via a multiplication per row of
$\mathbf{A}$ and summing the elements inside each product vector using the
rotate-and-sum procedure.

They also propose a _stacked-vector row-major_ method (which we call LoLa-
stacked) that requires input vector in the _stacked_ representation and
provides the output in the _interleaved_ representation. A vector $\mathbf{v}$
is stacked when represented as a single ciphertext that contains as many
copies of $\mathbf{v}$ as the ciphertext slot count permits, and is
interleaved when represented as a single ciphertext similar to the dense
representation - but the slots may be shuffled by some permutation. To obtain
the stacked representation, they first pack $k$ copies of $\mathbf{v}$ into a
single ciphertext where $k={N}/{\delta(n)}$ and
$\delta(n)=2^{\lceil\log_{2}n\rceil}$ is the smallest power of $2$ greater
than or equal to $n$. The stacked vector is then multiplied with corresponding
stacked rows of $\mathbf{A}$, before rotate-and-sum is applied. The main
drawback of this method is its reliance on the ability to pack many copies of
$\mathbf{v}$ into a ciphertext, which requires $N$ to be much larger than $n$.
For large convolutions, this is hard to achieve. We derive the number of
operations this method takes.

######  Remark 1.

The _stacked-vector row-major_ method proposed by Brutzkus et al. (2019)
requires
$\lceil\frac{m}{k}\rceil\left(k+\lceil\log_{2}n\rceil-1\right)+\lceil\frac{m}{k}\rceil-1$
rotations and $\lceil\frac{m}{k}\rceil$ multiplications.

###### Proof.

We have included this in Appendix A.3. ∎

Suppose we are passing a 4096-length representation into a fully connected
layer to map to a 64-length embedding, and let $N=16384$. LoLa-dense would
require $64\cdot\log_{2}4096=768$ rotations and $64$ multiplications; LoLa-
stacked would require $1023$ rotations and $64$ multiplications.

Figure 2: Comparison of LoLa’s matrix-vector product methods with the Halevi-
Shoup approach, in terms of the number of rotations required for computing a
fully-connected layer from $n$ inputs to $m$ outputs.

Now, the Halevi-Shoup product approach would require only $64+\log_{2}8192=77$
rotations and $64$ multiplications. Since rotations are the most expensive
operation, we compare the number of rotations required by each of the three
methods in Figure 2.

It can be shown that for any $n>1$, LoLa-stacked uses fewer rotations than
LoLa-dense. However, LoLa-dense can be used to compute two layers instead of
one. For large inputs and output layers (relative to $N$), however, both
methods require significantly more rotations than HS. LoLa-stacked relies on
$m\cdot\delta(n)/N$ being small whereas LoLa-dense relies on $\log_{2}n$ being
small. With HS, even if $n=N$, $\log_{2}n$ is insignificant compared to $m$.

In general, we believe that having the number of rotations be linear to only
$m$ is beneficial since neural networks typically down-sample or pool the data
to produce denser, higher-level representations, and so can expect $n\geq m$
generally. It should be noted that there are exceptions - such as bottleneck
layers.

##  4 Experiments and results

We conduct experiments on the MNIST and CIFAR-10 datasets. We first apply our
approach to the CryptoNets architecture used in LoLa, to create a model
CryptoNets-HS. The same architecture was shown to achieve close 98.95%
accuracy by Gilad-Bachrach et al. (2016), and we observe similar performance
using their training parameters. Training is conducted using TensorFlow (Abadi
et al., 2015). The architecture is then converted into a sequence of
homomorphic operations. We use the SEAL library (Chen et al., 2017) for this.
For reference, the original CryptoNets architecture is shown in Appendix A.1.
To reduce consumption of instruction depth, linear layers without activations
between them are composed together. For instance, each convolution-pooling
block of CryptoNets-HS is a single linear layer. This is done to enable our
baseline to match the architecture in LoLa. It should be noted that omitting
activations can reduce representational power and made not be ideal in
practice. In addition, we construct a further-optimised model ME that has
similar test accuracy to CryptoNets-HS but is designed in consideration of the
way we are computing the layers. We note that applying the HS method in
computing layer 6 of CryptoNets-HS requires setting $m=100$ and $n=845$.
Specifically, we reduce the kernel size of the first convolution from $5\times
5$ to $3\times 3$, and the size of the first dense layer from $100$ to $32$.
To account for the reduced representational power of the first convolution,
the stride is reduced from $2$ to $1$. This notably does not add any
homomorphic operations to the computation. We are able to achieve 98.7% test
accuracy using ME after 100 epochs of training with the Adam optimiser.

The models are implemented using operations provided by the SEAL library, and
inference is performed on a standard desktop processor. We utilise only a
single thread, and measure the number of homomorphic operations required per
single inference and model test accuracy as our evaluation metrics.

Layer | Total HOPs | Add PC | Add CC | Mul PC | Mul CC | Rot
---|---|---|---|---|---|---
|  $M$ |  $L^{\prime}$ |  $L$ |  $M$ |  $L^{\prime}$ |  $L$ |  $M$ |  $L^{\prime}$ |  $L$ |  $M$ |  $L^{\prime}$ |  $L$ |  $M$ |  $L^{\prime}$ |  $L$ |  $M$ |  $L^{\prime}$ |  $L$
Conv1 |  90 |  250 |  250 |  5 |  5 |  5 |  40 |  120 |  120 |  45 |  125 |  125 |  - |  - |  - |  - |  - |  -
Flat1 |  8 |  8 |  8 |  - |  - |  - |  4 |  4 |  4 |  - |  - |  - |  - |  - |  - |  4 |  4 |  4
Square1 |  1 |  1 |  1 |  - |  - |  - |  - |  - |  - |  - |  - |  - |  1 |  1 |  1 |  - |  - |  -
Conv2-Dense1 |  110 |  308 |  492 |  1 |  1 |  - |  38 |  103 |  246 |  32 |  100 |  13 |  - |  - |  - |  39 |  104 |  246
Square2 |  1 |  1 |  1 |  - |  - |  - |  - |  - |  - |  - |  - |  - |  1 |  1 |  1 |  - |  - |  -
Dense2 |  36 |  38 |  279 |  1 |  1 |  - |  12 |  13 |  139 |  10 |  10 |  10 |  - |  - |  - |  13 |  14 |  130
Total |  246 |  606 |  1031 |  7 |  7 |  5 |  94 |  240 |  509 |  87 |  235 |  148 |  2 |  2 |  2 |  56 |  122 |  380
Table 1: Comparison of operations in three different privacy-preserving MNIST
models. $L$ indicates the original LoLa-MNIST model, $L^{\prime}$ indicates
the CryptoNets-HS model, and $M$ indicates the ME model. ‘CC’ indicates an
operation between two ciphertexts and ‘PC’ indicates an operation between a
plaintext and a ciphertext.

Table 2 shows a breakdown of the operations for the models discussed so far.
In homomorphic encryption applied to neural networks, the most expensive
homomorphic operation is rotation, with a worst case time complexity of
performing both a number theoretic transform (NTT) and an inverse NTT on
vectors of length $N$. We observe that CryptoNets-HS requires a total of 122
rotations, as shown in Table 1. LoLa-MNIST requires a total of 380
rotations111Note that this is deduced to the best of our ability using the
descriptions supplied in their paper. for the same architecture. We notice
that at both the convolution-pooling blocks in the CryptoNets architecture,
the HS implementation has fewer rotations than LoLa. The reduction in
rotations is a direct result of applying the HS method to compute flattened
convolutions, in which the number of rotations scales with the size of the
output tensor, rather than the size of the input. LoLa uses the stacked-vector
row-major method for the first convolution, and the dense-vector row-major
method for the second convolution. In Section 3.2 we discussed that both of
these scale poorly compared to the HS method. In terms of latency, the
CryptoNets-HS model requires 2.7 seconds, whereas ME requires only 0.97
seconds. For reference, LoLa-MNIST requires 2.2 seconds per inference;
however, they utilise 8 cores on a server CPU whereas our homomorphic
operations are run on a single core of a desktop processor.

We also construct a model for the CIFAR-10 dataset, called CE. The
architecture details are shown in A.2. It achieves sightly lower test accuracy
($73.1\%$ vs. $74.1\%$) but requires less than $10\%$ the number of rotations
than LoLa’s CIFAR-10 model. This is due to both ensuring that intermediate
tensor sizes fit well into the number of available slots, and also the use of
the HS method. The number of operations for each layer is shown in A.4.

A big takeaway from the results is that the application of the HS method to
the framework proposed by LoLa can significantly reduce the number of
rotations, which are the most computationally expensive HE operation. In
addition, the optimised ME is significantly more efficient than LoLa whilst
achieving similar accuracy.

##  5 Conclusion

Privacy-preserving inference using homomorphic encryption is largely
constrained by the computational requirements of the operations. We propose
improvements over LoLa to achieve lower latencies computing intermediate
convolutions, resulting in over a two-fold reduction in the number of
rotations for the same MNIST architecture. It is clear that further
improvements can made along this direction, especially in the topic of
automatically selecting suitable layer parameters to set the trade-off between
inference latency and model accuracy (Lou et al., 2020).

## References

  * Abadi et al. (2015) Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng.  TensorFlow: Large-scale machine learning on heterogeneous systems, 2015\.  URL https://www.tensorflow.org/.  Software available from tensorflow.org.
  * Brakerski & Vaikuntanathan (2011) Zvika Brakerski and Vinod Vaikuntanathan.  Efficient fully homomorphic encryption from (standard) lwe.  In _Proceedings of the 2011 IEEE 52nd Annual Symposium on Foundations of Computer Science_ , pp. 97–106. IEEE Computer Society, 2011.
  * Brutzkus et al. (2019) Alon Brutzkus, Ran Gilad-Bachrach, and Oren Elisha.  Low latency privacy preserving inference.  In _Proceedings of the 36th International Conference on Machine Learning_ , volume 97, pp. 812–821. PMLR, 2019.
  * Chen et al. (2017) Hao Chen, Kim Laine, and Rachel Player.  Simple encrypted arithmetic library - seal v2.1.  In _Financial Cryptography and Data Security_ , pp. 3–18. Springer International Publishing, 2017.  ISBN 978-3-319-70278-0.
  * Cheon et al. (2017) Jung Cheon, Andrey Kim, Miran Kim, and Yongsoo Song.  Homomorphic encryption for arithmetic of approximate numbers.  In _Advances in Cryptology - ASIACRYPT 2017_ , pp. 409–437. Springer, 2017.
  * Gilad-Bachrach et al. (2016) Ran Gilad-Bachrach, Nathan Dowlin, Kim Laine, Kristin Lauter, Michael Naehrig, and John Wernsing.  Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy.  In _Proceedings of The 33rd International Conference on Machine Learning_ , volume 48, pp. 201–210. PMLR, 2016.
  * Halevi & Shoup (2019) Shai Halevi and Victor Shoup.  Algorithms in helib.  In _Advances in Cryptology_ , pp. 554–571. Springer, 2019.
  * Juvekar et al. (2018) Chiraag Juvekar, Vinod Vaikuntanathan, and Anantha Chandrakasan.  Gazelle: a low latency framework for secure neural network inference.  In _Proceedings of the 27th USENIX Conference on Security Symposium_ , pp. 1651–1668. USENIX, 2018.
  * Lou et al. (2020) Qian Lou, Song Bian, and Lei Jiang.  Autoprivacy: Automated layer-wise parameter selection for secure neural network inference.  In _Advances in Neural Information Processing Systems_ , volume 33, pp. 8638–8647. Curran Associates, Inc., 2020.
  * Mishra et al. (2020) Pratyush Mishra, Ryan Lehmkuhl, Akshayaram Srinivasan, Wenting Zheng, and Raluca Ada Popa.  Delphi: A cryptographic inference service for neural networks.  In _Proceedings of the 2020 Workshop on Privacy-Preserving Machine Learning in Practice_ , pp. 27–30. ACM, 2020.

##  Appendix A Appendix

###  A.1 CryptoNets architecture

Figure 3: The architecture used in LoLa. $k$ indicates the kernel size and $s$
indicates the stride of convolution and pooling. Note that the original
$28\times 28$ input is padded.

###  A.2 Details of our experimental models

Layer |  Description |  Parameters |  Input
---|---|---|---
1 |  Convolution |  $k=3,s=1$ |  (1, 30, 30)
- |  Square |  - |  (5, 28, 28)
2 |  Avg. Pool |  $k=3,s=2$ |  (5, 28, 28)
3 |  Convolution |  $k=3,s=1$ |  (50, 14, 14)
4 |  Avg. Pool |  $k=3,s=2$ |  (50, 12, 12)
5 |  Flatten |  - |  (50, 12, 12)
6 |  Dense |  $m=32$ |  (1250)
- |  Square |  - |  (32)
7 |  Dense |  $m=10$ |  (32)
- |  Softmax |  - |  (100)
(a) ME layer parameters.

Layer |  Description |  Parameters |  Input
---|---|---|---
1 |  Convolution |  $k=3,s=1$ |  (1, 32, 32)
- |  Square |  |  (18, 30, 30)
2 |  Average Pooling |  $k=2,s=2$ |  (18, 30, 30)
3 |  Sub-convolution |  $k=3,s=1$ |  (18, 10, 10)
4 |  Sub-convolution |  $k=1,s=1$ |  (13, 8, 8)
- |  Square |  |  (18, 30, 30)
5 |  Average Pooling |  $k=2,s=2$ |  (64, 8, 8)
6 |  Convolution |  $k=3,s=1$ |  (64, 4, 4)
7 |  Flatten |  - |  (256, 2, 2)
8 |  Dense |  $m=512$ |  (1024)
- |  Square |  |  (18, 30, 30)
9 |  Dense |  $m=10$ |  (256)
- |  Softmax |  - |  (10)
(b) CE layer parameters.

Table 2: Layer parameters for ME (left) and CE (bottom). $k$ and $s$ indicate
the kernel width and stride respectively. $m$ indicates the number of output
nodes of a fully-connected layer.

###  A.3 Proof of Remark 1

###### Proof.

The stacking in Brutzkus et al. (2019) is done is using $k-1$ rotations and
additions. Then the point-wise multiplication requires a single SIMD
multiplication between the ciphertext and $k$ (stacked) rows of $\mathbb{A}$.
Finally, $\lceil\log_{2}n\rceil$ rotations and additions are performed to
compute the $k$ elements of the product vector. Since $N$ is not always
sufficiently large to pack $m$ copies of $n$, this procedure has to be
performed $\lceil\frac{m}{k}\rceil$ times. To produce all elements of the
product vector,
$\lceil\frac{m}{k}\rceil\left(k+\lceil\log_{2}n\rceil-1\right)$ rotations and
$\lceil\frac{m}{k}\rceil$ multiplications are performed, before
$\lceil\frac{m}{k}\rceil-1$ rotations are performed to bring the
$\lceil\frac{m}{k}\rceil$ ciphertexts into a single one. Note that the
produced ciphertext(s) contain the product elements in an interleaved format,
i.e. a permutation. ∎

###  A.4 Performance on CIFAR-10

Layer |  HOPs |  Add PC |  Add CC |  Mul PC |  Mul CC |  Rot
---|---|---|---|---|---|---
Conv1 |  972 |  18 |  468 |  486 |  - |  -
Flat1 |  30 |  - |  15 |  - |  - |  15
Square1 |  3 |  - |  - |  - |  3 |  -
Pool1-Conv2 |  7506 |  1 |  2504 |  2496 |  - |  2505
Conv3 |  1677 |  64 |  768 |  832 |  - |  13
Flat2 |  126 |  - |  63 |  - |  - |  63
Square2 |  1 |  - |  - |  - |  1 |  -
Pool2-Dense1 |  778 |  1 |  260 |  256 |  - |  261
Square |  1 |  - |  - |  - |  1 |  -
Dense2 |  40 |  1 |  14 |  10 |  - |  15
Total |  11134 |  85 |  4092 |  4080 |  5 |  2872
Figure 4: Break-down of operations in CE.


# Semantics Altering Modifications
for Evaluating Comprehension in Machine Reading

Viktor Schlegel, Goran Nenadic and Riza Batista-Navarro

###### Abstract

Advances in NLP have yielded impressive results for the task of machine
reading comprehension (MRC), with approaches having been reported to achieve
performance comparable to that of humans. In this paper, we investigate
whether state-of-the-art MRC models are able to correctly process Semantics
Altering Modifications (SAM): linguistically-motivated phenomena that alter
the semantics of a sentence while preserving most of its lexical surface form.
We present a method to automatically generate and align challenge sets
featuring original and altered examples. We further propose a novel evaluation
methodology to correctly assess the capability of MRC systems to process these
examples independent of the data they were optimised on, by discounting for
effects introduced by domain shift. In a large-scale empirical study, we apply
the methodology in order to evaluate extractive MRC models with regard to
their capability to correctly process SAM-enriched data. We comprehensively
cover 12 different state-of-the-art neural architecture configurations and
four training datasets and find that – despite their well-known remarkable
performance – optimised models consistently struggle to correctly process
semantically altered data.

## Introduction

Machine Reading Comprehension (MRC), also commonly referred to as Question
Answering, is defined as finding the answer to a natural language question
given an accompanying textual context. State-of-the-art approaches build upon
large transformer-based language models (Vaswani et al. 2017) that are
optimised on large corpora in an unsupervised manner (Devlin et al. 2019) and
further fine-tuned on large crowd-sourced task-specific MRC datasets
(Rajpurkar et al. 2016; Yang et al. 2018; Trischler et al. 2017). They achieve
remarkable performance, consistently outperforming human baselines on multiple
reading comprehension and language understanding benchmarks (Lan et al. 2020;
Raffel et al. 2020).

More recently, however, research on “data biases” in NLP suggests that these
task-specific datasets exhibit various cues and spurious correlations between
input and expected output (Gururangan et al. 2018; Poliak et al. 2018;
Schlegel, Nenadic, and Batista-Navarro 2020). Indeed, data-driven approaches
such as the state-of-the-art models (described above) that are optimised on
these datasets learn to exploit these (Jia and Liang 2017; McCoy, Pavlick, and
Linzen 2019), thus circumventing the actual requirement to perform
comprehension and understanding.

P:_1 After the kickoff Naomi Daniel… _(B) Original: _curled in_(I1) Modal
negation: __couldn’t curl_ in_(I2) Adverbial Modification: __almost_ curled
in_(I3) Implicit Negation: __was prevented from_ curling in_(I4) Explicit
Negation: __didn’t succeed in_ curling in_(I5) Polarity Reversing: __lacked
the nerve to_ curl in_(I6) Negated Polarity Preserving: wouldn’t find
theopportunity to _curl in_ …a goal from 26 metres away following a decisive
coun- ter-attack. 2 Then Amanda Collins added more insult to the injury when
she slotted in from 23 metres after Linda Burger’s soft clearance. […] Q:_Who
scored the farthest goal?_ A:Naomi DanielA with SAM:Amanda Collins Figure 1:
Categories of SAM used in this paper with their implications on answering the
given question. Modifying the original _“Baseline”_ passage (B) by selecting
any _“Intervention”_ category (I1)$-$(I6), or removing the first sentence
(_“Control”_) changes the correct answer from “Naomi Daniel” located in
sentence 1 to “Amanda Collins” located in sentence 2.

For a (simplified) example, consider the question _“Who scored the farthest
goal?”_ illustrated in Figure 1. If a model is only exposed to examples where
the accompanying passage contains sentences similar to “X scored a goal from Y
metres” during training, a valid approximating decision based on this
information could be similar to _“select the name next to the largest number
and the word goal”_ without actually fully reading the passage.

Alarmingly, conventional evaluation methodology where the dataset is split
randomly into training and test data would not solve this issue. As both
splits still stem from the same generative process (typically crowd-sourcing),
the same types of cues are likely to exist in evaluation data, and a model can
achieve high performance by relying on exploiting them. These and other
problems suggest that the actual reading _comprehension_ of state-of-the-art
MRC models is potentially over-estimated.

In an attempt to present a more realistic estimate, we focus on the capability
to correctly process _Semantic Altering Modifications_ (SAM): minimal
modifications to the passage that change its meaning and therefore the
expected answer. On the one hand, it is important to know whether these
modifications are processed correctly by MRC models, as they drastically
change the meaning, for example if “X _almost_ scored a goal from Y metres”
then the goal effectively did not happen. On the other hand, distinguishing
between original and modified examples is hard by relying on lexical cues
only, as the modifications keep a similar lexical form. As a consequence, the
simplified decision rule hypothesised above would not apply anymore.

Manually curating evaluation data to incorporate SAM is expensive and requires
expert knowledge; also, the process must be repeated for each dataset resource
(Gardner et al. 2020). Automatically changing existing MRC data is not a
feasible strategy either, as the effects of a change on the meaning of the
passage cannot be traced through the process and will still need to be
verified manually. Instead, in this paper we propose a novel methodology to
generate SAM MRC challenge sets. We employ template-based natural language
generation to maintain control over the presence of SAM and their effect onto
the expected answer to a given question.

A problem that arises when evaluating models on challenge sets that were
optimised on different training data, as it is the case in this paper, is the
domain shift between training and evaluation data. For example, a model
trained to retrieve answers from Wikipedia paragraphs might have never
encountered a question involving comparing distances. In this case, wrong
predictions on SAM examples cannot be attributed to the presence of SAM alone.
To disentangle the effects introduced by the domain shift from the actual
capability of correctly processing examples featuring SAM, we introduce a
novel evaluation methodology with a corresponding metric, which we refer to as
Domain Independent Consistency Evaluation or $DICE$. This allows us to
precisely measure the capability of MRC models to process SAM of interest, and
therefore, evaluate comprehension and language understanding that cannot be
easily circumvented by relying on superficial cues. In a large-scale empirical
study111Code and Supplementary Materials SM1, SM2 and SM3 can be retrieved
from https://github.com/schlevik/sam, we evaluate the performance of state-of-
the-art transformer-based architectures optimised on multiple extractive MRC
datasets. We find that while approaches based on larger language models tend
to perform better, all investigated models struggle on the proposed challenge
set, even after discounting for domain shift effects.

## Semantics Altering Modifications

The task of (extractive) Machine Reading Comprehension is formalised as
follows: given a question $Q$ and a context $P$ consisting of words
$p_{0}\ldots p_{n}$, predict the start and end indices $s,e$ (where $s<e$)
that constitute the answer span $A=p_{s}\ldots p_{e}$ in $P$. A Semantics
Altering Modification (SAM) refers to the process of changing answer $A$ to
$A^{\prime}\neq A$ by applying a modification to the accompanying context $P$.
The rationale is to create a new _intervention_ instance
$(Q,P^{\prime},A^{\prime})$ that is lexically similar to the original but has
a different meaning and therefore a different expected answer for the same
question. Predicting both $A$ and $A^{\prime}$ given the question and the
respective passages becomes a more challenging task than predicting $A$ alone,
since it requires correctly processing and distinguishing both examples. Due
to their similarity, any simplifying heuristics inferred from training data
are more likely to fail.

Furthermore, this intuitive description aligns with one of the prevalent
linguistic definitions of modifiers as “an expression that combines with an
unsaturated expression to form another unsaturated expression of the same
[semantic] type” (McNally 2016). Particularly applicable to our scenario is
the pragmatic or discourse-related view, specifically the distinction between
modifiers that contribute to the content of a sentence with regard to a
specific issue, and those that do not. In the context of MRC, the issue is
whether the modification is relevant to finding the answer $A$ to the question
$Q$.

The linguistics literature is rich in reporting phenomena conforming with this
definition. In this paper we explore negation (Morante and Daelemans 2012),
(adverbial) restrictivity modification (Tenny 2000, Sec. 6), polarity
reversing verbs and expressions (Karttunen 1971, 2012) and expressions of
implicit negation (Iyeiri 2010). The categories with representative examples
are shown in Figure 1 and labelled _I1-I6_. They reflect our intuitive
definition as they involve relatively small edits to the original context, by
inserting between one and four words that belong to the most frequent parts of
speech classes of the English language, i.e. adverbials, modals, verbs and
nouns. Note, however, that this selection is non-exhaustive. Other linguistic
phenomena such as privative adjectives (Pavlick and Callison-Burch 2016), noun
phrase modification (Stanovsky and Dagan 2016) or—if one were to expand the
semantic types-based definition introduced above—corresponding discourse
relations, such as Contrast or Negative Condition (Prasad et al. 2008), or
morphological negation constitute further conceivable candidates. We leave it
for future work to evaluate MRC on other types of SAM.

## Domain Independent Consistency Evaluation

Consistency on “contrastive sets” (Gardner et al. 2020) was recently proposed
as a metric to evaluate the comprehension of NLP models beyond simplifying
decision rules. A contrastive set is—similar to SAM—a collection of similar
data points that exhibit minimal differences such that the expected prediction
(e.g. answer for MRC) differs for each member. Consistency is then defined as
the ratio of contrastive sets where the model yielded a correct prediction for
all its members to the total number of contrastive sets.

This notion requires that evaluation examples stem from the same generative
process as the training data, making the process of finding contrastive sets
dataset-dependent. If the processes are different however, as it is the case
with training set-independent challenge sets, this difference can be a
confounding factor for wrong predictions, i.e. a model might produce a wrong
prediction because the input differs too much from its training data and not
solely because it was not capable of solving the investigated phenomenon. As
we aim to establish an evaluation methodology independent of training data, we
propose the following approach in order to rightfully attribute the capability
to correctly process SAM even under domain shift.

We align each _baseline_ MRC instance consisting of question, expected answer
and context triple $B_{i}=(Q_{i},A_{i},P_{i})$ with an _intervention_ instance
$I_{i}=(Q_{i},A^{\prime}_{i},P^{\prime}_{i})$ s.t. $A^{\prime}_{i}\neq A_{i}$.
In practice, we achieve this by inserting a SAM in the sentence of $P_{i}$
that contains $A_{i}$ in order to obtain $P^{\prime}_{i}$. We further align a
_control_ instance where we completely remove the sentence that was modified
in $P^{\prime}_{i}$, i.e. $C_{i}=(Q_{i},A^{\prime}_{i},P^{\prime\prime}_{i})$.
Thus, an _aligned_ instance consists of the triple $(B_{i},I_{i},C_{i})$
sharing the question $Q$. The answer $A^{\prime}$ is equivalent for both
$I_{i}$ and $C_{i}$. $P,P^{\prime}$ and $P^{\prime\prime}$ are shown in Figure
1 by selecting original (B) for $P$, any of the alternatives (I1) through (I6)
for $P^{\prime}$ and completely removing the first sentence for
$P^{\prime\prime}$. Formulating the problem in this way is similar to the
Winograd Schema Challenge (Levesque, Davis, and Morgenstern 2012).

The goal is to establish first, whether the model under evaluation
“understood” the question and the accompanying context. Namely, if the model
predicted $A_{i}$ and $A^{\prime}_{i}$ correctly given $Q_{i},P_{i}$ and
$Q_{i},P^{\prime\prime}_{i}$, respectively, we conclude that the domain shift
is not pivotal for the prediction performance of this particular instance,
thus predicting the correct answer $A^{\prime}_{i}$ for $I_{i}$ can be
attributed to the model’s capability to correctly process the SAM in
$P^{\prime}_{i}$. Conversely, if the model fails to predict $A^{\prime}$ we
assume that the reason for this is is its incapability to process SAM (for
this instance), regardless of the domain shift.

Initial experiments showed that models sometimes struggle to predict the exact
span boundaries of the expected answer while retrieving the correct
information in principle (e.g. predicting “from 26 metres” vs. the expected
answer “26 metres”). Therefore we relax the usual _Exact Match_ measure $EM$
to establish the correctness of a prediction in the following way:
$rEM_{k}(\hat{A},A)=1$ if a $\hat{A}$ has at most $k$ words and $A$ is a
substring of $\hat{A}$ and $0$ otherwise, where $\hat{A}=f_{\theta}(Q,P)$ is
the answer prediction of an optimised MRC model $f_{\theta}$ given question
$Q$ and context $P$.

The metric $DICE$ is the number of examples the model predicted correctly in
their baseline, intervention and control version divided by the number of
those the model predicted correctly for the baseline and control version. This
notion reflects the ratio of those modified instances that the model processed
correctly regardless of the domain shift thus further evaluating the model’s
reading comprehension. Formally, for a challenge set
$\mathcal{N}=\mathcal{\\{B,I,C\\}}$ consisting of $N$ baseline, intervention
and control examples, let

$\displaystyle\begin{aligned} \mathcal{B^{+}}&=\\{i\ |\
rEM_{k}(f_{\theta}(Q_{i},P_{i}),A_{i})=1\\}_{i\in\\{1\ldots N\\}}\\\
\mathcal{I^{+}}&=\\{i\ |\
rEM_{k}(f_{\theta}(Q_{i},P^{\prime}_{i}),A^{\prime}_{i})=1\\}_{i\in\\{1\ldots
N\\}}\\\ \mathcal{C^{+}}&=\\{i\ |\
rEM_{k}(f_{\theta}(Q_{i},P^{\prime\prime}_{i}),A^{\prime}_{i})=1\\}_{i\in\\{1\ldots
N\\}}\end{aligned}$ (1)

denote the set of indices where an optimised model $f_{\theta}$ predicted a
correct answer for baseline, intervention and control instances, respectively.
Then

$DICE(f_{\theta})=\frac{|\mathcal{B^{+}}\cap\mathcal{I^{+}}\cap\mathcal{C^{+}}|}{|\mathcal{B^{+}}\cap\mathcal{C^{+}}|}\in[0,1].$
(2)

An inherent limitation of challenge sets is that they bear negative predictive
power only (Feng, Wallace, and Boyd-Graber 2019). Translated to our
methodology, this means that while low $DICE$ scores hint at the fact that
models circumvent comprehension, high scores do not warrant the opposite, as a
model still might learn to exploit some simple decision rules in cases not
covered by the challenge set. In other words, while necessary, the capability
of distinguishing and correctly processing SAM examples is not sufficient to
evaluate reading comprehension.

A limitation specific to our approach is that it depends on a model’s
capability to perform under domain shift, at least to some extent. If a model
performs poorly because of insufficient generalisation beyond training data or
if the training data are too different from that of the challenge set, the
sizes of $\mathcal{B^{+}},\mathcal{I^{+}}$ and $\mathcal{C^{+}}$ decrease and
therefore variations due to chance have a larger contribution to the final
result. Concretely, we found that if the question is not formulated in natural
language, as is the case for WikiHop (Welbl, Stenetorp, and Riedel 2018), or
the context does not consist of coherent sentences (with SearchQA (Dunn et al.
2017) as an example) optimised models transfer poorly. Having a formalised
notion of dataset similarity with respect to domain transfer for the task of
MRC would help articulate the limitations and application scenarios of the
proposed approach beyond pure empirical evidence.

## SAM Challenge Set Generation

We now present the methodology for generating and modifying passages at scale.
We aim to generate examples that require “reasoning skills” typically found in
state-of-the-art MRC benchmarks (Sugawara et al. 2017; Schlegel et al. 2020).
Specifically, we choose to generate football match reports as it intuitively
allows us to formulate questions that involve simple (e.g. _“Who scored the
first/last goal?”_) and more complex (e.g. _“When was the second/second to
last goal scored?”_) linear retrieval capabilities, bridging and understanding
the temporality of events (e.g. _“Who scored before/after X was fouled?”_) as
well as ordering (e.g. _“What was the farthest/closest goal?”_) and comparing
numbers and common properties (e.g. _“Who assisted the earlier goal, X or
Y?”_). Answer types for these questions are named entities (e.g. players) or
numeric event attributes (e.g. time or distance).

Selected Content Plan
---
1 (Order (Distance (Modified Goal) 0)
2 (Order (Distance (Just Goal) 1)
Q (Argselect Max Goal Distance Actor)
Generated Events
1 {actor: p4, distance: 26, mod: I2 …}
2 {actor: p2, distance: 23 …}
A: p4 A’: p2
Chosen Templates (Simplified)
1 %Con #Actor @SAM $V.Goal $PP.Distance…
2 #Actor %Con she $V.Score $PP.Distance…
Q Who scored the farthest goal ?
Generated Text
$P$: After the kickoff Naomi Daniel curled in a goal …
$P^{\prime}$: After the kickoff Naomi Daniel  almost curled in …
$P^{\prime\prime}$: Then Amanda Collins added more insult to the …
Figure 2: Stages of the generative process that lead to the question answer
and context in Figure 1. The _Content Plan_ describes the general constraints
that the question type imposes on the _Events_ (both sentences must describe
goal events, first sentence must contain SAM, distance attribute must be
larger in the modified sentence). Appropriate _Templates_ are chosen randomly
to realise the final Baseline $P$, Intervention $P^{\prime}$ and Control
$P^{\prime\prime}$ version of the passage.

To generate passages and questions, we pursue a staged approach, common in
Natural Language Generation (Gatt and Krahmer 2018). Note that we choose a
purely symbolic approach over statistical approaches in order to maintain full
control over the resulting questions and passages as well as the implications
of their modification for the task of retrieving the expected answer. Our
pipeline is exemplified in Figure 2 and consists of _(1)_ content
determination and structuring, followed by _(2)_ content generation (as we
generate the content from scratch) and finally _(3)_ lexicalisation and
linguistic realisation combining templates and a generative grammar.

Content planning and generation: The output of this stage is a structured
report of events that occurred during a fictitious match, describing event
properties such as actions, actors and time stamps. Furthermore each report is
paired with a corresponding question, an indication of which event is to be
modified, and the corresponding answers. The report is generated semi-
randomly, as the requirement to generate instances with a _meaningful_
modification–i.e. actually changing the valid answer to the question–imposes
constraints that depend on the type of the question. For example, for the
retrieval type question _”Who scored the farthest goal?”_ the report must
contain at least two events of the type “goal” and the distance attribute
associated with the event to be modified must be larger. We generate events of
the type “goal”, which are the target of the generated questions and
modifications, and “other” that diverify the passages. Furthermore, to prevent
repetition, we ensure that the order of the types of events is unique for each
report-question combination in the final set of generated reports.

Realisation: For the sake of simplicity, we choose to represent each event
with a single sentence, although it is possible to omit this constraint by
using sentence aggregation techniques and multi-sentence templates. Given a
structured event description, we randomly select a “seed” template suitable
for the event type. Seed templates consist of variables that are further
substituted by event properties and expressions generated by the grammar.
Thereby, we distinguish between context-free and context-sensitive
substitutions. For example `$PP.Distance` in Figure 2 is substituted by a
randomly generated prepositional phrase describing the distance (e.g. _“from
26 metes away”_) regardless of its position in the final passage. `%Con` in
the same figure is substituted by an expression that connects to the previous
sentence and depends on its content (e.g. _“After the kick-off”_ can only
appear in the first sentence of the paragraph). We collect the templates and
construct the grammar by combining various manual and automated measures
described in SM: A in more detail. Similarly to the content generation, we
ensure that the same template is not used more than once per report and the
permutation of templates used to realise a report is unique in the final set
of realised reports.

Data description: The challenge set used in the experiments to evaluate MRC
models trained on existing datasets consists of $4200$ aligned baseline,
intervention and control examples generated using the above process. The
modified intervention examples contain between one and three SAM from the six
categories described earlier. Using $25$ “seed” templates and a generative
grammar with $230$ production rules, we can realise an arbitrary event in
$4.8\times 10^{6}$ lexically different ways; for a specific event the number
is approx. $7.8\times 10^{5}$ on average (the difference is due to context-
sensitive parts of the grammar). When fine-tuning MRC models on our generated
data, we separate the seed templates in two distinct sets, in order to ensure
that the models do not perform well by just memorising the templates. These
template sets are used to generate a training (12000 instances) and an
evaluation (2400 instances) set with aligned baseline, intervention and
control instances. All reports consist of six events and sentences, the
average length of a realised passage is $174$ words, averaging $10.8$ distinct
named entities and $6.9$ numbers as answer candidates.

To estimate how realistic the generated MRC data is, we compare the paragraphs
to the topically most similar MRC data: the NFL subset of the DROP dataset
(Dua et al. 2019). We measure the following two metrics. _Lexical Similarity_
is the estimated Jaccard similarity between two paragraphs, i.e. the ratio of
overlapping words, with lower scores indicating higher (lexical) diversity. As
a rough estimate of _Naturality_ , we calculate the average of those sentence-
level cohesion indices that are reported to correlate with human judgements of
writing quality (Crossley, Kyle, and McNamara 2016; Crossley, Kyle, and
Dascalu 2019). For exact definitions and further results please consult SM1.
The results are shown below:

Data/Metric | Lex. Similarity $\downarrow$ | Naturality $\uparrow$
---|---|---
SAM ($n=200$) | $0.22$ | $0.65$
NFL ($n=188$) | $0.16$ | $0.68$

While not quite reaching the reference data due to its template-based nature
we conclude that the generated data is of sufficient quality for our purposes.

## Experiments Setup

Broadly, we address the following question: _How well does MRC perform on
Semantic Altering Modifications?_

In this study we focus our investigations on extractive MRC where the question
is in natural language, the context is one or more coherent paragraphs and the
answer is a single continuous span to be found within the context. To that end
we sample state-of-the-art (neural) MRC architectures and datasets and perform
a comparative evaluation. Scores of models with the same architecture
optimised on different data allow to compare how much these data enable models
to learn to process SAM, while comparing models with different architectures
optimised on the same data hints to which extent these architectures are able
to obtain this capability from data. Below we outline and motivate the choices
of datasets and models used in the study. For further details on data
preparation, model training and how we obtain predictions for evaluation,
please consult SM2.

Datasets: We select the following datasets in an attempt to comprehensively
cover various flavours of state-of-the-art MRC consistent with our definition
above.

  * •

SQuAD (Rajpurkar et al. 2016) is a widely studied dataset where the human
baseline is surpassed by the state of the art.

  * •

HotpotQA (Yang et al. 2018) in the “distractor” setting requires information
synthesis from multiple passages in the context connected by a common entity
or its property.

  * •

DROP (Dua et al. 2019) requires performing simple arithmetical tasks in order
to predict the correct answer.

  * •

NewsQA (Trischler et al. 2017) contains questions that were created without
having access to the provided context. The context is a news article,
different from the other datasets where contexts are Wikipedia excerpts.

Similar to Talmor and Berant (2019), we convert the datasets into the same
format for comparability and to suit the task definition of extractive MRC.
For HotpotQA we concatenate multiple passages into a single context and for
DROP and NewsQA we only include examples where the question is answerable and
the answer is a continuous span in the paragraph and refer to them as DROP’
and NewsQA’, respectively.

Models: The respective best-performing models on these datasets are employing
a large transformer-based language model with a task-specific network on top.
Note that we do not use architectures that make dataset-specific assumptions
(e.g. “Multi-hop” for HotpotQA) in order to maintain comparability of the
architectures across datasets. Instead, we employ a linear layer as the most
generic form of the task-specific network (Devlin et al. 2019). Following
common practice, we concatenate the question and context, and optimise the
parameters of the linear layer together with those of the language model to
minimise the cross-entropy loss between the predicted and expected start and
end indices of the answer span (and the answer sequence for the generative
model).

We are interested in the effects of various improvements that were proposed
for the original BERT transformer-based language model (Devlin et al. 2019).
Concretely, we compare the effects of more training data and longer training
for the language model (e.g. XLNet (Yang et al. 2019), RoBERTa (Liu et al.
2019)), parameter sharing between layers of the transformer (e.g. ALBERT (Lan
et al. 2020)) and utilising a unifying sequence-to-sequence interface (e.g.
BART (Lewis et al. 2020), T5 (Raffel et al. 2020)) and reformulating
extractive MRC as text generation conditioned on the question and passage. We
evaluate models of different sizes, ranging from base (small for T5) to large
(and xl and xxl for ALBERT). They describe specific configurations of the
transformer architecture, such as the number of the self-attention layers and
attention heads and the dimensionality of hidden vectors. For an in-depth
discussion please refer to Devlin et al. (2019) and the corresponding papers
introducing the architectures. For comparison, we also include the non-
transformer based BiDAF model (Seo et al. 2017). Finally, we train a model of
the best performing architecture on a combination of all four datasets
(*-comb) to investigate the effects of increasing training data diversity. For
this, we sample and combine 22500 instances from all four datasets to obtain a
training set that is similar in size to the others. The final selection
consists of the models reported in Table 1.

| Average | SQuAD | HotpotQA | NewsQA’ | DROP’
---|---|---|---|---|---
Architecture | $DICE$ | EM/F1 | $DICE$ | EM/F1 | $DICE$ | EM/F1 | $DICE$ | EM/F1 | $DICE$
bidaf | $11\pm 3$ | $67.2/76.9$ | $12\pm 4$ | $44.6/57.9$ | $4\pm 3$ | $40.0/54.3$ | $13\pm 5$ | $50.8/56.8$ | $18\pm 12$
bert-base | $13\pm 2$ | $76.3/84.9$ | $13\pm 3$ | $50.7/64.9$ | $17\pm 4$ | $46.6/62.5$ | $13\pm 3$ | $50.5/58.2$ | $10\pm 3$
bert-large | $15\pm 2$ | $81.9/89.4$ | $15\pm 3$ | $54.4/68.7$ | $14\pm 3$ | $49.1/65.7$ | $14\pm 4$ | $62.2/68.7$ | $16\pm 3$
roberta-base | $15\pm 2$ | $82.4/89.9$ | $8\pm 3$ | $51.9/66.4$ | $17\pm 4$ | $50.8/66.9$ | $14\pm 3$ | $63.5/69.3$ | $20\pm 3$
roberta-large | $18\pm 1$ | $86.4/93.3$ | $16\pm 3$ | $58.6/72.9$ | $21\pm 3$ | $54.4/71.1$ | $15\pm 3$ | $77.3/82.8$ | $20\pm 2$
albert-base | $14\pm 2$ | $82.8/90.3$ | $10\pm 3$ | $55.4/69.7$ | $17\pm 3$ | $49.7/65.7$ | $11\pm 3$ | $60.7/67.0$ | $18\pm 4$
albert-large | $16\pm 1$ | $85.4/92.1$ | $18\pm 3$ | $59.4/73.7$ | $12\pm 2$ | $52.5/68.9$ | $17\pm 3$ | $69.3/75.1$ | $18\pm 3$
albert-xl | $27\pm 2$ | $87.1/93.5$ | $19\pm 2$ | $62.4/76.2$ | $21\pm 3$ | $54.2/70.4$ | $29\pm 3$ | $76.4/81.8$ | $40\pm 3$
albert-xxl | $27\pm 1$ | $88.2/94.4$ | $29\pm 2$ | $65.9/79.5$ | $29\pm 3$ | $54.3/71.0$ | $25\pm 3$ | $78.4/84.5$ | $23\pm 2$
t5-small | $10\pm 1$ | $76.8/85.8$ | $13\pm 3$ | $51.8/65.6$ | $10\pm 3$ | $47.3/63.3$ | $8\pm 2$ | $60.4/66.1$ | $10\pm 3$
t5-base | $16\pm 1$ | $82.4/90.6$ | $16\pm 3$ | $61.0/74.4$ | $20\pm 3$ | $52.4/68.8$ | $14\pm 3$ | $69.0/74.9$ | $15\pm 2$
t5-large | $20\pm 1$ | $86.3/93.1$ | $21\pm 2$ | $65.0/78.5$ | $29\pm 3$ | $53.4/70.0$ | $16\pm 3$ | $70.1/75.3$ | $8\pm 2$
average | $19\pm 0$ | $76.4/83.2$ | $18\pm 1$ | $53.1/65.9$ | $20\pm 1$ | $47.1/62.1$ | $17\pm 1$ | $61.5/67.0$ | $20\pm 1$
albert-xl-comb | $20\pm 2$ | $85.3/92.2$ |  | $60.6/74.3$ |  | $53.6/70.4$ |  | $76.9/82.4$ |
random | $5\pm 0$ |
learned | $98\pm 0$ |
Table 1: $DICE$ and EM/F1 score on the corresponding development sets of the
evaluated models. Average $DICE$ scores are micro-averaged as it better shows
the average performance on processing SAM examples while EM/F1 are macro-
averaged as it reflects the average performance on the datasets (although the
difference between both averaging methods is small in practice).

Baselines: We implement a random baseline that chooses an answer candidate
from the pool of all named entities and numbers and an informed baseline that
chooses randomly between all entities matching the expected answer type (e.g.
person for “Who” questions). Finally, in order to investigate whether the
proposed challenge can be solved in general, we train a bert-base model on
12000 aligned baseline and intervention instances, each. We refer to this
baseline as learned. We train two more bert-base _partial baselines_ ,
masked-q and masked-p on the same data where, respectively, the question and
passage tokens (except for answer candidates) are replaced by out-of-
vocabulary tokens. Our motivation for doing this is to estimate the proportion
of the challenge set that can be solved due to regularities in the data
generation method, regardless of the realised lexical form to provide more
context to the performance of the learned baseline. Finally, we estimate the
human SAM performance by crowd-sourcing the manual annotation of 100
intervention examples.

## Results and Discussion

We present the main findings of our study here. For the obtained $DICE$ scores
we report the error margin as a confidence interval at $\alpha=0.05$ using
asymptotic normal approximation. Any comparisons between two $DICE$ scores
reported in this section are statistically significant ($p<0.05$) as
determined by performing the Fisher’s exact test.

Baseline | $\mathcal{B}$ | $\mathcal{I}$ | $\mathcal{C}$
---|---|---|---
learned | $81\pm 2$ | $79\pm 2$ | $76\pm 2$
masked-q | $20\pm 2$ | $28\pm 2$ | $26\pm 1$
masked-p | $29\pm 1$ | $5\pm 1$ | $1\pm 1$
random | $6\pm 1$ | $5\pm 1$ | $8\pm 1$
informed | $14\pm 1$ | $14\pm 1$ | $26\pm 2$
human | – | $87\pm 7$ | –
Table 2: Percentage of correct predictions of the introduced baselines under
the $rEM_{5}$ metric on aligned baseline $\mathcal{B}$, intervention
$\mathcal{I}$ and control $\mathcal{C}$ sets.

_SAM is learnable._ As expected, the learned baseline achieves high accuracy
on our challenge set, with 81% and 79% correctly predicted instances for
baseline and intervention examples, respectively, as seen in Table 2. The
results are in line with similar experiments on Recognising Textual Entailment
(RTE) and sentiment analysis tasks which involved aligned counterfactual
training examples (Kaushik, Hovy, and Lipton 2020). They suggest that neural
networks are in fact capable of learning to recognise and correctly process
examples with minimal yet meaningful differences such as SAM when explicitly
optimised to do so. Some part of this performance is to be attributed to
exploiting the regularity of the generation method rather than processing the
realised text only, however, as the partial baselines perform better than the
random baselines. This is further indicated by the slightly lower performance
on the _control_ set, where due to deletion the number of context sentences is
different compared to the baseline and intervention sets.

We note that the learned model does not reach 100% EM score on this
comparatively simple task, possibly due to the limited data diversity imposed
by the templates. Using more templates and production rules and a bigger
vocabulary will further enhance the diversity of the data.

_Pre-trained models struggle._ Table 1 reports the results of evaluating
state-of-the-art MRC. Trained models struggle to succeed on our challenge set,
with the best $DICE$ score of $40$ achieved by albert-xlarge when trained on
DROP’. There is a log-linear correlation between the effective size of the
language model (established by counting the shared parameters separately for
each update per optimisation step) and the SAM performance with Spearman’s
$r~{}=~{}0.93$. Besides the model size, we do not find any contribution that
leads to a considerable improvement in performance of _practical_
significance. We note that simply increasing the data diversity appears not
beneficial, as the score of albert-xl-comb that was optimised on the
combination of all four datasets is lower than the average score of the
corresponding albert-xl model. Humans appear to have little issues to find the
correct answer, with $87\%\pm 7\%$ of the intervention examples solved
correctly. This provides the worst-case estimate for the human $DICE$ score,
assuming all corresponding baseline and control examples would have been
solved correctly (more details on how we establish the human baseline in SM3).

The easiest SAM category to process is _I6: Explicit negation_ with all
optimised models scoring $26\pm 1.4$ on average. Models struggle most with
_I2: Adverbial Modification_ , with an average $DICE$ score of $14\pm 1$ (see
SM3 for breakdown by SAM category). A possible reason is that this category
contains _degree modifiers_ , such as _“almost”_. While they alter the
semantics in the same way as other categories for our purposes, generally they
act as a more nuanced modification (compare e.g. _“almost”_ with _“didn’t”_).
Finally, we note that the performance scales negatively with the number of SAM
present in an example. The average $DICE$ score on instances with a single SAM
is $23\pm 0.9$, while on instances with the maximum of three SAM it is $16\pm
0.8$ (and $19\pm 1.0$ for two SAM). This is reasonable, as more SAM requires
to process (and discard) more sentences, giving more opportunities to err.

We highlight that models optimised on HotpotQA and DROP’ perform slightly
better than models optimised on SQuAD and NewsQA’ (on average $20\%$ vs $18\%$
and $17\%$, respectively). This suggests that exposing models to training data
that require more complex (e.g. “multihop” and arithmetic) reasoning to deduce
the answer, as opposed to simple answer retrieval based on predicate-argument
structure (Schlegel et al. 2020), has a positive effect on distinguishing and
correctly processing lexically similar yet semantically different instances.

_Small improvements can be important._ Our results indicate that small
differences at the higher end of the performance spectrum can be of practical
significance for the comprehension of challenging examples, such as SAM.
Taking albert as an example, the relative performance improvements between the
base and xxlarge model when (macro) averaged over the EM and F1 scores on the
corresponding development sets are $15\%$ and $13\%$, respectively, while the
relative difference in average $DICE$ score is $93\%$. This is likely due to a
share of “easy” examples in MRC evaluation data (Sugawara et al. 2018) that
artificially bloat the (lower-end) performance scores to an extent.

_Meaningful training examples are missing._ One possible explanation for low
scores could be that the models simply never encountered the expressions we
use to modify the passages and thus fail to correctly process them. To
investigate this claim we count the occurrences of the expressions of the
worst performing category overall, _I2: Adverbial Modification_. The
expressions appear in $5\%,14\%,5\%$ and $22\%$ of the training passages of
SQuAD, HotpotQA, DROP’ and NewsQA’ respectively, showing that models do
encounter them during task-specific fine-tuning (not to mention during the
language-model pre-training). It is more likely that the datasets lack
examples where these expressions affect the search for the expected answer in
a meaningful way (Schlegel et al. 2020). In fact, after sampling $400$
passages and $647$ corresponding questions (100 passages from each dataset)
where the expression occurs within $100$ characters of the expected answer,
and manually annotating whether the modification would yield a different
answer, we find only one such case which we can thus consider as a naturally
occurring SAM. Worse yet, in $4\%$ of the cases the expected answer ignores
the presence of the SAM. This lends further credibility to the hypothesis that
current MRC struggles at distinguishing examples with minimal yet meaningful
changes such as SAM, if not explicitly incentivised during training. For more
details on this annotation task, see SM3.

An analysis of models’ errors suggests a similar conclusion: examining wrong
intervention predictions for those cases where the answers for baseline and
control were predicted correctly, we find that in $82\%\pm 1\%$ of those cases
the models predict the baseline answer. Models thus tend to ignore SAM, rather
than being “confused” by their presence (as if never encountered during
training) and predicting a different incorrect answer.

## Related Work

Systematically modified MRC data can be obtained by rewriting questions using
rule-based approaches (Ribeiro, Singh, and Guestrin 2018; Ribeiro, Guestrin,
and Singh 2019) or appending distracting sentences, e.g. by paraphrasing the
question (Jia and Liang 2017; Wang and Bansal 2018), or whole documents (Jiang
and Bansal 2019) to the context. Adversarial approaches with the aim to “fool”
the evaluated model, e.g. by applying context perturbations (Si et al. 2020)
fall into this category as well. These approaches differ from ours, however,
in that they aim to preserve the semantics of the modified example, therefore
the expected answer is unchanged. But the findings are similar: models
struggle to capture the semantic equivalence of examples after modification,
and rely on lexical overlap between question and passage (Jia and Liang 2017).
Our approach explores a complementary direction by generating semantically
altered passages.

Using rule-based NLG techniques for controlled generation of MRC data was
employed to obtain stories (Weston et al. 2015) that aim to evaluate the
learnability of specific reasoning types, such as inductive reasoning or
entity tracking. Further examples are TextWorld (Côté et al. 2018), an
environment for text-based role playing games with a dataset where the task is
to answer a question by interactively exploring the world (Yuan et al. 2019)
and extending datasets with unanswerable questions (Nakanishi, Kobayashi, and
Hayashi 2018). Similar to our approach, these generation methods rely on
symbolic approaches to maintain control over the semantics of the data.

Beyond MRC, artificially constructed challenge sets were established with the
aim to evaluate specific phenomena of interest, particularly for the RTE task.
Challenge sets were proposed to investigate neural RTE models for their
capabilities to perform logic reasoning (Richardson and Sabharwal 2019) and
lexical inference (Glockner, Shwartz, and Goldberg 2018), or understanding
language compositionality (Nie, Wang, and Bansal 2019; Geiger et al. 2019).

## Conclusion

We introduce a novel methodology for evaluating the reading comprehension of
MRC models by observing their capability to distinguish and correctly process
lexically similar yet semantically different input. We discuss lingustic
phenomena that act as Semantic Altering Modifications and present a
methodology to automatically generate and meaningfully modify MRC evaluation
data. In an empirical study, we show that while the capability to process SAM
correctly is learnable in principle, state-of-the-art MRC architectures
optimised on various MRC training data struggle to do so. We conclude that one
of the key reasons for this is the lack of challenging SAM examples in the
corresponding datasets.

Future work will include the search for and evaluation on further linguistic
phenomena suitable for the purpose of SAM, expanding the study from strictly
extractive MRC to other formulations such as generative or multiple-choice
MRC, and collecting a large-scale natural language MRC dataset featuring
aligned SAM examples (e.g. via crowd-sourcing) in order to investigate the
impact on the robustness of neural models when exposed to those examples
during training.

## Acknowledgements

The authors would like to thank the anonymous reviewers for valuable
suggestions as well as Alessio Sarullo for many fruitful discussions and help
with the design of the experiments. The computation-heavy aspects of this
paper were made possible due to access to the Computational Shared Facility at
The University of Manchester, for which the authors are grateful.

## References

  * Côté et al. (2018) Côté, M.-A.; Kádár, A.; Yuan, X.; Kybartas, B.; Barnes, T.; Fine, E.; Moore, J.; Tao, R. Y.; Hausknecht, M.; Asri, L. E.; Adada, M.; Tay, W.; and Trischler, A. 2018.  TextWorld: A Learning Environment for Text-based Games.  _arXiv preprint arXiv:1806.11532_ .
  * Crossley, Kyle, and Dascalu (2019) Crossley, S. A.; Kyle, K.; and Dascalu, M. 2019.  The Tool for the Automatic Analysis of Cohesion 2.0: Integrating semantic similarity and text overlap.  _Behavior Research Methods_ 51(1): 14–27.
  * Crossley, Kyle, and McNamara (2016) Crossley, S. A.; Kyle, K.; and McNamara, D. S. 2016.  The tool for the automatic analysis of text cohesion (TAACO): Automatic assessment of local, global, and text cohesion.  _Behavior Research Methods_ 48(4): 1227–1237.
  * Devlin et al. (2019) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 NAACL Conference_ , 4171–4186.
  * Dua et al. (2019) Dua, D.; Wang, Y.; Dasigi, P.; Stanovsky, G.; Singh, S.; and Gardner, M. 2019.  DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs.  In _Proceedings of the 2019 NAACL Conference_ , 2368–2378.
  * Dunn et al. (2017) Dunn, M.; Sagun, L.; Higgins, M.; Guney, V. U.; Cirik, V.; and Cho, K. 2017.  SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine.  _arXiv preprint arXiv 1704.05179_ .
  * Feng, Wallace, and Boyd-Graber (2019) Feng, S.; Wallace, E.; and Boyd-Graber, J. 2019.  Misleading Failures of Partial-input Baselines.  In _Proceedings of the 57th ACL Annual Meeting_ , 5533–5538.
  * Gardner et al. (2020) Gardner, M.; Artzi, Y.; Basmova, V.; Berant, J.; Bogin, B.; Chen, S.; Dasigi, P.; Dua, D.; Elazar, Y.; Gottumukkala, A.; Gupta, N.; Hajishirzi, H.; Ilharco, G.; Khashabi, D.; Lin, K.; Liu, J.; Liu, N. F.; Mulcaire, P.; Ning, Q.; Singh, S.; Smith, N. A.; Subramanian, S.; Tsarfaty, R.; Wallace, E.; Zhang, A.; and Zhou, B. 2020.  Evaluating NLP Models via Contrast Sets.  _Findings of the Association for Computational Linguistics: EMNLP 2020_ 1307–1323.
  * Gatt and Krahmer (2018) Gatt, A.; and Krahmer, E. 2018.  Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation.  _Journal of Artificial Intelligence Research_ 61: 65–170.
  * Geiger et al. (2019) Geiger, A.; Cases, I.; Karttunen, L.; and Potts, C. 2019.  Posing Fair Generalization Tasks for Natural Language Inference.  In _Proceedings of the 2019 EMNLP-IJCNLP Conference_ , 4484–4494.
  * Glockner, Shwartz, and Goldberg (2018) Glockner, M.; Shwartz, V.; and Goldberg, Y. 2018.  Breaking NLI Systems with Sentences that Require Simple Lexical Inferences.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , 650–655.
  * Gururangan et al. (2018) Gururangan, S.; Swayamdipta, S.; Levy, O.; Schwartz, R.; Bowman, S.; and Smith, N. A. 2018.  Annotation Artifacts in Natural Language Inference Data.  In _Proceedings of the 2018 NAACL Conference_ , 107–112.
  * Iyeiri (2010) Iyeiri, Y. 2010.  _Verbs of Implicit Negation and their Complements in the History of English_.  John Benjamins Publishing.
  * Jia and Liang (2017) Jia, R.; and Liang, P. 2017.  Adversarial Examples for Evaluating Reading Comprehension Systems.  In _Proceedings of the 2017 EMNLP Conference_ , 2021–2031.
  * Jiang and Bansal (2019) Jiang, Y.; and Bansal, M. 2019.  Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA.  In _Proceedings of the 57th ACL Annual Meeting_ , 2726–2736.
  * Karttunen (1971) Karttunen, L. 1971.  Implicative Verbs.  _Language_ 47(2): 340.
  * Karttunen (2012) Karttunen, L. 2012.  Simple and Phrasal Implicatives.  In _Proceedings of the 2012 *SEM conference_ , 124–131.
  * Kaushik, Hovy, and Lipton (2020) Kaushik, D.; Hovy, E.; and Lipton, Z. C. 2020.  Learning the Difference that Makes a Difference with Counterfactually-Augmented Data.  In _Proceedings of ICLR_.
  * Lan et al. (2020) Lan, Z.; Chen, M.; Goodman, S.; Gimpel, K.; Sharma, P.; and Soricut, R. 2020.  ALBERT: A Lite BERT for Self-supervised Learning of Language Representations.  In _Proceedings of ICLR_.
  * Levesque, Davis, and Morgenstern (2012) Levesque, H.; Davis, E.; and Morgenstern, L. 2012.  The Winograd Schema Challenge.  In _Proceedings of the Thirteenth KR Conference_ , 552–561.
  * Lewis et al. (2020) Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.; Stoyanov, V.; and Zettlemoyer, L. 2020.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.  In _Proceedings of the 58th ACL Annual Meeting_ , 7871–7880.
  * Liu et al. (2019) Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _arXiv preprint arXiv:1907.11692_ .
  * McCoy, Pavlick, and Linzen (2019) McCoy, T.; Pavlick, E.; and Linzen, T. 2019.  Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference.  In _Proceedings of the 57th ACL Annual Meeting_ , 3428–3448.
  * McNally (2016) McNally, L. 2016.  Modification.  In _The Cambridge Handbook of Formal Semantics_ , 442–464. Cambridge University Press.
  * Morante and Daelemans (2012) Morante, R.; and Daelemans, W. 2012.  ConanDoyle-neg: Annotation of negation cues and their scope in Conan Doyle stories.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation_ , 1563–1568.
  * Nakanishi, Kobayashi, and Hayashi (2018) Nakanishi, M.; Kobayashi, T.; and Hayashi, Y. 2018.  Answerable or Not: Devising a Dataset for Extending Machine Reading Comprehension.  In _Proceedings of the 27th COLING Conference_ , 973–983.
  * Nie, Wang, and Bansal (2019) Nie, Y.; Wang, Y.; and Bansal, M. 2019.  Analyzing Compositionality-Sensitivity of NLI Models.  In _Proceedings of the 33rd AAAI Conference_ , 6867–6874.
  * Pavlick and Callison-Burch (2016) Pavlick, E.; and Callison-Burch, C. 2016.  So-Called Non-Subsective Adjectives.  In _Proceedings of the 5th LEXSEM Conference_ , 114–119.
  * Poliak et al. (2018) Poliak, A.; Naradowsky, J.; Haldar, A.; Rudinger, R.; and Van Durme, B. 2018.  Hypothesis Only Baselines in Natural Language Inference.  In _Proceedings of the 7th LEXSEM Conference_ , 180–191.
  * Prasad et al. (2008) Prasad, R.; Dinesh, N.; Lee, A.; Miltsakaki, E.; Robaldo, L.; Joshi, A.; and Webber, B. 2008.  The Penn Discourse TreeBank 2.0.  In _Proceedings of the Sixth International Conference on Language Resources and Evaluation_ , 2961–2968.
  * Raffel et al. (2020) Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _Journal of Machine Learning Research_ 21(140): 1–67.
  * Rajpurkar et al. (2016) Rajpurkar, P.; Zhang, J.; Lopyrev, K.; and Liang, P. 2016.  SQuAD: 100,000+ Questions for Machine Comprehension of Text.  In _Proceedings of the 55th ACL Annual Meeting_ , 2383–2392.
  * Ribeiro, Guestrin, and Singh (2019) Ribeiro, M. T.; Guestrin, C.; and Singh, S. 2019.  Are Red Roses Red? Evaluating Consistency of Question-Answering Models.  In _Proceedings of the 57th ACL Annual Meeting_ , 6174–6184.
  * Ribeiro, Singh, and Guestrin (2018) Ribeiro, M. T.; Singh, S.; and Guestrin, C. 2018.  Semantically Equivalent Adversarial Rules for Debugging NLP models.  In _Proceedings of the 56th ACL Annual Meeting)_ , volume 1, 856–865.
  * Richardson and Sabharwal (2019) Richardson, K.; and Sabharwal, A. 2019.  What Does My QA Model Know? Devising Controlled Probes using Expert Knowledge.  _Transactions of the Association for Computational Linguistics_ 8: 572–588.
  * Schlegel, Nenadic, and Batista-Navarro (2020) Schlegel, V.; Nenadic, G.; and Batista-Navarro, R. 2020.  Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models.  _arXiv preprint arXiv:2005.14709_ .
  * Schlegel et al. (2020) Schlegel, V.; Valentino, M.; Freitas, A.; Nenadic, G.; and Batista-Navarro, R. 2020\.  A Framework for Evaluation of Machine Reading Comprehension Gold Standards.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , 5359–5369.
  * Seo et al. (2017) Seo, M. J.; Kembhavi, A.; Farhadi, A.; and Hajishirzi, H. 2017.  Bidirectional Attention Flow for Machine Comprehension.  In _Proceedings of ICLR_.
  * Si et al. (2020) Si, C.; Yang, Z.; Cui, Y.; Ma, W.; Liu, T.; and Wang, S. 2020.  Benchmarking Robustness of Machine Reading Comprehension Models.  _arXiv preprint arXiv 2004.14004_ .
  * Stanovsky and Dagan (2016) Stanovsky, G.; and Dagan, I. 2016.  Annotating and Predicting Non-Restrictive Noun Phrase Modifications.  In _Proceedings of the 54th ACL Annual Meeting_ , 1256–1265.
  * Sugawara et al. (2018) Sugawara, S.; Inui, K.; Sekine, S.; and Aizawa, A. 2018.  What Makes Reading Comprehension Questions Easier?  In _Proceedings of the 2018 EMNLP Conference_ , 4208–4219.
  * Sugawara et al. (2017) Sugawara, S.; Kido, Y.; Yokono, H.; and Aizawa, A. 2017.  Evaluation Metrics for Machine Reading Comprehension: Prerequisite Skills and Readability.  In _Proceedings of the 55th ACL Annual Meeting_ , 806–817.
  * Talmor and Berant (2019) Talmor, A.; and Berant, J. 2019.  MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension.  In _Proceedings of the 57th ACL Annual Meeting_ , 4911–4921.
  * Tenny (2000) Tenny, C. 2000.  Core events and adverbial modification.  In Tenny, C.; and Pustejovsky, J., eds., _Workshop on events as grammatical objects from the combined perspectives of lexical semantics, logical semantics, and syntax_ , 285–334.
  * Trischler et al. (2017) Trischler, A.; Wang, T.; Yuan, X.; Harris, J.; Sordoni, A.; Bachman, P.; and Suleman, K. 2017.  NewsQA: A Machine Comprehension Dataset.  In _Proceedings of the 2nd Workshop on Representation Learning for NLP_ , 191–200.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017.  Attention Is All You Need.  In _Advances in Neural Information Processing Systems 30_ , 5998–6008.
  * Wang and Bansal (2018) Wang, Y.; and Bansal, M. 2018.  Robust Machine Comprehension Models via Adversarial Training.  In _Proceedings of the 2018 NAACL Conference_ , 575–581.
  * Welbl, Stenetorp, and Riedel (2018) Welbl, J.; Stenetorp, P.; and Riedel, S. 2018.  Constructing Datasets for Multi-hop Reading Comprehension Across Documents.  _Transactions of the Association for Computational Linguistics_ 6: 287–302.
  * Weston et al. (2015) Weston, J.; Bordes, A.; Chopra, S.; Rush, A. M.; van Merriënboer, B.; Joulin, A.; and Mikolov, T. 2015.  Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks.  _arXiv preprint arXiv:1502.05698_ .
  * Yang et al. (2019) Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J.; Salakhutdinov, R. R.; and Le, Q. V. 2019\.  XLNet: Generalized Autoregressive Pretraining for Language Understanding.  In _Advances in Neural Information Processing Systems 32_.
  * Yang et al. (2018) Yang, Z.; Qi, P.; Zhang, S.; Bengio, Y.; Cohen, W.; Salakhutdinov, R.; and Manning, C. D. 2018.  HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering.  In _Proceedings of the 2018 EMNLP Conference_ , 2369–2380.
  * Yuan et al. (2019) Yuan, X.; Côté, M.-A.; Fu, J.; Lin, Z.; Pal, C.; Bengio, Y.; and Trischler, A. 2019.  Interactive Language Learning by Question Answering.  In _Proceedings of the 2019 EMNLP-IJCNLP Conference_ , 2796–2813.