# Learning Probabalistic Graph Neural Networks for Multivariate Time Series
Anomaly Detection

Saswati Ray∗ Sana Lakdawala∗, Mononito Goswami∗, Chufan Gao∗,
{saswatir, slakdawa, mgoswami<EMAIL_ADDRESS>∗ denotes equal
contribution

###### Index Terms:

Multivariate Times Series, Anomaly Detection, Structure Learning, Graph Neural
Networks

##  1 Introduction

An anomaly is loosely defined as any observation which deviates so much from
the remaining observations, so as to arouse suspicions that it was generated
by a different mechanism [1]. In most applications, data is generated by a
data generating process. Anomalies are generated when this process behaves
unusually. Therefore, anomaly detection can provide several useful application
specific insights, e.g. from detecting credit card fraud to tracking
interesting sensor events. Consequently, detecting anomalies in high-
dimensional multivariate time series (MTS) is an extremely important problem
which has been extensively studied in literature [2]. However, most prior work
on anomaly detection does not explicitly model the complex dependencies
between different variables, which significantly limits their ability to
detect anomalous events [3]. For instance, in a water treatment plant, if a
motorized valve is maliciously turned on, it can cause an overflow on a tank,
resulting in anomalous readings in most sensors associated with it111In the
WADI dataset [4], the first attack involves maliciously turning on motorized
valve 1_MV_001 which causes an overflow in the primary tank. This reflects in
sensors 1_LT_001 and 1_FIT_001 associated with the primary tank.. Recently,
Deng et al. [3] proposed the Graph Deviation Network (GDN) which automatically
learns variable dependencies and uses them to identify anomalous behaviour.
Like most Neural Networks (NNs), despite having an impressive accuracy, GDN
produces poor uncertainty estimates. Since overconfident yet incorrect
predictions may be harmful, precise uncertainty quantification is integral for
practical applications of such networks [5]. To this end, we propose GLUE (GDN
with Local Uncertainty Estimation) which not only automatically learns complex
dependencies between different variables in a MTS and uses them to detect
anomalous behaviour, but also models the uncertainty of predictions.

Results on two real world datasets reveal that GLUE performs on par with GDN,
outperforms most popular baseline models, and also learns meaningful
dependencies between variables (Sec. 5). The rest of the paper is organized as
follows. In Sec. 2 we briefly compare our work with prior work. Next, in Sec.
3 we describe the GDN and GLUE models in detail followed by an overview of our
baselines. Sec. 4 and 5 discuss our datasets, experimental setup and results.
We conclude the paper with Sec. 6.

####  1.0.1 Primary Contributions

Our primary contributions are as follows:

  1. 1.

Uncertainty estimation: GDN forecasts sensor readings as point estimates and
like most NNs is likely to produce poor uncertainty predictions. In order to
produce proper uncertainty estimates, GLUE instead predicts the mean $\mu(x)$
and variance $\sigma^{2}(x)$ of a heteroscedastic Gaussian distribution and
treats the observations as a sample from this distribution. As against GDN
which minimizes the Mean Squared Error (MSE) on the training set, GLUE
optimizes Gaussian negative log likelihood [6] of the observations, allowing
us to address prediction uncertainty and ensure interpretable thresholds [7,
8, 5].

  2. 2.

Validation on a new dataset: We evaluate our proposed approach against a suite
of increasingly complex and popular baselines on two real world datasets, the
WADI dataset from the original paper and a new NASA Turbofan dataset (Sec. 4)

  3. 3.

Open Source Implementation: We also release the code of our proposed GLUE
model to ensure reproducibility (github.com/chufangao/GLUE).

##  2 Related Work

###  2.1 Anomaly Detection

There has been much work on time series anomaly detection. Angiulli and
Pizzuti [9] proposed an unsupervised distance-based approach for outlier
detection using sum of distances from a point’s $k$-nearest neighbors. Later,
Shyu et al. [10] used an unsupervised linear-model approach for anomaly
detection using Principal Component Classifiers. More recently, Aggarwal [2]
discussed the use of the deep autoencoder-based reconstruction error as a
measure of anomalousness. While these methods are popular and yield
competitive results across a multitude of problems, they do not account for
the complex dependencies between variables. Recently, Graph Neural Networks
(GNNs) have been widely used to model structured data. For instance, Kipf and
Welling [11] used Graph Convolution Networks (GCNs) to model complex patterns
for classification in graph-structured data, whereas Ye et al. [12] approached
the time series forecasting problem in traffic domain using Spatio-Temporal
GCNs to capture comprehensive spatio-temporal correlations. In general, GNNs
assume that the state of a node is influenced by the states of its neighbour
nodes [3]. Both GLUE and GDN inherit this fundamental property of GNNs.
However, in contrast to GNNs, GLUE and GDN do not share the same set of
parameters for each node, which renders GNNs ineffective in modeling nodes
with different behaviour. Moreover, GNNs require a graph structure as input,
whereas GLUE learns the graph structure from the data.

###  2.2 Uncertainty Estimation

Quantifying uncertainty in NNs predictions is challenging and an as yet
unsolved problem [5]. Hence, GDN [3] like most NNs is likely to produce
overconfident uncertainty estimates 222We must note that GDN technically does
does not provide any uncertainty estimates and just forecasts point estimates
on its forecasted values. Hence, our argument is based on the fact that NNs in
general produce extremely poor uncertainty estimates.. In contrast, GLUE
predicts the mean and variance of a heteroscedastic multivariate Gaussian
distribution and treats the observed sensor readings as samples from this
distribution, following the seminal work by Nix and Weigend [6].

##  3 Approach

Figure 1: Graph Deviation Network with Local Uncertainty Uncertainty
Estimation (GLUE) learns complex dependencies between variables of a MTS and
uses it to identify anomalous behaviour.

Our primary goal is to learn the dependencies between the $N$ sensors of a
system (_structure learning_) and use them to learn a model $\mathcal{M}$ of
normal system operation. We then use $\mathcal{M}$ to flag significant
deviations from the expected sensor values as anomalies (_structured
prediction_). The key ingredients of GLUE are as follows–

1\. Learn sensor embeddings: Sensor embeddings
$\textbf{V}=\\{\textbf{v}_{1},\dots,\textbf{v}_{N}\\}$, capture the unique
characteristics of each of the $N$ sensors. Here, each
$\textbf{v}_{i}\in\mathbb{R}^{d},\forall i\in\\{1\dots N\\}$, where $d$
corresponds to the dimensionality of the embedding vector.

2\. Learn sensor dependencies: Next, we learn the relationships between
sensors using their embeddings and encode them as edges into a directed graph
$\mathcal{G}$ (neighbour graph). This is done by learning a sparse adjacency
matrix $\mathcal{A}$ using top $k$ neighbors for each sensor $i$. The
parameter $k$ controls the sparsity of the resulting neighbour graph
$\mathcal{G}$.

$\displaystyle e_{ji}$
$\displaystyle=\dfrac{\textbf{v}_{i}^{T}\textbf{v}_{j}}{\|\textbf{v}_{i}\|\cdot\|\textbf{v}_{j}\|},\
\forall j\in\\{1,\dots,N\\}$ (1) $\displaystyle\mathcal{A}_{ji}$
$\displaystyle=\mathbbm{1}\\{j\in\texttt{TopK}(\\{e_{ki}:k\in\mathcal{C}_{i}\\})\\}$
(2)

where $\mathcal{C}_{i}\subseteq\\{1,\dots,N\\}\setminus\\{i\\}$. $e_{ji}$ is
simply the cosine similarity between the embeddings of any two sensors $i$ and
$j$, whereas TopK returns the indices of the $k$ nearest nodes.
$\mathcal{C}_{i}$ refers to the number of candidate relationships for sensor
$i$. Domain experts can specify sensors which are more likely to be related to
each other. However, if we are not aware of any relationships apriori, we can
set $\mathcal{C}_{i}=\\{1,\dots,N\\}$.

3\. Predict sensor readings at time $t$ Given a fixed window size $w>0$ of
historical sensor readings from time steps $(t-w)$ to $(t-1)$, we predict
$\mathbf{\hat{s}}_{i}^{(t)}\sim\mathcal{N}(\mu_{\theta}(\mathbf{x}_{i}^{(t)}),\sigma_{\theta}^{2}(\mathbf{x}_{i}^{(t)}))$
i.e. the heteroscedastic Gaussian distribution around the future sensor
reading for $i$’th sensor at time $t$ based on an attention function over its
top-$k$ neighbors $\mathcal{N}(i)$ in $\mathcal{A}$. Here, $\theta$ represent
the GLUE parameters. In GLUE, as in GDN for each sensor $i\in\\{1,\dots,N\\}$,
we subject its historical readings,
$\mathbf{x}_{i}^{(t)}=[\mathbf{s}_{i}^{(t-w)},\dots,\mathbf{s}_{i}^{(t-1)}],\
\mathbf{x}_{i}^{(t)}\in\mathbb{R}^{w}$ to the following transformations. Let
$\textbf{W}\in\mathbb{R}^{d\times w}$ and
$\mathbf{a}_{i}\in\mathbb{R}^{4d\times 1}$ denote a set of weights and
attention mechanism coefficients, respectively. Instead of forecasting the
next $\mathbf{\hat{s}}_{i}^{(t)}$ as GDN does, GLUE predicts the mean
($\mu_{\theta}(\mathbf{x}_{i}^{(t)})$) and variance
($\sigma_{\theta}^{2}(\mathbf{x}_{i}^{(t)})$) of a gaussian distribution and
treats the future reading as a sample from this distribution. We compute
$\mu_{\theta}(\mathbf{x}_{i}^{(t)})$ and
$\sigma_{\theta}^{2}(\mathbf{x}_{i}^{(t)})$ as follows–

$\displaystyle\mathbf{g}_{i}^{(t)}$
$\displaystyle=\textbf{v}_{i}\oplus\textbf{W}\mathbf{x}_{i}^{(t)}$ (3)
$\displaystyle\pi(i,j)$
$\displaystyle=\texttt{LeakyReLU}(\mathbf{a}^{T}(\mathbf{g}_{i}^{(t)}\oplus\mathbf{g}_{j}^{(t)})),\
j\in\mathcal{N}(i)$ (4) $\displaystyle\alpha_{i,j}$
$\displaystyle=\dfrac{\exp(\pi(i,j))}{\sum_{k\in
N(i)\cup\\{i\\}}\exp(\pi(i,k))}$ (5) $\displaystyle\mathbf{z}_{i}^{(t)}$
$\displaystyle=\texttt{ReLU}\left(\alpha_{i,i}\textbf{W}\mathbf{x}_{i}^{(t)}+\sum_{j\in\mathcal{N}(i)}\alpha_{i,j}\textbf{W}\mathbf{x}_{j}^{(t)}\right)$
(6) $\displaystyle\mu_{\theta}(\mathbf{x}_{i}^{(t)}),\ $
$\displaystyle\sigma^{2}_{\theta}(\mathbf{x}_{i}^{(t)})=f_{\phi}(\textbf{v}_{i}\odot\mathbf{z}_{i}^{(t)})$
(7) $\displaystyle\hat{s}_{i}^{(t)}$
$\displaystyle:=\mu_{\theta}(\mathbf{x}_{i}^{(t)})$ (8)

where, $f_{\phi}$ are stacked fully-connected layers. The sensor prediction
for the $t^{th}$ timestep ($\hat{s}_{i}^{(t)}$) is defined as the Maximum
Likelihood Estimate (MLE) of the predicted Gaussian distribution i.e. its
mean.

4\. Identify deviations Finally, we identify anomalies as significant
deviations of the ground truth from the expected behaviour (explained in
Baselines and Metrics). In contrast to vanilla GDN which predicts point
estimates $\mathbf{\hat{s}}^{(t)}$, GLUE will produce distributions
$\mathbf{\hat{s}}^{(t)}$ with uncertainty estimates.

###  3.1 Baselines and Metrics

We compare GLUE with a host of popular baseline models: Principal Components
Analysis (PCA) [10], $k$-nearest neighbors (KNN) [9], and autoencoders (AE)
[2], which do not consider the complex structural relationships between the
sensors. Since GDN and GLUE essentially forecast the next sensor readings, we
also compare them with an advanced forecasting model Vector Autoregression
(VAR) [13].

We ran each of the baselines on the same time window as input. We classified
all test data points with a reconstruction error higher than the
$\left(1-\frac{\\#\text{num\\_anomalies}}{\text{total\\_points}}\right)^{th}$
quantile reconstruction error in the training data as anomalous. For GDN and
GLUE, we predicted anomalies on a robust standardized forecasting error
referred to as the Max Robust Error (MRE). The MRE at the $t^{th}$ timestep is
defined as the maximum robust forecasting error among all sensors:

$\displaystyle\text{MRE}(t)$
$\displaystyle=\max_{i\in\\{1,\dots,N\\}}\frac{|s_{i}^{(t)}-\hat{s_{i}}^{(t)}|-\tilde{\mu}_{i}}{\tilde{\sigma}_{i}}$
(9)

where $|s_{i}^{(t)}-\hat{s_{i}}^{(t)}|$ is the absolute value of the
difference between the observed and forecasted values for sensor $i$333Note
that while GDN forecasts the sensor value, the forecast for GLUE instead is
the MLE of the predicted Gaussian distribution. and $\tilde{\mu}_{i}$ and
$\tilde{\sigma}_{i}$ are the median and the interquartile range for a
particular sensor $i$. Data points with MREs above a threshold (determined
from the training data) were classified as anomalous (see Fig. 2).

|
---|---
(a) WADI train | (b) WADI test
|
(a) NASA train | (b) NASA test
Figure 2: Plots showing the max robust error (GLUE) across all sensors over
time for the WADI (top) and NASA (bottom) train (left)-test (right) datasets.
The red points show the true anomalies in the respective datasets (none in
WADI training data), whereas the blue points represent the normal points for
respective datasets. Cyan lines show the threshold (chosen as the proportion
of expected anomalies in the training dataset) above which data points have
been labeled as anomalies.

###  3.2 Training objective of GDN and GLUE

While GDN minimizes the mean squared error between the observed sensor
readings and the forecasted sensor values, i.e.
$(s_{i}^{(t)}-\hat{s_{i}}^{(t)})^{2}$, GLUE optimizes the negative Gaussian
log likelhood [6] of the observed readings based on its predicted mean
($\mu_{\theta}(\mathbf{x}_{i}^{(t)})$) and variance
($\sigma^{2}_{\theta}(\mathbf{x}_{i}^{(t)})$):

$\displaystyle loss$
$\displaystyle\propto\sum_{i}^{N}\frac{\log(\sigma_{\theta}^{2}(x_{i}))}{2}+\frac{(y-\mu_{\theta}(x_{i}))^{2}}{2\sigma_{\theta}(x_{i})^{2}}$

where $\mu_{\theta},\sigma_{\theta}$ are output by the GLUE model and $\theta$
represent the GLUE model parameters.

##  4 Datasets

Since large-scale real world datasets with labeled anomalies are scarce, we
used two datasets which realistically simulate both normal and anomalous
scenarios. For our experiments, we will consider the following two datasets.
In our experiments, both datasets were normalized for zero mean and a standard
deviation of 1. Additionally, NaNs were first forward-filled, then back-
filled, and then filled with zeroes to ensure stability and accuracy. To
obtain the time window, we take 5 time steps of each sensor as input.

###  4.1 WADI Dataset

The WADI dataset curated by the Singapore Public Utility board comes from a
realistic water treatment, storage and distribution network [4]. The data
comprises of $127$ sensor measurements at $1$Hz over $16$ days of continuous
operation: $14$ under normal operation and $2$ days with abnormal attack
scenarios resulting in total of $104,847$ training and $17,270$ testing data
points with an anomaly rate of $5.99\%$. In consonance with the original
paper, our results are reported for a downsampled version of the WADI dataset
with a rolling median of 10 seconds over the entire time period. Additionally,
we also filter out $30$ sensors with $0$ variance over the entire course of
the 2 week period.

###  4.2 NASA AMES’ Turbofan Dataset

The NASA Turbofan dataset 444https://data.nasa.gov/dataset/Turbofan-engine-
degradation-simulation-data-set/vrks-gjie[14] is derived from an engine
turbofan run under various climatic conditions until it failed. This dataset
consists of numerous trajectories of multiple time steps each (a total of
$157,523$ train points and $102,069$ test points, with about $10$% of the data
classified as anomalies, or, engine failures), for which readings of $24$
sensors are recorded. One of the challenges this dataset poses is
generalization over the climate conditions, as well as over degradation of
various parts of the turbofan. In this dataset, there are two distinct failure
modes, however our experiments will draw no distinction between the two, and
both modes will be classified as the anomalies to be detected.

Datasets | #Sensors | #Train | #Test | %Anomalies
---|---|---|---|---
WADI | 127 | 104,847 | 17,270 | 6.89
NASA Turbofan | 24 | 157,523 | 102,069 | 10.82
|  |  |  |
TABLE I: Statistics of datasets used for our experiments.

##  5 Experiments & Results

|  |  |
---|---|---|---
Figure 3: Plot of examples of individual WADI sensor predictions (red, with
95% confidence bounds)-vs-true values (blue) over time. The highlighted green
areas denote regions where the true anomalies occur in test data.

Figure 4: (Left) Plot of WADI Sensor Weights from GLUE using t-SNE [15]
illustrates that GLUE finds interesting relationships between the sensors and
clusters similar sensors together (having the same colour). (Right) Plot of
NASA Sensor Weights from GLUE using t-SNE [15] illustrates that our model
detects patterns and relationships between sensors.

We preprocessed the WADI and NASA datasets such that a single training example
is window with $w=5$ time steps. Table I summarizes the statistics of our
datasets i.e. number of instances, sensors etc. The results of GLUE along with
the other baselines (PCA, k-NN, AE, VAR, and GDN) are summarized in Tab. II
and III. Both GDN and GLUE were trained for a total of 25 epochs on a quad-
core Linux desktop (with hyper-threading enabled) with 16GB RAM. The models
are trained using Adam optimizer with learning rate $1\times 10^{-3}$ and
$(\beta_{1},\beta_{2})=(0.9,0.99)$. Through our experiments, we aim to answer
the following research questions:

  1. 1.

RQ1. How does GLUE compare with GDN and other baselines in terms of anomaly
detection on the NASA Turbofan and WADI datasets?

  2. 2.

RQ2. How does GLUE compare with GDN and VAR in terms of forecasting
performance?

  3. 3.

RQ3. Do the sensor embeddings learned by GLUE represent meaningful
relationships?

Below, we discuss our results while answering the aforementioned research
questions. Tables II and III show the anomaly detection results for the
different models measured using Precision, Recall and F1 metrics on the WADI
and NASA datasets respectively. These tables demonstrate that GLUE performs at
par with GDN on both the datasets while outperforming most baselines. On the
NASA Turbofan dataset, however, K-NN beats both GLUE and GDN in terms of the
F1 score. This may be due to stationary nature of the dataset, which makes it
extremely hard to distinguish between normal and anomalous data points based
on forecasting error.

Figure 3 shows examples of individual WADI sensor true readings and mean of
the forecasted predictions along with 95% confidence intervals. Here the first
and the fourth plots reveal circumstances when the true sensor readings
deviate out of the predicted confidence bands contributing towards the max
robust error (MRE) score of the system at that point in time. Second and third
plots are examples of sensors which do not show anomalous signal (since the
truth mostly lies within confidence bands of the predictions).

Figure 2 shows the Max Robust Errors computed over time on the WADI and NASA
train-test datasets. The red points show the true anomalies in the respective
datasets (none in WADI training data), whereas the blue points represent the
normal points for respective datasets. These plots help us visually see how
GLUE separates out most of the anomalies and normal data. To answer our second
research question, we compare the forecasting performance of VAR, GDN and GLUE
in Tab. IV for both the datasets. We can clearly see that on the WADI dataset,
GLUE has the lowest mean squared error (MSE) while having a slightly high mean
absolute error (MAE). VAR on the other hand has an orders of magnitude higher
MSE. On the NASA dataset, GDN has the lowest MSE and MAE.

| WADI
---|---
Models | Precision(%) | Recall(%) | F1
PCA | 0.36 | 0.52 | 0.42
K-NN | 0.16 | 0.17 | 0.04
Autoencoder | 0.36 | 0.53 | 0.43
VAR | 0.50 | 0.50 | 0.50
GDN | 0.63 | 0.66 | 0.64
GLUE | 0.60 | 0.57 | 0.58
|  |  |
TABLE II: Anomaly detection results obtained on WADI sensor datasets. GDN and GLUE perform significantly better than baselines and GLUE performs on par with GDN. | NASA Turbofan
---|---
Models | Precision(%) | Recall(%) | F1
PCA | 0.89 | 0.06 | 0.11
K-NN | 0.55 | 0.55 | 0.55
Autoencoder | 0.97 | 0.14 | 0.25
VAR | 0.50 | 0.50 | 0.50
GDN | 0.51 | 0.50 | 0.50
GLUE | 0.50 | 0.50 | 0.50
|  |  |
TABLE III: Anomaly detection results obtained on NASA sensor datasets. NASA dataset is so stationary that the anomalies don’t stand out. This could explain why baselines results do better, since they are simpler models. | WADI | NASA Turbofan
---|---|---
Models | MSE | MAE | MSE | MAE
VAR | 2497.52 | 0.65 | 129.92 | 0.99
GDN | 3.00 | 0.65 | 0.26 | 0.41
GLUE | 2.79 | 0.72 | 0.51 | 0.56
|  |  |  |
TABLE IV: Forecasting results obtained on two sensor datasets demonstrate that
GLUE outperforms the competitive VAR baseline, but generally slight worse than
GDN.

Figure 4 show TSNE plots of structural clusterings over sensors as learnt by
GLUE models on WADI and NASA datasets respectively. WADI sensors have been
colored by the available sensor grouping nomenclature. NASA dataset sensors
are unlabeled.

##  6 Conclusion

In this work, we propose GLUE (GDN with Local Uncertainty Estimation) building
on the recently proposed Graph Deviation Network (GDN). GLUE not only
automatically learns complex dependencies between variables and uses them to
better identify anomalous behaviour, but also quantifies its predictive
uncertainty, allowing us to account for the variation in the data as well to
have more interpretable anomaly detection thresholds. Results on two real
world datasets tell us that optimizing the negative Gaussian log likelihood is
reasonable because GLUE’s forecasting results are at par with GDN and in fact
better than the vector autoregressor baseline, which is significant given that
at least GDN directly optimizes the MSE loss. In summary, our experiments
demonstrate that GLUE is competitive with GDN at anomaly detection, with the
added benefit of uncertainty estimations, as shown in Figure 3. We also show
that GLUE learns meaningful sensor embeddings which clusters similar sensors
together.

For future work, we aim to evaluate the quality of predictive uncertainties,
which is a challenging task since there is no ‘ground truth’ uncertainty
estimate, in lines similar to a recent work by Lakshminarayanan et al [5]. We
also aim to increase the number of evaluation datasets and to look into more
precise uncertainty estimates.

## References

  * [1] D. M. Hawkins, _Identification of outliers_. Springer, 1980, vol. 11.
  * [2] C. C. Aggarwal, “Outlier analysis,” in _Data mining_. Springer, 2015, pp. 237–263.
  * [3] A. Deng and B. Hooi, “Graph neural network-based anomaly detection in multivariate time series,” 2021.
  * [4] C. M. Ahmed, V. R. Palleti, and A. P. Mathur, “Wadi: a water distribution testbed for research in the design of secure cyber physical systems,” in _Proceedings of the 3rd International Workshop on Cyber-Physical Systems for Smart Water Networks_ , 2017, pp. 25–28.
  * [5] B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” _arXiv preprint arXiv:1612.01474_ , 2016.
  * [6] D. A. Nix and A. S. Weigend, “Estimating the mean and variance of the target probability distribution,” in _Proceedings of 1994 ieee international conference on neural networks (ICNN’94)_ , vol. 1. IEEE, 1994, pp. 55–60.
  * [7] J. An and S. Cho, “Variational autoencoder based anomaly detection using reconstruction probability,” _Special Lecture on IE_ , vol. 2, no. 1, pp. 1–18, 2015.
  * [8] D. Park, Y. Hoshi, and C. C. Kemp, “A multimodal anomaly detector for robot-assisted feeding using an lstm-based variational autoencoder,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 3, pp. 1544–1551, 2018\.
  * [9] F. Angiulli and C. Pizzuti, “Fast outlier detection in high dimensional spaces,” in _European conference on principles of data mining and knowledge discovery_. Springer, 2002, pp. 15–27.
  * [10] M.-L. Shyu, S.-C. Chen, K. Sarinnapakorn, and L. Chang, “A novel anomaly detection scheme based on principal component classifier,” MIAMI UNIV CORAL GABLES FL DEPT OF ELECTRICAL AND COMPUTER ENGINEERING, Tech. Rep., 2003.
  * [11] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” _arXiv preprint arXiv:1609.02907_ , 2016.
  * [12] B. Yu, H. Yin, and Z. Zhu, “Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting,” _arXiv preprint arXiv:1709.04875_ , 2017.
  * [13] S. A. Ford, “A beginner’s guide to vector autoregression,” Tech. Rep., 1986.
  * [14] M. A. Chao, C. Kulkarni, K. Goebel, and O. Fink, “Aircraft engine run-to-failure dataset under real flight conditions for prognostics and diagnostics,” _Data_ , vol. 6, no. 1, p. 5, 2021.
  * [15] L. Van Der Maaten, “Accelerating t-sne using tree-based algorithms,” _The Journal of Machine Learning Research_ , vol. 15, no. 1, pp. 3221–3245, 2014.


Negative $\rightarrow$ Positive | 59.73 ($\pm$9.48) | 59.00 ($\pm$15.81) |  37.47 ($\pm$10.41) | 59.07 ($\pm$14.97) | 63.27 ($\pm$17.61) | 45.87 ($\pm$24.13) | 46.67 ($\pm$18.75)
Positive $\rightarrow$ Negative (w/ Distractors) | 32.20 ($\pm$14.65) | 35.13 ($\pm$1.91) | 35.00 ($\pm$16.52) | 40.93 ($\pm$4.31) |  19.00 ($\pm$8.66) | 29.13 ($\pm$10.60) | 38.00 ($\pm$8.47)
Entity | Replace Names |  0.70 ($\pm$0.14 | 1.91 ($\pm$0.71) | 1.11 ($\pm$0.51) | 1.61 ($\pm$0.62) | 0.81 ($\pm$0.14) | 1.91 ($\pm$1.51) | 1.01 ($\pm$0.75)
Replace Locations | 3.33 ($\pm$0.74) |  2.73 ($\pm$1.15) | 3.40 ($\pm$0.86) | 3.00 ($\pm$0.33) | 3.07 ($\pm$1.79) | 3.20 ($\pm$1.57) | 3.53 ($\pm$1.95)
Replace Numbers | 0.80 ($\pm$0.00) | 0.53 ($\pm$0.34) |  0.47 ($\pm$0.41) | 0.60 ($\pm$0.43) | 0.60 ($\pm$0.33) | 0.67 ($\pm$0.81) | 0.87 ($\pm$0.66)

Table 14: RQ1 - Functional Subtests for Sentiment Analysis (§5.4.4). For
sentiment analysis, we compare various ER rationale alignment criteria using
the IxG machine rationale extractor (as well as the No-ER baseline), with
respect to performance on a range of functional tests/subtests (OOD). For each
functional test, we report model performance on each of its individual
functional subtests. Performance is reported in terms of failure rate.

Functional Test | Functional Subtest | NLI
---|---|---
ANLP-NLI
Failure Rate ($\downarrow$)
No-ER | IxG+MSE | IxG+MAE | IxG+BCE | IxG+Huber | IxG+Order
Vocabulary | Antonym in Hypothesis | 71.66 ($\pm$20.98) | 64.77 ($\pm$21.97) | 84.55 ($\pm$11.53) | 65.88 ($\pm$21.40) | 74.77 ($\pm$20.41) |  62.55 ($\pm$13.16)
Synonym in Hypothesis | 32.61 ($\pm$7.41) |  24.11 ($\pm$7.62) | 30.11 ($\pm$6.42) | 25.88 ($\pm$6.86) | 30.77 ($\pm$7.07) | 29.27 ($\pm$6.95)
Supertype in Hypothesis | 24.44 ($\pm$15.95) | 11.00 ($\pm$3.62) | 13.77 ($\pm$6.71) | 9.31 ($\pm$5.90) |  8.77 ($\pm$8.06) | 13.55 ($\pm$7.10)
Robustness | Add Punctuation | 14.55 ($\pm$4.13) | 9.44 ($\pm$2.79) | 11.33 ($\pm$1.63) |  8.11 ($\pm$1.19) | 10.00 ($\pm$2.58) | 9.88 ($\pm$2.51)
Add One Typo | 15.88 ($\pm$3.44) | 10.22 ($\pm$3.04) | 12.33 ($\pm$1.63) |  9.66 ($\pm$2.10) | 10.88 ($\pm$2.68) | 10.77 ($\pm$2.52)
Add Two Typos | 15.33 ($\pm$3.68) | 9.77 ($\pm$1.81) | 12.00 ($\pm$1.76) |  9.44 ($\pm$2.31) | 11.11 ($\pm$2.99) | 10.00 ($\pm$2.66)
Add Contractions | 24.69 ($\pm$6.98) | 24.69 ($\pm$8.72) | 25.92 ($\pm$9.07) | 22.22 ($\pm$9.07) | 25.92 ($\pm$7.40) |  14.81 ($\pm$5.23)
Logic | Negate Hypothesis | 50.88 ($\pm$32.25) | 27.77 ($\pm$37.24) |  9.77 ($\pm$15.66) | 41.33 ($\pm$41.54) | 15.22 ($\pm$28.77) | 18.44 ($\pm$23.21)
Negate Premise | 99.88 ($\pm$0.31) | 98.54 ($\pm$3.78) | 91.69 ($\pm$20.37) | 98.65 ($\pm$2.56) |  98.42 ($\pm$4.44) | 99.88 ($\pm$0.31)
Hypothesis is Premise |  14.22 ($\pm$8.63) | 14.33 ($\pm$10.14) | 19.44 ($\pm$12.12) | 18.16 ($\pm$12.69) | 14.38 ($\pm$9.23) | 17.38 ($\pm$10.16)
Entity | Replace Entity in Hypothesis |  77.21 ($\pm$39.57) | 88.88 ($\pm$24.11) | 79.91 ($\pm$22.20) | 85.18 ($\pm$30.04) | 83.83 ($\pm$24.25) | 96.40 ($\pm$4.85)

Table 15: RQ1 - Functional Subtests for NLI (§5.4.4). For NLI, we compare
various ER rationale alignment criteria using the IxG machine rationale
extractor (as well as the No-ER baseline), with respect to performance on a
range of functional tests/subtests (OOD). For each functional test, we report
model performance on each of its individual functional subtests. Performance
is reported in terms of failure rate.

Functional Test | Functional Subtest | Sentiment Analysis
---|---|---
Flights
Failure Rate ($\downarrow$)
IxG+MAE | IxG+Huber
Instance-Level | Task-Level | Instance-Level | Task-Level
Vocabulary | Add Sentiment Words |  0.80 ($\pm$0.16) | 2.00 ($\pm$0.82) |  1.13 ($\pm$0.50) | 1.27 ($\pm$0.98)
Paraphrase Neutral Words | 5.87 ($\pm$0.34) |  5.67 ($\pm$0.41) |  5.67 ($\pm$0.74) | 6.00 ($\pm$0.59)
Add Intensifiers | 1.67 ($\pm$0.41) |  1.60 ($\pm$0.49) | 2.67 ($\pm$0.96) |  2.27 ($\pm$0.66)
Add Reducers | 55.03 ($\pm$32.90) |  30.25 ($\pm$21.54) |  17.46 ($\pm$13.65) | 35.89 ($\pm$1.61)
Add Positive Phrases |  0.60 ($\pm$0.33) | 1.47 ($\pm$1.52) | 2.33 ($\pm$1.76) |  0.67 ($\pm$0.57)
Add Negative Phrases | 20.47 ($\pm$5.33) |  19.67 ($\pm$3.79) |  20.67 ($\pm$3.35) | 21.00 ($\pm$5.94)
Robustness | Add Random URLs/Handles | 9.80 ($\pm$0.48) |  7.27 ($\pm$2.23) |  10.27 ($\pm$0.90) | 10.40 ($\pm$2.05)
Add Punctuation |  2.07 ($\pm$0.25) | 4.00 ($\pm$2.41) | 3.80 ($\pm$0.28) |  3.40 ($\pm$1.34)
Add One Typo |  2.40 ($\pm$0.86) | 2.47 ($\pm$0.41) |  2.60 ($\pm$0.75) | 2.87 ($\pm$0.25)
Add Two Typos |  3.87 ($\pm$0.94) | 4.47 ($\pm$0.93) | 4.60 ($\pm$0.43) |  4.33 ($\pm$0.52)
Add Contractions |  1.00 ($\pm$0.43) | 1.20 ($\pm$0.43) |  0.47 ($\pm$0.09) | 0.87 ($\pm$0.25)
Logic | Positive $\rightarrow$ Negative |  4.60 ($\pm$1.88) | 6.13 ($\pm$1.82) | 3.93 ($\pm$1.57) |  3.80 ($\pm$2.12)
Negative $\rightarrow$ Positive | 41.60 ($\pm$21.11) |  40.87 ($\pm$20.03) | 59.07 ($\pm$14.97) |  31.53 ($\pm$9.24)
Positive $\rightarrow$ Negative (w/ Distractors) |  44.40 ($\pm$6.78) | 49.80 ($\pm$10.22) | 40.93 ($\pm$4.31) |  32.13 ($\pm$13.47)
Entity | Replace Names |  0.91 ($\pm$0.49 | 1.11 ($\pm$0.38) |  1.61 ($\pm$0.62) | 1.91 ($\pm$0.14)
Replace Locations |  3.80 ($\pm$0.86) | 5.00 ($\pm$1.84) |  3.00 ($\pm$0.33) | 4.07 ($\pm$0.90)
Replace Numbers | 0.87 ($\pm$0.68) |  0.73 ($\pm$0.25) | 0.60 ($\pm$0.43) |  0.53 ($\pm$0.19)

Table 16: RQ2 - Functional Subtests for Sentiment Analysis (§5.4.4). For
sentiment analysis, we compare various ER rationale alignment criteria using
the IxG machine rationale extractor (as well as the No-ER baseline), with
respect to performance on a range of functional tests/subtests (OOD). For each
functional test, we report model performance on each of its individual
functional subtests. Performance is reported in terms of failure rate.

Functional Test | Functional Subtest | Sentiment Analysis
---|---|---
Flights
Failure Rate ($\downarrow$)
0% | 100% | Random | LC | HC | LIS | HIS
Vocabulary | Add Sentiment Words | 1.20 ($\pm$0.74) | 1.27 ($\pm$0.84) | 1.09 ($\pm$0.43) | 2.00 ($\pm$1.13) |  0.47 ($\pm$0.19) | 1.67 ($\pm$0.74) |  0.47 ($\pm$0.25)
Paraphrase Neutral Words | 5.59 ($\pm$0.16) |  5.40 ($\pm$0.28) | 5.89 ($\pm$1.25) | 6.73 ($\pm$2.04) | 7.07 ($\pm$1.33) | 5.67 ($\pm$0.68) | 5.67 ($\pm$0.84)
Add Intensifiers | 2.13 ($\pm$1.63) |  1.40 ($\pm$0.16) | 2.24 ($\pm$0.53) | 3.27 ($\pm$1.52) | 1.93 ($\pm$0.09) | 2.20 ($\pm$0.43) | 2.53 ($\pm$0.90)
Add Reducers | 23.85 ($\pm$7.18) | 27.38 ($\pm$5.95) | 45.42 ($\pm$29.46) |  20.96 ($\pm$11.10) | 66.27 ($\pm$24.67) | 42.59 ($\pm$14.58) | 43.73 ($\pm$39.84)
Add Positive Phrases | 1.40 ($\pm$0.28) | 0.67 ($\pm$0.50) | 0.80 ($\pm$0.43) | 1.20 ($\pm$0.28) | 0.80 ($\pm$0.49) |  0.07 ($\pm$0.09) | 1.73 ($\pm$1.39)
Add Negative Phrases | 22.86 ($\pm$7.43) | 20.67 ($\pm$4.07) |  20.47 ($\pm$0.50) | 21.60 ($\pm$6.65) | 21.13 ($\pm$4.72) | 24.80 ($\pm$2.55) | 23.33 ($\pm$6.38)
Robustness | Add Random URLs/Handles | 9.80 ($\pm$0.48) | 9.07 ($\pm$1.80) | 9.18 ($\pm$1.98) | 9.53 ($\pm$1.00) | 9.13 ($\pm$0.98) | 10.00 ($\pm$1.13) |  9.00 ($\pm$3.85)
Add Punctuation | 3.93 ($\pm$0.89) |  1.93 ($\pm$0.41) | 2.80 ($\pm$1.07) | 3.33 ($\pm$0.82) | 2.60 ($\pm$0.43) | 2.40 ($\pm$0.49) | 3.80 ($\pm$1.73)
Add One Typo | 2.60 ($\pm$0.90) | 2.60 ($\pm$0.57) | 2.49 ($\pm$0.57) | 2.47 ($\pm$0.34) | 2.80 ($\pm$0.33) |  1.93 ($\pm$0.77) | 2.33 ($\pm$0.34)
Add Two Typos |  3.93 ($\pm$0.65) | 4.27 ($\pm$0.50) | 4.07 ($\pm$0.54) | 4.20 ($\pm$1.77) | 5.13 ($\pm$0.34) |  3.93 ($\pm$0.52) | 4.27 ($\pm$0.09)
Add Contractions | 1.00 ($\pm$0.00) | 0.87 ($\pm$0.25) | 0.87 ($\pm$0.28) | 1.00 ($\pm$0.43) | 1.47 ($\pm$0.96) |  0.80 ($\pm$0.16) | 1.07 ($\pm$0.25)
Logic | Positive $\rightarrow$ Negative | 5.20 ($\pm$2.75) | 4.47 ($\pm$3.07) | 6.60 ($\pm$2.85) | 7.13 ($\pm$1.09) | 6.93 ($\pm$2.1) | 7.27 ($\pm$1.81) |  3.73 ($\pm$1.64)
Negative $\rightarrow$ Positive | 59.73 ($\pm$9.48) | 37.47 ($\pm$10.41) | 45.36 ($\pm$23.87) | 45.40 ($\pm$17.25) |  32.87 ($\pm$7.25) | 38.87 ($\pm$21.89) | 64.93 ($\pm$30.43)
Positive $\rightarrow$ Negative (w/ Distractors) |  32.20 ($\pm$14.65) | 35.00 ($\pm$16.52) | 47.89 ($\pm$9.33) | 52.27 ($\pm$7.17) | 46.27 ($\pm$17.01) | 48.07 ($\pm$16.15) | 40.47 ($\pm$5.17)
Entity | Replace Names | 0.70 ($\pm$0.14) | 1.11 ($\pm$0.51) | 1.07 ($\pm$0.61) | 1.51 ($\pm$0.25) |  0.50 ($\pm$0.14) | 0.81 ($\pm$0.28) | 1.31 ($\pm$0.38)
Replace Locations |  3.33 ($\pm$0.74) | 3.40 ($\pm$0.86) | 3.82 ($\pm$1.57) | 4.20 ($\pm$.91) | 4.53 ($\pm$1.39) | 4.27 ($\pm$1.51) | 3.73 ($\pm$1.76)
Replace Numbers | 0.80 ($\pm$0.00) |  0.47 ($\pm$0.41) | 0.93 ($\pm$0.55) | 0.93 ($\pm$0.19) | 1.27 ($\pm$0.38) | 1.00 ($\pm$0.33) | 0.60 ($\pm$0.16)

Table 17: RQ3 - Functional Subtests for Sentiment Analysis - $k$=5% (§5.4.4).
For sentiment analysis, we compare various ER rationale alignment criteria
using the IxG machine rationale extractor (as well as the No-ER baseline),
with respect to performance on a range of functional tests/subtests (OOD). For
each functional test, we report model performance on each of its individual
functional subtests. Performance is reported in terms of failure rate.

Functional Test | Functional Subtest | Sentiment Analysis
---|---|---
Flights
Failure Rate ($\downarrow$)
0% | 100% | Random | LC | HC | LIS | HIS
Vocabulary | Add Sentiment Words | 1.20 ($\pm$0.74) | 1.27 ($\pm$0.84) | 0.98 ($\pm$0.60) | 1.20 ($\pm$0.59) |  0.53 ($\pm$0.25) | 0.93 ($\pm$0.25) | 0.87 ($\pm$0.09)
Paraphrase Neutral Words | 5.59 ($\pm$0.16) |  5.40 ($\pm$0.28) | 6.60 ($\pm$1.35) | 6.27 ($\pm$1.36) | 5.93 ($\pm$0.98) | 5.47 ($\pm$0.94) | 6.07 ($\pm$0.74)
Add Intensifiers | 2.13 ($\pm$1.63) |  1.40 ($\pm$0.16) | 2.07 ($\pm$1.12) | 2.67 ($\pm$0.34) | 1.93 ($\pm$0.25) | 1.53 ($\pm$0.41) | 2.40 ($\pm$0.71)
Add Reducers | 23.85 ($\pm$7.18) | 27.38 ($\pm$5.95) | 42.96 ($\pm$29.27) | 34.97 ($\pm$27.96) |  16.67 ($\pm$16.67) | 42.86 ($\pm$20.2) | 35.35 ($\pm$24.78)
Add Positive Phrases | 1.40 ($\pm$0.28) | 0.67 ($\pm$0.50) |  0.49 ($\pm$0.59) | 1.07 ($\pm$0.66) | 0.53 ($\pm$0.50) | 0.80 ($\pm$0.71) | 1.73 ($\pm$1.61)
Add Negative Phrases | 22.86 ($\pm$7.43) | 20.67 ($\pm$4.07) | 23.51 ($\pm$3.31) | 18.47 ($\pm$0.90) | 17.20 ($\pm$4.96) | 20.00 ($\pm$4.26) |  12.33 ($\pm$2.02)
Robustness | Add Random URLs/Handles | 9.80 ($\pm$0.48) | 9.07 ($\pm$1.80) | 8.93 ($\pm$1.31) |  8.20 ($\pm$2.29) | 8.87 ($\pm$1.37) | 8.53 ($\pm$2.92) | 9.20 ($\pm$2.75)
Add Punctuation | 3.93 ($\pm$0.89) |  1.93 ($\pm$0.41) | 2.89 ($\pm$0.89) | 3.27 ($\pm$2.39) | 3.53 ($\pm$0.84) | 2.87 ($\pm$0.34) | 3.20 ($\pm$0.71)
Add One Typo | 2.60 ($\pm$0.90) | 2.60 ($\pm$0.57) | 2.56 ($\pm$0.55) | 2.40 ($\pm$0.98) | 3.27 ($\pm$0.25) |  2.07 ($\pm$0.19) | 2.73 ($\pm$0.34)
Add Two Typos | 3.93 ($\pm$0.65) | 4.27 ($\pm$0.50) | 4.60 ($\pm$0.49) | 3.93 ($\pm$0.50) |  3.67 ($\pm$0.66) | 4.27 ($\pm$1.36) | 3.87 ($\pm$0.25)
Add Contractions | 1.00 ($\pm$0.00) | 0.87 ($\pm$0.25) | 1.29 ($\pm$0.19) |  0.80 ($\pm$0.33) | 1.33 ($\pm$0.57) | 0.93 ($\pm$0.25) | 1.07 ($\pm$0.34)
Logic | Positive $\rightarrow$ Negative | 5.20 ($\pm$2.75) | 4.47 ($\pm$3.07) | 7.78 ($\pm$2.17) | 7.20 ($\pm$1.85) | 7.80 ($\pm$0.99) | 7.93 ($\pm$1.65) |  3.27 ($\pm$1.91)
Negative $\rightarrow$ Positives | 59.73 ($\pm$9.48) |  37.47 ($\pm$10.41) | 53.62 ($\pm$24.45) | 43.67 ($\pm$34.97) | 59.93 ($\pm$27.40) | 49.67 ($\pm$12.45) | 66.13 ($\pm$22.82)
Positive $\rightarrow$ Negative (w/ Distractors) |  32.20 ($\pm$14.65) | 35.00 ($\pm$16.52) | 49.29 ($\pm$13.80) | 33.53 ($\pm$13.96) | 53.20 ($\pm$11.44) | 57.20 ($\pm$7.69) | 38.60 ($\pm$14.01)
Entity | Replace Names |  0.70 ($\pm$0.14) | 1.11 ($\pm$0.51) | 1.01 ($\pm$0.74) |  0.70 ($\pm$1.00) | 1.21 ($\pm$0.25) | 1.11 ($\pm$0.38) | 1.61 ($\pm$0.57)
Replace Locations | 3.33 ($\pm$0.74) | 3.40 ($\pm$0.86) | 4.42 ($\pm$1.13) | 4.87 ($\pm$1.89) | 3.73 ($\pm$1.73) |  3.20 ($\pm$0.75) | 3.40 ($\pm$0.33)
Replace Numbers | 0.80 ($\pm$0.00) |  0.47 ($\pm$0.41) | 1.09 ($\pm$0.35) | 1.20 ($\pm$0.65) | 1.00 ($\pm$0.16) | 0.73 ($\pm$0.34) | 0.67 ($\pm$0.25)

Table 18: RQ3 - Functional Subtests for Sentiment Analysis - $k$=15% (§5.4.4).
For sentiment analysis, we compare various ER rationale alignment criteria
using the IxG machine rationale extractor (as well as the No-ER baseline),
with respect to performance on a range of functional tests/subtests (OOD). For
each functional test, we report model performance on each of its individual
functional subtests. Performance is reported in terms of failure rate.

Functional Test | Functional Subtest | Sentiment Analysis
---|---|---
Flights
Failure Rate ($\downarrow$)
0% | 100% | Random | LC | HC | LIS | HIS
Vocabulary | Add Sentiment Words | 1.20 ($\pm$0.74) | 1.27 ($\pm$0.84) |  1.02 ($\pm$0.55) | 3.93 ($\pm$2.12) | 2.27 ($\pm$1.43) | 1.33 ($\pm$0.09) | 1.60 ($\pm$1.13)
Paraphrase Neutral Words | 5.59 ($\pm$0.16) | 5.40 ($\pm$0.28) | 5.98 ($\pm$0.91) | 9.73 ($\pm$0.41) | 5.93 ($\pm$0.98) | 4.87 ($\pm$0.50) |  4.53 ($\pm$0.19)
Add Intensifiers | 2.13 ($\pm$1.63) |  1.40 ($\pm$0.16) | 2.07 ($\pm$1.50) | 3.27 ($\pm$1.67) | 1.93 ($\pm$0.25) | 2.60 ($\pm$0.57) | 3.53 ($\pm$1.68)
Add Reducers | 23.85 ($\pm$7.18) | 27.38 ($\pm$5.95) | 20.32 ($\pm$19.45) | 41.86 ($\pm$7.54) | 15.81 ($\pm$2.82) | 27.86 ($\pm$9.17) |  15.20 ($\pm$11.38)
Add Positive Phrases | 1.40 ($\pm$0.28) |  0.67 ($\pm$0.50) | 1.87 ($\pm$2.16) |  0.67 ($\pm$0.41) | 1.00 ($\pm$0.33) | 4.67 ($\pm$1.81) | 6.73 ($\pm$6.74)
Add Negative Phrases | 22.86 ($\pm$7.43) | 20.67 ($\pm$4.07) | 25.38 ($\pm$8.40) |  19.93 ($\pm$5.66) | 21.47 ($\pm$9.29) | 23.73 ($\pm$3.14) | 24.47 ($\pm$15.92)
Robustness | Add Random URLs/Handles | 9.80 ($\pm$0.48) |  9.07 ($\pm$1.80) | 10.00 ($\pm$3.02) | 11.27 ($\pm$3.36) | 12.27 ($\pm$4.34) | 9.67 ($\pm$1.39) | 9.40 ($\pm$7.64)
Add Punctuation | 3.93 ($\pm$0.89) |  1.93 ($\pm$0.41) | 4.11 ($\pm$1.75) | 4.67 ($\pm$1.67) | 3.27 ($\pm$0.84) | 4.33 ($\pm$0.66) | 3.80 ($\pm$2.83)
Add One Typo | 2.60 ($\pm$0.90) | 2.60 ($\pm$0.57) | 2.89 ($\pm$0.39) | 4.07 ($\pm$1.16) | 2.40 ($\pm$0.28) |  2.13 ($\pm$0.57) | 3.00 ($\pm$0.59)
Add Two Typos | 3.93 ($\pm$0.65) | 4.27 ($\pm$0.50) | 4.56 ($\pm$1.12) | 6.33 ($\pm$1.20) | 5.47 ($\pm$1.39) |  3.47 ($\pm$0.84) | 4.40 ($\pm$0.59)
Add Contractions | 1.00 ($\pm$0.00) | 0.87 ($\pm$0.25) | 1.00 ($\pm$0.28) | 1.13 ($\pm$0.41) | 1.00 ($\pm$0.28) | 0.93 ($\pm$0.09) |  0.40 ($\pm$0.16)
Logic | Positive $\rightarrow$ Negative | 5.20 ($\pm$2.75) | 4.47 ($\pm$3.07) | 6.09 ($\pm$1.47) | 10.13 ($\pm$1.64) | 8.00 ($\pm$0.85) | 3.80 ($\pm$0.65) |  2.00 ($\pm$0.75)
Negative $\rightarrow$ Positives | 59.73 ($\pm$9.48) |  37.47 ($\pm$10.41) | 68.09 ($\pm$22.22) | 41.73 ($\pm$25.73) | 59.93 ($\pm$27.40) | 69.80 ($\pm$11.61) | 85.27 ($\pm$10.52)
Positive $\rightarrow$ Negative (w/ Distractors) | 32.20 ($\pm$14.65) | 35.00 ($\pm$16.52) | 51.11 ($\pm$12.86) | 53.53 ($\pm$7.48) | 56.07 ($\pm$9.26) | 46.80 ($\pm$2.21) |  22.87 ($\pm$7.56)
Entity | Replace Names |  0.70 ($\pm$0.14) | 1.11 ($\pm$0.51) | 1.48 ($\pm$0.79) | 1.61 ($\pm$0.87) | 1.01 ($\pm$0.14) |  0.70 ($\pm$0.38) | 1.81 ($\pm$1.08)
Replace Locations | 3.33 ($\pm$0.74) | 3.40 ($\pm$0.86) | 5.00 ($\pm$1.04) | 6.27 ($\pm$1.73) | 5.07 ($\pm$1.37) | 3.67 ($\pm$1.09) |  3.00 ($\pm$2.14)
Replace Numbers | 0.80 ($\pm$0.00) |  0.47 ($\pm$0.41) | 1.27 ($\pm$0.45) | 1.87 ($\pm$0.57) | 1.40 ($\pm$0.00) | 0.80 ($\pm$0.43) | 0.93 ($\pm$0.41)

Table 19: RQ3 - Functional Subtests for Sentiment Analysis - $k$=50% (§5.4.4).
For sentiment analysis, we compare various ER rationale alignment criteria
using the IxG machine rationale extractor (as well as the No-ER baseline),
with respect to performance on a range of functional tests/subtests (OOD). For
each functional test, we report model performance on each of its individual
functional subtests. Performance is reported in terms of failure rate.

Functional Test | Functional Subtest | Sentiment Analysis
---|---|---
Flights
Failure Rate ($\downarrow$)
0 min | 10 min | 30 min | 5 hr | 24 hr | 48 hr
Vocabulary | Add Sentiment Words | 2.22 ($\pm$1.58) | 3.00 ($\pm$4.79) | 1.27 ($\pm$0.84) | 1.29 ($\pm$0.80) |  0.85 ($\pm$0.85) | 1.11 ($\pm$0.70)
Paraphrase Neutral Words | 4.91 ($\pm$1.66) | 6.15 ($\pm$1.52) | 5.40 ($\pm$0.28) |  4.27 ($\pm$0.93) | 5.52 ($\pm$1.02) | 6.46 ($\pm$0.89)
Add Intensifiers | 4.40 ($\pm$2.05) | 4.18 ($\pm$3.70) |  1.40 ($\pm$0.16) | 3.76 ($\pm$1.34) | 2.75 ($\pm$1.26) | 2.34 ($\pm$0.42)
Add Reducers | 21.68 ($\pm$16.20) |  13.11 ($\pm$11.58) | 27.38 ($\pm$5.95) | 18.61 ($\pm$18.37) | 25.78 ($\pm$32.43) | 28.25 ($\pm$14.76)
Add Positive Phrases | 2.80 ($\pm$2.56) | 4.80 ($\pm$4.04) |  0.67 ($\pm$0.50) | 3.27 ($\pm$2.08) | 2.05 ($\pm$1.94) | 1.06 ($\pm$1.01)
Add Negative Phrases | 20.31 ($\pm$11.60) | 23.78 ($\pm$12.68) | 20.67 ($\pm$4.07) | 16.44 ($\pm$7.70) |  14.65 ($\pm$3.36) | 16.17 ($\pm$4.42)
Robustness | Add Random URLs/Handles | 8.82 ($\pm$4.95) | 9.85 ($\pm$4.57) | 9.07 ($\pm$1.80) |  6.87 ($\pm$3.45) | 7.65 ($\pm$2.79) | 7.66 ($\pm$1.83)
Add Punctuation | 3.73 ($\pm$2.68) | 4.45 ($\pm$3.51) | 3.00 ($\pm$1.02) |  2.31 ($\pm$1.22) | 2.55 ($\pm$1.40) | 2.77 ($\pm$0.92)
Add One Typo |  2.09 ($\pm$0.60) | 2.60 ($\pm$0.77) | 2.60 ($\pm$0.57) | 2.18 ($\pm$0.46) | 2.68 ($\pm$0.45) | 2.37 ($\pm$0.61)
Add Two Typos | 4.56 ($\pm$1.79) | 5.00 ($\pm$0.78) | 4.27 ($\pm$0.5) |  3.91 ($\pm$0.99) | 4.40 ($\pm$1.03) | 4.20 ($\pm$0.88)
Add Contractions | 0.80 ($\pm$0.52) | 0.72 ($\pm$0.22) | 0.87 ($\pm$0.25) |  0.56 ($\pm$0.26) | 0.65 ($\pm$0.37) | 0.77 ($\pm$0.43)
Logic | Positive $\rightarrow$ Negative | 5.04 ($\pm$4.09) | 4.62 ($\pm$2.39) | 4.47 ($\pm$3.07) |  2.93 ($\pm$1.38) | 5.10 ($\pm$2.64) | 5.00 ($\pm$2.27)
Negative $\rightarrow$ Positive | 84.56 ($\pm$14.41) | 64.03 ($\pm$25.19) |  37.47 ($\pm$10.41) | 81.29 ($\pm$13.66) | 74.53 ($\pm$17.39) | 49.94 ($\pm$28.02)
Positive $\rightarrow$ Negative (w/ Distractors) | 30.40 ($\pm$21.11) | 34.20 ($\pm$17.76) | 35.00 ($\pm$16.52) |  25.53 ($\pm$18.38) | 32.88 ($\pm$15.44) | 38.20 ($\pm$14.73)
Entity | Replace Names | 1.28 ($\pm$0.91) | 1.96 ($\pm$1.33) | 1.11 ($\pm$0.51) | 1.11 ($\pm$0.38) | 1.02 ($\pm$0.45) |  0.91 ($\pm$0.65)
Replace Locations | 3.47 ($\pm$1.54) | 3.20 ($\pm$1.76) | 3.40 ($\pm$0.86) | 2.42 ($\pm$1.31) |  3.05 ($\pm$1.12) | 2.89 ($\pm$1.26)
Replace Numbers | 0.58 ($\pm$0.60) | 0.75 ($\pm$0.21) | 0.47 ($\pm$0.41) |  0.38 ($\pm$0.37) | 0.42 ($\pm$0.43) | 0.69 ($\pm$0.57)

Table 20: RQ4 - Functional Subtests for Sentiment Analysis - Label Only
(§5.4.4). For sentiment analysis, we compare various ER rationale alignment
criteria using the IxG machine rationale extractor (as well as the No-ER
baseline), with respect to performance on a range of functional tests/subtests
(OOD). For each functional test, we report model performance on each of its
individual functional subtests. Performance is reported in terms of failure
rate.

Functional Test | Functional Subtest | Sentiment Analysis
---|---|---
Flights
Failure Rate ($\downarrow$)
0 min | 10 min | 30 min | 5 hr | 24 hr | 48 hr
Vocabulary | Add Sentiment Words | 2.22 ($\pm$1.58) |  0.91 ($\pm$0.58) | 1.16 ($\pm$0.62) | 2.07 ($\pm$1.25) | 0.92 ($\pm$0.52) | 0.92 ($\pm$0.52)
Paraphrase Neutral Words | 4.91 ($\pm$1.66) | 5.27 ($\pm$1.47) |  4.82 ($\pm$1.05) | 5.47 ($\pm$2.25) | 5.82 ($\pm$2.47) | 5.82 ($\pm$2.47)
Add Intensifiers | 4.40 ($\pm$2.05) | 2.80 ($\pm$1.19) | 2.76 ($\pm$0.90) | 3.04 ($\pm$1.09) |  2.00 ($\pm$1.11) |  2.00 ($\pm$1.11)
Add Reducers | 21.68 ($\pm$16.20) | 11.06 ($\pm$9.21) |  8.64 ($\pm$11.13) | 18.38 ($\pm$10.12) | 29.43 ($\pm$19.41) | 29.43 ($\pm$19.41)
Add Positive Phrases | 2.80 ($\pm$2.56) | 2.49 ($\pm$1.66) | 2.76 ($\pm$1.80) | 1.76 ($\pm$1.18) |  1.52 ($\pm$0.62) |  1.52 ($\pm$0.62)
Add Negative Phrases | 20.31 ($\pm$11.60) | 18.51 ($\pm$9.87) | 21.49 ($\pm$8.73) |  16.76 ($\pm$5.49) | 19.45 ($\pm$8.40) | 19.45 ($\pm$8.40)
Robustness | Add Random URLs/Handles | 8.82 ($\pm$4.95) | 7.80 ($\pm$3.71) | 7.69 ($\pm$3.75) | 8.13 ($\pm$2.99) |  7.15 ($\pm$4.06) |  7.15 ($\pm$4.06)
Add Punctuation | 3.73 ($\pm$2.68) |  2.84 ($\pm$1.37) | 3.42 ($\pm$1.85) | 3.20 ($\pm$1.48) | 3.65 ($\pm$2.75) | 3.65 ($\pm$2.75)
Add One Typo |  2.09 ($\pm$0.60) | 2.56 ($\pm$0.67) | 2.51 ($\pm$0.82) | 2.62 ($\pm$1.01) | 2.78 ($\pm$0.64) | 2.78 ($\pm$0.64)
Add Two Typos | 4.56 ($\pm$1.79) | 4.76 ($\pm$1.11) |  4.09 ($\pm$0.96) | 4.49 ($\pm$1.18) | 4.65 ($\pm$1.02) | 4.65 ($\pm$1.02)
Add Contractions | 0.80 ($\pm$0.52) | 0.73 ($\pm$0.46) |  0.56 ($\pm$0.53) | 0.73 ($\pm$0.65) | 0.68 ($\pm$0.33) | 0.68 ($\pm$0.33)
Logic | Positive $\rightarrow$ Negative | 5.04 ($\pm$4.09) |  3.36 ($\pm$1.30) | 3.44 ($\pm$1.50) | 5.91 ($\pm$3.16) | 5.98 ($\pm$2.96) | 5.98 ($\pm$2.96)
Negative $\rightarrow$ Positive | 84.56 ($\pm$14.41) | 75.89 ($\pm$26.78) | 85.89 ($\pm$12.74) |  56.98 ($\pm$34.69) | 58.12 ($\pm$27.91) | 58.12 ($\pm$27.91)
Positive $\rightarrow$ Negative (w/ Distractors) | 30.40 ($\pm$21.11) | 28.38 ($\pm$8.88) |  24.91 ($\pm$11.84) | 40.73 ($\pm$23.29) | 46.75 ($\pm$20.49) | 46.75 ($\pm$20.49)
Entity | Replace Names | 1.28 ($\pm$0.91) | 1.17 ($\pm$0.97) | 1.28 ($\pm$1.03) |  1.01 ($\pm$0.55) | 1.21 ($\pm$0.52) | 1.21 ($\pm$0.52)
Replace Locations | 3.47 ($\pm$1.54) |  2.69 ($\pm$1.49) | 2.87 ($\pm$1.36) | 3.56 ($\pm$2.05) | 3.18 ($\pm$2.31) | 3.18 ($\pm$2.31)
Replace Numbers | 0.58 ($\pm$0.60) | 0.44 ($\pm$0.39) |  0.31 ($\pm$0.21) | 0.69 ($\pm$0.58) | 1.02 ($\pm$0.87) | 1.02 ($\pm$0.87)

Table 21: RQ4 - Functional Subtests for Sentiment Analysis - Expl Only
(§5.4.4). For sentiment analysis, we compare various ER rationale alignment
criteria using the IxG machine rationale extractor (as well as the No-ER
baseline), with respect to performance on a range of functional tests/subtests
(OOD). For each functional test, we report model performance on each of its
individual functional subtests. Performance is reported in terms of failure
rate.

Functional Test | Functional Subtest | Sentiment Analysis
---|---|---
Flights
Failure Rate ($\downarrow$)
0 min | 10 min | 30 min | 5 hr | 24 hr | 48 hr
Vocabulary | Add Sentiment Words | 2.22 ($\pm$1.58) | 2.07 ($\pm$2.44) | 2.84 ($\pm$4.05) | 1.38 ($\pm$0.73) | 1.53 ($\pm$1.15) |  1.07 ($\pm$0.83)
Paraphrase Neutral Words | 4.91 ($\pm$1.66) | 6.18 ($\pm$2.36) | 4.80 ($\pm$1.98) | 5.27 ($\pm$1.56) | 5.36 ($\pm$1.20) |  4.78 ($\pm$1.14)
Add Intensifiers | 4.40 ($\pm$2.05) | 3.00 ($\pm$2.20) | 3.51 ($\pm$3.21) | 4.04 ($\pm$4.98) | 2.89 ($\pm$1.08) |  2.07 ($\pm$0.85)
Add Reducers | 21.68 ($\pm$16.20) | 22.08 ($\pm$29.84) |  16.55 ($\pm$15.31) | 22.27 ($\pm$16.00) | 22.68 ($\pm$11.99) | 46.09 ($\pm$19.50)
Add Positive Phrases | 2.80 ($\pm$2.56) | 2.78 ($\pm$2.22) | 1.44 ($\pm$1.04) | 3.11 ($\pm$2.83) | 1.91 ($\pm$1.60) |  1.40 ($\pm$1.10)
Add Negative Phrases | 20.31 ($\pm$11.60) | 19.07 ($\pm$9.48) |  16.60 ($\pm$6.60) | 18.84 ($\pm$17.92) | 25.87 ($\pm$14.51) | 17.98 ($\pm$3.68)
Robustness | Add Random URLs/Handles | 8.82 ($\pm$4.95) | 8.78 ($\pm$4.78) |  7.27 ($\pm$2.48) | 9.02 ($\pm$7.59) | 10.02 ($\pm$5.74) | 8.44 ($\pm$2.37)
Add Punctuation | 3.73 ($\pm$2.68) | 3.22 ($\pm$1.58) | 3.27 ($\pm$2.24) | 3.91 ($\pm$5.02) | 3.93 ($\pm$2.24) |  2.38 ($\pm$0.71)
Add One Typo |  2.09 ($\pm$0.60) | 2.62 ($\pm$0.73) | 2.42 ($\pm$0.86) | 3.02 ($\pm$1.22) | 2.44 ($\pm$1.06) | 2.22 ($\pm$0.64)
Add Two Typos | 4.56 ($\pm$1.79) | 4.62 ($\pm$0.89) |  3.84 ($\pm$1.49) | 4.58 ($\pm$1.85) | 4.36 ($\pm$1.14) | 3.98 ($\pm$0.60)
Add Contractions | 0.80 ($\pm$0.52) | 0.82 ($\pm$0.38) |  0.62 ($\pm$0.57) | 0.82 ($\pm$0.63) | 0.80 ($\pm$0.23) | 0.73 ($\pm$0.25)
Logic | Positive $\rightarrow$ Negative | 5.04 ($\pm$4.09) | 8.31 ($\pm$10.47) | 4.47 ($\pm$3.07) |  4.13 ($\pm$2.99) | 4.84 ($\pm$3.00) | 5.82 ($\pm$2.81)
Negative $\rightarrow$ Positive | 84.56 ($\pm$14.41) | 65.76 ($\pm$21.49) | 68.91 ($\pm$28.29) | 81.29 ($\pm$13.66) | 70.98 ($\pm$18.14) |  63.73 ($\pm$22.93)
Positive $\rightarrow$ Negative (w/ Distractors) | 30.40 ($\pm$21.11) | 35.02 ($\pm$25.60) | 36.38 ($\pm$22.33) | 34.24 ($\pm$15.02) |  28.24 ($\pm$16.92) | 41.13 ($\pm$11.90)
Entity | Replace Names | 1.28 ($\pm$0.91) | 1.21 ($\pm$0.65) | 1.11 ($\pm$0.43) | 1.51 ($\pm$0.97) | 1.61 ($\pm$1.18) |  0.94 ($\pm$0.59)
Replace Locations | 3.47 ($\pm$1.54) | 3.53 ($\pm$1.96) |  2.44 ($\pm$0.78) | 3.38 ($\pm$1.92) | 3.07 ($\pm$2.12) | 2.89 ($\pm$1.02)
Replace Numbers | 0.58 ($\pm$0.60) | 0.73 ($\pm$0.70) | 0.71 ($\pm$0.52) |  0.47 ($\pm$0.28) | 0.56 ($\pm$0.41) | 0.69 ($\pm$0.39)

Table 22: RQ4 - Functional Subtests for Sentiment Analysis - Label+Expl
(§5.4.4). For sentiment analysis, we compare various ER rationale alignment
criteria using the IxG machine rationale extractor (as well as the No-ER
baseline), with respect to performance on a range of functional tests/subtests
(OOD). For each functional test, we report model performance on each of its
individual functional subtests. Performance is reported in terms of failure
rate.