# Combining Euclidean Alignment and Data Augmentation for BCI decoding

Gustavo H. Rodrigues1 Bruno Aristimunha2,3 Sylvain Chevallier2 Raphael Y. de
Camargo1

1 University of São Paulo, Sao Paulo, Brazil,
2 Université Paris-Saclay, Inria TAU team, LISN-CNRS, France,
3 Federal University of ABC (UFABC), Santo André, Brazil,

###### Abstract

Automated classification of electroencephalogram (EEG) signals is complex due
to their high dimensionality, non-stationarity, low signal-to-noise ratio, and
variability between subjects. Deep neural networks (DNNs) have shown promising
results for EEG classification, but the above challenges hinder their
performance. Euclidean Alignment (EA) and Data Augmentation (DA) are two
promising techniques for improving DNN training by permitting the use of data
from multiple subjects, increasing the data, and regularizing the available
data. In this paper, we perform a detailed evaluation of the combined use of
EA and DA with DNNs for EEG decoding. We trained individual models and shared
models with data from multiple subjects and showed that combining EA and DA
generates synergies that improve the accuracy of most models and datasets.
Also, the shared models combined with fine-tuning benefited the most, with an
overall increase of 8.41% in classification accuracy.

_K_ eywords Neural Networks, Brain-Computer Interfaces, Data Augmentation,
Euclidean Alignment

##  1 Introduction

The brain-computer interface (BCI) is an emerging technology of recent
decades, which establishes a direct link between external devices and brain
signals [1]. Traditionally, BCIs have been used for medical applications such
as emotion recognition, human-machine interactions, and, above all, neural
control of prosthetic limbs [2, 3, 4]. Among the various non-invasive
techniques, the electroencephalogram (EEG) is widely used to detect neural
activity since it is portable and has a low acquisition cost. Despite these
advantages, decoding EEG is still challenging due to the inherent complexity
of this type of data, resulting from the high dimensionality and low signal-
to-noise ratio of this type of signal.

To overcome the challenges of EEG decoding, researchers started to evaluate
the use of Deep Neural Networks (DNNs), which can extract important features
directly from raw data and have been shown to provide superior performance
compared to conventional methods [5, 6, 7, 8, 9]. Yet, despite the promising
results of DNNs, they require large training datasets. Two possibilities for
increasing the availability of data are training DNNs using data from multiple
subjects and using data augmentation techniques.

Data augmentation [10, 11] increases the training sets’ size by adding new
synthetic examples by performing transformations in the data while preserving
the labels. This strategy alleviates the problem of overfitting by providing
more training data while also providing regularization since the DNNs become
invariant to the imposed transformation. It has shown that data augmentation
improves the accuracy of EEG Decoding [12, 13].

The second way to improve the training of DNNs is to use data from multiple
subjects. Considering each subject as a domain, we must achieve greater
similarity between domains [14]. One approach is to use Euclidean alignment
(EA) [15], which is a pre-processing method that centers the average of the
covariance matrices of each subject EEG trials on the identity matrix.
Although little explored in the context of neural networks, this type of
alignment has already shown good results in recent studies [15] and is
generating increasing interest due to its computational efficiency and
simplicity.

Although these techniques have been studied separately [11, 16, 12, 17], using
both techniques together can generate positive interactions. For instance, the
regularization provided by the augmentation process may enhance the domain
adaptation provided by Euclidean alignment.

In this paper, we evaluate the combined use of Euclidean alignment and a data
augmentation strategy, called Segmentation and Reconstruction (S&R), for
training DNNs using data from single or multiple subjects. Our results show
that combining S&R with EA improved the accuracy of most models and datasets,
especially in the case of shared models, reaching up to $13\%$ of improvement
with fine-tuning.

##  2 Material and Methods

###  2.1 EEG Decoding

We assume EEG decoding is a machine-learning classification problem when
evaluating our proposal [18]. We consider signals as real-valued matrices
$\mathcal{X}$, where each matrix $\mathbf{x}_{i}\in\mathcal{X}$ is associated
with a cognitive task $\mathbf{y}_{i}\in\mathcal{Y}$. The matrix
$\mathbf{x}_{i}$ is an element of the set $\mathcal{X}=\mathbb{R}^{C\times
T}$, with $C$ as the number of channels (electrodes), $T$ is the number of
time points in the matrices and $i\in\\{1\ldots N\\}$ are in the index vector,
with $N$ as the total number of trials/matrices.

The decoding problem involves learning a neural network model to map each
matrix $\mathbf{x}$ to the associated label $\mathbf{y}$, in other words,
$f_{\theta}:\mathcal{X}\to\mathcal{Y}$. The model parameters $\theta$ are
optimized to minimize the average loss function $\ell$ across the training
dataset:

$\min_{\theta}\frac{1}{N}\sum\ell(f_{\theta}(\mathbf{x}_{i}),\mathbf{y}_{i})\enspace.$
(1)

In all our experiments, the loss function $\ell$ used is the balanced cross-
entropy. To analyze the robustness of the data augmentation with the Euclidean
Alignment, we employed four well-established neural networks for EEG decoding,
the _ShallowNet_ [8], _Deep4Net_ [8], _EEGNet_ [6] and _EEGConformer_ [16]
implemented on braindecode library v0.7.

###  2.2 Datasets

We selected three motor imagery datasets to validate our approach, BCI
Competition IV 2a [19], the Cho2015 [20], and Shin2017A [21], available at the
MOABB v1.0 library [22]. We chose these large datasets, in the number of
trials or the number of subjects, to avoid the common BCI practice of small
and single dataset bias [23, 24].

The first dataset, denoted in the rest of the paper as BNCI2014001, consists
of 9 subjects, with two sessions (different days) and four cognitive imagery
tasks: left hand, right hand, and both feet and tongue movements. The second
dataset, Cho2017, contains 52 subjects with two cognitive imagery tasks, left-
hand and right-hand, recorded with a single session. Finally, the Shin2017A
includes data from 29 subjects registered in three sessions, with two
cognitive imagery tasks, left hand and right hand.

Regarding the preprocessing steps applied to the datasets, we conduct a re-
scaling voltage from V to $\mu$V, band-pass filtering with interval $[4-38]$
Hz, exponential moving standardization [25], artifact rejection based on
annotations, and baseline correction when necessary. These steps were designed
and aligned with recent work on the importance of pre-processing to separate
the marks from the central nervous and peripheral systems [26].

###  2.3 Euclidean Alignment

The natural variability of brain signals represents a significant obstacle in
developing a model that can be effectively generalized to several individuals
or even the same subject between different days. To address this challenge, we
employ the Euclidean alignment (_EA_) technique [27, 28]. For each subject’s
run with $N$ trials, we calculate the arithmetic mean $\bar{R}$ of the
covariance matrices:

$\bar{R}=\frac{1}{N}\sum_{i=1}^{N}\mathbf{x}_{i}\mathbf{x}_{i}^{\top},$ (2)

Next, we perform the alignment as the square root of the inverse of $\bar{R}$
and apply it back to each matrix.

$\tilde{X}_{i}=\bar{R}^{-1/2}\mathbf{x}_{i},$ (3)

with these steps, the average covariance matrices in the data become equal to
the identity matrix. Consequently, the distributions of the covariance
matrices of different subjects become more similar, a desirable aspect of
improving the model’s generability.

###  2.4 Segment & Reconstruction – S&R

While the alignment technique can be seen as a domain generation component,
the data augmentation is considered a regularization transformation. In EEG
decoding, data augmentation has been known to enhance the brain decoding task
[17, 12, 11].

Because of the non-stationarity property present in the signal, the most
important segments of the signal do not have a fixed size or point in the
temporal dimension. Here, we employ one of the possible data augmentations
that capture this component and introduce more variability to the $f$ neural
network.

Given the associated $\mathbf{x}$ signal with dimensions $C\times T$ for each
class, we _segment_ the temporal component into $s$ parts. Then, across the
set of trials for each class, we randomly concatenate those segments to form
new synthetic _reconstructed_ trials, maintaining the original temporal order.
An illustration of the process is present in Figure 1. The Segment and
Reconstruction data augmentation increased the number of samples during the
batch size of each iteration, with the concatenation of the original trials
and the reconstructed synthetic trials.

Figure 1: Principle of synthetic EEG data generation using Segmentation and
Reconstruction \- $S\&R$ strategy in the time domain. The $s$ segmentations
are done randomly and the reconstruction preserves the labels of each trial
(i.e. there is only a mixing of trials from the same class).

##  3 Experimental Setup

Various tests were carried out to compare the accuracies and check whether
there were any significant changes when varying the training and data
processing conditions, as shown in the graphs in the next subsection. With
these results, the performance of the architectures was also compared.

As a first step, the comparative tests conducted initially considered the
effects of Euclidean alignment on the models, checking whether or not there
was an improvement in the accuracy values for the classes (tasks). In
addition, the contribution of data augmentation (S&R) to the model’s
performance in the classification task was also checked. After these cases,
the model was tested with EA and S&R together111Euclidean Alignment processing
is realized in an offline fashion, whereas S&R data augmentation is executed
online..

To examine its performance, we analyzed the models in two main situations.
Firstly, we used individual models, in which we evaluated the architecture
performance per subject (Figure 2b). In this case, for BNCI2014_001, we used
one session for training and another for testing. For Cho2017, which has 200
trials per subject, we used 70% of data for train, 15% for validation, and 15%
for testing. Finally, for Shin2017A, we used the first session for train, and
the other two for validation and test, respectively.

Secondly, the model was trained with data from all the subjects. The strategy
was based on choosing a random batch of subjects as the targets, which will be
the test set, and the other subjects will be used for training (Figure 2a).
For the experiments with fine-tuning, we followed the same split as in the
individual models with the test subjects sets for BNCI2014001 and Shin2017A.
In this case, the train data became the fine-tuning set, and the remaining was
used as the test set. For Cho2017, we split the target subject set in half,
using the first part for fine-tuning and the other for testing. The results
obtained follow in the next section.

Figure 2: (a) Cross-subject training, $5$-folds cross-validation over the
subjects data. (b) Individual training. Data from a single subject is used for
training and validation.

For both modalities (individual and shared), the models have trained $200$
epochs, using an early stopping of 75 iterations, learning rate of $0.000625$,
weight decay of $0.1$, and a batch size of $64$ trials222In the case of
individual training with Shin2017A dataset, due to the limited number of
trials, we used a batch size of 20.. The loss criterion and optimizer used
were CrossEntropyLoss and Adam, respectively, with beta values equal to $0.5$
and $0.999$. For fine-tuning, the models were trained for an additional 100
epochs, with a patience of 30. In all cases, we used $s$ in S&R data
augmentation set to 12. We used a one-tailed paired permutation test with and
without data transformation, and we combines the p-values resulting from each
dataset taken separately via Stouffer [23].

##  4 Results And Discussion

We evaluated the effectiveness of Euclidean Alignment and S&R data
augmentation using individual and shared models. Our aim was to determine
whether the use of these transformations, both individually and combined,
would help to improve BCI decoding accuracy.

For the individual models, trained using data from single subjects, using EA,
S&R, or EA and S&R did not improve the accuracy, as shown in the column
overall in Table 1. When looking for individual models, we note that for EEG-
Conformer and ShallowNet, using EA and S&R improved average performance in
$2.53$% and $4.05$%. However, for DeepNet, there was a decrease of $8.34\%$
when using EA and S&R. One possible explanation is that DeepNet is a larger
model, which may be overfitting the augmented data. Still, the results
indicate that using EA and S&R for training with data from single subjects
does not improve accuracy consistently.

When training shared models with data from all subjects in a dataset, using
either EA, S&R, and EA and S&R improved the accuracy of all models compared to
the models without EA and S&R (Table 1). Overall, the impact of using EA was
the most prominent, but combining EA and S&R resulted in further improvements.
When evaluating individual datasets, shown in Figure 3, using EA and S&R
generated improvements in all cases, but using only S&R reduced the accuracy
in some models in the BNCI2014001 dataset. We can see that using EA and S&R
enabled the shared models to perform similarly to the individual models (Table
1), even without using any target subject data for fine-tuning (although we
used a small amount of target data to adjust the Euclidean Alignment). Also,
the larger model, DeepNet, benefited the most from using data from multiple
subjects, enabled by the use of EA and S&R.

Figure 3: Prediction accuracy using EEG-Conformer, ShallowNet, DeepNet, and
EEGNet shared models with and without Euclidean alignment and data
augmentation (S&R). The statistic comparison between EA+S&R and without EA and
S&R shows significant difference with $p<0.001$.

When we fine-tuned the models using target subject data, using EA and S&R
resulted in the most prominent improvements compared to the base model (Table
1 and Figure 4). Still, using only EA or S&R individually also improved the
accuracies, except for the S&R with the ShallowNet. Moreover, compared to the
individual models, the fine-tuned models with EA, S&R, or EA and S&R improved
the overall accuracies for all models (Table 2). This shows that combining
these techniques, EA and S&R, enabled the effective training of shared models
with data from multiple subjects. Finally, combining EA and S&R generated
synergies in the shared and fine-tuned models, resulting in the best overall
accuracies, except for DeepNet, which worked better using only EA.

Figure 4: Prediction accuracy using EEG-Conformer, ShallowNet, DeepNet, and
EEGNet shared models after fine-tuning, with and without Euclidean alignment
and data augmentation (S&R). The statistic comparison between EA+S&R and
without EA and S&R shows significant difference with $p<0.001$. Table 1:
Average accuracy contribution of EA and S&R (mean $\pm$ standard deviation)
compared to the baseline model of each training paradigm (Individual, Shared,
and Fine-tuned)

Processing | Modality | Models | Overall
---|---|---|---
EEG-Conformer | DeepNet | ShallowNet | EEGNet
EA | Individual | 3.37 $\pm$ 9.86 | -3.54 $\pm$ 9.81 | 2.43 $\pm$ 6.83 | -1.06 $\pm$ 9.25 | 0.30 $\pm$ 8.94
Shared | 5.99 $\pm$ 3.85 | 7.90 $\pm$ 3.64 | 4.20 $\pm$ 2.53 | 5.53 $\pm$ 2.76 | 5.91 $\pm$ 3.19
Shared and Fine-tuning | 10.53 $\pm$ 6.35 |  7.94 $\pm$ 5.00 | 3.10 $\pm$ 1.92 | 5.20 $\pm$ 3.64 | 6.69 $\pm$ 4.23
S&R | Individual | 0.97 $\pm$ 5.51 | -3.89 $\pm$ 8.46 | 2.10 $\pm$ 6.43 | 1.24 $\pm$ 5.17 | 0.10 $\pm$ 6.39
Shared | 2.07 $\pm$ 4.47 | 2.40 $\pm$ 4.21 | 0.28 $\pm$ 2.03 | 1.23 $\pm$ 4.71 | 1.50 $\pm$ 3.86
Shared and Fine-tuning | 8.58 $\pm$ 5.31 | 4.55 $\pm$ 3.49 | -1.04 $\pm$ 3.69 | 1.79 $\pm$ 3.01 | 3.47 $\pm$ 3.87
EA and S&R | Individual | 2.53 $\pm$ 7.56 | -8.34 $\pm$ 10.56 | 4.05 $\pm$ 7.78 | -0.28 $\pm$ 8.32 | -0.51 $\pm$ 8.55
Shared | 8.08 $\pm$ 3.76 | 7.53 $\pm$ 4.27 |  5.07 $\pm$ 2.61 |  7.53 $\pm$ 3.72 | 7.05 $\pm$ 3.59
Shared and Fine-tuning |  13.60 $\pm$ 6.38 | 7.58 $\pm$ 6.58 | 5.00 $\pm$ 3.47 | 7.47 $\pm$ 2.94 |  8.41 $\pm$ 4.84
Bold for the best model accuracy improvement.

Table 2: Average accuracy of Individual and Cross-subject with fine-tuning
paradigms (mean $\pm$ standard deviation)

Model | Individual | Shared and Fine-tuned
---|---|---
EA | S&R | EA and S&R | EA | S&R | EA and S&R
EEG-Conformer | 46.53 $\pm$ 12.38 | 48.38 $\pm$ 14.58 | 47.45 $\pm$ 12.53 | 64.2 $\pm$ 7.59 | 58.65 $\pm$ 7.72 |  67.81 $\pm$ 6.86
DeepNet | 33.68 $\pm$ 8.20 | 32.52 $\pm$ 5.31 | 28.24 $\pm$ 5.90 |  62.74 $\pm$ 6.27 | 60.35 $\pm$ 9.88 | 60.90 $\pm$ 7.36
ShallowNet | 57.87 $\pm$ 10.91 | 54.05 $\pm$ 13.79 | 62.27 $\pm$ 10.29 | 68.30 $\pm$ 9.81 | 59.27 $\pm$ 9.39 |  72.36 $\pm$ 10.34
EEGNet | 48.26 $\pm$ 10.53 | 49.77 $\pm$ 9.92 | 49.31 $\pm$ 11.39 | 70.70 $\pm$ 7.99 | 60.94 $\pm$ 7.92 |  71.70 $\pm$ 7.15
Bold for the best model accuracy.

Other projects have also explored data augmentation to improve deep learning
models in the context of decoding EEG data. For instance, Rommel et al. [12]
made a systematic comparison of data augmentation methods in EEG data to
improve model performance. They obtained improvements of up to $25\%$ and
$45\%$, in low data regimes, for BNCI2014001 using the SmoothTimeMask and
FTSurrogate strategies, respectively. Ali et al. [13] used a new data
augmentation strategy and its proposed feature extraction method, reaching
significant improvements compared to state-of-the-art networks. Our
combination of EA and S&R showed an improvement in the average accuracy of
$13.60\%$ for EEG-Conformer, $7.58\%$ for DeepNet, $5.00\%$ for ShallowNet and
$7.47\%$ for EEGNet shared models regarding the three datasets.

Similarly, for Euclidean alignment, He and Dongrui [15] showed an increase of
up to $10\%$ when using EA with machine learning algorithms. More recently,
Junqueira, B. et al. [28] also showed an improvement of $4.33\%$ when applying
EA as a preprocessing step for deep learning architectures. We systematically
evaluated using both Euclidean Alignment and Data Augmentation in Deep
Learning models. We showed that using EA and S&R enabled the training of
shared models with data from all subjects in a dataset, resulting in better
overall accuracies in the evaluated datasets.

For individual models, we still see some inconsistency with the use of these
regularizations, and we should further investigate which conditions cause them
to improve or degrade performance. We could also evaluate other data
augmentation methods and which work better with Euclidean Alignment and with
which each model and dataset. Finally, we could perform hyperparameter
optimizations to improve the accuracy.

##  5 Acknowledgment

GHR and RYC are grateful to São Paulo Research Foundation (FAPESP) for the
financial support (grant 23/06715-3). The work of BA was supported by DATAIA
Convergence Institute as part of the “Programme d’Investissement d’Avenir”,
(ANR-17-CONV-0003) operated by LISN.

## References

  * [1] J. R. Wolpaw, N. Birbaumer, D. J. McFarland, G. Pfurtscheller, and T. M. Vaughan, “Brain–computer interfaces for communication and control,” Clinical Neurophysiology, vol. 113, pp. 767–791, jun 2002.
  * [2] J. Jin, H. Sun, I. Daly, S. Li, C. Liu, X. Wang, and A. Cichocki, “A Novel Classification Framework Using the Graph Representations of Electroencephalogram for Motor Imagery Based Brain-Computer Interface,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 30, pp. 20–29, 2022.
  * [3] B. Liu, X. Chen, N. Shi, Y. Wang, S. Gao, and X. Gao, “Improving the Performance of Individually Calibrated SSVEP-BCI by Task- Discriminant Component Analysis,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 29, pp. 1998–2007, 2021.
  * [4] J. W. Li, S. Barma, P. U. Mak, F. Chen, C. Li, M. T. Li, M. I. Vai, and S. H. Pun, “Single-Channel Selection for EEG-Based Emotion Recognition Using Brain Rhythm Sequencing,” IEEE Journal of Biomedical and Health Informatics, vol. 26, pp. 2493–2503, jun 2022.
  * [5] H. Dose, J. S. Møller, H. K. Iversen, and S. Puthusserypady, “An end-to-end deep learning approach to MI-EEG signal classification for BCIs,” Expert Systems with Applications, vol. 114, pp. 532–542, 2018.
  * [6] V. J. Lawhern, A. J. Solon, N. R. Waytowich, S. M. Gordon, C. P. Hung, and B. J. Lance, “EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces,” Journal of Neural Engineering, vol. 15, p. 056013, oct 2018.
  * [7] Y. Roy, H. Banville, I. Albuquerque, A. Gramfort, T. H. Falk, and J. Faubert, “Deep learning-based electroencephalography analysis: a systematic review,” Journal of Neural Engineering, vol. 16, p. 051001, aug 2019.
  * [8] R. T. Schirrmeister, J. T. Springenberg, L. D. J. Fiederer, M. Glasstetter, K. Eggensperger, M. Tangermann, F. Hutter, W. Burgard, and T. Ball, “Deep learning with convolutional neural networks for EEG decoding and visualization,” Human Brain Mapping, vol. 38, pp. 5391–5420, 2017.
  * [9] J. Paillard, J. F. Hipp, and D. A. Engemann, “GREEN: a lightweight architecture using learnable wavelets and Riemannian geometry for biomarker exploration,” bioRxiv, pp. 2024–05, 2024.
  * [10] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” in Advances in Neural Information Processing Systems, vol. 25, Curran Associates, Inc., 2012.
  * [11] F. Lotte, “Signal Processing Approaches to Minimize or Suppress Calibration Time in Oscillatory Activity-Based Brain–Computer Interfaces,” Proceedings of the IEEE, vol. 103, pp. 871–890, jun 2015.
  * [12] C. Rommel, J. Paillard, T. Moreau, and A. Gramfort, “Data augmentation for learning predictive models on EEG: a systematic comparison,” Journal of Neural Engineering, vol. 19, p. 066020, 11 2022.
  * [13] O. Ali, M. Saif-ur Rehman, S. Dyck, T. Glasmachers, I. Iossifidis, and C. Klaes, “Enhancing the decoding accuracy of EEG signals by the introduction of anchored-STFT and adversarial data augmentation method,” Scientific Reports, vol. 12, p. 4245, mar 2022.
  * [14] S. J. Pan and Q. Yang, “A Survey on Transfer Learning,” IEEE Transactions on Knowledge and Data Engineering, vol. 22, pp. 1345–1359, oct 2010.
  * [15] H. He and D. Wu, “Transfer learning for brain-computer interfaces: A Euclidean space data alignment approach,” IEEE Transactions on Biomedical Engineering, vol. 67, no. 2, pp. 399–410, 2020.
  * [16] Y. Song, Q. Zheng, B. Liu, and X. Gao, “Eeg conformer: Convolutional transformer for eeg decoding and visualization,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 31, pp. 710–719, 2022.
  * [17] C. Rommel, T. Moreau, J. Paillard, and A. Gramfort, “CADDA: Class-wise Automatic Differentiable Data Augmentation for EEG Signals,” in International Conference on Learning Representations, 2022.
  * [18] J.-R. King, L. Gwilliams, C. Holdgraf, J. Sassenhagen, A. Barachant, D. Engemann, E. Larson, and A. Gramfort, “Encoding and Decoding Framework to Uncover the Algorithms of Cognition,” in The Cognitive Neurosciences, The MIT Press, 05 2020.
  * [19] M. Tangermann, K.-R. Müller, A. Aertsen, B. N, C. Braun, B. C, R. Leeb, M. C, M. KJ, G. Müller-Putz, G. Nolte, G. Pfurtscheller, H. Preissl, G. Schalk, A. Schlögl, V. C, S. Waldert, and B. Blankertz, “Review of the BCI competition IV,” Frontiers in Neuroscience, vol. 55, 01 2012.
  * [20] H. Cho, M. Ahn, S. Ahn, M. Kwon, and S. C. Jun, “EEG datasets for motor imagery brain–computer interface,” GigaScience, vol. 6, jul 2017.
  * [21] J. Shin, A. von Luhmann, B. Blankertz, D.-W. Kim, J. Jeong, H.-J. Hwang, and K.-R. Muller, “Open Access Dataset for EEG+NIRS Single-Trial Classification,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 25, pp. 1735–1745, oct 2017.
  * [22] B. Aristimunha, I. Carrara, P. Guetschel, S. Sedlar, P. Rodrigues, J. Sosulski, D. Narayanan, E. Bjareholt, B. Quentin, R. T. Schirrmeister, E. Kalunga, L. Darmet, C. Gregoire, A. Abdul Hussain, R. Gatti, V. Goncharenko, J. Thielen, T. Moreau, Y. Roy, V. Jayaram, A. Barachant, and S. Chevallier, “Mother of all BCI Benchmarks v1.0.” doi.org/10.5281/zenodo.10034223, 2023.  DOI: 10.5281/zenodo.10034223.
  * [23] V. Jayaram and A. Barachant, “MOABB: trustworthy algorithm benchmarking for BCIs,” Journal of neural engineering, vol. 15, no. 6, p. 066011, 2018.
  * [24] S. Chevallier, I. Carrara, B. Aristimunha, P. Guetschel, S. Sedlar, B. Lopes, S. Velut, S. Khazem, and T. Moreau, “The largest EEG-based BCI reproducibility study for open science: the MOABB benchmark,” arXiv preprint arXiv:2404.15319, 2024.
  * [25] D. A. Engemann, A. Mellot, R. Höchenberger, H. Banville, D. Sabbagh, L. Gemein, T. Ball, and A. Gramfort, “A reusable benchmark of brain-age prediction from m/eeg resting-state signals,” Neuroimage, vol. 262, p. 119521, 2022.
  * [26] P. Bomatter, J. Paillard, P. Garces, J. Hipp, and D. Engemann, “Machine learning of brain-specific biomarkers from EEG,” bioRxiv, 2024.
  * [27] H. He and D. Wu, “Transfer learning for brain–computer interfaces: A euclidean space data alignment approach,” IEEE Transactions on Biomedical Engineering, vol. 67, no. 2, pp. 399–410, 2019.
  * [28] B. Junqueira, B. Aristimunha, S. Chevallier, and R. Y. de Camargo, “A systematic evaluation of euclidean alignment with deep learning for eeg decoding,” ArXiv, vol. abs/2401.10746, 2024.


# Perceived and Intended Sarcasm Detection with Graph Attention Networks

Joan Plepi  Lucie Flek
Conversational AI and Social Analytics (CAISA) Lab
Department of Mathematics and Computer Science, University of Marburg
http://caisa-lab.github.io

###### Abstract

Existing sarcasm detection systems focus on exploiting linguistic markers,
context, or user-level priors. However, social studies suggest that the
relationship between the author and the audience can be equally relevant for
the sarcasm usage and interpretation. In this work, we propose a framework
jointly leveraging (1) a user context from their historical tweets together
with (2) the social information from a user’s conversational neighborhood in
an interaction graph, to contextualize the interpretation of the post. We use
graph attention networks (GAT) over users and tweets in a conversation thread,
combined with dense user history representations. Apart from achieving state-
of-the-art results on the recently published dataset of 19k Twitter users with
30K labeled tweets, adding 10M unlabeled tweets as context, our results
indicate that the model contributes to interpreting the sarcastic intentions
of an author more than to predicting the sarcasm perception by others.

##  1 Introduction

Sarcasm is a form of non-literal language, in which the intended meaning of
the utterance differs from the literal meaning, fulfilling a social function
in a discourse Dews et al. (1995); Riloff et al. (2013). Sarcasm detection
poses a challenge for numerous NLP tasks, such as sentiment or stance
prediction Maynard and Greenwood (2014).

Early sarcasm detection systems are based on lexical and syntactic cues
Carvalho et al. (2009); Davidov et al. (2010); Tsur et al. (2010); González-
Ibáñez et al. (2011); Reyes et al. (2013); Ghosh et al. (2015). However,
sarcasm interpretation requires context, even for humans Wallace et al.
(2014). More recent works hence incorporate discourse information such as
contrast Riloff et al. (2013); Khattri et al. (2015); Joshi et al. (2015);
Rajadesingan et al. (2015); Tay et al. (2018), and contextualize the post by
using features from user history Bamman and Smith (2015); Amir et al. (2016);
Oprea and Magdy (2019); Hazarika et al. (2018). The relationship between an
author and the audience has been given comparably less attention, despite its
relevance for the sarcasm interpretation Rockwell and Theriot (2001); Gibbs
(2000); Dress et al. (2008); Marwick and Boyd (2011); Bamman and Smith (2015).
In this work, we propose a graph neural network framework jointly leveraging a
user context from their historical tweets together with the social information
from a user’s neighborhood modeled by heterogeneous graph structures.

The key contributions of this paper are:

(1) We present the first graph attention-based model to identify sarcasm on
social media by explicitly modeling users’ social and historical context
jointly, capturing complex relations between a sarcastic tweet and its
conversational context.

(2) We demonstrate that exploiting these relationships increases performance
in the sarcasm detection task, reaching state-of-the-art results on the recent
SPIRS dataset Shmueli et al. (2020), which we expand with user history. We
examine the impact of different parts of the context, captured by attention
weights, in modeling sarcastic utterances.

(3) We find that even with user-based models, detecting sarcastic intentions
of the author is easier than identifying the sarcasm perception by others.

##  2 Related Work

#### Leveraging user history

Several previous works contextualize a sarcastic post by using features from
user history - employing past tweets to identify a user’s behavioral traits
(Rajadesingan et al., 2015), encoding user sentiment priors over different
entities (Khattri et al., 2015), or manually crafting user interaction
features (Bamman and Smith, 2015). Amir et al. (2016) introduce the user2vec
model, applying paragraph2vec (Le and Mikolov, 2014) over user history.
Hazarika et al. (2018) propose an alternative user embedding approach,
encoding style and personality features.

#### Leveraging user network

An emerging line of research makes use of social interactions to encode
information about the user induced by neural architecture (Grover and
Leskovec, 2016; Qiu et al., 2018). Network information improves performance on
detecting cyberbullying (Mathew et al., 2019), abusive language use (Qian et
al., 2018), suicide ideation (Mishra et al., 2019) or fake news Chandra et al.
(2020). To the best of our knowledge, graph network based approaches have not
been used in the sarcasm detection task so far.

#### Perceived and intended sarcasm

Perceiving sarcasm in text is not trivial even for humans, not only due to the
lack of acoustic markers (Bänziger and Scherer, 2005; Woodland and Voyer,
2011) but also due to the sociocultural diversity (Rockwell and Theriot, 2001;
Dress et al., 2008) where in many cases the audience may misinterpret a
sarcastic statement as sincere. This has been only recently reflected in
sarcasm detection models Hazarika et al. (2018); Shmueli et al. (2020).

##  3 Proposed Approach

###  3.1 Tweet Embeddings

We denote the current tweet to be assessed $t_{i}\in
T=\\{t_{1},t_{2},\dots,t_{N}\\}$, where $N$ is the total number of tweets. We
utilize SentenceBERT embeddings Reimers and Gurevych (2019) to encode the
tweets. Formally, $\mathbf{t_{i}}^{\prime}=SentenceBERT(t_{i})$ where
$\mathbf{t_{i}}^{\prime}\in\mathbb{R}^{768}$, and SentenceBERT computes the
mean of all tokens’ representation. We forward this representation into a
linear layer to transform in dimension $d$,
$\mathbf{\widetilde{t_{i}}}\in\mathbb{R}^{d}$.

###  3.2 User Embeddings (Historical Context)

Let $u^{t_{i}}\in U=\\{u^{t_{1}},u^{t_{2}},\dots,u^{t_{M}}\\}$ be the author
of tweet $t_{i}$, from now on we keep only the index $i$ for brevity. Each
user $u^{i}$ is associated with a set of historical tweets
$\mathcal{H}^{i}=\\{(H_{1}^{i},\tau_{1}^{i}),\dots,(H_{m}^{i},\tau_{m}^{i})\\}$,
where $H_{j}^{i}$ is a historic tweet posted at a time $\tau_{j}^{i}$ by the
user $u^{i}$. We adopt user2vec Amir et al. (2016) to compute the initial user
representation $\widetilde{\mathbf{u}_{i}}\in\mathbb{R}^{d}$ of user $u^{i}$
based on their corresponding historical tweets $\mathcal{H}^{i}$, optimizing
the conditional probability of texts given the author.

Figure 1: An example of a heterogeneous user and tweet social graph extracted
from one conversation.

###  3.3 Social Graph (Network Context)

Apart from the importance of surrounding context to understand sarcasm Wallace
et al. (2014), certain understanding is needed between the audience and the
author Gibbs (2000); Dress et al. (2008). Our goal is to model relations
between users and their past tweets, interactions between users, and relations
between tweets in one conversation. We model these relationships as a graph
$\mathcal{G}=(V,E)$, where $V=\\{U\cup T\\}$ contains two types of nodes -
Users and Tweets (Figure 1). We use three edge types
$E=\\{\mathrm{e}^{U}\cup\mathrm{e}^{T}\cup\mathrm{e}^{C}\\}$, where
$\mathrm{e}^{U}$ represents the social interaction between users. This
involves quotes, mentions, or replies in the user history. $\mathrm{e}^{T}$
denotes the edges between tweets that are involved in one discussion thread,
with all tweets connected with each other, and $\mathrm{e}^{C}$ is the
relation between a tweet and its author.

#### Representation Learning:

We use Graph Attention Networks (GATs, Veličković et al. (2018)) to exploit
the neighborhood of each node to compute the final representations.111We ran
early experiments with Graph Convolutional Networks as well, obtaining
inferior and less interpretable results. GAT uses a self-attention mechanism
Bahdanau et al. (2015); Vaswani et al. (2017) to assign an importance score to
the connections that contribute more to the detection of sarcastic or non-
sarcastic tweets. We initialize the user and the tweet nodes of the GAT with
their corresponding embeddings $\widetilde{\mathbf{u}_{i}}$ and
$\widetilde{\mathbf{t}_{i}}$. The initial node representation of each node
$v\in V$ is linearly transformed by a weight matrix
$\mathbf{W}\in\mathbb{R}^{d^{\prime}\times d}$ into a vector
$\mathbf{h}_{v}\in\mathbb{R}^{d^{\prime}}$. Following, the attention weights
$e_{vn}$ of each node $v$ are computed as:

$e_{vn}=att(\mathbf{h}_{v}\|\mathbf{h}_{n})$ (1)

where $n\in\mathcal{N}(v)$ is a node in the neighborhood of $v$ and $att$ is
the attention mechanism function which is a single-layer feedforward neural
network, parameterized by a weight vector
$\vec{\mathbf{a}}\in\mathbb{R}^{2d^{\prime}}$ with a LeakyReLU nonlinearity.

The final node representation $\mathbf{h}_{v}^{\prime}\in\mathbb{R}^{K\cdot
d^{\prime}}$ is computed as:

$\mathbf{h}_{v}^{\prime}=\sigma\left(\frac{1}{K}\sum_{k=1}^{K}\sum_{n\in\mathcal{N}(v)}\alpha_{vn}^{k}\mathbf{W}^{k}\mathbf{h}_{n}\right)$
(2)

where $K$ is the number of attention heads, $\sigma$ is the ReLU nonlinear
function, $\mathbf{W}^{k}\in\mathbb{R}^{d^{\prime}\times d^{\prime}}$ a weight
matrix and $\alpha_{vn}^{k}=softmax(e_{vn}^{k})$ the normalized attention
weights from the $k$-th attention mechanism $att^{k}$.

###  3.4 Classification model

The user and tweet representations learned by GAT layer are concatenated and
forwarded through a two-layer feed-forward network parameterized by weight
matrices $\mathbf{W}_{1}^{c}\in\mathbb{R}^{d_{1}\times 2d^{\prime}}$ and
$\mathbf{W}_{2}^{c}\in\mathbb{R}^{o\times d_{1}}$, where $d_{1}$ is the
dimension of projected embeddings, and $o$ is equal to the number of classes.
The final prediction of the model is given by:

$\hat{y}=softmax\left(\mathbf{W}_{2}^{c}\left(\sigma\left(\mathbf{W}_{1}^{c}[h_{t}||h_{u}]\right)\right)\right)$
(3)

Figure 2: The social graph is initialized with user and tweet embeddings
(user2vec and sentence-BERT), and tuned by GAT to take into account
relationships between them. The output representations are then fed into the
classification layer.

##  4 Experimental Setup

###  4.1 Dataset

For our experiments, we use a recently published SPIRS sarcasm dataset Shmueli
et al. (2020). It utilizes cue tweets, conversation replies which point out
the sarcastic nature of a previous post. In addition, the dataset also
provides oblivious tweets, questioning the sarcastic nature of a given
example, and elicit tweets, being the original start of the conversation. Non-
sarcastic posts were collected randomly in equal numbers. The labeled dataset
contains in total 15,000 sarcastic tweets (10,000 self-reported and 5000
perceived cues), 15,000 non-sarcastic, 10,000 oblivious and 9156 elicit
tweets.

#### User context

We extend SPIRS with over 10 million past tweets of the authors in the dataset
in order to compute the user embeddings.

#### Social network

Our graph consists of the three types of connections described in Sec.3.3.To
avoid the bias coming from cue tweets, we exclude these from our graph. Our
final social network consists of 108K nodes with 0.00002 density and 32%
homophily, defined as the percentage of connections between authors of tweets
with the same label.

###  4.2 Comparison Baselines

The baselines introduced by Shmueli et al. (2020) are a Convolutional Neural
Network, a Bidirectional LSTM, and a fine-tuned pre-trained BERT model. We
compare our model with BERT, which performs the best of these. We add two
baselines which incorporate user information. First, we extend BERT by simply
concatenating the tweet embeddings with their respective user2vec author
representation (‘BERT + user2vec’). As a second baseline (‘BERT + user-only
GAT’), we build a social graph with only user nodes and their interactions
(quotes, mentions, or replies) $\mathrm{e}^{U}$ as edges, and apply the GAT
initialized with user2vec embeddings. The implementation of the models and the
results are made publicly available, to facilitate reproducibility and
reuse222https://github.com/caisa-lab/sarcasm_detection.

##  5 Results and Analysis

Our proposed GAT base model significantly outperforms all the baselines (Table
1) despite having fewer trainable parameters (500K) than the BERT model
(110M). First, by simply concatenating the user2vec embeddings to BERT, we
obtain 3.4% f1 score improvement on the BERT model, indicating the importance
of user context in sarcasm detection. Moreover, we introduce the GAT module in
the model. We first experiment with only tweet to tweet connections in the
graph based on the conversations on Twitter and trained on top of the fine-
tuned BERT. In this case, the GAT layer only bring 0.2% improvement due to the
sparse and disconnected nature of the constructed graph. In addition, we
replace user2vec with GAT embeddings tuned on user-only social graph, and we
achieve 6.1% improvement on BERT and 3% over ‘BERT + user2vec’, presumably
thanks to exploiting the homophily relations between users. Finally, applying
GAT on the full heterogeneous user and tweet graph (as per Figure 2) provides
a large performance boost thanks to incorporating the conversational thread
context between tweets.

Sarcasm Detection
---
Model | P | R | F1
BERT | 70.1% | 69.7% | 69.9%
BERT + user2vec | 73.6% | 73.2% | 73.4%
BERT + tweet-tweet GAT | 70.4% | 69.9% | 70.1%
BERT + user-only GAT | 74.2% | 78.1% | 76.1%
User+tweet GAT (no cues) |  84.7% |  83.7% |  84.2%
User+tweet GAT, no elicit | 83.2% | 80.8% | 82.0%
User+tweet GAT, no oblivious | 82.4% | 80.4% | 81.4%
User+tweet GAT + cue tweets | 94.7% | 94.3% | 94.5%
Table 1: Mean overall precision (P), recall (R), and F1 score (F1) of each
model over 10 runs with varying seeds, detecting sarcasm on the SPIRS dataset.

#### User representation

We compare the initial user embeddings initialized by user2vec with the final
representations computed from the GAT. The representations are projected in
2-dimensional space using T-SNE Van der Maaten and Hinton (2008). In Figure 3
and 4 we visualize the initial representations with user2vec and computed
representation by GAT layer respectively. While in user2vec representations
sarcastic users cannot be distinguished from non-sarcastic ones, in the GAT
representations we can observe communities of users sharing the same sarcastic
tendency.

Figure 3: Initial representations of users (user2vec) projected in 2D space
with T-SNE. Red color denotes sarcastic users, blue non-sarcastic. Figure 4:
Learned representations by our social network module (GAT) projected in 2D
space with T-SNE. Red color denotes sarcastic users, blue non-sarcastic.

#### Conversation context

For comparison, we construct three more social graphs where: 1) We remove the
elicit tweets which triggered the sarcastic comment (GAT - elicit tweets), 2)
We remove the oblivious tweets which interpreted the comment as serious (GAT -
oblivious tweets), 3) We add the original cue tweets, revealing that the post
was sarcastic (GAT + cue tweets). As expected, adding the cue tweets in the
social graph leads to an almost perfect F1 score of 94.5%. Removing oblivious
and elicit tweets causes just a small performance drop (2-3%). In the way the
SPIRS dataset is annotated, an oblivious tweet typically triggers a cue tweet
(“c mon, dude, it was just sarcasm”). We hypothesize that even with the cue
tweets removed, the model is able to learn the predictive relation between
oblivious and sarcastic tweets. This is in line with the original paper (i.e.
without user context), where a 3.4% drop in prediction accuracy was observed,
when the oblivious tweets were removed.

#### Attention weights

The attention mechanism of GAT is able to assign varied weights to different
nodes in the neighborhood, dynamically encoding of the user by their homophily
relations, which boosts the effect of authors in tweet representations Flek
(2020). We confirm this by examining users with a larger number of tweets in
the dataset. When users tend to be sarcastic in most of the posts, the
attention weight of their non-sarcastic tweets is smaller. In these cases, the
attention weights give more importance to the surrounding user context over
the conversation thread. Overall, the largest source of information for the
model are the user nodes and the tweet that is being classified. We note the
normalized attention weights are smaller for the oblivious and elicit tweet
edges, and higher for the edges that connect tweets with their respective
author. In other words, the conversational context only plays a decisive role
in case of insufficient or inconsistent user-level priors.

#### Sarcasm Perception

Cue tweets can be either authored by the same user as the sarcastic post
(intended sarcasm) or a different one (perceived sarcasm).We observe that in
the sarcasm detection task, the error rate on perceived sarcasm is 20% while
in the self-reported sarcasm it is only around 15%. We therefore test our
model on distinguishing between perceived and self-reported sarcasm. Our GAT
model brings an improvement of 2.2% over the BERT baseline, with the perceived
sarcasm being harder to detect (F1 56%) than the self-reported one (F1 84%).
These results are aligned with the conclusions from Oprea and Magdy (2019). In
most cases, the perceived sarcasm is misclassified as self-reported, which is
present more often (70%) in the data. Perceived sarcasm is dependent on the
readers rather than the author of the tweet, therefore we hypothesize that
modeling the authors’ context is less useful. It could be of benefit to model
more robust recipient user profiles as well, to better predict how each
individual will react.

Sarcasm Perception
---
Model | P | R | F1
BERT | 73.2% | 68.0% | 69.0%
User+tweet GAT (no cues) | 75.0% | 67.7% | 71.2%
Table 2: Mean overall precision (P), recall (R), and F1 score (F1) over 10
runs classifying self-reported (intended) and perceived sarcasm on the SPIRS
dataset.

#### Limitations

Modeling the social networks with GAT is affected by several factors. First,
the low graph density, as the original dataset wasn’t collected by following
relationships between users, hence many users across different conversation
threads are not related to each other. Second, the homophily degree is only
32%, users with sarcastic tendency have few connections among them.

##  6 Conclusions

In this work, we explore social networks of user interactions, and contextual
information to interpret sarcastic intentions in social media. We propose a
graph attention-based model, which combines contextual information of users,
linguistic features, and social networks. The heterogeneous social network
modeling dynamically exploits relationships between users and tweets in a
conversation and significantly improves the state-of-the-art results.

## Acknowledgements

This work has been supported by the German Federal Ministry of Education and
Research (BMBF) as a part of the Junior AI Scientists program under the
reference 01-S20060. We would like to thank Flora Sakketou and all the
anonymous reviewers for their valuable input.

## Ethical Considerations

The ability to automatically approximate personal characteristics of online
users in order to improve natural language classification algorithms requires
us to consider a range of ethical concerns, including: (1) privacy and user
consent, (2) representativeness of the data for generalization, and (3) user
vulnerability to a potential model or data misuse or misinterpretation.

Use of any user data for personalization shall be transparent, and limited to
the given purpose, no individual posts shall be republished (Hewson and
Buchanan, 2013). Researchers are advised to take account of users’
expectations (Williams et al., 2017; Shilton and Sayles, 2016; Townsend and
Wallace, 2016) when collecting public data such as Twitter. In this case, when
we expand the original dataset with more extensive user history, we utilize
publicly available Twitter data in a purely observational (Norval and
Henderson, 2017), and non-intrusive manner. All user data is kept separately
on protected servers, linked to the raw text and network data only through
anonymous IDs.

Shah et al. (2020) identify four different sources of bias in NLP models:
selection bias, label bias, model overamplification, and semantic bias. While
we can’t exclude any of those, the selection bias should be kept in mind in
particular, when reusing the presented model, as it is unclear to which extent
the augmented SPIRS dataset with user history represents a sample of the
overall population on Twitter. The user selection was based solely on the
available sarcasm annotations, and doesn’t include any sociodemographic
information.

In addition, any user-augmented classification efforts risk invoking
stereotyping and essentialism, as the algorithm may lean towards label people
rather than posts (e.g. “this is a sarcastic person”). Such stereotypes can
cause harm even if they are accurate on average differences (Rudman and Glick,
2008). These can be emphasized by the semblance of objectivity created by the
use of a computer algorithm (Koolen and van Cranenburgh, 2017). It is
important to be mindful of these effects when interpreting the model results
in an own end-application context.

## References

  * Amir et al. (2016) Silvio Amir, Byron C. Wallace, Hao Lyu, Paula Carvalho, and Mário J. Silva. 2016\.  Modelling context with user embeddings for sarcasm detection in social media.  In _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_ , pages 167–177, Berlin, Germany. Association for Computational Linguistics.
  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. 2015.  Neural machine translation by jointly learning to align and translate.  In _3rd International Conference on Learning Representations, ICLR 2015_.
  * Bamman and Smith (2015) David Bamman and Noah Smith. 2015.  Contextualized sarcasm detection on twitter.  In _Proceedings of the International AAAI Conference on Web and Social Media_ , volume 9.
  * Bänziger and Scherer (2005) Tanja Bänziger and Klaus R. Scherer. 2005.  The role of intonation in emotional expressions.  _Speech Communication_ , 46(3):252–267.  Quantitative Prosody Modelling for Natural Speech Description and Generation.
  * Carvalho et al. (2009) Paula Carvalho, Luís Sarmento, Mário J. Silva, and Eugénio de Oliveira. 2009.  Clues for detecting irony in user-generated contents: Oh…!! it’s "so easy" ;-).  In _Proceedings of the 1st International CIKM Workshop on Topic-Sentiment Analysis for Mass Opinion_ , TSA ’09, page 53–56, New York, NY, USA. Association for Computing Machinery.
  * Chandra et al. (2020) Shantanu Chandra, Pushkar Mishra, Helen Yannakoudakis, Madhav Nimishakavi, Marzieh Saeidi, and Ekaterina Shutova. 2020.  Graph-based modeling of online communities for fake news detection.  _arXiv preprint arXiv:2008.06274_.
  * Davidov et al. (2010) Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010.  Semi-supervised recognition of sarcasm in Twitter and Amazon.  In _Proceedings of the Fourteenth Conference on Computational Natural Language Learning_ , pages 107–116, Uppsala, Sweden. Association for Computational Linguistics.
  * Dews et al. (1995) Shelly Dews, Joan Kaplan, and Ellen Winner. 1995.  Why not say it directly? the social functions of irony.  _Discourse Processes_ , 19(3):347–367.
  * Dress et al. (2008) Megan L. Dress, Roger J. Kreuz, Kristen E. Link, and Gina M. Caucci. 2008.  Regional variation in the use of sarcasm.  _Journal of Language and Social Psychology_ , 27(1):71–85.
  * Flek (2020) Lucie Flek. 2020.  Returning the N to NLP: Towards contextually personalized classification models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7828–7838, Online. Association for Computational Linguistics.
  * Ghosh et al. (2015) Debanjan Ghosh, Weiwei Guo, and Smaranda Muresan. 2015.  Sarcastic or not: Word embeddings to predict the literal or sarcastic meaning of words.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1003–1012, Lisbon, Portugal. Association for Computational Linguistics.
  * Gibbs (2000) Raymond W. Gibbs. 2000.  Irony in talk among friends.  _Metaphor and Symbol_ , 15(1-2):5–27.
  * González-Ibáñez et al. (2011) Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. 2011\.  Identifying sarcasm in Twitter: A closer look.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 581–586, Portland, Oregon, USA. Association for Computational Linguistics.
  * Grover and Leskovec (2016) Aditya Grover and Jure Leskovec. 2016.  Node2vec: Scalable feature learning for networks.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’16, page 855–864, New York, NY, USA. Association for Computing Machinery.
  * Hazarika et al. (2018) Devamanyu Hazarika, Soujanya Poria, Sruthi Gorantla, Erik Cambria, Roger Zimmermann, and Rada Mihalcea. 2018.  CASCADE: Contextual sarcasm detection in online discussion forums.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 1837–1848, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Hewson and Buchanan (2013) Claire Hewson and Tom Buchanan. 2013.  Ethics guidelines for internet-mediated research.  The British Psychological Society.
  * Joshi et al. (2015) Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015.  Harnessing context incongruity for sarcasm detection.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 757–762, Beijing, China. Association for Computational Linguistics.
  * Khattri et al. (2015) Anupam Khattri, Aditya Joshi, Pushpak Bhattacharyya, and Mark Carman. 2015.  Your sentiment precedes you: Using an author’s historical tweets to predict sarcasm.  In _Proceedings of the 6th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis_ , pages 25–30, Lisboa, Portugal. Association for Computational Linguistics.
  * Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba. 2015.  Adam, a method for stochastic optimization.  In _Proceedings of the 3rd International Conference on Learning Representations (ICLR)_ , volume 1412.
  * Koolen and van Cranenburgh (2017) Corina Koolen and Andreas van Cranenburgh. 2017.  These are not the stereotypes you are looking for: Bias and fairness in authorial gender attribution.  In _Proceedings of the First ACL Workshop on Ethics in Natural Language Processing_ , pages 12–22, Valencia, Spain. Association for Computational Linguistics.
  * Le and Mikolov (2014) Quoc Le and Tomas Mikolov. 2014.  Distributed representations of sentences and documents.  In _Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32_ , ICML’14, page II–1188–II–1196. JMLR.org.
  * Marwick and Boyd (2011) Alice E. Marwick and Danah Boyd. 2011.  I tweet honestly, i tweet passionately: Twitter users, context collapse, and the imagined audience.  _New Media & Society_, 13(1):114–133.
  * Mathew et al. (2019) Binny Mathew, Ritam Dutt, Pawan Goyal, and Animesh Mukherjee. 2019.  Spread of hate speech in online social media.  In _Proceedings of the 10th ACM Conference on Web Science_ , WebSci ’19, page 173–182, New York, NY, USA. Association for Computing Machinery.
  * Maynard and Greenwood (2014) Diana Maynard and Mark Greenwood. 2014.  Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 4238–4243, Reykjavik, Iceland. European Language Resources Association (ELRA).
  * Mishra et al. (2019) Rohan Mishra, Pradyumn Prakhar Sinha, Ramit Sawhney, Debanjan Mahata, Puneet Mathur, and Rajiv Ratn Shah. 2019.  SNAP-BATNET: Cascading author profiling and social network graphs for suicide ideation detection on social media.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop_ , pages 147–156, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Norval and Henderson (2017) Chris Norval and Tristan Henderson. 2017.  Contextual consent: Ethical mining of social media for health research.  _CoRR_ , abs/1701.07765.
  * Oprea and Magdy (2019) Silviu Oprea and Walid Magdy. 2019.  Exploring author context for detecting intended vs perceived sarcasm.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2854–2859, Florence, Italy. Association for Computational Linguistics.
  * Qian et al. (2018) Jing Qian, Mai ElSherief, Elizabeth Belding, and William Yang Wang. 2018.  Leveraging intra-user and inter-user representation learning for automated hate speech detection.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 118–123, New Orleans, Louisiana. Association for Computational Linguistics.
  * Qiu et al. (2018) Jiezhong Qiu, Yuxiao Dong, Hao Ma, Jian Li, Kuansan Wang, and Jie Tang. 2018.  Network embedding as matrix factorization: Unifying deepwalk, line, pte, and node2vec.  In _Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining_ , WSDM ’18, pages 459–467, New York, NY, USA. ACM.
  * Rajadesingan et al. (2015) Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015.  Sarcasm detection on twitter: A behavioral modeling approach.  In _Proceedings of the Eighth ACM International Conference on Web Search and Data Mining_ , WSDM ’15, page 97–106, New York, NY, USA. Association for Computing Machinery.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Reyes et al. (2013) Antonio Reyes, Paolo Rosso, and Tony Veale. 2013.  A multidimensional approach for detecting irony in twitter.  _Lang. Resour. Eval._ , 47(1):239–268.
  * Riloff et al. (2013) Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013.  Sarcasm as contrast between a positive sentiment and negative situation.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 704–714, Seattle, Washington, USA. Association for Computational Linguistics.
  * Rockwell and Theriot (2001) Patricia Rockwell and Evelyn M. Theriot. 2001.  Culture, gender, and gender mix in encoders of sarcasm: A self-assessment analysis.  _Communication Research Reports_ , 18(1):44–52.
  * Rudman and Glick (2008) Laurie A Rudman and Peter Glick. 2008.  The social psychology of gender: How power and intimacy shape gender relations.
  * Shah et al. (2020) Deven Santosh Shah, H. Andrew Schwartz, and Dirk Hovy. 2020.  Predictive biases in natural language processing models: A conceptual framework and overview.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5248–5264, Online. Association for Computational Linguistics.
  * Shilton and Sayles (2016) Katie Shilton and Sheridan Sayles. 2016.  " we aren’t all going to be on the same page about ethics": Ethical practices and challenges in research on digital and social media.  In _2016 49th Hawaii International Conference on System Sciences (HICSS)_ , pages 1909–1918. IEEE.
  * Shmueli et al. (2020) Boaz Shmueli, Lun-Wei Ku, and Soumya Ray. 2020.  Reactive Supervision: A New Method for Collecting Sarcasm Data.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2553–2559, Online. Association for Computational Linguistics.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ , 15(1):1929–1958.
  * Tay et al. (2018) Yi Tay, Anh Tuan Luu, Siu Cheung Hui, and Jian Su. 2018.  Reasoning with sarcasm by reading in-between.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1010–1020, Melbourne, Australia. Association for Computational Linguistics.
  * Townsend and Wallace (2016) Leanne Townsend and Claire Wallace. 2016.  Social media research: A guide to ethics.  _University of Aberdeen_ , 1:16.
  * Tsur et al. (2010) Oren Tsur, Dmitry Davidov, and Ari Rappoport. 2010.  Icwsm — a great catchy name: Semi-supervised recognition of sarcastic sentences in online product reviews.  _Proceedings of the International AAAI Conference on Web and Social Media_ , 4(1):162–169.
  * Van der Maaten and Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-sne.  _Journal of machine learning research_ , 9(11).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17, page 6000–6010, Red Hook, NY, USA. Curran Associates Inc.
  * Veličković et al. (2018) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018.  Graph Attention Networks.  _International Conference on Learning Representations_.  Accepted as poster.
  * Wallace et al. (2014) Byron C. Wallace, Do Kook Choe, Laura Kertz, and Eugene Charniak. 2014.  Humans require context to infer ironic intent (so computers probably do, too).  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 512–516, Baltimore, Maryland. Association for Computational Linguistics.
  * Williams et al. (2017) Matthew L Williams, Pete Burnap, and Luke Sloan. 2017.  Towards an ethical framework for publishing twitter data in social research: Taking into account users’ views, online context and algorithmic estimation.  _Sociology_ , 51(6):1149–1168.
  * Woodland and Voyer (2011) Jennifer Woodland and Daniel Voyer. 2011.  Context and intonation in the perception of sarcasm.  _Metaphor and Symbol_ , 26(3):227–239.

## Appendix

##  Appendix A Configurations

We perform a stratified 90/10 train-test split. We sample $10\%$ of the
training data for validation. All splits have the same class distribution and
different sets of tweet authors. We use 3 GAT layers, with number of heads
$K=4$. The initial dimension is $d=400$ and the final output dimension
$d^{\prime}=100$. To train our model we set learning rate to $1e-4$, and
dropout 0.4 Srivastava et al. (2014), and use the Adam optimization algorithm
Kingma and Ba (2015) for 500 training epochs with early stopping. For the GAT
layers, we compute the mean of the outputs from each attention head instead of
concatenation. All experiments are run in Nvidia A100 40 GB GPUs.

##  Appendix B User Context

To incorporate user context, we first extract all user IDs for all the tweets
in the dataset. In the dataset, due to different tweet types with different
users, we get in total 57K users. We fetch the tweet post timeline for each
user, and we end up with a total of 104M tweets, in average 1800 posts per
user. For user2vec training, we take into account only the users with a
minimum of 50 posts in their timeline, and we limit the total number of posts
to 1000. After filtering, the amount of tweets in the context is 10M. Every
tweet is pre-processed by removing all links, user mentions are replaced with
"@user", emojis and hashtags are cleared. We train user2vec Amir et al. (2016)
for 12 epochs, with learning rate 1e-4. For those users which are filtered, or
we cannot extract history, we initialize them as the mean representation of
his user neighbors in the social network. We used the history tweets only for
creating the user-to-user edges, and those are not present in the constructed
graph, but are already encoded in the initial user representation. We
experimented with various history length settings, and found almost no
difference in the performance between using interactions throughout all
history and interactions during the last year. Hence, we omitted older
interactions to ease the computations.