# Guilt Detection in Text: A Step Towards Understanding Complex Emotions

Abdul Gafar Manuel Meque  Instituto Politécnico Nacional (IPN), Centro de
Investigación en Computación (CIC), Mexico City, Mexico  Catholic University
of Mozambique, Faculdade de Economia e Gestao, Beira, 2100, Mozambique  Nisar
Hussain  Instituto Politécnico Nacional (IPN), Centro de Investigación en
Computación (CIC), Mexico City, Mexico  Grigori Sidorov  Instituto Politécnico
Nacional (IPN), Centro de Investigación en Computación (CIC), Mexico City,
Mexico<EMAIL_ADDRESS>Alexander Gelbukh  Instituto Politécnico Nacional
(IPN), Centro de Investigación en Computación (CIC), Mexico City, Mexico

###### Abstract

We introduce a novel Natural Language Processing (NLP) task called Guilt
detection, which focuses on detecting guilt in text. We identify guilt as a
complex and vital emotion that has not been previously studied in NLP, and we
aim to provide a more fine-grained analysis of it. To address the lack of
publicly available corpora for guilt detection, we created VIC, a dataset
containing 4622 texts from three existing emotion detection datasets that we
binarized into guilt and no-guilt classes. We experimented with traditional
machine learning methods using bag-of-words and term frequency-inverse
document frequency features, achieving a 72% f1 score with the highest-
performing model. Our study provides a first step towards understanding guilt
in text and opens the door for future research in this area.

## Introduction

In this study, we introduce Guilt detection, a novel task in Natural Language
Processing aimed at detecting guilt in text. We also developed a dataset and a
set of baseline classifiers for this task. Guilt is an emotion or feeling that
arises when one contemplates past wrongdoings or thoughts regarding the
morality of future actions, whether real or imagined. Throughout their lives,
individuals experience a wide range of emotions and strongly need to connect
with others and share their feelings with peers.

Sharing emotional experiences is a crucial component of the emotional process,
as observed by [1]; furthermore, in [2], the authors found that sharing
emotional experiences on the internet has become a crucial part of daily life
on contemporary cultures. This can explain the popularity of emotion-related
research in many scientific disciplines, including computer science,
particularly Natural Language Processing (NLP), to determine the presence of,
assess the intensity and polarity of, and compare various emotional
experiences shared in the form of written text. Despite the considerable
interest in emotion detection in general, there is still a sizable research
vacuum regarding the in-depth examination of particular emotions, such as
guilt.

Guilt comes in many forms, namely anticipatory, existential, and reactive
guilt [3], and it is frequently felt when people feel responsible for
wrongdoing or failing to uphold their own moral standards [4], as a result, it
frequently indicates an understanding of the experiences of others and a
desire to correct any perceived interpersonal flaws [5]. As with other self-
conscious emotions, when in excess, guilt can have negative implications for
one’s mental health [6], thus attracting the interest of researchers in
various disciplines, from Psychology to neuroscience to Computer Science, in
the latter, particularly as a sub-field in Natural Language Processing, guilt
has been featured as a class in Emotion Detection sub-tasks in numerous NLP
research works.

Considering the mental health implications of excessive guilt as mentioned in
[6], and numerous research studies have shown a link between guilt and
suicidality in clinical populations, as discussed in [5], which cites two
studies [7] [8], identifying guilt as one of the risk factors in suicidal
ideation (SI), there is clear importance and value in studying ways to
identify if, when and how an individual experiences guilt. Considering the
finds of [1] and [2], social media texts can be a great source for studying
user-shared guilt experiences and feelings. However, to our knowledge, guilt
has only been studied as a class in a multi-class emotion detection task and
never as the primary subject of study in NLP, hence our interest in it. As a
first step toward closing the research gap, we examined existing textual
datasets from previous emotion detection tasks in which guilt was featured to
build a binary guilt detection dataset. As detailed below, we intentionally
selected data shared by the experiencers through different media and in
different settings. In addition, we present a set of baseline models based on
Traditional and Deep Learning methods.

The following are the main contributions of this paper:

  * •

A study of guilt detection from text using NLP techniques

  * •

Development of a multi-source dataset for binary guilt detection,

  * •

Development of baseline models for the proposed dataset,

  * •

An in-depth analysis of the dataset and models

The remainder of the paper is comprised of a Literature Review section
showcasing works on emotion detection, especially those featuring guilt,
Dataset Development section describing the techniques used in acquiring and
building the dataset, Benchmark Experiments which present the baseline model
experiment setups, followed by ”Results and discussion” where an initial
analysis of the baseline results is made, concluding with Conclusion and
Future Work, where we hint at possible future directions of our research work
on guilt.

## Literature Review

To the best of our knowledge, there has not been any comprehensive research
work done targeting Guilt, nor is there any dataset dedicated to guilt
emotion, at least not one that is publicly available. Although Guilt itself
has not been the main subject of research, it has been featured in various
studies, such as in [9], [10], where the authors explore a way and propose an
approach to detect different types of emotions, using a Common Sense Knowledge
EmotiNet, which they extended and improved in [11].
In this section, we present related work regarding both datasets and
techniques for Emotion Detection in general and fine-grained detection of a
specific emotion or emotion-related affective states featuring Guilt.

Due to its popularity, there exist a considerable number of datasets on
Emotion Detection available publicly and featured in numerous studies, such as
Vent [12], XED [13], GoEmotions [14], EmotionLines [15], Corpus of Emotion
Annotated Suicide notes in English (CEASE) [10], The “International Survey on
Emotion Antecedents and Reactions” (ISEAR) [16], deISEAR [17], to mention a
few. It is worth noting that many existing emotion detection datasets use
emotion labels based on Ekman’s [18, 19] or Plutchik’s [20] theory, or a
variation thereof, often including the neutral label. In our study, we
specifically focused on three datasets [10, 12, 16] that contain instances of
guilt.

Vent dataset[12], a large annotated dataset of texts (texts are from 33M
posts), emotions, and social connections from the Vent social media platform.
In Vent, each post is associated with emotion, self-annotated with emotion by
the post’s author at the posting time. In [12] no experiments on ED were
reported, but in [21] the authors ran ED experiments using Naive Bayes (NB),
Random Forests (RF) and Logistic Regression (LR), Deep Neural Networks and Bi-
LSTM. They employed two sets of feature representations, based on the type of
method. For each of the traditional ML methods, they ran two sets of
experiments, one using Bag-of-Words and then using TF-IDF feature vectors. The
same procedure was used for Neural Network methods, but now using FastText and
BERT embeddings as features. This resulted in ten (10) combinations of methods
and features, with their best model achieving 19% and 21% in average macro and
micro F1-score respectively, which is not surprising, considering the number
of classes in Vent.

The CEASE dataset [10], consisting of 2393 sentences extracted from around 205
English language suicide notes, collected from various websites and annotated
for 15 emotion classes. For their experiments, they first employed three deep
learning (DNN) models (Convolutional Neural Network[22], Long Short Term
Memory[23], and Gated Recurrent Unit model[24]), the three DNN models were
later combined to create two additional models (one majority vote ensemble and
one multi-layer perceptron-based ensemble model). For comparison purposes,
they trained and tested various combinations of traditional ML classifiers
(Multinomial-NB, RF, LR, and Support Vector Machine (SVC)) with different sets
of features. The MLP ensemble and LTSM models achieved the best results, with
an F1-Score of 59% on average for all classes, and 48% for guilt, 4 points
less than the performance of the majority vote ensemble on this particular
class. Among the traditional ML models, LR was the best-performing model, with
an average F1-score of 48% and 35% for the guilt class.

The ISEAR dataset [16], a collection of 7669 sentences extracted from
questionnaire answers from respondents from different countries, in different
languages, as part of the ISEAR project in the ’90s, the sentences were first
translated and then annotated for 7 emotion classes. This dataset inspired the
creation of two new datasets, the enISEAR (English ISEAR) deISEAR and (Deutsch
ISEAR)[17], both comprising 1001 descriptions obtained from the study
participants after being presented with emotion and asked to describe an event
in which they felt that particular emotion. Both datasets have the same number
of descriptions, but are different in length, thus resulting in deISEAR being
divided into 1084 sentences and a vocabulary size of 2613 distinct tokens, and
enISEAR into 1366 sentences and a vocabulary of 3066. In [17] the authors
trained a Maximmum Entropy (MaxEnt) classifier with unigram features on
ISEAR[16] and applied it to enISEAR and a translated version of deISEAR,
achieving an average micro F1-Score of 47%, on both datasets, 42% on guilt
class on deISEAR, and 41% for the guilt class on enISEAR.

## Dataset Development

For this research, three existing datasets were used as a starting point: the
Vent [12], ISEAR[16] and CEASE [10] Emotion datasets. The selection of these
specific datasets was mainly for two reasons, (1) they are each from a
different domain (source of where the texts were collected), and (2) they all
contain guilt as one of the classes. A full description is provided below.

### Dataset Preparation

Initially, we considered all samples from each dataset because the Vent
dataset contains 33M samples, we only selected samples from the feelings
category, which is the category that contains the guilt subclass. This first
sampling resulted in 4.358.680 samples from Vent, 7666 from ISEAR, and 2393
from CEASE. Table 1 shows the emotion distribution based on this selection.
Emotion classes that are present in at least two original data sources are
labeled with the original name, and those only appearing in one data source
are labeled as “others” in the table. As shown in Table 1, there were a total
of 4,368,739 samples, of which 135,601 were texts labelled with guilt, and
4,233,138 with joy/happiness, anger, sadness/sorrow, fear, disgust, shame, and
other emotions.

| Dataset
---|---
Emotion | Vent | ISEAR | CEASE
guilt | 134434 | 1093 | 74
joy/happiness | - | 1094 | 38
anger | - | 1096 | 79
sadness/sorrow | - | 1096 | 305
fear | - | 1095 | 29
disgust | - | 1096 | -
others | 4224246 | - | 1868
shame | - | 1096 |
Total | 4358680 | 7666 | 2393
Table 1: Initial Data sampling result

With the initial selection, we proceeded with a second round of selection, to
deal with the enormous differences in size. For this, we first limited the
choice of non-guilt samples to the number of guilt samples found in each
original dataset, giving us 135,604 guilt and 135,604 not-guilt samples,
totaling 271.208 samples.

Because we are dealing with data from three different sources, we consider a
few different steps for cleaning the data, taking into account its origin. The
following steps were taken to identify and fix problems with the dataset:

The vent dataset comes from automatically scraping millions of social media
posts, so it stands to reason that there might be some issues with some of the
samples, and a quick look into the texts, revealed some samples consisting of
only a single character space, special character, or emojis, which are not
helpful to our task. The second step was to identify duplicated text-label
combinations, followed by text language identification. Since ISEAR was
compiled from a questionnaire answers, we check whether there are samples with
no “no response”, “blank” or similar values, which are a clear indication of
the absence of an answer from the respondents. As an extra cleaning step for
the ISEAR, we also checked for samples with “No response.”

### VIC Dataset Statistics

With the dataset fairly cleaned, we proceeded to sample all guilt instances
from each dataset and set the returned number of samples as a cap for
selecting non-guilt samples, which resulted in 245,826 samples, which were
then binarized (all non-guilt labels converted to “no guilt”), followed by a
random selection of up to 1200 samples per class per origin, which led to the
final dataset comprised of 4622 samples, with a balanced class distribution.

Text |  Origin Label |  Binary Label
---|---|---
why do i always go back to christian why does he accept every time i fuck up and go back to him |  guilt |  guilt
its two am and all i can think about is you i want you here i need you now i dont think i can ever walk away from you i wish i could move on but how do you move on from something youve never really had ugh |  Tired |  no guilt
my mom brought me and sister home because we were hungry and tired so i’m back in my nice bed |  Sleepy |  no guilt
vent would be more useful if i could actually use words to describe my feelings |  Tired |  no guilt
on embarking on university life i came from a different city and did not know anybody at the uni i was frightened because my well known and loved friends also all my security had been taken away |  fear |  no guilt
fear paralysing that i would not be accepted by the god who i believed to be there because i was morally bankrupt before becoming a christian and realising that that was why christ came to free us from sin and to forgive us |  fear |  no guilt
i am sorry to the people that i love but i can not fucking take it anymore |  guilt |  guilt
on a day like this everyone seeks the company of beloved ones |  love |  no guilt
i am sorry once again |  guilt |  guilt
i angered a close friend and he was injured |  guilt |  guilt

Table 2:  A sample of texts and labels, with origin label from dataset VIC

Our resulting dataset (VIC) contains 126 samples from CEASE, 2,096 from ISEAR,
and 1248 from Vent, Table 2 presents a sample of texts in VIC and Table 3
details the label distribution by sample origin and Figure 1 depicts the
distribution of non-guilt class samples distribution for each origin and their
distribution in VIC. As shown in Table 3, initial data exploration revealed
that guilt samples generally have a slightly smaller average word length when
compared to the non-guilt samples, and the largest texts are of non-guilt
samples.

Figure 1: Original label distribution for each subset Class | # of Samples | Sample Origin
---|---|---
Guilt | 63 | CEASE
No Guilt | 63
Guilt | 1048 | ISEAR
No Guilt | 1048
Guilt | 1200 | Vent
No Guilt | 1048
Table 3: A summary of label and origin distribution in the resulting dataset
VIC

##

## Benchmark Experiments

To provide a baseline for the guilt detection task, we conducted two sets of
experiments, one using traditional machine learning and the second using
neural network-based methods. For the traditional machine learning methods, we
first conducted hyperparameter tuning using a Grid Search algorithm and then
trained and tested models with the best parameters from the grid search which
was then compared with the results from neural networks models without tuning
them. The full details of the experiments are detailed in the following
subsection. To properly assess the effects of the dataset origin in the task,
for each combination of method and features, we tested on subsets of the
dataset based on the sample origin and finally on the whole dataset.

### Methods

Traditional Machine Learning: we have chosen three methods, namely Support
Vector Machine (SVM) a machine learning algorithm widely used in
classification tasks. It has the property of being able to learn the correct
labeling of data points that belong to different classes, given a set of
training examples. Multinomial Naive Bayes, a Bayesian learning approach also
widely used in natural language processing tasks, and Logistic Regression, a
binary classifier that uses sigmoid activation in the output layer to predict
the label. For each method, we experimented using two different features type,
BoW and tf-idf vectors.
Neural Networks:  for neural network methods, we selected Convolutional Neural
Networks, and inspired by the experiments in [25] and [26], we added Bi-
directional Long Short Term Memory (BiLSTM) to our choices of neural network
models. All of our Sequential models start with an Embedding layer, with 64
dimensions and the vocabulary and input length based on the training data,
with a final Dense layer with sigmoid activation.

  * •

CNN is a feed-forward neural network called commonly used to analyze visual
images by processing data in a grid-like architecture. According to [26] CNNs
are able to automatically learn features from the input text and have been
successful in many natural language processing tasks. in addition to the
layers in common to all the other models, our CNN has a 1-D Convolutional
layer, a Global Max Pooling layer, and a Dense layer with relu activation.

  * •

Bi-LSTM [27], a form of recurrent neural network that uses LSTM cells (i.e.
cell state representations) to adaptively change the size of its step length
between input. This mechanism makes it possible for the network to learn long
and short-term dependencies, as well as dynamics that span multiple time steps
and hidden states. this NN is made of a Bidirectional LSTM layer and Dropout
layers.

### Feature engineering

Bag of Words: each instance of text in our corpus is treated as a collection
of words, that is then vectorized using a Count Vectorizer using sklearn
library’s feature extraction package, with n={1-4}.
TFIDF: as a second feature option, we used the n-gram term frequency and
inverse document frequency Vectors with n={1-4},

### Hyperparameter tuning with GridSearch

We used TF-IDF as features and trained classical ML classifiers for the first
set of experiments. We used GridSearch to find the best hyperparameters (C for
SVM and LR and alpha for MNB for example) and feature combinations for each ML
model. The entire hyperparameter search space is presented in Table 4.

Method | Hyperparameter | Search Space
---|---|---
LR | solver | [newton-cg, lbfgs,sag, liblinear, saga]
C | [0.01, 0.1, 1, 10, 100]
penalty | [’l1’, ’l2’, ’elasticnet’]
SVC | kernel | linear, rbf]
C | [0.01, 0.1, 1, 10, 100]
gamma | [0.001, 0.0001]
GB | n_estimators | [16, 32]
learning_rate | [0.8, 1.0]
DT & RF | max_depth | [5,10,25,None]
min_samples_leaf | [5, 10, 20, 50, 100]
criterion | [gini, entropy]
RF | n_estimators | [16, 32]
MNB | alpha | [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]
fit_prior | [True, False]
class_prior | [None, [.1,.9],[.2, .8]]

Table 4: Hyper parameter search configuration

### Dataset experiment setup

Origin-based subsetting : We selected the samples from VIC based on their
origin, resulting in three sub-datasets, and ran experiments on each of them
separately, using the standard train and test split, with 10% samples for
testing and 90% for training each time.
Combined and Shuffled: with the help of the pandas library, we shuffled the
entire VIC dataset and conducted experiments on it, also using the 90-10 train
and test splits.
Leave-one-out Training and Testing: In addition to the experiments detailed
above, we ran train and test experiments using examples from two origins for
training and the third one for testing.

## Results and discussion

We started by running a hyperparameter search for the traditional ML models
(which resulted in 478 candidate model configurations, from combining methods,
hyperparameters, and features) on the VIC dataset. The resulting models were
then ranked based on their overall performance, and for each method, we
selected their best parameters for the subsequent tests. As shown in Table 5,
for the Tfidf feature type, the best performing algorithm is MNB with an
F1-score of 0.72, using alpha=0.9, class_prior=None, and fit_prior=True. LR
and SVC also have similar performances, with F1 scores of 0.68 and 0.67,
respectively. The best-performing RF and DT have F1 scores of 0.67 and 0.65,
respectively.

For the Bow feature type, MNB also has the highest F1-score of 0.72, using
alpha=0.7, class_prior=None, and fit_prior=True. LR has an F1-score of 0.69,
while SVC has an F1-score of 0.67. The best-performing RF and DT have F1
scores of 0.67 and 0.66, respectively. The best-performing GB has an F1-score
of 0.63, using n_estimators=32 and lr=1.

Table 5 shows the hyperparameter results for different machine learning
algorithms and feature types. The evaluation metric used is F1-score, and the
hyperparameters selected by the tuning process are listed as well.

Features | Estimator | min | mean | max | std dev | Hyperparameters
---|---|---|---|---|---|---
Tfidf | Multinomial Naive Bayes | 0.71 | 0.72 | 0.75 | 0.015 |  $\alpha=1.0$, class_prior=None, fit_prior=True
Support Vector Machine | 0.68 | 0.69 | 0.71 | 0.009 |  $C=100$, kernel=linear
Logistic Regression | 0.68 | 0.69 | 0.70 | 0.007 |  $C=100$, penalty=l2, solver=saga
Random Forest | 0.33 | 0.58 | 0.67 | 0.127 |  $n\\_estimators=32$, criterion=gini, max_depth=None, min_samples_leaf=100
Gradient Boosting | 0.60 | 0.62 | 0.64 | 0.014 |  $n\\_estimators=32$, lr=0.8
Decision Tree | 0.60 | 0.61 | 0.63 | 0.010 | criterion=entropy, max_depth=None, min_samples=10
BoW | Multinomial Naive Bayes | 0.70 | 0.71 | 0.73 | 0.011 |  $\alpha=0.7$, class_prior={0.1,0.9}, fit_prior=True
Logistic Regression | 0.65 | 0.67 | 0.69 | 0.016 |  $C=10$, penalty=l2, solver=sag
Random Forest | 0.47 | 0.61 | 0.69 | 0.079 |  $n\\_estimators=32$, criterion=gini, max_depth=None, min_samples_leaf=10
Support Vector Machine | 0.65 | 0.66 | 0.67 | 0.008 |  $C=10$, kernel=linear
Gradient Boosting | 0.63 | 0.65 | 0.67 | 0.014 |  $n\\_estimators=32$, lr=0.8
Decision Tree | 0.52 | 0.57 | 0.62 | 0.034 | criterion=entropy, max_depth=None, min_samples=10

Table 5: Hyperparameter search results for different machine learning
algorithms and feature types using F1-score as the evaluation metric.

### Experiments on VIC dataset

Based on the Hyperparameter search results detailed in Table 5, we again
trained and tested twelve (12) classical ML classifiers with their best-
performing hyperparameters, using a K-Fold cross-validation strategy, with
k=5, thus training on 80% of the data and testing on the remainder 20%, on
each fold. Multinomial Naive Bayes achieved the best performance, 72% F-Score
when trained using Tfidf vectors as features and 71% with BoW feature vectors,
as expected, as per the result of our hyperparameter tuning. Additionally, we
ran tests using neural network classifiers, a CNN and BiLSTM as described in
the previous section, equally trained and tested with the same K-Fold strategy
as our classical ML models. Our CNN model achieved an F1-Score of 68%, 4% less
than our best-performing MNB, which can be attributed at the fact that the
proposed NN models were not subject to any fine-tuning. Regardless of the
feature used, RF models had the worse performance amongst all the classifiers
we experimented on, with an F1-Score of 52% overall. In general, Tfidf feature
vectors gave better results for all methods when compared with BoW which is in
line with the findings of previous research, with the only exception being the
RF classifier, and the detailed results are depicted in Table 6.

Feature | Model | Acc | Precision | Recall | F1-Score
---|---|---|---|---|---
Traditional Machine Learning Models
tfidf | DT | 0.61 | 0.62 | 0.61 | 0.61
RF | 0.52 | 0.53 | 0.75 | 0.58
MNB | 0.68 | 0.64 | 0.83 | 0.72
LR | 0.70 | 0.72 | 0.67 | 0.69
GB | 0.65 | 0.68 | 0.56 | 0.61
SVC | 0.70 | 0.72 | 0.67 | 0.69
BoW | DT | 0.62 | 0.63 | 0.59 | 0.61
RF | 0.53 | 0.55 | 0.59 | 0.52
MNB | 0.66 | 0.62 | 0.83 | 0.71
LR | 0.66 | 0.66 | 0.69 | 0.67
GB | 0.65 | 0.70 | 0.52 | 0.59
SVC | 0.68 | 0.70 | 0.63 | 0.67
Neural Network Models
text tokens | CNN | 0.68 | 0.67 | 0.70 | 0.68
bilstm | 0.64 | 0.64 | 0.66 | 0.64

Table 6: Detailled results of different combinations of ML methods and
features on the entire VIC dataset

### Origin-based subsetting

To better understand the effects of sample origin on the results, we conducted
training and testing on each subset of our data based on the sample origin.
For each method+feature combination, on each subset, we calculated the
accuracy, precision, recall, and F1 scores, which are presented as summarized
in Table 7. This result tells us that:

| Vent | ISEAR | CEASE
---|---|---|---
Feature | Model | Acc. | Prec. | Rec. | F1-Score | Acc. | Prec. | Rec. | F1-Score | Acc. | Prec. | Rec. | F1-Score
Traditional Machine Learning Models
Tfidf | DT | 0.59 | 0.59 | 0.58 | 0.59 | 0.62 | 0.63 | 0.61 | 0.62 | 0.68 | 0.73 | 0.62 | 0.66
RF | 0.52 | 0.54 | 0.46 | 0.45 | 0.52 | 0.22 | 0.26 | 0.24 | 0.49 | 0.40 | 0.80 | 0.53
MNB | 0.66 | 0.62 | 0.82 | 0.71 | 0.72 | 0.69 | 0.82 | 0.75 | 0.77 | 0.72 | 0.91 | 0.80
LR | 0.68 | 0.69 | 0.65 | 0.67 | 0.73 | 0.75 | 0.71 | 0.73 | 0.75 | 0.71 | 0.84 | 0.77
GB | 0.64 | 0.67 | 0.55 | 0.60 | 0.67 | 0.70 | 0.61 | 0.65 | 0.79 | 0.91 | 0.65 | 0.75
SVC | 0.68 | 0.69 | 0.64 | 0.67 | 0.73 | 0.74 | 0.71 | 0.73 | 0.76 | 0.84 | 0.65 | 0.73
BoW | DT | 0.62 | 0.63 | 0.60 | 0.61 | 0.65 | 0.66 | 0.63 | 0.64 | 0.72 | 0.75 | 0.69 | 0.71
RF | 0.52 | 0.54 | 0.42 | 0.43 | 0.51 | 0.20 | 0.15 | 0.13 | 0.49 | 0.40 | 0.80 | 0.53
MNB | 0.64 | 0.61 | 0.82 | 0.70 | 0.70 | 0.66 | 0.83 | 0.74 | 0.60 | 0.56 | 0.95 | 0.71
LR | 0.65 | 0.64 | 0.71 | 0.67 | 0.73 | 0.73 | 0.72 | 0.73 | 0.75 | 0.74 | 0.75 | 0.74
GB | 0.66 | 0.72 | 0.52 | 0.61 | 0.69 | 0.73 | 0.61 | 0.66 | 0.76 | 0.90 | 0.60 | 0.71
SVC | 0.66 | 0.68 | 0.60 | 0.64 | 0.73 | 0.74 | 0.71 | 0.72 | 0.73 | 0.82 | 0.59 | 0.68
Neural Network Models
text tokens | CNN | 0.63 | 0.68 | 0.61 | 0.66 | 0.71 | 0.71 | 0.73 | 0.71 | 0.74 | 0.74 | 0.77 | 0.74
BiLSTM | 0.64 | 0.64 | 0.65 | 0.65 | 0.65 | 0.65 | 0.67 | 0.67 | 0.66 | 0.69 | 0.68 | 0.67

Table 7: Performance of different combinations of ML methods and features on
every origin subset.

  * •

On the Vent subset, MNB with BoW feature vector outperformed all other models,
with an F1-Score of 71%, followed by MNB with tf-idf and CNN models, with 70%
and 66% F1-Score, respectively. Here RF was the worst-performing classifier,
regardless of the features used.

  * •

On ISEAR subset, both MNB+Tfidf and MBN+BoW achieved highest F1-Scores of 75%
and 74% respectively, but MNB+BoW however achieving a better recall. Our CNN
model achieved an F1-Score of 71%, and the RF classifier exhibited the worse
results.

  * •

For the CEASE subset, the MNB classifier trained with Tfidf features achieved
an F1-Score of 80%, followed by GB with tf-idf and the CNN model with 75% and
74% F1-Score respectively.

From Table 7 we can conclude the following:

  1. 1.

Overall, classifiers with the best hyperparameters tuned on VIC achieved
better when trained and tested on the smaller subset (CEASE), and on average,
they performed a little better on each subset than in the overall dataset,
telling us that the models are fairly good at generalizing and that they are
more suitable for short texts classifications. Note that the average sample
length in Vent is higher than in ISEAR which is higher than in CEASE.

  2. 2.

Every model performed better with Tfidf, compared to BoW feature vectors
across all subsets, except for RF ( with virtually the same result regardless
of the feature type) and LR and GB where in general, training with BoW feature
vectors.

  3. 3.

At first glance, similarly to the classical ML classifiers, for CNN models the
shorter the sample texts length the better the performance, while BiLSTM seems
unaffected by the sample text length.

### Leave-one-out Training and Testing

When leaving one subset for testing and using the other two for training, as
depicted in Table 8, the results show that models trained on VENT+ISEAR
perform better, in almost all instances of train and testing, except the SVM
model with Bag-of-Words, which showed a poor F1-score (45%). These results can
naively be explained by the fact that VENT+ISEAR models benefited from having
a large number of training samples(~4.5K) and greater average sample length
than those in the testing set (CEASE subset).

|  | Test Set
---|---|---
|  | Vent | ISEAR | CEASE
Feature | Model | Acc | Precision | Recall | F1-Score | Acc | Precision | Recall | F1-Score | Acc | Precision | Recall | F1-Score
tfidf | DT | 0.56 | 0.57 | 0.45 | 0.50 | 0.57 | 0.60 | 0.42 | 0.50 | 0.65 | 0.63 | 0.71 | 0.67
RF | 0.50 | 0.50 | 1.00 | 0.67 | 0.50 | 0.50 | 1.00 | 0.67 | 0.48 | 0.36 | 0.06 | 0.11
MNB | 0.57 | 0.55 | 0.82 | 0.66 | 0.57 | 0.54 | 0.89 | 0.67 | 0.61 | 0.57 | 0.90 | 0.70
LR | 0.58 | 0.56 | 0.66 | 0.61 | 0.61 | 0.60 | 0.67 | 0.63 | 0.71 | 0.65 | 0.87 | 0.75
GB | 0.57 | 0.58 | 0.49 | 0.53 | 0.57 | 0.59 | 0.48 | 0.53 | 0.63 | 0.63 | 0.60 | 0.62
SVC | 0.57 | 0.56 | 0.63 | 0.59 | 0.61 | 0.60 | 0.65 | 0.62 | 0.68 | 0.64 | 0.86 | 0.73
BoW | DT | 0.56 | 0.56 | 0.50 | 0.53 | 0.57 | 0.58 | 0.48 | 0.53 | 0.6 | 0.59 | 0.62 | 0.60
RF | 0.50 | 0.50 | 1.00 | 0.67 | 0.50 | 0.50 | 1.00 | 0.67 | 0.51 | 0.57 | 0.06 | 0.11
MNB | 0.53 | 0.52 | 0.89 | 0.65 | 0.54 | 0.52 | 0.93 | 0.67 | 0.52 | 0.51 | 0.89 | 0.65
LR | 0.55 | 0.56 | 0.52 | 0.54 | 0.58 | 0.60 | 0.51 | 0.55 | 0.67 | 0.63 | 0.79 | 0.70
GB | 0.57 | 0.64 | 0.34 | 0.44 | 0.59 | 0.67 | 0.34 | 0.45 | 0.7 | 0.75 | 0.60 | 0.67
SVC | 0.58 | 0.61 | 0.43 | 0.51 | 0.57 | 0.61 | 0.40 | 0.49 | 0.67 | 0.67 | 0.70 | 0.68

Table 8: Summarization of the best performing classical ML models using
different combinations of two subsets for training and one for testing.

## Error Analysis

Error analysis is a crucial step in any Natural Language Processing (NLP)
study, as it provides insight into the strengths and weaknesses of the model
and helps identify areas for improvement. In this section, we present an error
analysis of our guilt detection models, in which we analyze the most common
types of errors made by our models and attempt to understand the causes of
these errors. By examining these errors, we hope to gain a deeper
understanding of the nature of the task of detecting guilt in text and to
identify potential avenues for improving the performance of our models in
future work. A sample of misclassified examples is presented in Table 9, and
each instance of misclassification is explored in detail in this section.

Text | Label | Predicted
---|---|---
I really can’t anymore |  no guilt |  guilt
me: hey this person hasn’t talked to me in a bit my brain: they’re dead me: well no they could be driving or with friends or… my brain: they hate you and they’re dead me: no -my brain: THEYRE DEAD THEYRE DEAD THEYRE DEAD |  no guilt |  guilt
My mother had for some time been trying to separate me from a ágood friend who, she thought, was not good company for me. áFinally, at breakfast one day, we had an argument and I tried to ádefend my friend. |  no guilt |  guilt
A twitter Stan asked me if I have some pills to gargleTried clicking on the notifications and it said they got deleted;) |  no guilt |  guilt
Disordered eating impliedThis is harder than i anticipated considering how hungry we were ashfkdjsj camera-emoji |  guilt |  no guilt
Someone told a lie that I had stolen his money. |  no guilt |  guilt
Heard that my girl-friend was chosen for the English lectures and áI was not. I lost my temper and she is very upset now. |  guilt |  no guilt
NSFWSo I should have prepped my ass a lot more than that lmao |  guilt |  no guilt
When two drug addicts tried to take away my money. |  no guilt |  guilt
When one of my lovers told me that I was a flirt. |  guilt |  no guilt

Table 9:  A sample of misclassified examples from dataset VIC, using MNB+Tfidf

  1. 1.

The text "I really can’t anymore" seems to express a feeling of exhaustion or
despair, which could be interpreted as a negative emotional state and lead to
a prediction of guilt. However, without more context, it is difficult to say
for certain whether guilt is an appropriate label. It is possible that the
model may have been influenced by the word "can’t" which could be interpreted
as a form of guilt for not being able to handle a situation as could also be
possible that the word "anymore" triggered the model to predict guilt, as it
suggests some kind of frustration or negative emotion. It is also possible
that the model may not have encountered enough examples of this type of
language and therefore is not able to accurately predict the label.

  2. 2.

"me: hey this person hasn’t talked to me in a bit my brain: they’re dead me:
well no they could be driving or with friends or… my brain: they hate you and
they’re dead me: no -my brain: THEYRE DEAD THEYRE DEAD THEYRE DEAD". The
predicted label is guilt while the actual label is no guilt

    1. (a)

The model might have misinterpreted the use of all caps as an expression of
strong negative emotions, leading to the prediction of guilt. However, in this
context, the person is simply describing an internal dialogue, and there is no
actual event that would cause guilt.

    2. (b)

The sample seems to be a humorous or exaggerated representation of intrusive
or anxious thoughts, so the predicted label of guilt may indicate that the
speaker feels guilty for having these thoughts or for being anxious about
their relationship with the person who has not contacted them.

  3. 3.

"My mother had for some time been trying to separate me from a good friend
who, she thought, was not good company for me. Finally, at breakfast one day,
we had an argument, and I tried to defend my friend." - The predicted label is
guilt, while the actual label is no guilt.

    1. (a)

The text describes a disagreement between the speaker and their mother over
whether their friend is a good influence, but there is no indication of the
speaker feeling guilty.

    2. (b)

It is possible that the model picked up on the word "argument" and predicted
guilt based on the assumption that arguments are usually associated with
negative emotions. However, in this context, the person is simply defending
their friend, and there is no indication of guilt.

    3. (c)

This sample describes a conflict between the speaker and their mother over a
friend. The predicted label of guilt may indicate that the speaker feels
guilty for not being able to resolve the conflict or for causing their mother
stress.

  4. 4.

"A twitter Stan asked me if I have some pills to gargle…Tried clicking on the
notifications and it said they got deleted;)": The true label is "no guilt"
and the predicted label is "guilt". This sample is difficult to interpret
without more context, but it may indicate that the speaker is being asked for
a favor or being made an uncomfortable request. The predicted label of guilt
may indicate that the speaker feels guilty for not being able to fulfill the
request, but it is unclear why the model predicted guilt in this instance, as
there is no indication of negative emotions or events.

  5. 5.

"Disordered eating implied This is harder than i anticipated considering how
hungry we were ashfkdjsj camera-emoji": The true label is "no guilt" and the
predicted label is "no guilt".

    1. (a)

The text expresses frustration about hunger, but there is no indication of
guilt.

    2. (b)

This sample seems to indicate that the speaker is struggling with disordered
eating and finding it difficult to manage their hunger. The predicted label of
no guilt may indicate that the speaker does not feel guilty about their
struggle with disordered eating or their difficulty managing their hunger.

    3. (c)

It is possible that the model did not pick up on the implication of disordered
eating in this text, leading to misclassification.

  6. 6.

"Someone told a lie that I had stolen his money." - The predicted label is
guilt while the actual label is no guilt. It is possible that the model picked
up on the word "stolen" and predicted guilt based on the assumption that
stealing is associated with negative emotions. However, in this context, the
person is simply reporting an accusation.

  7. 7.

"Heard that my girl-friend was chosen for the English lectures and I was not.
I lost my temper and she is very upset now.": The predicted label is "no
guilt", but the true label is "guilt".

    1. (a)

It’s possible that the model did not pick up on the guilt in the text, which
is implied through the author’s loss of temper and the upset caused to the
girlfriend.

    2. (b)

The model may have also been confused by the use of the word "good" in the
text, which could have made it think that the text was more related to "no
guilt" than "guilt."

    3. (c)

The model may have failed to pick up on the implication that the person’s loss
of temper was a negative event, leading to misclassification.

  8. 8.

"NSFWSo I should have prepped my ass a lot more than that lmao" - The
predicted label is no guilt while the actual label is guilt.

    1. (a)

Here, the model could not pick up on the implication of a negative or
embarrassing event, leading to misclassification.

    2. (b)

A second possibility is that the model did not pick up on the guilt in the
text, which is implied through the author’s use of a phrase that suggests a
lack of preparation.

    3. (c)

The model may have also been confused by the use of the NSFW tag (usually used
to implicate something of mature or sexually explicit nature), which could
have made it think that the text was more related to "no guilt" than "guilt".

  9. 9.

"When two drug addicts tried to take away my money." - The predicted label is
guilt while the actual label is no guilt.

    1. (a)

The model may have incorrectly predicted guilt for this text due to the
mention of "drug addicts" and "taking away my money." It’s possible that the
model assumed the author must have done something wrong to have encountered
this situation when they were simply the victim of an attempted robbery.

    2. (b)

Ambiguity: The text is brief and lacks a lot of context, so it’s possible that
the author did something that could be interpreted as guilt-inducing in this
situation. Perhaps they were carrying a large amount of money and were in an
area known for drug activity, or maybe they responded to the situation with
aggression or violence. And the lack of further context may have led the model
to fail.

  10. 10.

"When one of my lovers told me that I was a flirt" - The predicted label is ni
guilt while the actual label is guilt. Possible reasons for the
misclassification could be:

    1. (a)

Ambiguity in the text: The phrase "one of my lovers" could indicate that the
person has multiple partners, which might be interpreted as flirting. The
classifier might have assigned a guilt label based on this ambiguity, while in
reality, the person might not have been flirting at all.

    2. (b)

Lack of context: The classifier might not have had enough information about
the situation to judge accurately. The text doesn’t provide details about the
nature of the relationship or the context of the accusation, which could have
influenced the label.

    3. (c)

Personal bias: The classifier might be influenced by personal bias or
assumptions about what constitutes "flirting". Due to the training data, the
model might have learned a biased interpretation of the term "flirt" that
doesn’t align with the writer’s intentions.

In general, misclassification can occur for various reasons, including
ambiguity in the text, errors in the training data or model, and differences
in how people interpret or understand language. It’s important to continually
evaluate and improve models to minimize misclassifications and improve
accuracy.

## Conclusion

In this study, we developed a new binary classification dataset for guilt
detection by extracting guilt and non-guilt samples from three existing multi-
class emotion detection datasets. By doing so, we provided a unique resource
for researchers and practitioners to study and develop machine learning models
for guilt detection, a critical application in the fields of studies at the
intersection of psychology and computational linguistics. We trained classical
machine learning models and two neural network models, CNN and BiLSTM, to
establish baselines using both Tfidf and BoW feature vectors.

Our results showed that, in general, every model performed better with Tfidf
compared to BoW feature vectors, except for RF. LR and GB achieved better
results when trained with BoW feature vectors. Moreover, CNN models performed
better with shorter sample text lengths, while BiLSTM models were not affected
by the text length.

Regarding subset-specific performance, on the Vent subset, the MNB model with
BoW features achieved the highest F1-Score of 71%, outperforming all other
models. On the ISEAR subset, both MNB with Tfidf and MNB with BoW achieved the
highest F1-Scores of 75% and 74%, respectively, with the latter achieving a
better recall. On the CEASE subset, the MNB classifier trained with Tfidf
features achieved the highest F1-Score of 80%, followed by GB and the CNN
model, achieving F1-Scores of 75

Overall, our study provides a valuable benchmark for future research in guilt
detection. Our error analysis also revealed some of the weaknesses of our
models, which could be addressed in future studies to improve performance. Our
work highlights the importance of carefully selecting the training data and
the choice of machine learning models when developing systems for guilt
detection. Furthermore, our leave-one-out analysis demonstrated that our
models can generalize to new data with reasonable accuracy.

## References

  * [1] Rimé, B., Mesquita, B., Boca, S. & Philippot, P.  Beyond the emotional event: Six studies on the social sharing of emotion.  _Cognition and Emotion_ 5, 435–465, DOI: 10.1080/02699939108411052 (1991).  https://doi.org/10.1080/02699939108411052.
  * [2] Berger, J. & Milkman, K.  What makes online content viral.  _Journal of Marketing Research_ 49, 192–205, DOI: 10.2139/ssrn.1528077 (2009).
  * [3] Rawlings, E. I.  Reactive guilt and anticipatory guilt in altruistic behavior.  In Macaulay, J. & Berkowitz, L. (eds.) _Altruism and Helping Behavior_ , 163–177 (Academic Press, New York, 1970).
  * [4] Kugler, K. E. & Jones, W. H.  On conceptualizing and assessing guilt.  _Journal of Personality and Social Psychology_ 62, 318–327 (1992).
  * [5] Kealy, D., Treeby, M. S. & Rice, S. M.  Shame, guilt, and suicidal thoughts: The interaction matters.  _British Journal of Clinical Psychology_ 60, 414–423, DOI: https://doi.org/10.1111/bjc.12291 (2021).  https://bpspsychub.onlinelibrary.wiley.com/doi/pdf/10.1111/bjc.12291.
  * [6] Tangney, J. & Dearing, R.  _Shame and guilt_ (Guilford Press, New York, 2002).
  * [7] Bryan, C. J. _et al._ Guilt as a mediator of the relationship between depression and posttraumatic stress with suicide ideation in two samples of military personnel and veterans.  _International Journal of Cognitive Therapy_ 8, 143–155, DOI: 10.1521/ijct.2015.8.2.143 (2015).  https://doi.org/10.1521/ijct.2015.8.2.143.
  * [8] Cunningham, K. C. _et al._ A model comparison approach to trauma-related guilt as a mediator of the relationship between ptsd symptoms and suicidal ideation among veterans.  _Journal of Affective Disorders_ 221, 227–231, DOI: https://doi.org/10.1016/j.jad.2017.06.046 (2017).
  * [9] Balahur, A., Hermida, J. M. & Montoyo, A.  Detecting implicit expressions of sentiment in text based on commonsense knowledge.  In _Proceedings of the 2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis (WASSA 2.011)_ , 53–60 (Association for Computational Linguistics, Portland, Oregon, 2011).
  * [10] Ghosh, S., Ekbal, A. & Bhattacharyya, P.  CEASE, a Corpus of Emotion Annotated Suicide notes in English.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , 1618–1626 (European Language Resources Association, Marseille, France, 2020).
  * [11] Balahur, A., Hermida, J. M. & Montoyo, A.  Building and exploiting emotinet, a knowledge base for emotion detection based on the appraisal theory model.  _IEEE Transactions on Affective Computing_ 3, 88–101, DOI: 10.1109/T-AFFC.2011.33 (2012).
  * [12] Lykousas, N., Patsakis, C., Kaltenbrunner, A. & Gómez, V.  Sharing Emotions at Scale: The Vent Dataset.  _Proceedings of the International AAAI Conference on Web and Social Media_ 13, 611–619, DOI: 10.1609/icwsm.v13i01.3361 (2019).
  * [13] Öhman, E., Pàmies, M., Kajava, K. & Tiedemann, J.  XED: A multilingual dataset for sentiment analysis and emotion detection.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , 6542–6552, DOI: 10.18653/v1/2020.coling-main.575 (International Committee on Computational Linguistics, Barcelona, Spain (Online), 2020).
  * [14] Demszky, D. _et al._ GoEmotions: A dataset of fine-grained emotions.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 4040–4054, DOI: 10.18653/v1/2020.acl-main.372 (Association for Computational Linguistics, Online, 2020).
  * [15] Hsu, C.-C., Chen, S.-Y., Kuo, C.-C., Huang, T.-H. & Ku, L.-W.  EmotionLines: An emotion corpus of multi-party conversations.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ (European Language Resources Association (ELRA), Miyazaki, Japan, 2018).
  * [16] Scherer, K. R. & Wallbott, H. G.  Evidence for universality and cultural variation of differential emotion response patterning: Correction.  _Journal of Personality and Social Psychology_ 67, 55–55, DOI: 10.1037/0022-3514.67.1.55 (1994).
  * [17] Troiano, E., Padó, S. & Klinger, R.  Crowdsourcing and validating event-focused emotion corpora for german and english.  _arXiv preprint arXiv:1905.13618_ (2019).
  * [18] Ekman, P.  Universals and cultural differences in facial expressions of emotion.  In _Nebraska symposium on motivation_ (University of Nebraska Press, 1971).
  * [19] Ekman, P.  An argument for basic emotions.  _Cognition and Emotion_ 6, 169–200, DOI: 10.1080/02699939208411068 (1992).  https://doi.org/10.1080/02699939208411068.
  * [20] Plutchik, R.  A general psychoevolutionary theory of emotion.  _Theories of emotion_ 1, 3–31 (1980).
  * [21] Alvarez-Gonzalez, N., Kaltenbrunner, A. & Gómez, V.  Uncovering the limits of text-based emotion detection.  _CoRR_ abs/2109.01900 (2021).  2109.01900.
  * [22] Kim, Y.  Convolutional neural networks for sentence classification.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 1746–1751, DOI: 10.3115/v1/D14-1181 (Association for Computational Linguistics, Doha, Qatar, 2014).
  * [23] Hochreiter, S. & Schmidhuber, J.  Long shortterm memory.  _Neural computation_ 9, 1735–1780.
  * [24] Cho, K., van Merriënboer, B., Bahdanau, D. & Bengio, Y.  On the properties of neural machine translation: Encoder–decoder approaches.  In _Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation_ , 103–111, DOI: 10.3115/v1/W14-4012 (Association for Computational Linguistics, Doha, Qatar, 2014).
  * [25] Balouchzahi, F., Sidorov, G. & Gelbukh, A.  Polyhope: Two-level hope speech detection from tweets, DOI: 10.48550/ARXIV.2210.14136 (2022).
  * [26] Balouchzahi, F., Butt, S., Sidorov, G. & Gelbukh, A.  Reddit: Regret detection and domain identification from text, DOI: 10.48550/ARXIV.2212.07549 (2022).
  * [27] Schuster, M. & Paliwal, K.  Bidirectional recurrent neural networks.  _IEEE Transactions on Signal Processing_ 45, 2673–2681, DOI: 10.1109/78.650093 (1997).

## Acknowledgements

The work was done with partial support from the Mexican Government through the
grant A1-S-47854 of CONACYT, Mexico, grants 20220852 and 20220859 of the
Secretaría de Investigación y Posgrado of the Instituto Politécnico Nacional,
Mexico. The authors thank the CONACYT for the computing resources brought to
them through the Plataforma de Aprendizaje Profundo para Tecnologías del
Lenguaje of the Laboratorio de Supercómputo of the INAOE, Mexico and
acknowledge the support of Microsoft through the Microsoft Latin America Ph.D.
Award.

## Author contributions statement

A.M. conceived and conducted the experiment(s), analyzed the results, and
conducted error analysis; N.H. conducted the literature review and analyzed
the results; and A.G. and G.S. supervised the entire research. All authors
reviewed the manuscript.

## Data availability

The ISEAR dataset is publicly available at
https://www.unige.ch/cisa/index.php/download_file/view/395/296/ and the source
paper introducing this dataset is cited correctly in this manuscript. The
CEASE dataset is available upon reasonable request to the authors of the paper
that first presented it, and a form for requesting access is available at
https://www.iitp.ac.in/ ai-nlp-ml/resources.html. Portions of the Vent Dataset
are publicly available, while the texts are only available upon reasonable
request to the paper’s corresponding author who first introduced it. The
resulting dataset used in this study, which only contains readers and labels,
is also available upon reasonable request due to compliance constraints on
some of the samples because of their origin.

##  1 Ethics statement

The research presented in this manuscript involves using three different
datasets, namely the ISEAR, CEASE, and Vent datasets. All ethical
considerations for each dataset were addressed in each paper that initially
introduced them. The dataset comprises texts of three types, social media
posts, suicide notes, and psychological study questionnaire answers. The
resulting dataset used in this study only contains texts and labels, with no
identifying or sensitive information. The authors obtained access to the
datasets through reasonable requests to each of the dataset’s authors/creators
and fulfilled and followed all the compliance requests set forth by the
authors. The authors also ensured that identifying and sensitive information
was removed from the resulting dataset to protect the individual’s privacy.
The resulting dataset is available upon reasonable request due to compliance
constraints on some of the samples because of their origin.

##  2 Competing interests

The authors declare no competing interests.