# Data Augmentation for Modeling Human Personality: The Dexter Machine

Yair Neuman, Vladyslav Kozhukhov and Dan Vilenchik

###### Abstract

Modeling human personality is important for several AI challenges, from the
engineering of artificial psychotherapists to the design of persona bots.
However, the field of computational personality analysis heavily relies on
labeled data, which may be expensive, difficult or impossible to get. This
problem is amplified when dealing with rare personality types or disorders
(e.g., the anti-social psychopathic personality disorder). In this context, we
developed a text-based data augmentation approach for human personality
(PEDANT). PEDANT doesn’t rely on the common type of labeled data but on the
generative pre-trained model (GPT) combined with domain expertise. Testing the
methodology on three different datasets, provides results that support the
quality of the generated data.

##  1 Introduction

Personality concerns the individual’s relatively stable pattern of thoughts,
emotions and behaviors [1]. There are various personality theories from the
Big Five [2] to Affective Neuroscience [3] and Mischel’s contextual approach
to personality [4]. In this paper, we adhere to the clinical approach
represented by the Psychodynamic Diagnostic Manual (PDM) [5] and SWAP [6],
which is highly relevant for diagnosis, and research [7]. According to the PDM
approach, personality types are stable configurations characterized by key
features such as the individual’s core beliefs about self and others. For
instance, a depressive personality is characterized by self-criticism and
accompanied by the belief that ”something is essentially bad about me”.

Current computational personality research is almost exclusively focused on
features’-based data-driven classification involving the prediction of a
personality class label. Accomplishing such tasks relies on the availability
of a large amount of high-quality labeled data (e.g., [8, 9, 10]). However,
obtaining such data may be expensive, difficult, or impossible for various
reasons. For instance, the prevalence of the anti-social psychopathic
personality disorder in the population is low [11, 12, 13], and it is
currently impossible to gain access to a massive dataset of labeled texts
produced by clinically diagnosed psychopaths. High-quality diagnostic
procedures, such as SWAP, are costly as they require human expertise and
significant time to complete. While self-reported questionnaires for
personality assessment are available, they rely on the collaboration of the
diagnosed individual and their ability to provide a valid self-assessment,
which in the case of the anti-social personality disorder, for instance, is
not trivial to gain.

In the face of these challenges, a natural solution for data scarcity is data
augmentation, intensively developed in computer vision but ”relatively under-
explored” in NLP, where the generation of effective augmented examples is
”less obvious” [14, 15]. To illustrate the challenges in textual data
augmentation, we ran the SOTA data-augmentation pipeline LAMBADA [16] to
generate 200 sentences out of a seed set of 20 sentences (see appendix)
expressing a clear psychopathic signature. This attempt to produce artificial
”psychopathic” sentences resulted in only 100 unique sentences, where the vast
majority of sentences were either one of the seed sentences or a simple
paraphrasing thereof.

Data augmentation is typically viewed as the process of increasing the amount
of data by adding slightly modified copies of _already existing_ labeled data.
In some cases, there is no labeled data at all or a very small quantity which
precludes proper augmentation (as our experiment with LAMBADA suggests). In
this paper, we offer a solution to these cases by using unlabelled data and
adding domain expert input to compensate for the absence of labeled data (once
can view labeled data as domain expert knowledge).

A constructive approach to personality modeling may be found in the
revolutionary large language models recently introduced to NLP (e.g., GPT-2)
[17, 18]. Recently, [19] and [20] showed that the GPT model, once fine-tuned,
can be useful in the domain of personal conversations. Their approach led to
substantial improvements in the PersonaChat data set, showcasing the potential
of exploiting large pre-trained generative models in the conversational
domain. However, these advancements do not naively imply anything for modeling
personality types, as the poor results obtained from LAMBADA, which is based
on GPT technology, show. Indeed, personalized chit-chat models, [21] use the
notion of personalization (e.g., age) which is different from the
psychodynamic approach used in this paper.

###  1.1 Our contribution.

We present a novel personality data augmentation approach, PEDANT (PErsonality
Data AugmeNTation), using (1) a generative pre-trained model (GPT) combined
with (2) domain expertise (the domain expert is the first author who has
intensively studied and published about personality) while relying only on (3)
unlabeled text.

PEDANT operates in two phases. In the first phase, unlabeled data relevant to
the selected personality type is harvested from online resources; this data is
then used to train a generative language model. In the second phase, the
language model is repeatedly prompted to complete a set of seed sentences
carefully crafted by the domain expert. All these completions are then
filtered and ranked according to a scoring function that the domain expert
pre-defined; the top $k$ sentences are the output of PEDANT.

We implement PEDANT with regard to a specific personality type: the anti-
social psychopathic personality [6]; we call this particular pipeline Dexter.
This type of personality is suitable for validating our approach as the
prevalence of psychopathic personality disorder is extremely low, and a
labeled corpus of naturally produced texts of diagnosed psychopaths does not
exist. The texts that we harvest for the first phase of PEDANT come from a few
fictive characters from the cinema and TV (e.g., Dexter the psychopath from
the TV series ”Dexter”) and from Reddit forums such as r/psychopath. The
second phase, where domain expertise is used, is described in detail in
Sections 3.2 and 3.4.

We validated Dexter using a downstream text classification task, as common in
other works that deal with the evaluation of data augmentation pipelines [22,
16]. We used the data generated by Dexter to train a classifier and then
tested it on three offensive-speech datasets that cover different dimensions
of the psychopathic personality (e.g., lack of empathy, toxicity, and being
manipulative).

For comparison, we tested two SOTA data augmentation pipelines, LAMBADA [16],
and LeakGAN [23]. The Dexter dataset produced a classifier that ranked first
(by a large gap) in all three datasets. The complete detail of both
experiments appears in Section 5.2.

###  1.2 Illustration

It is a non-trivial task to evaluate the extent to which the resulting
generative model (the outcome of Dexter) reflects the psychopathic mind. One
way mentioned above is via a downstream task that uses data generated by the
model. Another way is to have a host of personality domain experts chat with
the model. While we did not have the resources to perform this expensive and
laborious task, we invite the reader to peek into such a possible Q&A session
and to judge for herself the change in personality.

Below is a comparison of the output of a GPT model before and after fine-
tuning on the harvested psychopathic-related texts. In the examples below, GPT
is prompt with psychopathic statements chosen by the domain expert, and its
completion is recorded; a couple more examples are given in the appendix.

Statement: I act impulsively, without regard for the consequences of my
actions.
GPT-2 before fine-tuning: I know when someone is suffering and I feel bad.
GPT-3 davinci: I have difficulty organizing tasks and activities. I have
difficulty with time management. .
GPT-3 curie: My friends are more important to me than my family. I dress in a
way that is not in keeping with the standards of my community.
chatGPT: It’s natural to act impulsively at times, but if you find that you
frequently act without considering the consequences of your actions, it may be
helpful to try to develop more self-control. GPT-2 after fine-tuning: I have
an abundance of rage, and I can turn to it with no consideration for
consequences.

Statement: I feel like a hunter seeking a prey.
GPT-2 before fine-tuning: I don’t want to be a bad person.
GPT-3 davinci: I want to let my heart feel.
GPT-3 curie: I’m lost in this forest ! I’ll be crushed by this forest!
chatGPT: It’s important to remember that other people are not prey to be
hunted. It’s never okay to view or treat others as if they are objects or
targets to be pursued or exploited.
GPT-2 after fine-tuning: I don’t care about it, I just want to kill and eat.

##  2 Related work

Data augmentation methods have been explored to address the imbalance of
datasets challenge in supervised classification tasks. Noise injection or
attribute modification techniques were commonly applied to generate synthetic
data for image and sound classification tasks [24, 25, 26]. However, such
techniques do not extend to text due to the categorical nature of words and
the sequential nature of text.

We locate our work in the context of text data augmentation, [14, 27, 16].
“Previous-generation” textual data augmentation approaches focus on sample
alteration [28, 29, 30, 31, 32], in which a single sentence is altered to
generate a new sentence primarily by synonyms replacements. Presumably,
methods that make only local changes will produce sentences with a structure
similar to the original ones, thus yielding low corpus-level variability.

Other recent approaches to textual data augmentation that generate whole
sentences rather than making a few local changes include using variational
autoencoding [33], paraphrasing [34] and methods based on generative
adversarial networks [35, 23, 36].

Recent progress in NLP has been marked by the emergence of large language
models (i.e., transformers) such as GPT-2 [18]. GPT-based language models
scored high in open-domain dialogue generation tasks [20, 19, 37]. The data-
augmentation pipeline presented in [16] uses GPT technology to generate themed
synthetic text. The idea behind [16] involves fine-tuning a GPT model to a
specific task using existing labeled data. Using the fine-tuned model and
given a class label, new sentences for the class are generated. The sentences
are filtered with a classifier trained on the original data.

While our pipeline is similar to [16] in flavor (a fine-tuning step followed
by a filtering step), it is different in two key aspects. We use unlabelled
data for the fine-tuning step. This allows us to fine-tune the GPT model with
a large amount of, possibly slightly lower quality, data. [16] use labeled
data for the fine-tuning step; thus the quality of the augmentation depends on
the amount of available text. Second, our filtering is also done in an
unsupervised manner, replacing the need for labeled data for training a
classifier with the knowledge of a domain expert. These two key differences
make our pipeline useful for data generation for rare classes, such as rare
personality types, where labeled data is scarce or non-existent. Indeed,
comparing the performance of [16] to Dexter corroborates the latter.

Figure 1: Pipeline for PEDANT. GPT-2 is trained and prompted to complete a
carefully chosen seed of sentences. The completions are filtered and ranked
using similarity to relevant words chosen using domain expertise.

##  3 Methodology

The pipeline for generating data for a given personality type, illustrated in
Figure 1, is composed of the following stages (the actual parameter values
that we’ve used are given in this general description):

  1. 1.

Texts produced by a few fictive characters (e.g., Dexter) and secondary
sources (e.g., Reddit forums discussing the personality style) are collected
to form a preliminary dataset. Let $\cal{D}$ be that preliminary dataset.

  2. 2.

A pre-trained language model is fine-tuned on $\cal{D}$. Let $\cal{G}$ be the
obtained model

  3. 3.

A domain-expert hand-crafted set of $s=40$ seed sentences representing the
personality’s beliefs about self/others is prepared; $\cal{G}$ is prompted to
complete each seed $c=200$ times, for a total of $n=s\cdot c=8000$ candidate
sentences.

  4. 4.

Based on domain expertise, a hand-crafted vector $\cal{F}$, containing $f$
words that are typical of the personality type, is assembled. The $n$
candidate sentences are being filtered and ranked according to their cosine
similarity with $\cal{F}$.

  5. 5.

The top $k=2000$ sentences compose the output.

We now describe how we customized this general pipeline to the psychopathic
personality.

###  3.1 The preliminary datasets

Following [38, 39] we used data from movie scripts – the text produced by
three well-known fictive psychopathic characters: The Joker in the movie ”The
Joker”, Bateman in the movie ”American Psycho” and Dexter from the TV series
”Dexter”. In addition, we collected all texts from Reddit discussion groups
dealing with psychopathy (r/psychopath, r/sociopath, r/antisocial).

After cleaning the data by applying a spell checker [40], removing emojis,
duplicates, hyperlinks, and spam messages [41], the preliminary cleaned
dataset consisted of 1,320,552 tokens.

###  3.2 The sentences completion seed set

Our domain expert manually prepared 20 seed sentences representing the
psychopath’s ”beliefs about self” (e.g., ”I take advantage of others whenever
I can”) and 20 seed sentences representing the psychopath’s ”beliefs about
others” (e.g., ”Human beings are weak”). The complete list of seed sentences
appears in the appendix.

The number ’20’ is somewhat arbitrary. While testing with other seed sizes, we
found that 20 was the minimal number that gave good results, considering the
computational constraints such as space and running time.

###  3.3 Training GPT-2 and generating sentences

Our starting point is the pre-trained GPT-2 with 1.5B parameters accessed via
the popular HuggingFace API [42]. We chose GPT-2 as it is currently one of the
most useful language generation model. (The newer GPT-3 is still not open
source, and it’s harder to work with, e.g., fine-tuning on a large text like
our preliminary dataset). Next, we fine-tuned the pre-trained GPT-2 model on
the entire preliminary dataset (the fictive characters text and the Reddit
data) using the task of predicting the next word of the sentence [18]. The
parameters we used were: learning rate=0.0001, model name=‘1558M’, batch
size=4, optimizer = ‘adafactor’, steps = 10000 and the cross entropy loss
function.

We prompted the fine-tuned GPT-2 model on each of the 40 seed sentences ( see
the appendix for the full list), producing 200 sentence completions for each
sentence. Using the experts’ judgment of two psychologists, we qualitatively
evaluated a random sample of these sentences. We concluded that the best
completions were obtained with the following parameters:
length=50,temperature=0.7, top $k$ = 50, top $p=0.90$.

We used the free Google Colab resources with Tesla P100 GPUs for this part.

###  3.4 Filtering and Ranking

We applied filtering and ranking to the 8000 sentence completions. First, we
removed sentences that (1) include the trivial words: psychopath, antisocial,
and sociopath; (2) are duplicates of other sentences; (3) contain less than
three words; (4) end with a stop word; (5) are emotionally neutral or have a
higher positive than a negative sentiment (we used NLTK to estimate the
sentiment); (6) are simple paraphrases of each other (via [43]).

For the ranking task, we identified words significantly collocated with the
target word ”psychopath” in the iWeb repository [44]. The domain expert
selected 28 to form a ”psychopathic vector” (see the appendix). Next, we used
the vectorial semantic approach for personality assessment [45] and measured
the cosine similarity between each filtered candidate sentence completion and
the psychopathy vector. For each of the 40 seed sentences, we selected the 50
completions that scored highest on the cosine similarity test. The output was
a set of 1735 synthetically generated sentences that are supposed to represent
a psychopathic mind (if for some seed less than 50 completions passed the
filtering step, we took all of them).

##  4 Data

Our setting inherently precludes the existence of a large labeled bench-
marking dataset of text written by clinically diagnosed psychopaths. However,
as the antisocial psychopathic personality is composed of several dimensions
(e.g., lack of empathy), we tested our approach on labeled datasets
hypothesized to share one dimension or more with this personality type.

###  4.1 Test Data

We now describe the three datasets that we’ve used to evaluate the performance
of Dexter and two other data augmentation pipelines.

Sexual predators. Sexual predators share with psychopaths at least two
psychological dimensions: being manipulative and lacking empathy, as indicated
by the correlation between sexual offending and psychopathy [46, 47, 48]. Our
first dataset was a labeled data set of texts produced by 142 sexual predators
and 97,689 non-predators [49].

Empathy. Psychopaths are characterized by a lack of empathy. Our second data
set consists of interactions between help seekers in an online call center for
mental health support [50]. Labeled texts of the mental health supporters
(responders) are provided. Responders are tagged according to three increasing
levels of empathy: “0” ($N=2037$), “1” ($N=895$), and “2” ($N=152$). Unlike
the other two datasets, the empathy dataset does not contain a natural
positive class, as an empathy score of 0 does not necessarily imply a strong
negative personality.

Cyberbullying. Cyberbullying may have a clear psychopathic signature, given
the reported association between the psychopathic mind and sadism [51, 52]. We
have used the labeled toxic-text subset of the cyberbullying dataset [53] that
contains 12,168 toxic vs. 14,874 non-toxic texts. Unlike the previous two
datasets, this one is labeled at the message level. Each message consists of
several sentences, and the entire message is assigned the label “toxic” if
there are “enough” toxic sentences (the exact labeling procedure is described
in the original paper [53]).

###  4.2 Train Data

We now describe the datasets we used to fine-tune the BERT-base-uncased model
[54], which we then ran to classify the aforementioned test datasets. The data
statistics is summarized in Table 1.

The Dexter dataset. This dataset contains 3400 sentences; 1700 sentences are
the output of Dexter, which serve as the positive class, and 1700 sentences
from various Reddit discussion groups that serve as the negative class. These
sentences were selected by first sampling 8000 random sentences from various
Reddit groups, then filtering and cleaning them according to the same
procedure that was applied to the psychopathic texts. Finally, the 1700
sentences with the lowest psychopathic score were chosen.

The Dexter-minus and the PRELIM dataset. These two datasets allow us to
evaluate the importance of the different stages in the Dexter pipeline (Figure
1). The Dexter-minus dataset follows the same pipeline as Dexter just that the
fine-tuning of the GPT is skipped. The PERLIM dataset shortcuts the GPT step
altogether and proceeds directly to the filtering and ranking step.

Dataset | #Pos. | #Neg
---|---|---
Sexual Predators | 142 | 97,689
Empathy | 2,037 | 152
Cyberbullying | 12,168 | 14,784
Table 1: Summary statistics for the three datasets described in Section 4. The
number of samples in the positive and negative class are shown.

To compare the performance of Dexter against the two SOTA data-augmentation
pipelines, we created the following two synthetic datasets.

The LAMBADA dataset. To train LAMBADA, we used the Papers-With-Code
recommended implementation of LAMBADA [55]. We augmented GPT-2 with two new
classes, ”#beliefs about others” and ”# beliefs about self”. Each class was
seeded with the 20 sentences that our domain expert crafted (Section 3.2). All
the training parameters and the code are available at [55].

We then used the LAMBADA pipeline to generate 17,000 sentences, out of which
we chose the best 1700 (this was the recommendation of the authors [16], to
generate 10x more sentences than needed). Specifically, we generated 8500 from
the “#beliefs about others” class and 8500 from “#beliefs about self” (we used
the following parameters for GPT-2: max length=50, top $k=10$, $p=0.85$). We
ranked the sentences the same way we ranked ours: using cosine similarity to
the psychopathic vector of words (Section 3.4).

The negative class of the LAMBADA dataset is the same as Dexter’s.

The LeakGAN dataset. We trained LeakGAN using the official LeakGAN
implementation [56]. LeakGAN is trained with the target class text. We used
all the text in the preliminary dataset to that end (Section 3.1) and the
default parameters from the official implementation. We generated 1700
sentences using LeakGAN to serve as the positive class, and the negative class
was the same as Dexter’s.

External competition datasets. This collection contains three gold-standard
competition offensive speech datasets: OffenseEval [57], HatEval [58], and
AbuseEval [59]. Each dataset contains roughly 10,000 labeled texts. We call
the union of all three datasets the Golden dataset.

##  5 Evaluation

To evaluate Dexter we’ve created a family of models using the aforementioned
training datasets of Section 4.2. Each model is named X@BERT, which means that
the BERT-base-uncased model [54] was fine-tuned using dataset X. If a ’+’ is
appended, X@BERT+, this means that a preceding step of fine-tuning on
OffenseEval [57] was taking place.

###  5.1 Evaluation procedure

All the test datasets mentioned in Section 4.1, Table 1, are imbalanced to
different degrees (the sexual predators dataset contained merely 0.001% sexual
predators). To allow comparison across the three datasets while avoiding
misleading artifacts that such imbalanced data introduces, we down-sampled the
majority class to obtain balanced sets.

The output of a BERT model is a number in $[0,1]$, the result of the last
layer activation unit (soft-max in our case). This number may be thought of as
the probability that BERT assigns the instance to belong to the positive class
(in our case, ”psychopath”). We define the PsychoScore of a user as the
average output of the model over all the sentences produced by that user (each
sentence is scored separately by the model). It is common practice to feed the
BERT score into a simple classifier, like SVM, to find the optimal cut-off for
the binary classification task [60].

To evaluate the model on each test data set, we computed the 5-fold cross-
validation F1 and Macro F1 scores. Each fold consisted of $n=100$ randomly
sampled instances from each class, and split into 80% train and 20% test. We
trained a soft-margin kernel SVM (we used the default Python sklearn module
parameters $C=1$, kernel = RBF) on the users’ PsychoScores and the
corresponding label.

Model | Pred. | Emp. | Cyber. | Avg Rank
---|---|---|---|---
Dexter@BERT+ | 1 | 1 | 2 | 1.33
OffenseEval@BERT | 3 | 2 | 1 | 2
Dexter@BERT | 2 | 4 | 5 | 3.66
HateEval@BERT | 6 | 3 | 6 | 5
Dexter-@BERT | 4 | 5 | 7 | 5.33
Golden@BERT | 7 | 7 | 3 | 5.66
AbuseEval@BERT | 7 | 7 | 4 | 6
PRELIM@BERT | 8 | 8 | 8 | 8
Table 2: Summary of Table 4. The Dexter@BERT variants are in bold. Model | Pred. | Emp. | Cyber. | Avg Rank
---|---|---|---|---
Dexter@BERT+ | 1 | 1 | 1 | 1
Dexter@BERT | 2 | 2 | 3 | 2.33
LAMBADA@BERT+ | 4 | 4 | 2 | 3.33
Dexter-@BERT | 3 | 3 | 5 | 3.66
LeakGAN@BERT+ | 5 | 5 | 4 | 4.66
LAMBADA@BERT | 7 | 6 | 6 | 6.33
LeakGAN@BERT | 6 | 7 | 7 | 6.66
Table 3: Summary of Table 5. The Dexter@BERT variants are in bold.

###  5.2 Results

The results of running the X@BERT models on the test datasets of Section 4.1
are summarized in Tables 4 and 5. Table 4 reprots the comparison against the
pre-trained offensive speech models, while Table 5 reports the comparison
against LeakGAN and LAMBADA. Table 2 summarizes Table 4 with the overall
average ranking across the three datasets and similarly Table 3 summarizes
Table 5. Both show that the model Dexter@BERT+ ranked first, and Dexter@BERT
came second (Table 5) and third (Table 4) . The following key conclusions are
read from the tables:

Data set | Model | Precision | Recall | F1 score | Macro F1 score
---|---|---|---|---|---
| Dexter@BERT+ | $0.92$ | ${0.87}$ | $0.89$ | ${0.91}\pm{0.029}$
| Dexter@BERT | 0.91 | 0.86 | 0.88 | $0.90\pm 0.037$
Sexual Predator Identification | OffenseEval@BERT | 0.89 | 0.87 | 0.88 | $0.90\pm 0.035$
Competition [49] | Dexter-@BERT | 0.80 | 0.93 | 0.86 | $0.88\pm 0.043$
| HateEval@BERT | 0.95 | 0.5 | 0.65 | $0.75\pm 0.011$
| Golden@BERT | 0.88 | 0.50 | 0.63 | $0.69\pm 0.095$
| AbuseEval@BERT | 0.73 | 0.58 | 0.51 | $0.53\pm 0.133$
| PRELIM@BERT | 0.51 | 1.00 | 0.68 | $0.38\pm 0.024$
| Dexter@BERT+ | 0.66 | 0.80 | 0.72 | ${0.70}\pm 0.083$
| OffenseEval@BERT | 0.61 | 0.81 | 0.69 | $0.64\pm 0.081$
Empathy [50] | HateEval@BERT | 0.59 | 0.65 | 0.61 | $0.59\pm 0.063$
| Dexter@BERT | 0.55 | 0.88 | 0.67 | $0.54\pm 0.077$
| Dexter-@BERT | 0.51 | 0.65 | 0.57 | $0.52\pm 0.075$
| Golden@BERT | 0.42 | 0.93 | 0.58 | $0.38\pm 0.061$
| AbuseEval@BERT | 0.44 | 0.85 | 0.58 | $0.37\pm 0.018$
| PRELIM@BERT | 0.16 | 0.27 | 0.22 | $0.27\pm 0.080$
| OffenseEval@BERT | 0.92 | 0.80 | 0.85 | $0.88\pm 0.044$
| Dexter@BERT+ | 0.96 | 0.72 | 0.83 | ${0.87}\pm 0.048$
| Golden@BERT | 0.84 | 0.87 | 0.85 | $0.87\pm 0.041$
Cyberbullying [53] | AbuseEval@BERT | 0.89 | 0.78 | 0.82 | $0.83\pm 0.050$
| Dexter@BERT | 0.93 | 0.60 | 0.72 | $0.77\pm 0.051$
| HateEval@BERT | 0.86 | 0.61 | 0.71 | $078\pm 0.075$
| Dexter-@BERT | 0.91 | 0.56 | 0.68 | $0.77\pm 0.080$
| PRELIM@BERT | 0.80 | 0.57 | 0.67 | $0.70\pm 0.012$
Table 4: Results of the various models on the test data sets sorted according to macro F1 score. Dataset | Model | Precision | Recall | F1 score | Macro F1 score
---|---|---|---|---|---
| Dexter@BERT+ | $0.92$ | ${0.87}$ | $0.89$ | ${0.91}\pm{0.029}$
| Dexter@BERT | 0.91 | 0.86 | 0.88 | $0.90\pm 0.037$
Sexual Predator Identification | Dexter-@BERT | 0.80 | 0.93 | 0.86 | $0.88\pm 0.043$
Competition [49] | LAMBADA@BERT+ | 0.85 | 0.73 | 0.77 | $0.80\pm 0.074$
| LeakGAN@BERT+ | 0.55 | 0.99 | 0.71 | $0.51\pm 0.035$
| LeakGAN@BERT | 0.53 | 0.85 | 0.65 | $0.51\pm 0.041$
| LAMBADA@BERT | 0.47 | 0.80 | 0.51 | $0.30\pm 0.019$
| Dexter@BERT+ | 0.66 | 0.80 | 0.72 | ${0.70}\pm 0.083$
| Dexter@BERT | 0.55 | 0.88 | 0.67 | $0.54\pm 0.077$
Empathy [50] | Dexter-@BERT | 0.51 | 0.65 | 0.57 | $0.52\pm 0.075$
| LAMBADA@BERT+ | 0.40 | 0.67 | 0.50 | $0.33\pm 0.071$
| LeakGAN@BERT+ | 0.30 | 0.76 | 0.41 | $0.31\pm 0.122$
| LAMBADA@BERT | 0.81 | 0.35 | 0.40 | $0.50\pm 0.145$
| LeakGAN@BERT | 0.28 | 0.53 | 0.36 | $0.30\pm 0.040$
| Dexter@BERT+ | 0.96 | 0.72 | 0.83 | ${0.87}\pm 0.048$
| LAMBADA@BERT+ | 0.95 | 0.70 | 0.80 | $0.83\pm 0.060$
| Dexter@BERT | 0.93 | 0.60 | 0.72 | $0.77\pm 0.051$
Cyberbullying [53] | LeakGAN@BERT+ | 0.91 | 0.56 | 0.69 | $0.71\pm 0.064$
| Dexter-@BERT | 0.91 | 0.56 | 0.68 | $0.77\pm 0.080$
| LAMBADA@BERT | 0.88 | 0.54 | 0.66 | $0.71\pm 0.078$
| LeakGAN@BERT | 0.97 | 0.47 | 0.62 | $0.68\pm 0.114$
Table 5: Results of the various models that were trained by different text
augmentation techniques, sorted according to macro F1 score.

  * •

The results for the sexual predators place Dexter@BERT+ and Dexter@BERT at the
top two both with respect to the other data augmentation pipelines (Table 5)
and with respect to the abusive speech BERT models (Table 4). In fact, both
LAMBADA@BERT+ and LeakGAN@BERT+ obtained worse results than the baseline
OffenseEval@BERT. (F1 score of 0.88 vs 0.8 and lower).

  * •

The results for the empathy dataset in Table 4 show that Dexter@BERT+ obtained
the highest F1 and macro F1 score. In Table 5 we see that the performance of
Dexter@BERT and its derivatives is far better than the other two pipelines. We
also observe a poorer overall performance than the other two datasets, in
accordance with the absence of a natural positive class.

  * •

The results for the cyberbullying dataset in Table 4 show that Dexter@BERT+
scored at the top (Macro F1 score 0.87) together with OffenseEval@BERT (0.88)
and AbuseEval@BERT (0.87). In Table 5 we again see that Dexter@BERT+ came
first, although this time, the gap from LAMBADA@BERT+ is small.

  * •

The performance of Dexter@BERT+ is similar to OffenseEval@BERT on the sexual
predators and cyberbullying data sets. This is to be expected as these
datasets have a clear offensive speech element. The more telling result is the
larger gap for the empathy dataset, 0.7 vs 0.64 in Macro F1 score. Indeed lack
of empathy has more to do with the psychopathic mind than offensive speech.

  * •

One can look at our results through the lens of transfer learning, where our
dataset was successfully used to facilitate transfer learning from the task of
offensive speech detection to the task of predicting various aspects
associated with the psychopathic personality.

##  6 Discussion

This paper presents a new unsupervised approach for personality data
augmentation (PEDANT), trading labeled data with domain expertise. We
implement it in a specific pipeline that generates sentences with a
psychopathic signature (Dexter). One could ask whether it is feasible to
assemble a labeled dataset via platforms such as Amazon Turk. The answer is
probably no, as domain expertise is required in the field of personality to
correctly label the data. Our work offers a scalable and feasible data
augmentation pipeline that circumvents such caveats by taking input from a
domain expert in the later stages of the pipeline rather than at the beginning
(the data collection step).

The clear conclusion from the evaluation experiments we ran is that our
pipeline produced synthetic data with better quality than the other two
pipelines ([16], and [23]) thus highlighting the point that not all data
augmentation tasks were born equal. The task of generating synthetic data
about flight and travel issues (the examples from the LAMBADA paper) is not
the same as generating personality-type text. The same way generating a
synthetic dog picture is not the same as generating a CT-scan picture of a
brain with a tumor in order to train med students to read such images.

We expect that our pipeline can be adapted to other domains where high-quality
labeled data is lacking and hard to obtain by crowd-sourcing: suicide, school
shooters, etc.

## References

  * [1] Philip J Corr and Gerald Ed Matthews.  The Cambridge handbook of personality psychology.  Cambridge University Press, 2020.
  * [2] Lewis R Goldberg.  An alternative” description of personality”: the big-five factor structure.  Journal of personality and social psychology, 59(6):1216, 1990.
  * [3] Kenneth L Davis and Jaak Panksepp.  The brain’s emotional foundations of human personality and the affective neuroscience personality scales.  Neuroscience & Biobehavioral Reviews, 35(9):1946–1958, 2011.
  * [4] Walter Mischel.  Toward an integrative science of the person.  Annu. Rev. Psychol., 55:1–22, 2004.
  * [5] Vittorio Lingiardi and Nancy McWilliams.  Psychodynamic diagnostic manual: PDM-2.  Guilford Publications, 2017.
  * [6] Jonathan Shedler and Drew Westen.  The shedler–westen assessment procedure (swap): making personality diagnosis clinically meaningful.  Journal of personality assessment, 89(1):41–55, 2007.
  * [7] Vittorio Lingiardi, Nancy McWilliams, Robert F Bornstein, Francesco Gazzillo, and Robert M Gordon.  The psychodynamic diagnostic manual version 2 (pdm-2): Assessing patients for improved clinical practice and research.  Psychoanalytic Psychology, 32(1):94, 2015.
  * [8] Yair Neuman.  Computational personality analysis: Introduction, practical applications and novel directions.  Springer, 2016.
  * [9] David E Losada and Fabio Crestani.  A test collection for research on depression and language use.  In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 28–39. Springer, 2016.
  * [10] Yoram Bachrach, Michal Kosinski, Thore Graepel, Pushmeet Kohli, and David Stillwell.  Personality and patterns of facebook usage.  In Proceedings of the 4th annual ACM web science conference, pages 24–32, 2012.
  * [11] Ana Sanz-García, Clara Gesteira, Jesús Sanz, and María Paz García-Vera.  Prevalence of psychopathy in the general adult population: A systematic review and meta-analysis.  Frontiers in Psychology, page 3278, 2021.
  * [12] Kimberly B Werner, Lauren R Few, and Kathleen K Bucholz.  Epidemiology, comorbidity, and behavioral genetics of antisocial personality disorder and psychopathy.  Psychiatric annals, 45(4):195–199, 2015.
  * [13] Katherine J Holzer, Michael G Vaughn, Travis M Loux, Michael A Mancini, Noelle E Fearn, and Cara L Wallace.  Prevalence and correlates of antisocial personality disorder in older adults.  Aging & Mental Health, pages 1–10, 2020.
  * [14] Steven Y Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy.  A survey of data augmentation approaches for nlp.  arXiv preprint arXiv:2105.03075, 2021.
  * [15] Tri Dao, Albert Gu, Alexander Ratner, Virginia Smith, Chris De Sa, and Christopher Ré.  A kernel theory of modern data augmentation.  In International Conference on Machine Learning, pages 1528–1537. PMLR, 2019.
  * [16] Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling.  Do not have enough data? deep learning to the rescue!  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7383–7390, 2020.
  * [17] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.
  * [18] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [19] Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue.  Transfertransfo: A transfer learning approach for neural network based conversational agents.  arXiv preprint arXiv:1901.08149, 2019.
  * [20] Sergey Golovanov, Rauf Kurbanov, Sergey Nikolenko, Kyryl Truskovskyi, Alexander Tselousov, and Thomas Wolf.  Large-scale transfer learning for natural language generation.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6053–6058, 2019.
  * [21] Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston.  Personalizing dialogue agents: I have a dog, do you have pets too?  arXiv preprint arXiv:1801.07243, 2018.
  * [22] Sebastian Ruder and Barbara Plank.  Strong baselines for neural semi-supervised learning under domain shift.  arXiv preprint arXiv:1804.09530, 2018.
  * [23] Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang.  Long text generation via adversarial training with leaked information.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * [24] Connor Shorten and Taghi M Khoshgoftaar.  A survey on image data augmentation for deep learning.  Journal of big data, 6(1):1–48, 2019.
  * [25] Toan Tran, Trung Pham, Gustavo Carneiro, Lyle Palmer, and Ian Reid.  A bayesian data augmentation approach for learning deep models.  Advances in neural information processing systems, 30, 2017.
  * [26] Justin Salamon and Juan Pablo Bello.  Deep convolutional neural networks and data augmentation for environmental sound classification.  IEEE Signal processing letters, 24(3):279–283, 2017.
  * [27] Connor Shorten, Taghi M Khoshgoftaar, and Borko Furht.  Text data augmentation for deep learning.  Journal of big Data, 8(1):1–34, 2021.
  * [28] Sosuke Kobayashi.  Contextual augmentation: Data augmentation by words with paradigmatic relations.  arXiv preprint arXiv:1805.06201, 2018.
  * [29] Jonas Mueller and Aditya Thyagarajan.  Siamese recurrent architectures for learning sentence similarity.  In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
  * [30] Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu.  Conditional bert contextual augmentation.  In International Conference on Computational Science, pages 84–95. Springer, 2019.
  * [31] Georgios Rizos, Konstantin Hemker, and Björn Schuller.  Augment to prevent: short-text data augmentation in deep learning for hate-speech classification.  In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 991–1000, 2019.
  * [32] Mai Ibrahim, Marwan Torki, and Nagwa El-Makky.  Imbalanced toxic comments classification using data augmentation and deep learning.  In 2018 17th IEEE international conference on machine learning and applications (ICMLA), pages 875–878. IEEE, 2018.
  * [33] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [34] Ashutosh Kumar, Satwik Bhattamishra, Manik Bhandari, and Partha Talukdar.  Submodular optimization-based diverse paraphrasing and its effectiveness in data augmentation.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3609–3619, 2019.
  * [35] Fabio Henrique Kiyoiti dos Santos Tanaka and Claus Aranha.  Data augmentation using gans.  arXiv preprint arXiv:1904.09135, 2019.
  * [36] Rui Cao and Roy Ka-Wei Lee.  HateGAN: Adversarial generative-based data augmentation for hate speech detection.  In Proceedings of the 28th International Conference on Computational Linguistics, pages 6327–6338, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics.
  * [37] Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan.  Dialogpt: Large-scale generative pre-training for conversational response generation.  arXiv preprint arXiv:1911.00536, 2019.
  * [38] Iulian Vlad Serban, Ryan Lowe, Peter Henderson, Laurent Charlin, and Joelle Pineau.  A survey of available corpora for building data-driven dialogue systems.  arXiv preprint arXiv:1512.05742, 2015.
  * [39] Glen Coppersmith, Mark Dredze, and Craig Harman.  Quantifying mental health signals in twitter.  In Proceedings of the workshop on computational linguistics and clinical psychology: From linguistic signal to clinical reality, pages 51–60, 2014.
  * [40] Filip Sondej.  Autocorrect.  https://github.com/filyp/autocorrect, 2022.
  * [41] Lorenz Leitner.  redditcleaner.  https://github.com/LoLei/redditcleaner, 2020.
  * [42] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.  Huggingface’s transformers: State-of-the-art natural language processing.  arXiv preprint arXiv:1910.03771, 2019.
  * [43] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  arXiv preprint arXiv:1908.10084, 2019.
  * [44] Mark Davies and Jong-Bok Kim.  The advantages and challenges of” big data”: Insights from the 14 billion word iweb corpus.  Linguistic Research, 36(1):1–34, 2019.
  * [45] Yair Neuman and Yochai Cohen.  A vectorial semantics approach to personality assessment.  Scientific reports, 4(1):1–6, 2014.
  * [46] Heather M Gretton, Michelle McBride, Robert D Hare, Roy O’Shaughnessy, and Gary Kumka.  Psychopathy and recidivism in adolescent sex offenders.  Criminal justice and behavior, 28(4):427–449, 2001.
  * [47] Sonja Krstic, Craig S Neumann, Sandeep Roy, Carrie A Robertson, Raymond A Knight, and Robert D Hare.  Using latent variable-and person-centered approaches to examine the role of psychopathic traits in sex offenders.  Personality Disorders: Theory, Research, and Treatment, 9(3):207, 2018.
  * [48] Ji Seun Sohn, Adrian Raine, and Soo Jung Lee.  The utility of the psychopathy checklist-revised (pcl-r) facet and item scores in predicting violent recidivism.  Aggressive behavior, 46(6):508–515, 2020.
  * [49] Giacomo Inches and Fabio Crestani.  Overview of the international sexual predator identification competition at pan-2012.  In CLEF (Online working notes/labs/workshop), volume 30, 2012.
  * [50] Ashish Sharma, Adam S Miner, David C Atkins, and Tim Althoff.  A computational approach to understanding empathy expressed in text-based mental health support.  arXiv preprint arXiv:2009.08441, 2020.
  * [51] Myrthe Meere and Vincent Egan.  Everyday sadism, the dark triad, personality, and disgust sensitivity.  Personality and Individual Differences, 112:157–161, 2017.
  * [52] Natalie Sest and Evita March.  Constructing the cyber-troll: Psychopathy, sadism, and empathy.  Personality and Individual Differences, 119:69–72, 2017.
  * [53] Fatma Elsafoury.  Cyberbullying datasets.  Mendeley Data, 2020.
  * [54] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [55] Edward Ma.  Nlp augmentation.  https://github.com/makcedward/nlpaug, 2019.
  * [56] Jiaxian Guo.  Leakgan.  https://github.com/CR-Gjx/LeakGAN, 2018.
  * [57] Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar.  Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval).  arXiv preprint arXiv:1903.08983, 2019.
  * [58] Valerio Basile, Cristina Bosco, Elisabetta Fersini, Nozza Debora, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, Manuela Sanguinetti, et al.  Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter.  In 13th International Workshop on Semantic Evaluation, pages 54–63. Association for Computational Linguistics, 2019.
  * [59] Tommaso Caselli, Valerio Basile, Jelena Mitrović, Inga Kartoziya, and Michael Granitzer.  I feel offended, don’t be abusive! implicit/explicit messages in offensive and abusive language.  In Proceedings of the 12th language resources and evaluation conference, pages 6193–6202, 2020.
  * [60] Jennifer S Raj, J Vijitha Ananthi, et al.  Recurrent neural networks and nonlinear prediction in support vector machines.  Journal of Soft Computing Paradigm (JSCP), 1(01):33–40, 2019.

##  Appendix A Sentence completion examples

Each example consists of a statement that servers as a prompt for GPT-2,
GPT-3, and its completion. Two completions are given, before fine-tuning
(GPT-2,GPT-3) and after the fine-tuning on the preliminary dataset (GPT-2).
The statements were chosen to entail a natural discriminatory completion
between a normal and a psychopathic personality. The statement is first, and
the completions follow.

I take advantage of others whenever I can

(GPT-2 before) I know how to get out of trouble.
(GPT-3 davinci) said he, ”but they generally take advantage of me. It is the
way of the world.” ”I am glad to hear you say so,” said Holmes, laughing; ”it
is a most comforting reflection.”.
(GPT-3 curie) and I am a very strong negotiator. Yes, I wouldn’t say that I am
perfect, but I do try to do the right thing by all of my partners.I am not a
fan of people who are selfish, narcissistic or narcissistic.

(GPT-2 after) I have a very low empathy for others and I am constantly
manipulating people to get what I want.

People are violent

(GPT-2 before) I don’t think I was ever taught to be violent.
(GPT-3 davinci) People are not moral. People are amoral. People are depraved.”
The senator was bullish on the prospects of the Republican Party. “The future
is ours,” he said, to loud applause. “
(GPT-3 curie) and stupid, but they’re not all violent and stupid, and they’re
not all crazy. I don’t know what the hell happened. But I do know that this is
not a good thing. Things have been getting better in this country

(GPT-2 after) They are the ones who kill animals for fun, rape and torture
people, and steal and rob.

People are selfish

(GPT-2 before) I don’t think we are selfish.
(GPT-3 davinci) they don’t think about what they’re doing,” said the 17-year-
old. “They take the time to stop and think about what they’re doing, maybe
this wouldn’t happen
(GPT-3 curie) It’s a part of human nature. But, most importantly, it’s a part
of the human condition.” ”This is the fundamental question of all human
history,” he continued. ”Do I believe that people are fundamentally selfish?”

(GPT-2 after) It’s just that, when we see someone in pain, we don’t really
want to help him.

##  Appendix B Psychopathic Vector

28 words that are significantly collocated with the target word “psychopath”
in the iWeb. These words constitute the “psychopathic vector” used for
ranking: psychopath, sociopath, killer, criminal, murderer, victim, empathy,
prison, monster, liar, rapist, violent, dangerous, sexual, antisocial,
sadistic, narcissistic, evil, insane, manipulative, charming, kill,
manipulate, torture, hunt, rape, terrorize, terrify.

##  Appendix C Seed Sentences

40 seed sentences for completion by GPT. 20 sentences about self, and 20 about
others.

### Beliefs about self

I take advantage of others whenever I can.
I experience no remorse for harm or injury I have caused to others.
I enjoy manipulating others’ emotions to get what I want.
I am unconcerned with the consequences of my actions.
I feel invulnerable and that nothing can hurt me.
I have no interest in the rights, property, or safety of others.
I feel little empathy to the feelings of others.
I don’t understand others’ needs and feelings.
I act impulsively, without regard for the consequences of my actions.
I am unreliable and irresponsible.
I engage in unlawful or criminal behaviour.
I tend to be angry or hostile toward others.
I tend to get into power struggles with other people.
I gain pleasure and satisfaction by being sadistic or aggressive toward
others.
I blame others for my failures or shortcomings and believe my problems are
caused by external factors.
I don’t understand my behaviour and motives.
I don’t feel shame or remorse.
I am seeking thrill and excitement.
I usually feel empty and bored.
I feel like a hunter seeking a prey.

#### Beliefs about others

People are selfish.
Human beings are greedy.
The majority of people are cruel.
The world is full of inconsiderate people
Most people are childish
Most people are arrogant
People I know are irresponsible
People are manipulative
Human beings are deceptive
The majority of people are abusive
The majority of people are dangerous
Most people are exploitative
Most people are untrustworthy
People are violent
People are vulnerable
Human beings are weak
The majority of people are helpless
People are predatory
Most people are an easy prey
The human condition is weak and vulnerable to predation