# Zero-Shot Ranking Socio-Political Texts with Transformer Language Models to
Reduce Close Reading Time

Kiymet Akdemir
Boğaziçi University
<EMAIL_ADDRESS>
Ali Hürriyetoğlu
KNAW Humanities Cluster DHLab
<EMAIL_ADDRESS>

###### Abstract

We approach the classification problem as an entailment problem and apply
zero-shot ranking to socio-political texts. Documents that are ranked at the
top can be considered positively classified documents and this reduces the
close reading time for the information extraction process. We use Transformer
Language Models to get the entailment probabilities and investigate different
types of queries. We find that DeBERTa achieves higher mean average precision
scores than RoBERTa and when declarative form of the class label is used as a
query, it outperforms dictionary definition of the class label. We show that
one can reduce the close reading time by taking some percentage of the ranked
documents that the percentage depends on how much recall they want to achieve.
However, our findings also show that percentage of the documents that should
be read increases as the topic gets broader.

##  1 Introduction

For the information retrieval process positively labeled documents in a
dataset are important and should not be missed, therefore achieving high
recall is extremely important. However, there is generally a large number of
documents that are relevant or not to the concerned topic and doing close
reading for all documents and annotating them requires lots of time and
resources Hürriyetoğlu et al. (2016); Hürriyetoǧlu et al. (2017). Therefore,
ranking documents according to relevance to the investigated class may help to
reduce close reading time and decrease the likelihood of missing critical
information.

Baeza-Yates and Ribeiro-Neto (1999) propose ranking documents in decreasing
order of being relevant to a given query to accelerate the information
retrieval process. Halterman et al. (2021) apply this method with Natural
Language Understanding (NLU) models for binary classification problems using
the entailment probabilities of a document and a declarative form of the
label. Therefore, to catch a high percentage of positively labeled documents,
reading some percentage of documents would be enough since documents that are
relevant would be at the top with a high probability. However, their dataset
India Police Events focuses on a relatively specific task in information
retrieval that is police actions like killing, arresting, failing to
intervene, etc. Besides, they apply this method at the sentence level and as
they also stated their model suffers from understanding multi-sentence context
that increases the false negative rate.

We apply this approach to ProtestNews dataset (Hürriyetoğlu et al., 2021)
along with the India Police Events dataset (Halterman et al., 2021) and
investigate whether sentence level evaluation or document level evaluation
ranks positive documents at the higher level measured by different evaluation
metrics. We further investigate whether using the dictionary definition of a
class or the declarative form of a class for the query performs this task
better. We compare two NLU models DeBERTa-Large-MNLI (He et al., 2020) and
RoBERTa-Large-MNLI (Liu et al., 2019) in terms of recall and mean average
precision.

We present the related work in Section 2. Next, we introduce two datasets we
used in our experiments in Section 3. Then we explain our methodology and list
all queries used in this work in Section 4. We detail our experiments for both
datasets and present results in Section 5. Finally, we conclude this work in
Section 6 and state what can be done as future work in Section 7.

##  2 Related Work

#### Protest Event Detection

Protest event extraction holds an important place in political social sciences
and detection of protest events is generally the first step of the extraction.
Due to the cost of manual event extraction, besides the presence of digital
news articles and enhancing machine learning methods; automated event
extraction comes into play.
Hanna (2017) presents MPEDS, an automated system for protest event extraction
that contains an ensemble of shallow machine learning classifiers (SVM, SGD
and Logistic Regression) to detect protest-related documents. Caselli et al.
(2021) proposes Domain Adaptive Retraining for Transformer Language Models and
shows that further training BERT with domain-specific dataset improves the
performance. They present PROTEST-ER by retraining pre-trained BERT with
protest related data from TREC Washington Post Corpus. Wiedemann et al. (2022)
classifies protest related documents in German local news using Pretrained
Language Models. They attempt to improve performance and generalizability by
eliminating protest-unrelated sentences with keyword search and also by
masking named entities with the idea of models may overfit on data by
recognizing actors, organizatons and places.
Elsafoury (2019) focuses on both protest events and police actions i.e.
protest repression events in Twitter with Machine Learning models with the
claim of news articles suffer from bias, censorship and duplication. Won et
al. (2017) detects and analyze protest events in geotagged tweets and
associated images with Convolutional Neural Networks.

#### Ranking Documents with Transformer Language Models

Yates et al. (2021) presents a comprehensive survey of how BERT (Devlin et
al., 2019) works, ranking documents with BERT, retrieve and rerank approach
with monoBERT, ranking metrics, etc. One of the most remarkable works in the
survey is monoBERT and duoBERT, a multistage ranking approach with transformer
language models proposed by Nogueira et al. (2019). The first stage retrieves
the candidate documents with BM25 by treating the query as a bag of words and
later, documents are reranked with their relevance score with BERT. DuoBERT
also takes into account one document being more relevant than the other at a
third stage. However, we rank the documents with a language model at one
stage.

Halterman et al. (2021) rank documents with RoBERTa-Large-MNLI (Liu et al.,
2019) on sentence level by being relevant to a police activity. Yet sentence
level evaluation does not take into consideration the relationship between the
sentences. Moreover, the task of extracting police events is a relatively
specific topic in political event extraction. We apply this method with
different document sizes and test on datasets in different topic
specificities.

#### Transformer Language Models DeBERTa and RoBERTa

DeBERTa-Large-MNLI (DLM) (He et al., 2020) and RoBERTa-Large-MNLI (RLM) (Liu
et al., 2019) are pre-trained language models that improve BERT. Both models
are pre-trained on Wikipedia (English Wikipedia dump3; 12GB), BookCorpus
(6GB), OPENWEBTEXT (38GB), and STORIES (a subset of CommonCrawl (31GB) and
fine-tuned for MNLI task. RLM has a token limitation of 512 whereas DLM has a
limitation of theoretically 24,528. We limit the inputs to 512 tokens for both
models to be able to compare them fairly. Ye and Manoharan (2021) find that
DLM achieves a better performance in different sentence similarity tasks with
respect to RLM and BERT. He et al. (2020) also show that DeBERTa outperforms
RoBERTa in a variety of NLP tasks even when DeBERTa is trained on half of the
training data. Therefore, we use DLM and compare it with RLM for our task.

#### Transferring Question Answering to Entailment Problem

Khot et al. (2018) and Demszky et al. (2018) transfer the question answering
problem to the entailment problem by forming the question into a declarative
form. Clark et al. (2019) transfer yes/no question answering to entailment
problem by training supervised models on entailment datasets and treating
entailment probabilities as the probability of the answer being yes. They also
use pre-trained ELMo, BERT, and OpenAI GPT as unsupervised models and show
that fine-tuning BERT on entailment dataset MultiNLI boosts the performance.
The problem of any binary classification can be also transferred to an
entailment problem similar to the yes/no question answering, by considering
the probability of entailment as the probability of data belonging to the
positive class.

##  3 Data

We carried out the experiments on two different datasets: India Police Events
dataset111Data and code are provided at
https://github.com/slanglab/IndiaPoliceEvents. (Halterman et al., 2021) and
the ProtestNews dataset of the workshop CASE @ ACL-IJCNLP 2021222Information
and data are provided at https://github.com/emerging-welfare/case-2021-shared-
task.  (Hürriyetoğlu et al., 2021).

India Police Events dataset includes 1,257 articles about the Indian state
Gujarat from The Times of India and from March 2002. The articles are in
English and contain 21,391 sentences in total. Each sentence is classified
into 5 different labels regarding police activity: kill, arrest, fail, force,
and any action. Question form of the each event type is given in Table 1. A
document belongs to a class if any of its sentences belongs to that class.
Table 2 illustrates the number of positive documents and the proportion of the
positive documents for each event class. Note that one document may belong to
one class, several classes or none of them.

Event type | Question
---|---
kill | Did police kill someone?
arrest | Did police arrest someone?
fail | Did police fail to intervene?
force | Did police use force or violence?
any action | Did police do anything?
Table 1: Question form of each event type. Event type | Positive Documents
---|---
kill | 50 (3.98%)
arrest | 128 (10.17%)
fail | 114 (9.05%)
force | 90 (7.15%)
any action | 457 (36.24%)
Table 2: Number of positive documents for each event class (India Police Events Dataset). Dataset | Positive Documents
---|---
ProtestNews Dataset | 1,912 (20.51%)
ProtestNews Subset | 268 (21.32 %)
Table 3: Number of positive documents for ProtestNews Dataset and its subset.

ProtestNews dataset includes local news articles of countries India, China,
Argentina, and Brazil. These articles are in English, Spanish, Portuguese, and
Hindi. For this work, we have only used English articles. There are 9,327
English documents but to equalize data sizes with the India Police Events
Dataset we randomly selected 1,257 articles among those. Documents that
contain past or ongoing protest events are labeled as positive (Duruşan et
al., 2022). Number and proportion of positive documents are given in Table 3.

##  4 Method

First, the probability of entailment for each document and a query is
calculated with NLU models from Huggingface333http://huggingface.co, and
documents are ranked by the decreasing probability of being relevant to the
query. Thus we expect the documents that are more relevant are ranked at the
top.

Entailment probabilities are evaluated on both sentence and document levels.
At sentence level evaluation, entailment probabilities of sentences in a
document with the given query are calculated and the largest probability among
the sentences is considered as the probability of the document being relevant.
For the document level evaluation since RLM is limited to 512 tokens, we
divided documents into parts such that each part does not exceed 512 tokens.
Similar to the sentence-level approach, probabilities of each part are
calculated and the one with the largest probability is considered as the
probability of the document. After getting the probabilities for all
documents, they are ranked in the decreasing probability.

We compare the results by checking how much recall is achieved when a
specified proportion of data is read from the ranked documents following
Halterman et. al. (2021) and also by calculating the mean average precision.
We release our code publicly444https://github.com/kiymetakdemir/zero-shot-
entailment-ranking.

###  4.1 Models

We focused on the performances of two multilingual NLU models that are
RLM555https://huggingface.co/roberta-large-mnli (Liu et al., 2019) and
DLM666https://huggingface.co/microsoft/deberta-large-mnli (He et al., 2020)
which are pre-trained on the same datasets (Wikipedia and BookCorpus). We
conduct experiments with both models and compare the results.

Query type |  Query
---|---
Declarative query |  There is a protest.
Definitional query |  There is a strong complaint expressing disagreement, disapproval, or opposition. (definition of protest999https://dictionary.cambridge.org/dictionary/english/protest)
Social protest definition (Annotation Manual) |  Individuals, groups, or organizations voice their objections, oppositions, demands or grievances to a person or institution of authority.
Contentious politics event definition (Annotation Manual) |  There is a politically motivated collective action event.
’protest’ + definitional query |  Protest, there is a strong complaint expressing disagreement, disapproval, or opposition.
Protest definition + opposition definition |  There is a strong complaint expressing disagreement, disapproval, or opposition. Disagreement with something, often by speaking or fighting against it, or (esp. in politics) the people or group who are not in power. (definition of opposition101010https://dictionary.cambridge.org/dictionary/english/opposition)
Protest definition + disapproval definition |  There is a strong complaint expressing disagreement, disapproval, or opposition. The feeling of having a negative opinion of someone or something. (definition of disapproval111111https://dictionary.cambridge.org/dictionary/english/disapproval)
Table 4: Queries used for the ProtestNews dataset. Event type | Declarative query |  Definitional query
---|---|---
kill | Police killed someone. |  Police caused someone or something to die. (definition of kill121212https://dictionary.cambridge.org/dictionary/english/kill)
arrest | Police arrested someone. |  Police used legal authority to catch and take someone to a place where the person may be accused of a crime. (definition of arrest131313https://dictionary.cambridge.org/dictionary/english/arrest)
fail | Police failed to intervene. |  Police failed to have an effect. (definition of act141414https://dictionary.cambridge.org/dictionary/english/act)
force | Police used violence. |  Police used actions or words that are intended to hurt people. (definition of violence151515https://dictionary.cambridge.org/dictionary/english/violence)
any action | Police did something. |  Police have an effect. (definition of act)
Table 5: Queries for the India Police Events dataset.

###  4.2 Queries

We have experimented with different types of queries: definitional queries,
extended definitional queries and declarative queries.

We used the Cambridge Dictionary777https://dictionary.cambridge.org and form
the definitional queries by using the definitions of the class name (protest,
kill, arrest, etc.). Annotation manual may possibly be a good resource to find
the definition of the investigated class. For this reason, we also
experimented with definitions from Annotation
Manual888https://github.com/emerging-
welfare/general_info/tree/master/annotation-manuals (Duruşan et al., 2022). On
the other hand, a declarative query is a sentence that simply describes the
class. For instance, we use “There is a protest.” as the declarative query for
the ProtestNews dataset. For the India Police Events dataset, we use
declarative queries proposed by Halterman et al. (2021).

We also extended protest dictionary definition by concatenating it with the
definitions of words that pass in the query (see last 3 rows in Table 4). In
one of the queries, the ‘protest’ word is added to the beginning of the
protest definition. In the other one, definition of opposition is concatenated
with the protest definition. In the third one, definitions of protest and
definition of disapproval are concatenated and used as a query. Note that we
used the definitions of opposition and disapproval since they occur in the
protest definition. All queries used for both datasets are listed in Table 4
and 5.

##  5 Experiments & Results

(a) Sentence level evaluation.

(b) Document level evaluation.

(c) Extended definitions.

(d) Different definitions.

Figure 1: ProtestNews dataset tested on two models: RLM and DLM.

#### ProtestNews Dataset

is tested with declarative queries, definitional queries and extended
definitions on models DLM and RLM and results are presented in Figure 1(a) for
the sentence level evaluation. The x-axis represents what percentage of the
data is read and the y-axis represents how much recall is achieved at that
stage. One can investigate what percentage of the data should be read to
achieve a specified recall. We see that both models yield similar results when
the same query is given but positive documents are accumulated at more top
with the declarative query compared to the definitional query.

For document level evaluation, Figure 1(b) illustrates the comparison of the
models. DLM achieves higher recall scores than RLM, however, the query type
does not affect the performance of the model at the document level
significantly.

We compare the extended and Annotation Manual definitions at document level
using the DLM model since the DLM achieves higher recall compared to RLM at
the document level as in Figure 1(b). However, from Figure 1(c) we see that
extending the protest definition performs slightly worse than using the only
dictionary definition. Also, Annotation Manual definitions do not perform
better than the dictionary definition as we see from Figure 1(d).

| ProtestNews | India Police Events
---|---|---
| - |  kill |  arrest |  fail |  force | any action
DLM |  decl-sent | 0.64 |  0.96 |  0.94 |  0.65 |  0.91 | 0.89
DLM |  decl-doc | 0.60 |  0.80 |  0.75 |  0.25 |  0.75 | 0.80
DLM |  def-sent | 0.35 |  0.89 |  0.63 |  0.47 |  0.71 | 0.69
DLM |  def-doc | 0.41 |  0.62 |  0.42 |  0.21 |  0.21 | 0.65
RLM |  decl-sent | 0.65 |  0.55 |  0.91 |  0.34 |  0.66 | 0.42
RLM |  decl-doc | 0.51 |  0.18 |  0.44 |  0.18 |  0.27 | 0.36
RLM |  def-sent | 0.38 |  0.36 |  0.26 |  0.23 |  0.18 | 0.38
RLM |  def-doc | 0.34 |  0.11 |  0.15 |  0.16 |  0.11 | 0.37
Table 6: mAP scores for DLM and RLM models with different document lengths and
queries.

#### India Police Events Dataset

is tested with declarative and definitional queries on RLM and DLM as in
ProtestNews dataset. For all event types, we see from Figure 2 and Figure 3
that DLM with declarative query gives the best result that is positive
documents are accumulated at more top-level, whereas RLM with a definitional
query stays behind other combinations of model and queries.

(a) kill

(b) arrest

(c) fail

(d) force

(e) any action

Figure 2: India Police Events dataset sentence level evaluation tested on RLM
and DLM.

(a) kill

(b) arrest

(c) fail

(d) force

(e) any action

Figure 3: India Police Events dataset document level evaluation tested on RLM
and DLM.

#### Mean Average Precision (mAP)

is calculated for each ranking and reported in Table 6. Query and document
length combination that gives the highest mAP is marked in bold for each
dataset and event type.

For the ProtestNews dataset we observe that using models DLM or RLM, and
document lengths do not differ significantly. Whereas using a declarative
query gives much better mAP than the definitional query. For the India Police
Events dataset for all event types DLM and declarative query with the sentence
level evaluation yield the highest score rather than the definitional or
document level evaluation. Besides, note that there is a large difference with
the other combinations. For example for event type force, sentence level
evaluation with DLM and the declarative query gives 0.91 mAP whereas document
level evaluation with RLM and the definitional query yields 0.11 mAP.
As the topic gets broader, we see that performance gets worse in Table 6. For
instance, kill is a more specific topic than any action since any action event
type also includes kill events. When 20% of the data read, 90% recall is
achieved for event type kill, on the other hand, even 60% recall is not
reached for any action.
We take the average sentence and document level mAP scores for each model and
present in Table 7. For ProtestNews dataset, sentence or document level does
not differ in mAP when DLM is used. However, for India Police Events dataset
sentence level evaluation achieves much higher mAP than document level
evaluation (0.24 mAP increase for DLM and 0.21 increase for RLM). For both
sentence and document level, DLM reaches higher mAP than RLM.

| ProtestNews | India Police Events
---|---|---
DLM | RLM | DLM | RLM
sent | 0.50 | 0.52 | 0.77 | 0.43
doc | 0.50 | 0.42 | 0.53 | 0.22
Table 7: Average mAP on ProtestNews and India Police Events Dataset for all
event types.

##  6 Conclusion

We investigate the performances of two Transformer Language Models (DLM and
RLM), different query types (declarative and definitional) in different
document lengths (document and sentence level). Our experiments that conclude
DLM achieves higher mAP scores than RLM are consistent with the findings of Ye
and Manoharan (2021) and He et al. (2020). In general, we find that the
combination of DLM with a declarative query in sentence level outperforms
other combinations in mAP score. However, scores decrease as the topic or
event type gets broader where protest events can be considered broader than
specific police actions.

##  7 Future Work

We plan to analyze results more for example by considering subcategories of
protest events for the ProtestNews dataset. Future work can extend this work
to a different political event classification dataset and further investigate
the association between the broadness of the topic and metric scores.
Experiments in languages other than English are also left as future work.

## References

  * Baeza-Yates and Ribeiro-Neto (1999) Ricardo A. Baeza-Yates and Berthier A. Ribeiro-Neto. 1999.  _Modern Information Retrieval_.  ACM Press / Addison-Wesley.
  * Caselli et al. (2021) Tommaso Caselli, Osman Mutlu, Angelo Basile, and Ali Hürriyetoğlu. 2021\.  PROTEST-ER: Retraining BERT for protest event extraction.  In _Proceedings of the 4th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2021)_ , pages 12–19, Online. Association for Computational Linguistics.
  * Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019.  BoolQ: Exploring the surprising difficulty of natural yes/no questions.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2924–2936, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Demszky et al. (2018) Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018.  Transforming question answering datasets into natural language inference datasets.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Duruşan et al. (2022) Fırat Duruşan, Ali Hürriyetoğlu, Erdem Yörük, Osman Mutlu, Çağrı Yoltar, Burak Gürel, and Alvaro Comin. 2022.  Global contentious politics database (glocon) annotation manuals.
  * Elsafoury (2019) Fatma Elsafoury. 2019.  _Detecting protest repression incidents from tweets_.  Ph.D. thesis, University of Glasgow.
  * Halterman et al. (2021) Andrew Halterman, Katherine Keith, Sheikh Sarwar, and Brendan O’Connor. 2021.  Corpus-level evaluation for event QA: The IndiaPoliceEvents corpus covering the 2002 Gujarat violence.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4240–4253, Online. Association for Computational Linguistics.
  * Hanna (2017) Alex Hanna. 2017.  Mpeds: Automating the generation of protest event data.
  * He et al. (2020) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020.  Deberta: Decoding-enhanced bert with disentangled attention.
  * Hürriyetoğlu et al. (2021) Ali Hürriyetoğlu, Osman Mutlu, Erdem Yörük, Farhana Ferdousi Liza, Ritesh Kumar, and Shyam Ratan. 2021.  Multilingual protest news detection - shared task 1, CASE 2021.  In _Proceedings of the 4th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2021)_ , pages 79–91, Online. Association for Computational Linguistics.
  * Hürriyetoǧlu et al. (2017) Ali Hürriyetoǧlu, Nelleke Oostdijk, Mustafa Erkan Başar, and Antal van den Bosch. 2017.  _Supporting Experts to Handle Tweet Collections About Significant Events_, pages 138–141. Springer International Publishing, Cham.
  * Hürriyetoğlu et al. (2016) Ali Hürriyetoğlu, Christian Gudehus, Nelleke Oostdijk, and Antal van den Bosch. 2016.  Relevancer: Finding and labeling relevant information in tweet collections.  In _Social Informatics - 8th International Conference, SocInfo 2016, Bellevue, WA, USA, November 11-14, 2016, Proceedings, Part II_ , volume 10047 of _Lecture Notes in Computer Science_ , pages 210–224.
  * Khot et al. (2018) Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018.  Scitail: A textual entailment dataset from science question answering.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 32(1).
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.
  * Nogueira et al. (2019) Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy J. Lin. 2019.  Multi-stage document ranking with bert.  _ArXiv_ , abs/1910.14424.
  * Wiedemann et al. (2022) Gregor Wiedemann, Jan Matti Dollbaum, Sebastian Haunss, Priska Daphi, and Larissa Daria Meier. 2022.  A generalized approach to protest event detection in German local news.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 3883–3891, Marseille, France. European Language Resources Association.
  * Won et al. (2017) Donghyeon Won, Zachary C Steinert-Threlkeld, and Jungseock Joo. 2017.  Protest activity detection and perceived violence estimation from social media images.  In _Proceedings of the 25th ACM international conference on Multimedia_ , pages 786–794.
  * Yates et al. (2021) Andrew Yates, Rodrigo Nogueira, and Jimmy Lin. 2021.  Pretrained transformers for text ranking: Bert and beyond.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’21, page 2666–2668, New York, NY, USA. Association for Computing Machinery.
  * Ye and Manoharan (2021) Xinfeng Ye and Sathiamoorthy Manoharan. 2021.  Performance comparison of automated essay graders based on various language models.  In _2021 IEEE International Conference on Computing (ICOCO)_ , pages 152–157.


# Improving Recommendation Systems with User Personality Inferred from Product
Reviews

Xinyuan Lu1,2 Min-Yen Kan2
1Integrative Sciences and Engineering Programme (ISEP), NUS Graduate School
2School of Computing, National University of Singapore, Singapore
<EMAIL_ADDRESS><EMAIL_ADDRESS>

(2023)

###### Abstract.

Personality is a psychological factor that reflects people’s preferences,
which in turn influences their decision-making. We hypothesize that accurate
modeling of users’ personalities improves recommendation systems’ performance.
However, acquiring such personality profiles is both sensitive and expensive.
We address this problem by introducing a novel method to automatically extract
personality profiles from public product review text. We then design and
assess three context-aware recommendation architectures that leverage the
profiles to test our hypothesis.

Experiments on our two newly contributed personality datasets — Amazon-beauty
and Amazon-music — validate our hypothesis, showing performance boosts of
3–28%. Our analysis uncovers that varying personality types contribute
differently to recommendation performance: open and extroverted personalities
are most helpful in music recommendation, while a conscientious personality is
most helpful in beauty product recommendation.

Recommendation Systems, Psychology, Personality, Review Texts

††copyright: acmcopyright††journalyear: 2023††doi:
XXXXXXX.XXXXXXX††conference: In Proceedings of the Workshop on Interactive
Recommender Systems of the 16th ACM International Conference on Web Search and
Data Mining; February 27–March 3, 2023; Singapore††booktitle: Workshop on
Interactive Recommender Systems of the 16th ACM International Conference on
Web Search and Data Mining (IRS@WSDM’ 23), February 27-March 3, 2023,
Singapore††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Information
systems Recommender systems††ccs: Applied computing Psychology

##  1\. Introduction

Online recommendation systems are algorithms that help users to find their
favorite items. In recommendation systems, the user’s profile is important as
people with different ages, educational backgrounds exhibit different
preferences. Besides static attributes such as gender, the user’s
psychological factors, especially personality, can be viewed as a user’s
dynamic profile are also vital in recommendations.

People with similar personalities are more likely to have similar interests
and preferences (Yang and Huang, 2019). Therefore, accurate modeling of the
user’s personality plays a vital role in recommendation systems. For example,
in movie recommendation, an outgoing person may favour watching comedic movies
over romantic ones (Yang and Huang, 2019). Other studies (Mariappan et al.,
2012) have shown that in music recommendation, a user’s degree of openness
strongly determines their preference for energetic music genres. These
examples show that personality traits can influence users’ preferences.

While we can see that personality traits motivate users’ preferences, there
are challenges that need to be solved before one can utilize the traits in the
recommendation. First, collecting personality data is time-consuming. The
current best practice for collecting personality data requires conducting a
user study via an ethically-cleared questionnaire with informed consent.
Subsequent training of assessors is also needed. The entire collection process
can take months (Stajner and Yenikent, 2020) and also be an expensive process
in terms of effort.

Second, processing personality data raises sensitivity and privacy concerns.
If handled incorrectly, such data can be misused by users intentionally,
resulting in a violation of privacy protection policies and biased performance
of the recommendation systems. For example, a scandal emerged when a Facebook
app illegally collected 87 million users’ personality information to
manipulate their voting choices in the U.S. presidential election in March
2018 (Hinds et al., 2020). Such risks make the balance between collecting and
utilizing users’ personality information challenging. This issue has stalled
progress in this emerging field of research.

Due to the problem above, the third challenge is a lack of personality-
grounded datasets in the existing work. One notable exception is the website
myPersonality111https://sites.google.com/michalkosinski.com/mypersonality,
which contained personality data and the likes of Facebook users. However, in
2018, myPersonality’s founders decided to discontinue the project as
“complying with various regulations [had] become too burdensome”. To the best
of our knowledge, there are thus few datasets suitable for testing the effect
of personality factors in recommendation systems research.

In this study, we explore methods to overcome these challenges discussed above
and contribute to personality-based recommendation research. We identify a new
source for inferring a user’s personality traits: user-generated content,
specifically e-commerce review texts. Studies show that review texts can
reflect a user’s personality since individuals manifest their personality
through their choice of words (Pennebaker and King, 1999). For example, when
showing their dislikes on the same shampoo, an agreeable person may comment “I
bought this shampoo for my husband. The smell is not good.”, while a neurotic
and aggressive person might comment “Arrived opened and leaking all over the
box. Tried shampoo but didn’t help at all. Still so itchy!!!”. In addition,
review text is easy to obtain and made publicly available by users in full
disclosure on online commercial websites, which helps to solve both time and
privacy issues.

In our experiments, we explore the possibility of automatically inferring
users’ personality traits from their review texts and then use this
information to help recommendations. We do this by leveraging an Application
Programming Interface (API) to automatically analyze the user’s personality.
There already exist deployed, production-level APIs for automatic personality
detection, such as IBM Personality
Insights222https://cloud.ibm.com/docs/personality-insights, Humantic
AI333https://humantic.ai/, and Receptiviti444https://www.receptiviti.com/ that
purport to yield personality profiles. In our work here, we use the
Receptiviti API, because it is a widely-validated and widely-used psychology-
based language analysis platform for understanding human emotion, personality,
motivation, and psychology from language. Receptiviti’s API outputs scores for
the commonly-used OCEAN personality model: five values, one for each of the
five personality aspects of Openness, Conscientiousness, Extroversion,
Agreeableness, and Neuroticism (each corresponding to one letter of “OCEAN”).
Finally, this inferred personality is fed as input to a recommendation system,
to test whether it can improve recommendation performance.

To conduct our study, we first construct two new datasets extending from an
existing Amazon review dataset, in the beauty and music domains. We first
extract the user reviews that are between 30 to 80 words. Afterward, we
concatenate all the valid review texts of each user and input their
concatenation to the Receptiviti API to output each user’s inferred
personality scores. As a quality check, we evaluate the accuracy of
personality detection, by plotting the personality distribution for each
dataset. We observe the users with extremely high/low personality scores and
find that these are reliable indicators of personality, and use the such
confidently labeled output as ground truth (silver data).

We incorporate these personality scores into the recommendation process and
investigate their effect on current neural-based recommendation systems. We
observe consistent improvements on the performance of such recommendation
systems. When we consider different personality groups, we find that
extroversion and agreeableness benefit the recommendation performances across
all the domains. However, we lack an in-depth understanding of how these
personalities affect recommendations and users’ behavior. This points to
future directions in utilizing other auxiliary information to infer users’
personality traits, e.g., users’ browsing histories.

In summary, our contributions are:

  * •

We construct two new datasets in the music and beauty domains that combine
users’ public product reviews alongside automatically inferred personality
scores. This directly addresses the lack of personality-based datasets for
recommendation, while avoiding privacy issues by using public data.

  * •

We conduct empirical experiments over these datasets, finding that leveraging
personality information indeed improves the recommendation performance, from
3% to 28%.

  * •

We analyze the influence of personality traits in these domains and find the
personality traits of extroversion and agreeableness improve the
recommendation performance across all domains.

##  2\. Related Work

The current study investigates how to extract personality traits from texts
and how personality traits can be utilized in recommendation systems.
Therefore, the review below focuses on the literature that discusses
personality detection and personality-based recommendation systems. We first
give an introduction of the OCEAN personality models (Section 2.1) before
reviewing two topics related to our work: personality detection (Section 2.2)
and personality-based recommendation systems (Section 2.3).

###  2.1. The OCEAN Model

Personality involves a pattern of behavior that is not likely to change over a
short period of time (Allport, 1961). It can be detected either explicitly by
a questionnaire or implicitly by observing user behaviors (McCrae and John,
1992). The most commonly-used model describing personality traits is the OCEAN
model (McCrae and John, 1992), which we use to model a user’s personality
traits. The five fundamental personality dimensions defined by OCEAN are:

  1. (1)

Openness to Experience (O), which describes the breadth and depth of people’s
life, including the originality and complexity of their experiences.
Individuals with high openness tend to be knowledgeable, analytical, and more
investigative.

  2. (2)

Conscientiousness (C) This trait involves how individuals control, regulate
and direct their impulses. For example, highly conscientious people are
usually cautious.

  3. (3)

Extroversion (E) Extroversion indicates how much people are in touch with the
outside world. Extroverts are more willing to talk to others about their
thoughts.

  4. (4)

Agreeableness (A) This trait reflects individual differences and social
harmony in cooperation. Highly agreeable people are more willing to share
tasks than to complete tasks independently.

  5. (5)

Neuroticism (N) This refers to the tendency of experiencing negative emotions.
People with high neuroticism are often in a bad mood, therefore they prefer to
respond emotionally.

###  2.2. Personality Detection

There are two common ways to measure a person’s personality traits using a
personality model: personality assessment questionnaires and automatic
personality detection.

####  2.2.1. Personality Assessment Questionnaires

Self-reporting personality questionnaires are commonly used to reveal
personality differences among individuals. Responses to questions usually take
the form of a five-point Likert scale (strongly agree, agree, disagree, and
strongly disagree). Such personality inventories differ with respect to the
number and content of their questions. Common long questionnaires include the
NEO Five-Factor Inventory (60 items) (Aluja et al., 2005), NEO-Personality-
Inventory Revised (240 items) (Costa Jr and McCrae, 2008), and the Big-Five
Inventory (BFI, 44 items) (Rammstedt and John, 2007). Practitioners prefer
using shorter instruments, such as the BFI-10 and Ten-Item Personality
Inventory (TIPI) (Gosling et al., 2003; Topolewska et al., 2014), as they are
time-saving and easier to fill out.

However, using questionnaires for self-report has two major drawbacks. First,
questions that assess personality are often quite subjective such as “Do you
easily get nervous?”. Answers for such questions are easily affected by a
self-bias (Pedregon et al., 2012) or reference-group effects (Youyou et al.,
2017). For example, an introverted engineer might think he is an extrovert if
he/she is working with a group of individuals that may be more introverted.
Consequently, the results of questionnaires are often hard to reproduce.
Second, assessing a personality by questionnaires can be inconvenient, as the
subjects are necessary to participate in the studies.

####  2.2.2. Automatic Personality Detection

To make personality detection more convenient and reproducible, practitioners
prefer automated personality detection, which infers a personality type based
on user data. Such methods are less accurate than personality questionnaires —
as it relies on the input user data manifesting personality traits — but has
the advantage of not requiring inputs to be answers to questionnaires. For
example, social media posts that exhibit opinions and viewpoints are a prime
source of text data useful for personality detection. Individuals have
different language use behaviors that reveal personality traits (Hirsh and
Peterson, 2009). Automatic, text-based personality detection infer users’
personalities by analyzing users’ word choice (lexical selection) and sentence
structure (grammatical selection). Such technology has been sufficiently
proven, making them commonplace and deployed at scale in production, and
available as a service through cloud-based application APIs.

We study whether knowing users’ personality information can lead to better
recommendations, and also, how users’ personality information be best modeled
in recommendation systems to improve performance. While large-scale
recommendation datasets exist, they universally lack users’ personality
information. It is infeasible to ask to obtain this information via
questionnaires since the identity of users is usually confidential. Therefore,
we utilize automatic personality detection to infer personality from product
reviews written by users. Product reviews are ideal: they are widely available
on online commercial websites, they often demonstrate personality traits, and
they are public (the texts are meant to be read by others). Hence, they can
serve as a good source for automatically detecting personality in an economic
but accurate way.

###  2.3. Personality-based Recommendation Systems

Since personality traits are characteristics that do not change sharply over
time and do not depend on a certain context or stimulus, they are more easily
used to create personalized recommendation systems. Earlier work by Winoto and
Tang (Winoto and Tang, 2010) (Ayata et al., 2018) focused on extending Matrix
Factorization by adding a personality latent factor. Their model used implicit
feedback data, such as user–item interactions, beyond just ratings. They only
considered the OCEAN scores as one attribute, so the effects that are
attributable just to personality are not clear. Besides, personality traits
have been used to determine a neighborhood of similar users by calculating
personality similarity. Thus, for example, Asabere et al. (Asabere et al.,
2018) proposed a recommendation system for attendees of conferences that
integrates personality traits and social bonds of attendees.

In their work, user similarity was equated as personality similarity,
calculated by Pearson’s correlation between two users’ OCEAN scores. They
demonstrated that the system accuracy improves with a larger user base, due to
the higher likelihood of finding other users with similar personalities (high
correlation).

Researchers have also associated user personality scores with items. Yang and
Huang (Yang and Huang, 2019) attributed items (here, computer games) with a
personality that is an average of its users. This latent representation can
then be used to recommend items to users with a similar personality as that of
the other users of that item. This may make sense when certain items are used
primarily by certain personality types (as in computer games) but are less
compelling for items that may be used by many personality types. Lastly, in
social media recommendations, Wu et al. (Wu et al., 2018) proposed an approach
for recommending interest groups by integrating personality. The personality-
based similarity was defined as the Euclidean distance between two users’
personality scores. However, it combines the personality signal linearly in
the recommendation process, which we feel may be limiting.

In summary, compared with other context attributes (e.g., purchase history),
personality information helps to capture the users’ potential interests rather
than recommending a similar purchased item. However, the previous works used
the OCEAN personality as a linear similar score which lacks the capability of
capturing more nuanced information latent in personality scores. Different
from the methods above, we propose two novel ways of adding personality
features into the recommendation system: 1) taking the most salient
personality trait as a learnable vector and 2) calculating a user’s
personality embedding as a weighted sum of a user’s OCEAN personality
features, which is a learnable embedding within the recommendation system.

##  3\. Dataset Construction

We construct two new datasets, as extensions of the existing, well-known
Amazon review dataset. We first automatically infer users’ personality traits
from users’ review texts as review texts can reflect the personality through
word usage. They are also publicly-available text on online commercial
websites, allowing researchers to have legal access to textual data where
experimentation can be replicated. Based upon the parent Amazon review
dataset, we construct two new domain-specific datasets: an Amazon-beauty and
an Amazon-music dataset. These contain Amazon reviews of products in the
beauty and music domains, alongside their posting users’ inferred personality
scores.

###  3.1. Data Source

The Amazon dataset555https://nijianmo.github.io/amazon/index.html (Ni et al.,
2019) is widely used for training and evaluating recommendation systems. It
contains a large number of item descriptions, ratings, and product reviews
collected from the Amazon online commercial website. The Amazon dataset is
divided according to the domain. In our study, we choose two domains: beauty
and music. We construct datasets separately for these two domains since we
want to study whether personality has different influences on users’
behaviours for different domains. Studies have shown that people with
different personalities prefer different kinds of music (Mariappan et al.,
2012). For example, people with a high degree of openness like to listen to
rock music, while neurotic people like jazz. Therefore, we choose music as one
of the domains to be studied. In order to study the role of personality in
different domains, we randomly select beauty for comparison. Table 1 shows a
sample of the original Amazon dataset, which contains the user (reviewerID,
reviewerName), the product’s Amazon Standard Identification Number (asin), the
review text for the product (reviewText), and the overall rating given to the
product (overall).

reviewerID | A2SUAM1J3GNN38
---|---
asin | 0000013714
reviewerName | J.McDonald
vote | 5
style | Format:Hardcover
reviewText |  | I bought this for my husband who plays the piano. He is
---
having a wonderful time playing these old hymns. The
music is at times hard to read because we think the book
was published for singing from more than playing form.
Greate purchase though!
overall | 5.0

Table 1. An example of Receptiviti score for a specific, anonymized user.

###  3.2. Dataset Construction

Since we do not know the personality for each user in the Amazon dataset, we
need to infer them. We first retrieve each user’s review texts and then use
the Receptiviti API666https://www.receptiviti.com/, a computational language
psychology platform for understanding human behavior, to infer a personality.
The API can take a long piece of human-written text (more than 300 words), and
output a faceted personality score with 35 factors, including OCEAN scores.

For each user that wrote reviews in either of the two domains, we collect all
his/her review texts and concatenate them together into a single document.
Afterward, we send the concatenated text to Receptiviti to infer a
personality. We select the personality scores corresponding to the five-
dimensional personality traits defined in the OCEAN model (McCrae and John,
1992) (Table 2). Each personality score is normalized to a range from 1 to
100. The higher the score, the more overt the personality trait. Note that
each of the five OCEAN scores is independent of the other.

User ID | AGR | CON | NEU | EXT | OPEN
---|---|---|---|---|---
A2GBIFL43U1LKJ | 54.05 | 34.87 | 25.96 | 54.39 | 42.71

Table 2. An example of Receptiviti score for a specific, anonymized user.

To improve the personality prediction process, we only analyze the personality
traits for active users who bought many products and wrote a sufficient number
of product reviews. To be specific, we select users that 1) wrote product
reviews for at least 10 different items they purchased, and where 2) each
product review contains between 30 to 80 words. Table 3 shows the statistics
after the filtration. For example, using these criteria, 1,791 active users
are selected for the Amazon-music dataset. Each user in the Amazon-music
dataset has an average of 990.48 review words over all of his/her reviews,
averaging 51.01 words for each review.

###  3.3. Dataset Statistics

Aside from our constructed Amazon-beauty and Amazon-music dataset, we also
include an existing dataset Personality 2018 in our study. Personality
2018777https://grouplens.org/datasets/personality-2018/ (Nguyen et al., 2018)
is a version of the MovieLens dataset that includes each user’s personality
information obtained through questionnaires. It contains 21,776 movies,
339,000 ratings, and 678 users with the OCEAN personality questionnaire scores
from 1 to 7. This dataset is included to study the difference between
questionnaire-based personality trait scores with our review-based automatic
personality trait detection scores.

Table 3 shows the final statistics of the datasets used in our study. We can
observe that the Amazon-beauty / Amazon-music dataset has the largest /
smallest percentage of interactions. The Personality2018 dataset contains the
largest number of items and the smallest number of users. We can see that
these datasets differ in domains, number of users, items, and interactions,
which facilitates the study of personality-based recommendation across a wide
spectrum of settings.

Dataset | Amazon-beauty | Amazon-music | Personality’18
---|---|---|---
# of items | 85 | 8,895 | 21,776
# of users | 991 | 1,791 | 678
# of ratings | 5,269 | 28,399 | 339,000
# of interactions | 6.26% | 0.18% | 2.30%
| Avg. words/user
---
990.48 | 466.43 | -
| Avg. words/review
---
51.01 | 51.18 | -

Table 3. Statistics of the three datasets used in our study.

##  4\. Methods

Based on our constructed dataset, we conduct experiments to study whether the
recommendation system can benefit from incorporating personality traits. We
choose the Neural Collaborative Filtering (NCF) (He et al., 2017) as the
foundation model of our study because it is the fundamental neural-based model
for the recommendation. Specifically, we design a personality-enhanced version
of NCF (He et al., 2017) to compare with the vanilla NCF, alongside several
other baselines.

###  4.1. Neural Collaborative Filtering (NCF)

NCF (He et al., 2017) is the first deep-learning-based recommendation
algorithm. Different from traditional collaborative filtering algorithms, the
model encodes the user and item into latent vectors and then projects them
through a Multi-layer Perceptron (MLP) to predict a probability score,
representing the probability that a user would buy a target item. In our
implementation, we use a 4-layer MLP and a 16-dimensional user and item
embedding.

###  4.2. Personality-enhanced NCF

We then propose three different ways to incorporate the personality
information into the NCF model, as shown in Fig. 1. We first design NCF+Most
salient Personality model by adding the most salient personality trait as
input into NCF. We also design NCF + Soft-labeled Personality and NCF + Hard-
coded Personality to incorporate all the five personality traits of OCEAN. The
difference between the two latter versions is that the personality vector in
NCF + Soft-labeled Personality is learnable, while in NCF + Hard-coded
Personality, the vector is predetermined and fixed.

Figure 1. The overall structure of our model. In this example, the user’s
OCEAN score is {30,70,50,30,20}. TheNCF + Most salient personality selects the
personality with the highest score, i.e., conscientiousness as the personality
embedding vector. NCF + Soft-labeled personality takes all five OCEAN scores
as a personality embedding matrix. NCF + Hard-coded personality predetermines
and fixes the personality vector as {0.3,0.7,0.5,0.3,0.2}

1\. NCF + Most Salient Personality. In this model, we introduce a
4-dimensional personality vector for each of the five types of personalities,
which are learned during training. We treat the most salient personality as
the user’s personality label and concatenate the corresponding personality
vector with the user’s latent vector.

2\. NCF + Soft-labeled Personality. In this model, we make full use of all
five personality trait scores. We first apply a Softmax function to map the
personality scores into a probability distribution of personality. Afterward,
the probability distribution is used as the weight to calculate the weighted
sum of the five personality vectors. The output vector is then concatenated
with the user’s latent vector as the input of the MLP.

3\. NCF + Hard-coded Personality. This model also considers all the user’s
five personality traits information. However, instead of introducing learnable
personality vectors, we directly scale each personality score to sum to a unit
value (here, 100) to get a hard-coded 5-dimensional vector to represent the
user’s personality information. This vector is concatenated with the user’s
latent vector, but is fixed during training.

##  5\. Experiments

We evaluate our proposed method on our three datasets by answering the
following four research questions. We first evaluate whether we can accurately
detect personality from texts (RQ1, Section 5.1). Afterward, we analyze the
distribution of personality in review texts (RQ2, Section 5.2). Then, we
explore whether adding personality information can improve recommendation
performance (RQ3, Section 5.3). Finally, we analyze the influence of
personality information on different domains (RQ4, Section 5.4).

###  5.1. Can we accurately detect personality from texts? (RQ1)

To evaluate whether we can accurately detect personality traits from texts, we
analyze the personality scores inferred by the Receptiviti API for each user.
Since there are over 2,500 users in total in our two constructed datasets, it
is time-consuming to manually evaluate them all. As a compromise, we choose to
manually examine the users that receive extremely high scores for certain
personality traits. We believe those examples are more easily evaluated by
humans. Specifically, for each personality trait, we select the users that
receive the top 10 highest scores on this type. We analyze both the Amazon-
beauty and the Amazon-music datasets, resulting in a total of 100 samples.
These samples are evaluated by two graduate students. Both were trained with a
detailed explanation of the OCEAN personality model. We ask them to choose
whether the sampled review texts accurately match their inferred personality,
choosing between three options of yes, no, or not sure. We then calculate the
accuracy of the samples and the inter-annotator agreement between the two
annotators using Cohen’s Kappa (Cohen, 1968). We find that the inferred
personality matches with the review text in 81% of the Amazon-beauty samples,
and 79% of the samples from Amazon-music. The average Cohen’s Kappa is 0.70.
We take this to indicate that the Receptiviti API can indeed infer users’
personality traits from review texts with generally high accuracy.

Table 4 shows examples of review texts with their inferred personality scores.
We observe that people with different personalities have different language
habits. For example, extroverts tend to use the words “love” and exclamation
marks because they are characterized by a strong tendency to express their
affection. People who are agreeable are usually bought items for other people,
e.g., “my kids” and “my wife”, perhaps due to their inclusiveness.
Conscientious people usually talk about their own experience and feelings
before recommending the items to others, e.g., “I have had this shower gel
once before” or “Don’t just take my word for it”. This is perhaps because they
are usually cautious.

Personality label |  | Personality
---
Score
Review Texts
Openness | 63.07 |  | Near perfect exfoliating gloves my only complaint is a matter of preference rather than product
---
defect.
I prefer the harder surface area to use on round areas of the body or
potentially harder like the feet,
elbows, etc.
Openness | 62.62 |  | Azur is always my favorite in the Thymes collection because of its clean, fresh scent.
---
I like that my skin feels moisturized when using this product in the shower.
Conscientiousness | 75.38 |  | I have had this shower gel once before, and it’s amazing. Hard to find, too.
---
One of The Body Shop’s best scents, and it’s usually only available
seasonally!
wish they sold it in bigger bottles, but I was happy to find it.
Conscientiousness | 71.02 |  | Don’t just take my word for it, you must try it.
---
A dear friend got me this from Italy 12 years ago and has been using it since,
very hard to find it in the US.
This shower cream will transform your shower experience.
Extroversion | 75.06 | Love this shampoo! Recommended by a friend! The color really lasts!!!
Extroversion | 72.90 |  | Looked all over to find where to purchase this product and we are very happy to
---
be able to finally find it.
The PRELL Conditioner is by far the best you can buy. We love it!!
Agreeableness | 80.06 | Great product - my wife loves it
Agreeableness | 78.18 |  | Great deal and leaves my kids smelling awesome!
---
I bought a box of them years ago and we still have some left!!!
Great deal and leaves my kids smelling awesome!
Neuroticism | 67.81 |  | Too expensive for such poor quality.
---
There was no improvement and I am starting to think my scalp is worse off than
it was before
I started using this product.
I do agree with other reviews that it feels watered.
Neuroticism | 62.28 |  | Nope. It smells like artificial bananas, and this smell does linger.
---
It’s pure liquid, there is no thickness to it at all, it’s like pouring banana
water on your head
that lathers.
It does not help with an itchy scalp either.
Table 4. The data sample of extreme personality cases to the annotators. Each
data sample contains the user’s personality labels, personality scores, and
review texts.

###  5.2. What is the distribution of users’ personalities? (RQ2)

We further analyze the personality distribution for all users by plotting the
score histograms for each personality trait in the Amazon-beauty dataset and
the Amazon-music dataset in Fig. 2.

Figure 2. Distribution of personality traits in Amazon-beauty and Amazon-music
datasets. The x-axis represents the score for each trait; the y-axis
represents the number of users. The red line represents the median for each
trait.

We observe a similar trend in both domains: agreeable people have the highest
median score, and neurotic people have the lowest median score. A possible
reason is that neurotic people are more introverted and are less likely to
publish their opinions publicly, while agreeable people are more willing to
share their thoughts. Another observation is that the personalities of each
dataset are generally bell-curved. This indicates that each personality trait
is normally distributed.

We also examine the difference in personality distributions between the two
domains. In the Amazon-music dataset, the average scores of extroversion and
openness are higher than those in the Amazon-beauty dataset. This indicates
that the personality characteristics of extroverts and open people are more
obvious in the music domain than in the beauty domain.

From the above figures, we draw the following conclusions. First, the
personality traits of users are not evenly distributed. There are more
instances of people with certain personality traits (e.g., agreeableness) than
others (e.g., neuroticism). A possible reason is that people with certain
personalities are more willing to write product reviews. 2) The distributions
for the two domains are generally the same, with higher agreeable scores and
lower neurotic scores. However, there is a slight difference. For example, the
scores of extroverts in music are generally higher than that in the beauty
domain. This could be explained by the possibility that people who are
passionate about music may be more emotional.

###  5.3. Does incorporating personality improve recommendation performance?
(RQ3)

Next, we want to explore whether adding the induced user personality benefits
the recommendation quality. To this end, we compare the personality-enhanced
NCF with the following two baseline models that do not utilize personality
information.

1\. NCF with random personality (NCF + Random). We randomly assign each user
with a random personality label, regardless of his/her original, inferred
personality scores.

2\. NCF with same personality (NCF + Same). We assign each user to a single
personality trait. To be specific, we assume every user is “open” and assign
the corresponding personality vector to NCF. Although the personality vector
does not provide any additional signal to the model in this case, it can serve
as a place-holder to keep the network structure identical to the personality-
enhanced model, resulting in a fair comparison.

Evaluation Metrics. We use two metrics to measure the performance of our
proposed recommendation models: Hit Rate (HR) @ K and Normalized Discounted
Cumulative Gain (NDCG) @ K (K = 3, 5, 10). Larger HR and NDCG demonstrate
better accuracy.

Experiment Results. Table 5 shows the experimental results in the Amazon-
beauty and the Amazon-music, and Personality2018 datasets, respectively. In
Amazon-beauty and Amazon-music, we find that the three personality-enhanced
NCF models outperform the two baseline models, in terms of both NDCG and HR.
Especially, the first three rows show that the NCF with the most salient
personality label outperforms NCF with the same or random personality label.
This indicates that adding personality information into NCF improves
recommendation performance. From the last three rows, we further find that NCF
+ Soft-labeled/Hard-coded outperforms NCF + Most salient personality in terms
of NDCG. This shows that utilizing all five personality traits is better than
using the most salient personality trait in NCF.

In the Personality 2018 dataset, the trend in the Amazon-beauty and Amazon-
music also holds for it. For example, the NCF + Soft-labeled model outperforms
the other models, showing that adding personality information improves
performance. However, the improvement in the Personality 2018 is less obvious
than that in the Amazon-beauty dataset. We hypothesize the reason might be due
to the difference in the sizes of the datasets. Since Amazon-beauty is a small
dataset, adding personality information may better help to address the data
sparsity problem, therefore exhibiting a better performance gain.

Algorithms | Amazon-beauty | Amazon-music | Personality2018
---|---|---|---
Rating | H@3 | H@5 | H@10 | N@3 | N@5 | N@10 | H@3 | H@5 | H@10 | N@3 | N@5 | N@10 | H@3 | H@5 | H@10 | N@3 | N@5 | N@10
NCF+Random | 0.923 | 0.965 | 0.975 | 0.675 | 0.605 | 0.660 | 0.159 | 0.224 | 0.339 | 0.117 | 0.143 | 0.171 | 0.510 | 0.628 | 0.777 | 0.406 | 0.454 | 0.504
NCF+Same | 0.918 | 0.967 | 0.975 | 0.683 | 0.630 | 0.662 | 0.160 | 0.224 | 0.340 | 0.122 | 0.149 | 0.167 | 0.511 | 0.622 | 0.777 | 0.403 | 0.454 | 0.502
NCF+Most-Salient | 0.939 | 0.969 | 0.977 | 0.714 | 0.676 | 0.707 | 0.156 | 0.226 | 0.343 | 0.164 | 0.145 | 0.174 | 0.516 | 0.631 | 0.795 | 0.415 | 0.463 | 0.511
NCF+Soft-labeled | 0.936 | 0.965 | 0.973 | 0.810 | 0.867 | 0.831 | 0.156 | 0.225 | 0.348 | 0.113 | 0.141 | 0.175 | 0.528 | 0.656 | 0.805 | 0.421 | 0.471 | 0.511
NCF+Hard-Coded | 0.948 | 0.961 | 0.977 | 0.849 | 0.826 | 0.848 | 0.175 | 0.232 | 0.345 | 0.147 | 0.160 | 0.189 | 0.503 | 0.622 | 0.758 | 0.398 | 0.447 | 0.498

Table 5. Hit Rate(H) and NDCG(N) @K in the Amazon-beauty, Amazon-music, and
Personality 2018 datasets. The best performance is bolded.

###  5.4. How does personality information improve the performance of
recommendation system? (RQ4)

To gain a better understanding of the improvement brought by incorporating
personality, we separately evaluate the HR and NDCG for the five personality
traits, as shown in Table 6 . “+” represents the NCF+Soft-labeled model (with
personality information), and “-” represents the NCF+Same model (without
personality information). We make two major observations.

First, the improvement brought by adding personality is prominent for the
Amazon-beauty dataset, over all five personality traits. In particular, the
trait of conscientiousness (CON) has the highest gain in terms of both HR
(+21%) and NDCG (+57%). However, in the Amazon-music dataset, openness (+27%),
agreeableness (+10%), extroversion (+5%) improve while neuroticism (–18%) and
conscientiousness (–12%) decreases.

Second, for the Personality2018 dataset, the improvement brought by adding
personality is not obvious: only conscientiousness, extroversion, and
agreeableness have shown minor performance gain. From the above breakdown
analysis, we find that adding personality information can benefit certain
personality traits better than others. However, the personality trait that
improves the most differs greatly across the three datasets. This indicates
that although improvements are observed in terms of empirical results, the
mechanism of how personality influences the recommendation still deserves more
in-depth investigation.

|  | Amazon-beauty | Amazon-music | Personality2018
---|---|---|---|---
Trait |  | HR | NDCG | HR | NDCG | HR | NDCG
OPEN | + | 0.833 | 0.729 | 0.330 | 0.205 | 0.535 | 0.420
- | 0.750 | 0.545 | 0.313 | 0.161 | 0.547 | 0.422
CON | + | 0.883 | 0.769 | 0.228 | 0.132 | 0.475 | 0.358
- | 0.727 | 0.490 | 0.279 | 0.150 | 0.441 | 0.361
EXT | + | 0.970 | 0.882 | 0.319 | 0.181 | 0.611 | 0.412
- | 0.872 | 0.600 | 0.317 | 0.169 | 0.556 | 0.411
AGR | + | 0.968 | 0.878 | 0.332 | 0.198 | 0.621 | 0.512
- | 0.864 | 0.593 | 0.308 | 0.185 | 0.552 | 0.430
NEU | + | 0.933 | 0.835 | 0.397 | 0.230 | 0.489 | 0.390
- | 0.833 | 0.536 | 0.397 | 0.254 | 0.511 | 0.415

Table 6. HR and NDCG results group by 5 personality traits in Amazon-beauty,
Amazon-music, and Personality2018 datasets. “+” represents the NCF+Soft-
labeled model (with personality information), and “-” represents the NCF+Same
model (without personality information).The best performance is in bold.

##  6\. Discussion

In this work, we make a preliminary attempt to explore how to automatically
infer users’ personality traits from product reviews and how the inferred
traits can benefit the state-of-the-art automated recommendation processes.
Although we observe that recommendation performance is indeed boosted by
incorporating personality information, we believe there are several
limitations. In the following, we discuss these limitations with potential
future directions.

First, we believe capturing personality from the review texts may lead to
selective bias. Introverts are less likely to share their thoughts online
while extroverts are more likely to share experiences. This results in an
imbalanced personality distribution in our collected data. As shown in the
analysis in RQ2 (Section 5.2), extroversion is the most common personality
trait of users in our datasets. To address this, in future works, we could
utilize other context information to infer users’ personalities such as a
user’s purchase history. Such user behaviours can also reflect personality;
for example, open people are more likely to follow popular trends which can be
reflected in their purchase history.

Second, we only conduct experiments on a single basic model, NCF, which may
loss of generalization. More advanced models graph recommendation models can
be used in the future. Third, we conduct empirical experiments on whether
personality information benefits recommendation. However, more in-depth
investigation is necessary on how personality affects recommendation and
users’ behavior. In the future, we could conduct a user study to find the
causal relationship between personality and recommendation. To be specific, we
can develop different marketing strategies for users with different
personalities. By observing the effects of different strategies on users’
behavior, we can gain a better understanding of how personality affects
recommendation. Fourth, we find that the openness, conscientiousness and
neuroticism features do not have a noticeable impact on the recommendation
performance. A possible reason is that OCEAN only contains five types of
personality, which might be insufficient to provide enough useful signals to
recommendations. A possible solution is to use a more fine-grained personality
model than OCEAN; e.g., the MBTI personality model which has a richer,
16-facet personality profile.

Last, the five personalities are encoded independently of each other in our
model. But there is a correlation between these personality traits in real
life; e.g., a majority of extroverts are also open. In the future, we can make
use of the relationship between personalities, perhaps by defining a
hierarchical structure of personality traits and employing graph-based neural
networks to encode them.

##  7\. Conclusion and Future Works

In this work, we explore a new way of automatically extracting personality
information from review texts and applying it to recommendation systems. We
first construct two new datasets based on the Amazon dataset in the beauty and
music domains and include OCEAN personality scores automatically inferred by
the Receptiviti API, a commercial service. We then analyze the accuracy of
using texts to obtain personality profiles and output personality score
distributions. To explore the effectiveness of using personality in current
recommendation systems, we conduct a few experiments with the standard neural
collaborative filtering (NCF) recommendation algorithm and our variants,
finding that incorporating personality information improves recommendation
performance by 3% to 28%. In terms of the relationship between personality and
domain, we find that openness, extroversion, and agreeableness are helpful in
music recommendation, while conscientiousness is most helpful in the beauty
recommendation.

In the future, more advanced models graph recommendation models can be used in
the experiments. In addition, collecting more information beyond review texts
(e.g., purchase history, browsing history) is a potential direction. Moreover,
except for the accuracy-based performance, it is possible to improve the
fairness by using the OCEAN model (Melchiorre et al., 2020). To explore the
inner relationship between personality and recommendation systems, doing a
user study is also a possible way to further validate the findings.

## Acknowledgement

We sincerely appreciate Dr. Liangming Pan’s efforts in his help in
proofreading this work.

## References

  * (1)
  * Allport (1961) G.W. Allport. 1961\.  _Pattern and Growth in Personality_.  Holt, Rinehart and Winston.  https://books.google.com.sg/books?id=GVRAAAAAIAAJ
  * Aluja et al. (2005) Anton Aluja, Oscar Garcia, Jerome Rossier, and Luis F. Garcia. 2005.  Comparison of the NEO-FFI, the NEO-FFI-R and an alternative short version of the NEO-PI-R (NEO-60) in Swiss and Spanish samples.  _Personality and Individual Differences_ 38, 3 (2005), 591–604.  https://doi.org/10.1016/j.paid.2004.05.014
  * Asabere et al. (2018) Nana Yaw Asabere, Amevi Acakpovi, and Mathias Bennet Michael. 2018\.  Improving Socially-Aware Recommendation Accuracy Through Personality.  _IEEE Trans. Affect. Comput._ 9, 3 (2018), 351–361.  https://doi.org/10.1109/TAFFC.2017.2695605
  * Ayata et al. (2018) Deger Ayata, Yusuf Yaslan, and Mustafa E Kamasak. 2018\.  Emotion based music recommendation system using wearable physiological sensors.  _IEEE transactions on consumer electronics_ 64, 2 (2018), 196–203.
  * Cohen (1968) J. Cohen. 1968.  Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit.  _Psychological bulletin_ 70 4 (1968), 213–20.
  * Costa Jr and McCrae (2008) Paul T Costa Jr and Robert R McCrae. 2008.  _The Revised Neo Personality Inventory (neo-pi-r)._ Sage Publications, Inc.
  * Gosling et al. (2003) Samuel D Gosling, Peter J Rentfrow, and William B Swann Jr. 2003\.  A very brief measure of the Big-Five personality domains.  _Journal of Research in personality_ 37, 6 (2003), 504–528.
  * He et al. (2017) Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017\.  Neural Collaborative Filtering. In _Proceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017_ , Rick Barrett, Rick Cummings, Eugene Agichtein, and Evgeniy Gabrilovich (Eds.). ACM, 173–182.  https://doi.org/10.1145/3038912.3052569
  * Hinds et al. (2020) Joanne Hinds, Emma J. Williams, and Adam N. Joinson. 2020\.  ”It wouldn’t happen to me”: Privacy concerns and perspectives following the Cambridge Analytica scandal.  _Int. J. Hum. Comput. Stud._ 143 (2020), 102498.  https://doi.org/10.1016/j.ijhcs.2020.102498
  * Hirsh and Peterson (2009) Jacob B Hirsh and Jordan B Peterson. 2009.  Personality and language use in self-narratives.  _Journal of research in personality_ 43, 3 (2009), 524–527.
  * Mariappan et al. (2012) Mahesh Babu Mariappan, Myunghoon Suk, and Balakrishnan Prabhakaran. 2012.  FaceFetch: A User Emotion Driven Multimedia Content Recommendation System Based on Facial Expression Recognition. In _2012 IEEE International Symposium on Multimedia, ISM 2012, Irvine, CA, USA, December 10-12, 2012_. IEEE Computer Society, 84–87.  https://doi.org/10.1109/ISM.2012.24
  * McCrae and John (1992) Robert R McCrae and Oliver P John. 1992.  An introduction to the five-factor model and its applications.  _Journal of personality_ 60, 2 (1992), 175–215.
  * Melchiorre et al. (2020) Alessandro B Melchiorre, Eva Zangerle, and Markus Schedl. 2020\.  Personality bias of music recommendation algorithms. In _Fourteenth ACM conference on recommender systems_. 533–538.
  * Nguyen et al. (2018) Tien T. Nguyen, F. Maxwell Harper, Loren Terveen, and Joseph A. Konstan. 2018.  User Personality and User Satisfaction with Recommender Systems.  _Inf. Syst. Frontiers_ 20, 6 (2018), 1173–1189.  https://doi.org/10.1007/s10796-017-9782-y
  * Ni et al. (2019) Jianmo Ni, Jiacheng Li, and Julian J. McAuley. 2019.  Justifying Recommendations using Distantly-Labeled Reviews and Fine-Grained Aspects. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for Computational Linguistics, 188–197.  https://doi.org/10.18653/v1/D19-1018
  * Pedregon et al. (2012) Cynthia A. Pedregon, Roberta L. Farley, Allison Davis, James M. Wood, and Russell D. Clark. 2012\.  Social desirability, personality questionnaires, and the “better than average” effect.  _Personality and Individual Differences_ 52, 2 (2012), 213–217.  https://doi.org/10.1016/j.paid.2011.10.022
  * Pennebaker and King (1999) James W Pennebaker and Laura A King. 1999.  Linguistic styles: language use as an individual difference.  _Journal of personality and social psychology_ 77, 6 (1999), 1296.
  * Rammstedt and John (2007) Beatrice Rammstedt and Oliver P John. 2007.  Measuring personality in one minute or less: A 10-item short version of the Big Five Inventory in English and German.  _Journal of research in Personality_ 41, 1 (2007), 203–212.
  * Stajner and Yenikent (2020) Sanja Stajner and Seren Yenikent. 2020.  A Survey of Automatic Personality Detection from Texts. In _Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020_ , Donia Scott, Núria Bel, and Chengqing Zong (Eds.). International Committee on Computational Linguistics, 6284–6295.  https://doi.org/10.18653/v1/2020.coling-main.553
  * Topolewska et al. (2014) Ewa Topolewska, Ewa Skimina, WŁODZIMIERZ Strus, Jan Cieciuch, and Tomasz Rowiński. 2014\.  The short IPIP-BFM-20 questionnaire for measuring the Big Five.  _Roczniki Psychologiczne_ 17, 2 (2014), 385–402.
  * Winoto and Tang (2010) Pinata Winoto and Tiffany Ya Tang. 2010.  The role of user mood in movie recommendations.  _Expert Syst. Appl._ 37, 8 (2010), 6086–6092.  https://doi.org/10.1016/j.eswa.2010.02.117
  * Wu et al. (2018) Wen Wu, Li Chen, and Yu Zhao. 2018.  Personalizing recommendation diversity based on user personality.  _User Model. User Adapt. Interact._ 28, 3 (2018), 237–276.  https://doi.org/10.1007/s11257-018-9205-x
  * Yang and Huang (2019) Hsin-Chang Yang and Zi-Rui Huang. 2019.  Mining personality traits from social messages for game recommender systems.  _Knowl. Based Syst._ 165 (2019), 157–168.  https://doi.org/10.1016/j.knosys.2018.11.025
  * Youyou et al. (2017) Wu Youyou, David Stillwell, H. Andrew Schwartz, and Michal Kosinski. 2017.  Birds of a Feather Do Flock Together: Behavior-Based Personality-Assessment Method Reveals Personality Similarity Among Couples and Friends.  _Psychological Science_ 28, 3 (2017), 276–284.  https://doi.org/10.1177/0956797616678187 arXiv:https://doi.org/10.1177/0956797616678187  PMID: 28059682.