# Lumen: A Machine Learning Framework to Expose Influence Cues in Texts

Hanyu Shi*, Mirela Silva*, Daniel Capecci, Luiz Giovanini, Lauren Czech,
Juliana Fernandes, and Daniela Oliveira  H. Shi, M. Silva (corresponding
author), D. Capecci, L. Giovanini, L. Czech, and D. Oliveira are with the
Department of Electrical and Computer Engineering, University of Florida,
Gainesville, FL, USA, 32611 (E-mail: msilva1@ufl.edu).J. Fernandes is with the
Department of Advertising, University of Florida, Gainesville, FL, USA,
326011.* The first two authors have equal contribution.

###### Abstract

Phishing and disinformation are popular social engineering attacks with
attackers invariably applying influence cues in texts to make them more
appealing to users. We introduce Lumen, a learning-based framework that
exposes influence cues in text: (i) persuasion, (ii) framing, (iii) emotion,
(iv) objectivity/subjectivity, (v) guilt/blame, and (vi) use of emphasis.
Lumen was trained with a newly developed dataset of 3K texts comprised of
disinformation, phishing, hyperpartisan news, and mainstream news. Evaluation
of Lumen in comparison to other learning models showed that Lumen and LSTM
presented the best F1-micro score, but Lumen yielded better interpretability.
Our results highlight the promise of ML to expose influence cues in text,
towards the goal of application in automatic labeling tools to improve the
accuracy of human-based detection and reduce the likelihood of users falling
for deceptive online content.

###### Index Terms:

Influence cues, machine learning, natural language processing, phishing,
misinformation, deception detection.

##  1 Introduction

The Web has increasingly become an ecosystem for deception. Beyond social
engineering attacks such as phishing which put Internet users and even
national security at great peril [1, 2], false information is greatly shaping
the political, social, and economic landscapes of our society, exacerbated and
brought to light in recent years by social media. Recent years have
undoubtedly brought to light the dangers of selective exposure111A theory akin
to confirmation bias and often used in Communication research pertaining to
the idea that individuals favor information that reinforces their prior
beliefs [6]., and false content can increase individuals’ beliefs in the
falsehood [3]. These deceptive and divisive misuses of online media have
evolved the previously seemingly tacit political lines to the forefront of our
very own individual identities [4], thus raising concern for the anti-
democracy effects caused by this polarization of our society [5].

A key invariant of deceptive content is the application of influence cues in
the text. Research on deception detection [7, 8, 9, 10, 11] reveals that
deceivers apply influence cues in messages to increase their appeal to the
recipients. We posit several types of influence cues that are relevant and
prevalent in deceptive texts: (i) the principle of persuasion applied [7, 12]
(e.g., authority, scarcity), (ii) the framing of the message as either
potentially causing a gain or a loss [10, 11], (iii) the positive/negative
emotional salience/valence of the content [8, 9],  (iv) the subjectivity or
objectivity of sentences in the text, (v) attribution of blame/guilt, and (vi)
the use of emphasis.

Additionally, works such as Ross et al. [3] found that the ability to think
deliberately and analytically (i.e., “System 2” [11]) is generally associated
with the rejection of disinformation, regardless of the participants’
political alignment—thus, the activation of this analytical thinking mode may
act as an “antidote” to today’s selective exposure. We therefore advocate that
interventions should mitigate deceptive content via the exposure of influence
cues in texts. Similar to the government and state-affiliated media account
labels on Twitter [13], bringing awareness to the influence cues present in
misleading texts may, in turn, aid users by providing additional context in
the message, thus helping users think analytically, and benefit future work
aimed at the automatic detection of deceptive online content.

Towards this goal, we introduce Lumen222From Latin, meaning “to illuminate.”,
a two-layer learning framework that exposes influence cues in text using a
novel combination of well-known existing methods: (i) topic modeling to
extract structural features in text; (ii) sentiment analysis to extract
emotional salience; (iii) LIWC 333LIWC [14] is a transparent text analysis
program that counts words in psychologically meaningful categories and is
widely used to quantify psychometric characteristics of raw text data.to
extract dictionary features related to influence cues; and (iv) a
classification model to leverage the extracted features to predict the
presence of influence cues. To evaluate Lumen’s effectiveness, we leveraged
our dataset of 2,771 diverse pieces of online texts, manually labeled by our
research team according to the influence cues in the text using standard
qualitative analysis methods. We must, however, emphasize that Lumen is _not_
a consumer-focused end-product, and instead is insomuch as a module for
application in future user tools that we shall make publicly available to be
leveraged by researchers in future work (as described in Sec. 6.2).

Our newly developed dataset is comprised of nearly 3K texts, where 1K were
mainstream news articles, and 2K deceptive or misleading content in the form
of: Russia’s Internet Research Agency’s (IRA) propaganda targeting Americans
in the 2016 U.S. Presidential Election, phishing emails, and fake and
hyperpartisan news articles. Here, we briefly define these terms, which we
argue fall within the same “deceptive text umbrella.” _Disinformation_
constitutes any any _purposefully_ deceptive content aimed at altering the
opinion of or confusing an individual or group. Within disinformation, we find
instances of _propaganda_ (facts, rumors, half-truths, or lies disseminated
manipulatively for the purpose of influencing public opinion [15]) and _fake
news_ (fabricated information that mimics real online news [3] and
considerably overlaps with hyperpartisan news [5]). Misinformation’s subtler,
political form is _hyperpartisan news_ , which entails a misleading coverage
of factual events through the lens of a strong partisan bias, typically
challenging mainstream narratives [3, 5]. _Phishing_ is a social engineering
attack aimed at influencing users via deceptive arguments into an action
(e.g., clicking on a malicious link) that will go against the user’s best
interests.  Though phishing differs from disinformation in its modus operandi,
we argue that it overlaps with misleading media in their main purpose—to
galvanize users into clicking a link or button by triggering the victim’s
emotions [5], and leveraging influence and deception.

We conducted a quantitative analysis of the dataset, which showed that
authority and commitment were the most common principles of persuasion in the
dataset (71% and 52%, respectively), the latter of which was especially common
in news articles. Phishing emails had the largest occurrence of scarcity
(65%). Framing was a relatively rare occurrence (13% gain and 7% loss), though
gain framing was predominantly prevalent in phishing emails (41%). The dataset
invoked an overall positive sentiment (VADER compound score of $0.232$), with
phishing emails containing the most positive average sentiment ($0.635$) and
fake news with the most negative average sentiment ($-0.163$). Objectivity and
subjectivity occurred in over half of the dataset, with objectivity most
prevalent in fake news articles (72%) and subjectivity most common in IRA ads
(77%). Attribution of blame/guilt was disproportionately frequent for fake and
hyperpartisan news (between 38 and 45%). The use of emphasis was much more
common in informal texts (e.g., IRA social media ads, 70%), and less common in
news articles (e.g., mainstream media, 17%).

We evaluated Lumen in comparison with other traditional ML and deep learning
algorithms. Lumen presented the best performance in terms of its $F1$-micro
score (69.23%), performing similarly to LSTM (69.48%). In terms of $F1$-macro,
LSTM (64.20%) performed better than Lumen (58.30%); however, Lumen presented
better interpretability for intuitively understanding of the model, as it
provides both the relative importance of each feature and the topic structure
of the training dataset without additional computational costs, which cannot
be obtained with LSTM as it operates as a black-box. Our results highlight the
promise of exposing influence cues in text via learning methods.

33footnotetext: Lumen and dataset will be available upon publication.

This paper is organized as follows. Section 2 positions this paper’s
contributions in comparison to related work in the field. Section 3 details
the methodology used to generate our coded dataset. Section 4 describes
Lumen’s design and implementation, as well as Lumen’s experimental evaluation.
Section 5 contains a quantitative analysis of our dataset, and Lumen’s
evaluation and performance. Section 6 summarizes our findings and discusses
the limitations of our work, as well as recommendations for future work.
Section 7 concludes the paper.

##  2 Related Work

This section briefly summarizes the extensive body of work on machine learning
methods to automatically detect disinformation and hyperpartisan news, and
initial efforts to detect the presence of influence cues in text.

###  2.1 Automatic Detection of Deceptive Text

####  2.1.1 Phishing and Spam

Most anti-phishing research has focused on automatic detection of malicious
messages and URLs before they reach a user’s inbox via a combination of
blocklists [16, 17] and ML [18, 19]. Despite yielding high filtering rates in
practice, these approaches cannot prevent zero-day phishing444A new, not-yet
reported phishing email.from reaching users because determining maliciousness
of text is an open problem and phishing constantly changes, rendering learning
models and blocklists outdated in a short period of time [19]. Unless the same
message has been previously reported to an email provider as malicious by a
user or the provider has the embedded URL in its blocklist, determining
maliciousness is extremely challenging. Furthermore, the traditional approach
to automatically detect phishing takes a binary standpoint (phishing or
legitimate, e.g., [20, 21, 22]), potentially overlooking distinctive nuances
and the sheer diversity of malicious messages.

Given the limitations of automated detection in handling zero-day phishing,
human detection has been proposed as a complementary strategy. The goal is to
either warn users about issues with security indicators in web sites, which
could be landing pages of malicious URLs [23, 24] or train users into
recognizing malicious content online [25]. These approaches are not without
their own limitations. For example, research on the effectiveness of SSL
warnings shows that users either habituate or tend to ignore warnings due to
false positives or a lack of understanding about the warning message [26, 27].

####  2.1.2 Fake & Hyperpartisan Media

The previously known “antidote” to reduce polarization and increase readers’
tolerance to selective exposure was via the use of counter-dispositional
information [5]. However, countering misleading texts with mainstream or high-
quality content in the age of rapid-fire social media comes with logistical
and nuanced difficulties. Pennycook and Rand [28] provide a thorough review of
the three main approaches employed in fighting misinformation: automatic
detection, debunking by field experts (which is not scalable), and exposing
the publisher of the news source.

Similar to zero-day phishing, disinformation is constantly morphing, such that
“zero-day” disinformation may thwart already-established algorithms, such was
the case with the COVID-19 pandemic [28]. Additionally, the final
determination of a _fake, true_ , or _hyperpartisan_ label is fraught with
subjectivity. Even fact-checkers are not immune—their agreement rates plummet
for ambiguous statements [29], calling into question their efficacy in
hyperpartisan news.

We posit that one facet of the solution lies within the combination of human
and automated detection. Pennycook and Rand [28] conclude that lack of careful
reasoning and domain knowledge is linked to poor truth discernment, suggesting
(alongside [3, 30]) that future work should aim to trigger users to to think
slowly and analytically [11] while assessing the accuracy of the information
presented. Lumen aims to fulfill the first step of this goal, as our framework
exposes influence cues in texts, which we hypothesize is disproportionately
leveraged in deceptive content.

###  2.2 Detecting Influence in Deceptive Texts

####  2.2.1 Phishing

We focus on prior work that has investigated the extent to which Cialdini’s
principles of persuasion (PoP) [12, 7] (described in Sec. 3) are used in
phishing emails [31, 32, 33] and how users are susceptible to them [34, 35].

Lawson et al. [35] leveraged a personality inventory and an email
identification task to investigate the relationship between personality and
Cialdini’s PoP. The authors found that _extroversion_ was significantly
correlated with increased susceptibility to _commitment, liking,_ and the pair
_(authority, commitment)_ , the latter of which was found in 41% of our
dataset. Following Cialdini’s PoP, after manually labeling $\sim$200 phishing
emails, Akbar [36] found that _authority_ was the most frequent principle in
the phishing emails, followed by _scarcity_ , corroborating our findings for
high prevalence of _authority_. However, in a large-scale phishing email study
with more than 2,000 participants, Wright et al. [37] found that _liking_
receives the highest phishing response rate, while _authority_ received the
lowest. Oliveira et al. [34, 38] unraveled the complicated relationship
between PoP and Internet user age and susceptibility, finding that young users
are most vulnerable to _scarcity_ , while older ones are most likely to fall
for _reciprocation_ , with _authority_ highly effective for both age groups.
These results are promising in highlighting the potential usability of
exposing influence cues to users.

####  2.2.2 Fake & Hyperpartisan News

Contrary to phishing, few studies have focused on detecting influence cues or
analyzing how users are susceptible to them in the context of fake or highly
partisan content. Xu et al. [39] stands out as the authors used a mixed-
methods analysis, leveraging both manual analysis of the textual content of
1.2K immigration-related news articles from 17 different news outlets, and
computational linguistics (including, as we did, LIWC). The authors found that
moral frames that emphasize that support _authority/respect_ were shared/liked
more, while the opposite occurred for _reciprocity/fairness_. Whereas we
solely used trained coders, they measured the aforementioned frames by
applying the moral foundations dictionary [40].

To the best of our knowledge, no prior work has investigated or attempted to
automatically detect influence cues in texts in such a large dataset,
containing multiple types of deceptive texts. In this work, we go beyond
Cialdini’s principles to also detect gain and loss framing, emotional
salience, subjectivity and objectivity, and the use of emphasis and blame.
Further, no prior work has made available to the research community a dataset
of deceptive texts labeled according to the influence cues applied in the
text.

##  3 Dataset Curation & Coding Methodology

This section describes the methodology to generate the labeled dataset of
online texts used to train Lumen, including the definition of each of the
influence cues labels.

###  3.1 Curating the Dataset

We composed a diverse dataset by gathering different types of texts from
multiple sources, split into three groups: (Deceptive Texts) $1,082$ pieces of
text containing disinformation and/or deception tactics, (Hyperpartisan News)
$1,003$ hyperpartisan media news from politically right- and left-leaning
publications, and (Mainstream News) $974$ center mainstream media news. Our
dataset therefore contained $3,059$ pieces of text in total.

For the Deceptive Texts Group, we mixed $492$ Facebook ads created by the
Russian Internet Research Agency (IRA), $130$ known fake news articles, and
$460$ phishing emails:

Facebook IRA Ads. We leveraged a dataset of 3,517 Facebook ads created by the
Russian IRA and made publicly available to the U.S. House of Representatives
Permanent Select Committee on Intelligence [41] by Facebook after internal
audits. These ads were a small representative sample of over 80K organic
content identified by the Committee and are estimated to have been exposed to
over 126M Americans between June 2015 and August 2017. After discarding ads
that did not have text entry, the dataset was decreased to 3,286 ads, which
were mostly (52.8%) posted in 2016 (U.S. election year).We randomly selected
$492$ for inclusion.

Fake News. We leveraged a publicly available555https://ieee-dataport.org/open-
access/fnid-fake-news-inference-dataset#filesdataset of nearly 17K news
labeled as _fake_ or _real_ collected by Sadeghi et al. [42] from
PoliticFact.com, a reputable source of fact-finding. We randomly selected
$130$ _fake_ news ranging from $110-200$ words dated between 2007 to 2020.

Phishing Emails. To gather our dataset, we collected approximately 15K known
phishing emails from multiple public sources [43, 44, 45, 46, 47, 48, 49, 50].
The emails were then cleaned and formatted to remove errors, noise (e.g.,
images, HTML elements), and any extraneous formatting so that only the raw
email text remained. We randomly selected $460$ of these emails ranging from
$50-150$ words to be included as part of the Deceptive Texts.

For the Hyperpartisan News and Mainstream News Groups, we used a public
dataset666https://components.one/datasets/all-the-news-2-news-articles-
dataset/comprised of 2.7M news articles and essays from 27 American
publications dated from 2013 to early 2020. We first selected articles ranging
from $50-200$ words and then classified them as _left_ , _right_ , or _center_
news according to the AllSides Bias Rating777https://www.allsides.com/media-
bias/media-bias-ratings. For inclusion in the Hyperpartisan News Group, we
randomly selected $506$ _right_ news and $497$ _left_ news; the former were
dated from 2016 to 2017 and came from two publications sources (Breitbart and
National Review) while the latter were dated from 2016 to 2019 and came from
six publications (Buzzfeed News, Mashable, New Yorker, People, VICE, and Vox).
To compose Mainstream News Group, we randomly selected $974$ _center_ news
from all seven publications (Business Insider, CNBC, NPR, Reuters, TechCrunch,
The Hill, and Wired) dated from 2014 to 2019.

####  3.1.1 Coding Process

We then developed coding categories and a codebook based on Cialdiani’s
principles of influence [51], subjectivity/objectivity, and gain/loss framing
[52]. These categories have been used in prior works (e.g., [32, 34], and were
adapted for the purposes of this study, as well as with the additional
emphasis and blame/guilt attribution categories. Next, we held an initial
training session with nine undergraduate students. The training involved a
thorough description of the coding categories, their definitions and
operationalizations, as well as a workshop-style training where coders labeled
a small sample of the texts to get acquainted with the coding platform, the
codebook, and the texts. Coders were instructed to read the text at least
twice before starting the coding to ensure they understood it. After that,
coders were asked to share their experiences labeling the texts and to discuss
any issues or questions about the process. After this training session, two
intercoder reliability pretests were conducted; in the first pretest, coders
independently co-coded a sample of $20$ texts, and in the second pretest,
coders independently co-coded a sample of $40$ texts. After each one of these
pretests, a discussion and new training session followed to clarify any issues
with the categories and codebook.

Following these additional discussion and training sessions, coders were then
instructed to co-code $260$ texts which served as our intercoder reliability
sample. To calculate intercoder reliability, we used three indexes. _Cohen’s
kappa_ and _Percent of Agreement_ ranged from $0.40$ to $0.90$, and 66% to
99%, respectively, which was considered moderately satisfactory. Due to the
nature of the coding and type of texts, we also opted to use _Perrault and
Leigh’s index_ because (a) it has been used in similar studies that also use
nominal data [53, 54, 55, 56]; (b) it is the most appropriate reliability
measure for $0/1$ coding (i.e., when coders mark for absence or presence of
given categories), as traditional approaches do not take into consideration
two zeros as agreement and thus penalize reliability even if coders disagree
only a few times [57]; and (c) indexes such as Cohen’s kappa and Scott’s pi
have been criticized for being overly conservative and difficult to compare to
other indexes of reliability [58]. Perrault and Leigh’s index ($I_{r}$)
returned a range of $0.67$ to $0.99$, which was considered satisfactory.
Finally, the remaining texts were divided equally between all coders, who
coded all the texts independently using an electronic coding sheet in
Qualtrics. Coders were instructed to distribute their workload equally over
the coding period to counteract possible fatigue effects. This coding process
lasted three months.

####  3.1.2 Influence Cues Definitions

The coding categories were divided into five main concepts: _principles of
influence_ , _gain/loss framing_ , _objectivity/subjectivity_ , _attribution
of guilt_ , and _emphasis_. Coders marked for the absence ($0$) or presence
($1$) of each of the categories. Definitions and examples for each influence
are detailed in Appendix A, leveraged from the coding manual we curated to
train our group of coders.

Principles of Persuasion (PoP). _Persuasion_ refers to a set of principles
that influence how people concede or comply with requests. The principles of
influence were based on Cialdini’s marketing research work [7, 12], and
consist of the following six principles: (i) _authority 888e.g., people tend
to comply with requests or accept arguments made by figures of authority._ or
expertise, (ii) _reciprocation_ , (iii) _commitment_ and consistency, (iv)
_liking_ , (v) _scarcity_ , and (vi) _social proof_. We added subcategories to
the principles of commitment (i.e., indignation and call to action) and social
proof (i.e., admonition) because an initial perusal of texts revealed
consistent usage across texts.

Framing. Framing refers to the presentation of a message (e.g., health
message, financial options, and advertisement) as implying a possible gain
(i.e., possible benefits of performing the action) vs. implying a possible
loss (i.e., costs of not performing a behavior) [10, 59, 11]. Framing can
affect decision-making and behavior; work by Kahneman and Tversky [11] on loss
aversion supports the human tendency to prefer avoiding losses over acquiring
equivalent gains.

Slant. Slant refers to whether a text is written subjectively or objectively;
_subjective_ sentences generally refer to a personal opinion/judgment or
emotion, whereas _objective_ sentences fired to factual information that is
based on evidence, or when evidence is presented. It is important to note that
we did not ask our coders to fact check, instead asking them to rely on
sentence structure, grammar, and semantics to determine the label of
_objective_ or _subjective_.

Attribution of Blame/Guilt. Blame or guilt refers to when a text references
“another” person/object/idea for wrong or bad things that have happened.

Emphasis. Emphasis refers to the use of all caps text, exclamation points
(either one or multiple), several question marks, bold text, italics text, or
anything that is used to call attention in text.

##  4 Lumen Design and Implementation

This section describes the design, implementation and evaluation of Lumen, our
proposed two-level learning-based framework to expose influence cues in texts.

###  4.1 Lumen Overview

Exposing presence of persuasion and framing is tackled as a multi-labeling
document classification problem, where zero, one, or more labels can be
assigned to each document. Due to recent developments in natural language
processing, emotional salience is an input feature that Lumen exposes
leveraging sentiment analysis. Note that Lumen’s goal is not to distinguish
deceptive vs. benign texts, but to expose different influence cues applied in
different types of texts.

Figure 1 illustrates Lumen’s two-level hierarchical learning- based
architecture. On the first level, the following features are extracted from
the raw text: (i) topical structure inferred by topic modeling, (ii) LIWC
features related to influence keywords [14], and (iii) emotional salience
features learned via sentiment analysis [60]. On the second level, a
classification model is used to identify the influence cues existing in the
text.

Figure 1: Lumen’s two-level architecture. Pre-processed text undergoes
sentiment analysis for extraction of emotional salience, LIWC analysis for
extraction of features related to influence keywords, and topic modeling for
structural features. These features are inputs to ML analysis for prediction
of influence cues applied to the message.

###  4.2 Topic Structure Features

Probabilistic topic modeling algorithms are often used to infer the topic
structure of unstructured text data [61, 62], which in our case are deceptive
texts, hyperpartisan news, and mainstream news. Generally, these algorithms
assume that a collection of documents (i.e., a corpus) are created following
the generative process.

Suppose that there are $D$ documents in the corpus $\mathcal{C}$ and each
document $d=1,...,D$ has length $m_{d}$. Also suppose that there are in total
$K$ different topics in the corpus and the vocabulary $\mathcal{V}$ includes
$V$ unique words. The relations between documents and topics are determined by
conditional probabilities $P(t|d)$, which specify the probability of topic
$t=1,...,K$ given document $d$. The linkage between topics and unique words
are established by conditional probabilities $P(w|t)$, which indicate the
probability of word $w=1,...,V$ given topic $t$. According to the generative
process, for each token $w(i_{d})$, which denotes the $i_{d}$-th word in
document $d$, we will first obtain the topic of this token, $z(i_{d})=t$,
according to $P(t|d)$. With the obtained $z(i_{d})$, we then draw the a word
$w(i_{d})=w$ according to $P(w|t=z(i_{d}))$ .

In this work, we leveraged Latent Dririchlet Allocation (LDA), one of the most
widely used topic modeling algorithms, to infer topic structure in texts [63].
In LDA, both $P(w|t)$ and $P(t|d)$ are assumed to have Dirichlet prior
distributions. Given our dataset, which is the evidence to the probabilistic
model, the goal of LDA is to infer the most likely conditional distribution
$\hat{P}(w|t)$ and $\hat{P}(t|d)$, which is usually done by either variational
Bayesian approach [63] or Gibbs Sampling [64]. In Lumen, the conditional
probabilities $\hat{P}(t|d)$ represent the topic structure of the dataset.

###  4.3 LIWC Influence Features

We use language to convey our thoughts, intentions, and emotions, with words
serving as the basic building blocks of languages. Thus, the way different
words are used in unstructured text data provide meaningful information to
streamline our understanding of the use of influence cues in text data. Lumen
thus leverages LIWC, a natural language processing framework that connects
commonly-used words with categories [65, 14] to retrieve influence features of
texts to aid ML classification. LIWC includes more than 70 different
categories in total, such as Perceptual Processes, Grammar, and Affect, and
more than 6K common words.

However, not all the categories are related to influence. After careful
inspection, we manually selected seven categories as features related to
influence for Lumen. For persuasion, we selected the category _time_ (related
to _scarcity_); for emotion, we selected the categories _anxiety, anger,_ and
_sad_ ; and for framing, we selected the categories _reward_ and _money_
(gain), and _risk_ (loss).

We denote the collection of the chosen LIWC categories as set $\mathcal{S}$.
Given a text document $d$ with document length $m_{d}$ from the corpus
$\mathcal{C}$, to build the LIWC feature $X^{LIWC}_{i,d},\forall
i\in\mathcal{S}$, we first count the number of words in the text $d$ belonging
to the LIWC category $i$, denoted as $n_{i,d}$, then normalize the raw word
count with the document length:

$X^{LIWC}_{i,d}=\frac{n_{i,d}}{m_{d}},\forall
i\in\mathcal{S},d\in\mathcal{C}.$ (1)

###  4.4 Emotional Salience by Sentiment Analysis

Emotional salience refers to both valence (positive to negative) and arousal
(arousing to calming) of an experience or stimulus [8, 66, 9], and research
has shown that deception detection is reduced for emotional compared to
neutral stimuli [66]. Similarly, persuasion messages that generate high
(compared to low) arousal lead to poorer consumer decision-making [9].
Emotional salience may impair full processing of deceptive content and high
arousal content may trigger System 1, the fast, shortcut-based brain
processing mode [67].

In this work, we used a pre-trained rule-based model, VADER, to extract the
emotional salience and valence from a document [60]. Both levels of emotion
range from 0 to 1, where a small value means low emotional levels and a large
number means high emotional levels. Therefore, emotional salience is both an
input feature to the learning model and one of Lumen’s outputs (see Fig. 1).

###  4.5 Machine Learning to Predict Persuasion & Framing

Lumen’s second level corresponds to the application of a general-purpose ML
algorithm for document classification. Although Lumen is general enough to
allow application of any general-purpose algorithm, in this paper, we applied
Random Forest (RF) because it can provide the level of importance for each
input predicative feature without additional computational cost, which aids in
model understanding. Another advantage of RF is its robustness to the
magnitudes of input predicative features, i.e., RF does not need feature
normalization. We use the grid search approach to fine-tune the parameters in
the RF model and follow the cross-validation to overcome any over-fitting
issues of the model.

####  4.5.1 Dataset Pre-Processing

As described previously, Lumen generates three types of features at its first
hierarchical level (emotional salience, LIWC categories, and topic structure),
which serve as input for the learning-based prediction algorithm (Random
Forest, for this analysis) at Lumen’s second hierarchical level (Fig. 1);
these features rely on the unstructured texts in the dataset. However,
different features need distinct preprocessing procedures. In our work, we
used the Natural Language Toolkit (NLTK) [68] to pre-process the dataset. For
all three types of features, we first removed all the punctuation, special
characters, digital numbers, and words with only one or two characters. Next,
we tokenized each document into a list of lowercase words.

For topic modeling features, we removed stopwords (which provide little
semantic information) and applied stemming (replacing a word’s inflected form
with its stem form) to further clean-up word tokens. For LIWC features, we
matched each word in each text with the pre-determined word list in each LIWC
category; we also performed stemming for LIWC features. We did not need to
perform pre-processing for emotional salience because we applied NLTK [60],
which has its own tokenization and pre-processing procedures.

Additionally, we filtered out documents with less than ten words since topic
modeling results for extremely short documents are not reliable [69]. We were
then left with 2,771 cleaned documents, with 183,442 tokens across the corpus,
and 14,938 unique words in the vocabulary.

####  4.5.2 Training and Testing

Next, we split the the 2,771 documents into a training and a testing set. In
learning models, hyper-parameters are of crucial importance because they
control the structure or learning processing of the algorithms. Lumen applies
two learning algorithms: an unsupervised topic modeling algorithm, LDA, on the
first hierarchical level and RF on the second level. Each algorithm introduces
its own types of hyper-parameters; for LDA, examples include the number of
topics and the concentration parameters for Dirichlet distributions, whereas
for RF are the number of trees and the maximum depth of a tree. We also used
the grid search approach to find a better combination of hyper-parameters.
Note that due to time and computational power constraints, it is impossible to
search all hyper-parameters and all their potential values. In this work, we
only performed the grid search for number of topics (LDA) and the number of
trees (RF). The results show that the optimal number of topics is 10 and the
optimal number of trees in RF is 200. Note also that the optimal result is
limited by the grid search space, which only contains a finite size of
parameter combinations.

If we only trained and tested Lumen on one single pair of training and testing
sets, there would be high risk of overfitting. To lower this risk, we used
5-fold cross-validation, wherein the final performance of the learning
algorithm is the average performance over the five training and testing pairs.

####  4.5.3 Evaluation Metrics

To evaluate our results (Sec. 5), we compared Lumen’s performance in
predicting the influence cues applied to a given document with three other
document classification algorithms: (i) Labeled-LDA, (ii) LSTM, and (iii)
naïve algorithm.

Labeled-LDA is a semi-supervised variation of the original LDA algorithm [70,
71]. When training the Labeled-LDA, both the raw document and the human coded
labels for influence cues were input into the model. Compared to Lumen,
Labeled-LDA only uses the word frequency information from the raw text data
and has a very rigid assumption of the relation between the word frequency
information and the coded labels, which limits its flexibility and prediction
ability.

Long Short-Term Memory (LSTM) takes the input data recurrently, regulates the
flow of information, and determines what to pass on to the next processing
step and what to forget. Since neural networks mainly deal with vector
operations, we used 50-dimensional word embedding matrix to transfer each word
into vector space [72]. The main shortcoming of neural network is that it
works as a blackbox, making it difficult to understand the underlying
mechanism.

The naïve algorithm served as a base line for our evaluation. We randomly
generated each label for each document according to a Bernoulli distribution
with equal probabilities for two outcomes.

As shown in Table I, we used $F1$-score (following the work by Ramage et al.
[70] and van der Heijden et al. [71]), and accuracy rate to quantify the
performance of the algorithms. We note that the comparison of $F$-scores is
only meaningful under the same experiment setup. It would be uninformative to
compare F-scores from distinctive experiments in different pieces of work in
the literature due to varying experiment conditions. $F1$-score can be easily
calculated for single-labeling classification problems, where each document
will only be assigned to one label. However, in our work, we are dealing with
a multi-labeling classification problem, which means that no limit is imposed
on how many labels each document can include. Thus, we employed two variations
of the $F1$-score to quantify the overall performance of the learning
algorithm: macro and micro $F1$-scores.

##  5 Results

This section details Lumen’s evaluation. First, we provide a quantitative
analysis of our newly developed dataset used to train Lumen, and the results
of Lumen classification in comparison to other ML algorithms.

###  5.1 Quantitative Analysis of the Dataset

We first begin by quantifying the curated dataset of 2,771 deceptive,
hyperpartisan, or mainstream texts, hand-labeled by a group of coders. When
considering all influence cues, most texts used between three and six cues per
texts; only 3% of all texts leveraged a single influence cue, and 2% used zero
cues ($n=58$).

When considering the most common pairs and triplets between all influence
cues, slant (i.e., _subjectivity_ or _objectivity_) and principles of
persuasion (PoP) dominated the top 10 most common pairings and triplets. As
such, the most common pairs were _(authority, objectivity)_ and _(authority,
subjectivity)_ , occurring for 48% and 45% of all texts, respectively. The
most common (PoP, PoP) pairing was between _authority_ and _commitment_ , co-
occurring in 41% of all texts. _Emphasis_ appeared once in the top 10 pairs
and twice in the top triplets: _(emphasis, subjectivity)_ occurring for 29% of
texts, and _(emphasis, authority, subjectivity)_ and _(emphasis, commitment,
subjectivity)_ for 20% and 19% of texts, respectively. _Blame/guilt_ appeared
only once in the top triplets as _(authority, blame/guilt, objectivity)_ ,
representing 19% of all texts. _Gain_ framing appeared only as the 33rd most
common pair _(gain, subjectivity)_ and 18th most common triplet _(call to
action, scarcity, gain)_ , further emphasizing its scarcity in our dataset.

####  5.1.1 Principles of Persuasion

We found that most texts in the dataset contained one to four principles of
persuasion, with only 4% containing zero and 3% containing six or more PoP
labels; 29% of texts apply two PoP and 23% leverage three PoP. Further, Fig. 2
shows that _authority_ and _commitment_ were the most prevalent principles
appearing, respectively, in 71% and 52% of the texts; meanwhile,
_reciprocation_ and _indignation_ were the least common PoP (5% and 9%,
respectively).

Almost all types of texts contained every PoP to varying degrees; the only
exception is _reciprocation_ (the least-used PoP overall) which was not at all
present in fake news texts (in the Deceptive Texts Group) and barely present
($n=3$, 0.6%) in right-leaning hyperpartisan news. _Authority_ was the most-
used PoP for all types of texts, except phishing emails (most: _call to
action_) and IRA ads (most: _commitment_), both of which are in the Deceptive
Texts Group.

,

Figure 2: The total for each text type, broken down based on principles of
persuasion.

Deceptive Texts. Fake news was notably reliant on _authority_ (92% of all fake
news leveraged the _authority_ label) compared to phishing emails (45%) and
the IRA ads (32%); however, fake news used _liking, reciprocation_ , and
_scarcity_ (5%, 0%, 3%, respectively) much less often than phishing emails
(27%, 8%, 65%) or IRA ads (41%, 10%, 24%). Interestingly, _admonition_ was
most used by fake news (35%), though overall, _admonition_ was only present in
14% of all texts. Phishing emails were noticeably more reliant on _call to
action_ (80%) and _scarcity_ (65%) compared to fake news (33%, 3%) and IRA ads
(40%, 24%), yet barely used _indignation_ (0.4%) compared to the same (13% for
fake news and 17% for IRA ads). The IRA ads relied on _indignation, liking,
reciprocation_ , and _social proof_ much more than the others; note again that
_reciprocation_ was the least occurring PoP (5% overall), but was most
commonly occurring in IRA ads (10%).

Hyperpartisan News. Right-leaning texts had nearly twice as much _call to
action_ and _indignation_ than left-leaning texts (61% and 19% vs. 31% and 8%,
respectively). Meanwhile, left-leaning hyperpartisan texts had noticeably more
_liking_ (30% vs. 13%), _reciprocation_ (8% vs. 0.6%), and _scarcity_ (27% vs.
13%) than right-leaning texts.

Mainstream News. _Authority_ (88%) and _commitment_ (43%) were the most
frequently appearing PoP in center news, though this represents the third
highest occurrence of _authority_ and lowest use of _commitment_ across all
six text type groups. Mainstream news also used very little _indignation_ (3%)
compared the the other text types except phishing emails (0.4%), and also
demonstrated the lowest use of _social proof_ (7%).

_Authority_ and _commitment_ were the most common PoP in the dataset, with the
former most common in fake news articles. Phishing emails had the largest
occurrence of _scarcity_).

####  5.1.2 Framing

There were few _gain_ or _loss_ labels for the overall dataset (only 13% and
7%, respectively). Very few texts (18%) were framed exclusively as either
_gain_ or _loss_ , 81% did not include any framing at all, and only 1% of the
texts used both _gain_ and _loss_ framing in the same message. We also found
that _gain_ was much more prevalent than _loss_ across all types of texts,
except for fake news, which showed an equal amount (1.5% for both _gain_ and
_loss_). Notably, phishing emails had significantly more _gain_ and _loss_
than any other text type (41% and 29%, respectively); mainstream center news
and IRA ads showed some use of _gain_ framing (10% and 13%, respectively)
compared to the remaining text types.

Next, we investigated how persuasion and framing were used in texts by
analyzing the pairs and triplets between the two influence cues. _Gain_
framing most frequently occurred with _call to action_ and _commitment_ ,
though these represent only 9% of pairings. _Gain, call to action_ and
_scarcity_ was the the most common triplet between PoP and framing, occurring
for 7% of all texts—this is notable as phishing emails had _call to action_
and _scarcity_ as its top PoP, and _gain_ framing was also most prevalent in
phishing. Also of note is that _loss_ appeared in even fewer common pairs and
triplets compared to _gain_ (e.g., _loss_ and _call to action_ appeared in
just 5% of texts).

Framing was a relatively rare occurrence in the dataset, though predominantly
present in phishing emails, wherein _gain_ was invoked $1.5\times$ more often
than _loss_.

####  5.1.3 Emotional Salience

We used VADER’s compound sentiment score ($E$, wherein $E\geq 0.05$, ,
$E\leq-0.05$, and $-0.05<E<0.05$ denote positive, negative, and neutral
sentiment, respectively) and LIWC’s positive and negative emotion word count
metrics to measure sentiment. Overall, our dataset was slightly positive in
terms of average compound sentiment ($\mu=0.23$) and with an average of 4.0
positive emotion words and 1.7 negative emotion words per text.

In terms of specific text types, fake news contained the only negative average
compound sentiment ($-0.163$), and right-leaning hyperpartisan news had the
only neutral average compound sentiment ($0.015$); all other text types had,
on average, positive sentiment, with phishing emails as the most positive text
type ($0.635$). Left-leaning hyperpartisan news had the highest average
positive emotion word count ($5.649$) followed by phishing emails ($4.732$),
whereas fake news had the highest average negative word count ($2.892$)
followed by left hyperpartisan news ($2.796$).

We also analyzed whether emotional salience has indicative powers to predict
the influence cues. Most influence cues and LIWC categories had an average
positive sentiment, with _liking_ and _gain_ framing having the highest levels
of positive emotion. _Anxiety_ and _anger_ (both LIWC categories) showed the
only neutral sentiment, whereas _admonition, blame/guilt_ , and _indignation_
as the only negative sentiment (with the latter being the most negative out of
all categories). Interestingly, items such as _loss_ framing and LIWC’s _risk_
both had positive sentiment.

The dataset invoked an overall positive sentiment, with phishing emails
containing the most positive average sentiment and fake news with the most
negative average sentiment.

####  5.1.4 Slant

The _objective_ and _subjective_ labels were present in 52% and 64% of all
texts in the dataset, respectively. This $>50$% frequency for both categories
was present in all text types except phishing emails and IRA ads, where
_subjectivity_ was approximately $2.5\times$ more common than _objectivity_.
The most subjective text type were IRA ads (77%) and the most objective texts
were fake news (72%); inversely, the least objective texts were phishing
emails (27%) and least subjective were mainstream center news (58%).

More notably, there was an overlap between the slants, wherein 29% of all
texts contained both _subjective_ and _objective_ labels. This could reflect
mixing factual (objective) statements with subjective interpretations of them.
Nonetheless, _objectivity_ and _subjectivity_ were independent variables,
$\chi^{2}(4,N=2,998)=72.0,p\approx 0$. The parings _(objectivity, authority)_
and _(subjectivity, authority)_ were the the top two most common pairs
considering PoP and slant; these pairs occurred at nearly the same frequency
within the dataset (48% and 45%, respectively). This pattern repeats itself
for other (PoP, slant) pairings and triplets, insofar as _(objectivity,
subjectivity, authority)_ is the third most commonly occurring triplet. When
comparing just (PoP, slant) triplets, slant is present in $9/10$ top triplets,
with _(subjectivity, authority, commitment)_ and _(objectivity, authority,
commitment)_ as the two most common triplets (30% and 27%, respectively).

_Objectivity_ and _subjectivity_ occurred over half of the dataset, with the
latter was much more common in phishing emails and IRA ads, while the former
was most common in fake news articles.

####  5.1.5 Attribution of Blame & Guilt

Twenty-nine percent of all texts contained the _blame/guilt_ label.
Interestingly, nearly the same proportions of fake news (45.4%) and right-
leaning hyperpartisan news (45.0%) were labeled with _blame/guilt_ , followed
by left-leaning hyperpartisan news (38%). Phishing emails, IRA ads, and
mainstream center media used _blame/guilt_ at the lowest frequencies (ranging
from 15% to 25%).

_Blame/guilt_ was somewhat seen in the top 10 pairs with PoP, only pairing
with _authority_ (4th most common pairing with 26% frequency) and _commitment_
(6th most common, 18%). However, _blame/guilt_ appeared more frequently
amongst the top 10 triplets with PoP, co-occurring with _authority,
commitment, call to action,_ and _social admonition_.

_Blame/guilt_ was disproportionately frequent for fake and hyperpartisan news,
commonly co-occurring with _authority_ or _commitment_.

####  5.1.6 Emphasis

_Emphasis_ was used in nearly 35% of all texts in the dataset. Among them, all
news sources (fake, hyperpartisan, and mainstream) appeared with the smallest
use of _emphasis_ (range: 17% to 26%). This follows as news (regardless of
veracity) likely is attempting to purport itself as legitimate. On the other
hand, phishing emails and IRA ads were both shared on arguably more informal
environments of communication (email and social media), and were thus often
found to use emphasis (over 54% for both categories). Additionally, similar to
previous analyses for other influence cues, _emphasis_ largely co-occurred
with _authority, commitment,_ and _call to action_.

The use of emphasis was much more common in informal text typed (phishing
emails and IRA social media ads), and less common in news-like sources (fake,
hyperpartisan, or mainstream).

####  5.1.7 LIWC Features of Influence

We also explored whether LIWC features have indicative powers to predict the
influence cues. Table 1 in Appendix B shows that _indignation_ and
_admonition_ had the highest average _anxiety_ feature, while _liking_ and
_gain_ framing had the lowest. _Indignation_ also scored three times above the
overall average for the _anger_ feature, as well as for _sadness_ (alongside
_blame/guilt_), whereas _gain_ had the lowest average for both _anger_ and
_sadness_. The _reward_ feature was seen most in _liking_ and in _gain_ ,
while _risk_ was slightly more common in _loss_ framing. The _time_ category
had the highest overall average and was most common in _blame/guilt_ , while
_money_ had the second largest overall average and was most common in _loss_.

We also saw that that left-leaning hyperpartisan news had the highest average
_anxiety_ , _sadness_ , _reward_ , and _time_ counts compared to all text
types, whereas right-leaning hyperpartisan news averaged slightly higher than
left-leaning media only in the _risk_ feature. Note, however, that LIWC is
calculated based on word counts and is therefore possibly biased towards
longer length texts; it should thus be noted that while hyperpartisan left
media had the highest averages for four of the seven LIWC features,
hyperpartisan media also had the second largest average text length compared
to other text types.

For the Deceptive Texts Group, phishing emails had the largest _risk_ and
_money_ averages over all text types, while averaging lowest in _anxiety_ ,
_anger_ , and _sadness_. Fake news was highest overall in _anger_ , though it
was slightly higher in _anxiety_ , _sadness_ , and _time_ compared to phishing
emails and IRA ads. On the other hand, the IRA ads were lowest in _reward_ ,
_risk_ , _time_ , and _money_ compared to the its group.

Lastly, mainstream center media had no LIWC categories in either high or low
extremities—most of its average LIWC values were close to the overall averages
for the entire dataset.

LIWC influence features varied depending on the type of text. Left
hyperpartisan news had the highest averages for four features (_anxiety_ ,
_sadness_ , _reward_ , and _time_). Phishing evoked risk and money, while fake
news evoked anger.

###  5.2 Lumen’s Multi-Label Prediction

TABLE I: Evaluation metric results for different learning algorithms in detecting influence cues. Algorithm |  $F1$-macro |  $F1$-micro | Overall Accuracy
---|---|---|---
Lumen | 58.30% | 69.23% | 72.43%
Labeled-LDA | 52.35% | 60.55% | 64.22%
LSTM | 64.20% | 69.48% | 72.34%
Naïve | 43.55% | 46.80% | 49.58%

This section describes our results in evaluating Lumen’s multi-label
prediction using the dataset. We compared Lumen’s performance against three
other ML algorithms: Labeled-LDA, LSTM, and a naïve algorithm. The former two
learning algorithms and Lumen performed much better than the naïve algorithm,
which shows that ML is promising for retrieval of influence cues in texts.
From Table I we can see that Lumen’s performance is as good as the state-of-
the-art prediction algorithm LSTM in terms of $F1$-micro score and overall-
accuracy (with $<$ 0.25% difference between each metric). On the other hand,
LSTM outperformed Lumen in terms of $F1$-macro, which is an unweighted mean of
the metric for each labels, thus potentially indicating that Lumen
underperforms LSTM in some labels although both algorithms share similar
overall prediction result (accuracy). Nonetheless, Lumen presented better
interpretability than LSTM (discussed below). Finally, both Lumen and LSTM
presented better performance than Labeled-LDA in both $F1$-scores and
accuracy, further emphasizing that additional features besides topic
structures can help improve the performance of the prediction algorithm.

To show Lumen’s ability to provide better understanding to practitioners
(i.e., interpretability), we trained it with our dataset and the optimal
hyper-parameter values from grid search. After training, Lumen provided both
the relative importance of each input feature and the topic structure of the
dataset without additional computational costs, which LSTM cannot provide
because it operates as a black-box.

TABLE II: The top-five most important features for Lumen’s prediction. Topic features are related to LDA topic modeling results. Input Feature | Importance | Keywords
---|---|---
Topic-1 | 0.073 | account, bank, security, time
Positive sentiment | 0.071 | N/A
Negative sentiment | 0.070 | N/A
Topic-8 | 0.065 | report, share, billion, source, profit
Topic-2 | 0.062 | black, people, trump, police, twitter

Table II shows the top-five important features in Lumen’s prediction decision-
making process. Among these features, two are related to sentiment, and the
remaining three are topic features (related to bank account security, company
profit report, and current events tweets), which shows the validity for the
choice of these types of input features. Positive and negative sentiment had
comparable levels of importance to Lumen, alongside the bank account security
topic.

##  6 Discussion

In this paper, we posit that interventions to aid human-based detection of
deceptive texts should leverage a key invariant of these attacks: the
application of influence cues in the text to increase its appeal to users. The
exposure of these influence cues to users can potentially improve their
decision-making by triggering their analytical thinking when confronted with
suspicious texts, which were not flagged as malicious via automatic detection
methods. Stepping towards this goal, we introduced Lumen, a learning framework
that combines topic modeling, LIWC, sentiment analysis, and ML to expose the
following influence cues in deceptive texts: persuasion, gain or loss framing,
emotional salience, subjectivity or objectivity, and use of emphasis or
attribution of guilt. Lumen was trained and tested on a newly developed
dataset of 2,771 texts, comprised of purposefully deceptive texts, and
hyperpartisan and mainstream news, all labeled according to influence cues.

###  6.1 Key Findings

Most texts in the dataset applied between three and six influence cues; we
hypothesize that these findings may reflect the potential appeal or popularity
of texts of moderate complexity. Deceptive or misleading texts constructed
without any influence cues are too simple to convince the reader, while texts
with too many influence cues might be far too long or complex, which are in
turn more time-consuming to write (for attackers) and to read (for receivers).

Most texts also applied _authority_ , which is concerning as it has been shown
to be one of the most impactful in user susceptibility to phishing studies
[34]. Meanwhile, _reciprocation_ was the least used principle at only 5%; this
may be an indication that _reciprocation_ does not lend itself well to be
applied in text, as it requires giving something to the recipient first and
expecting an action in return later. Nonetheless, _reciprocation_ was most
common in IRA ads (10%); these ads were posted on Facebook, and social media
might be a more natural and intuitive location to give gifts or compliments.
We also found that the application of the PoP was highly imbalanced with
_reciprocation, indignation, social proof,_ and _admonition_ each being
applied less than 15% the texts during the coding process.

The least used influence cue were _gain_ and _loss_ framing, appearing in only
13% and 7% of all texts. Though Kahneman and Tversky [11] posited that _loss_
is more impactful than the possibility of a gain, our dataset indicates that
_gain_ was more prevalent than _loss_. This is especially the case in phishing
emails, wherein the framing frequencies increase to 41% and 29%; this
difference suggests that in phishing emails, attackers might be attempting to
lure users to potential financial gain. We further hypothesize that phishing
emails exhibited these high rates of framing because successful phishing
survives only via a direct action from the user (e.g., clicking a link), which
may therefore motivate attackers to implement framing as a key influence
method. Phishing emails also exhibited the most positive average sentiment
(0.635) compared to other text types, possibly related to its large volume of
_gain_ labels, which were also strongly positive in sentiment (0.568).

Interestingly, texts varied among themselves in terms of influence cues even
within their own groups. For example, within the Deceptive Texts Group, fake
news used notably more _authority_ , _objectivity_ , and _blame/guilt_
compared to phishing emails and IRA ads, and was much lower in sentiment
compared to the latter two. Though phishing emails and IRA ads were more
similar, phishing was nonetheless different in its use of higher positive
sentiment, _gain_ framing, _scarcity_ , and lower _blame/guilt_. This was also
evident within the Hyperpartisan News Group—while right-learning news had a
higher frequency of _commitment_ , _call to action_ , and _admonition_ than
left-learning news, the opposite was also true for _liking_ , _reciprocation_
, and _scarcity_. Even comparing among all news types (fake, hyperpartisan,
and mainstream), this diversity of influence cues still prevailed, with the
only resounding agreement in a relative lack of use of _emphasis_. This
diversity across text types gives evidence of the highly imbalanced
application of influence cues in real deceptive or misleading campaigns.

We envision the use of Lumen (and ML methods in general) to expose influence
cues as a promising direction for application tools to aid human detection of
cyber-social engineering and disinformation. Lumen presented a comparable
performance compared to LSTM in terms of the $F1$-micro. Lumen’s
interpretability can allow a better understanding of both the dataset and the
decision-making process Lumen undergoes, consequently providing invaluable
insights for feature selection.

###  6.2 Limitations & Future Work

Dataset. One of the limitations of our work is that the dataset is unbalanced.
For example, our coding process revealed that some influence (e.g.,
_authority_) were disproportionately more prevalent than others (e.g.,
_reciprocation_ , framing). Even though an unbalanced dataset is not ideal for
ML analyses, we see this as part of the phenomenon. Attackers and writers
might find it more difficult to construct certain concept via text, thus
favoring other more effective and direct influence cues such as _authority_.
Ultimately, our dataset is novel in that each of the nearly 3K items were
coded according to 12 different variables; this was a time-expensive process
and we shall test the scalability of Lumen in future work. Nevertheless, we
plan to alleviate this dataset imbalance in our future work by curating a
larger, high-quality labeled dataset by reproducing our coding methodology,
and/or with the generation of synthetic, balanced datasets. Though we predict
that a larger dataset will still have varying proportions of certain influence
cues, it will facilitate machine learning with a larger volume of data points.

Additionally, our dataset is U.S.-centric, identified as a limitation in some
prior work (e.g., [4, 73, 74]). All texts were ensured to be in the English
language and all three groups of data were presumably aimed at an American
audience. Therefore, we plan future work to test Lumen in different cultural
contexts.

ML Framework. Lumen, as a learning framework, has three main limitations.
First, although the two-level architecture provides high degree of flexibility
and is general enough to include other predictive features in the future, it
also introduces complexity and overhead because tuning the hyper-parameters
and training the model will be more computationally expensive.

Second, topic modeling, a key component of Lumen, generally requires a large
number of documents of a certain length (usually thousands of documents and
hundreds of words in each document, such as a collection of scientific paper
abstracts) for topic inference. This will limit Lumen’s effectiveness on short
texts or when the training data is limited.

Third, some overlap between the LIWC influence features and emotional salience
might exist (e.g., the _sad_ LIWC category may correlate with the negative
emotional salience), which may negatively impact the prediction performance of
the machine learning algorithm used in Lumen. In other words, correlation of
input features makes machine learning algorithms hard to converge in general.

##  7 Conclusion

In this paper, we introduced Lumen, a learning-based framework to expose
influence cues in text by combining topic modeling, LIWC, sentiment analysis,
and machine learning in a two-layer hierarchical architecture. Lumen was
trained and tested with a newly developed dataset of 2,771 total texts
manually labeled according to the influence cues applied to the text.
Quantitative analysis of the dataset showed that authority was the most
prevalent influence cue, followed by subjectivity and commitment; gain framing
was most prevalent in phishing emails, and use of emphasis commonly occurred
in fake, partisan, and mainstream news articles. Lumen presented comparable
performance with LSTM in terms of $F1$-micro score, but better
interpretability, providing insights of feature importance. Our results
highlight the promise of ML to expose influence cues in text with the goal of
application in tools to improve the accuracy of human detection of cyber-
social engineering threats, potentially triggering users to think
analytically. We advocate that the next generation of interventions to
mitigate deception expose influence to users, complementing automatic
detection to address new deceptive campaigns and improve user decision-making
when confronted with potentially suspicious text.

## Acknowledgments

The authors would like to thank the coders for having helped with the labeling
of the influences cues in our dataset. This work was support by the University
of Florida Seed Fund award P0175721 and by the National Science Foundation
under Grant No 2028734. This material is based upon work supported by (while
serving at) the National Science Foundation.

## References

  * [1] R. S. Mueller, “Report on the investigation into Russian interference in the 2016 presidential election,” _US Dept. of Justice. Washington, DC_ , 2019\.
  * [2] “Fact check: Courts have dismissed multiple lawsuits of alleged electoral fraud presented by Trump campaign,” _Reuters_ , Feb. 2021.
  * [3] R. M. Ross, D. G. Rand, and G. Pennycook, “Beyond ”fake news”: Analytic thinking and the detection of false and hyperpartisan news headlines,” _Judgement and Decision Making_ , vol. 16, no. 2, pp. 484–504, 2021.
  * [4] B. Kalsnes and A. O. Larsson, “Facebook News Use During the 2017 Norwegian Elections—Assessing the Influence of Hyperpartisan News,” _Journalism Practice_ , vol. 15, no. 2, pp. 209–225, Feb. 2021.
  * [5] M. Barnidge and C. Peacock, “A third wave of selective exposure research? The challenges posed by hyperpartisan news on social media,” _Media and Communication_ , vol. 7, no. 3, pp. 4–7, Jul. 2019.
  * [6] N. J. Stroud, “Selective Exposure Theories,” _The Oxford Handbook of Political Communication_ , May 2014.
  * [7] R. B. Cialdini, _Influence: The Psychology of Persuasion_. Morrow New York, 1993.
  * [8] J. A. Russell, “A Circumplex Model of Affect,” _Journal of Personality and Social Psychology_ , vol. 39, no. 6, pp. 1161–1178, 1980.
  * [9] K. Kircanski, N. Notthoff, M. DeLiema, G. R. Samanez-Larkin, D. Shadel, G. Mottola, L. L. Carstensen, and I. H. Gotlib, “Emotional Arousal May Increase Susceptibility to Fraud in Older and Younger Adults,” _Psychol. Aging_ , vol. 33, no. 2, pp. 325–337, Mar. 2018.
  * [10] A. J. Rothman and P. Salovey, “Shaping Perceptions to Motivate Healthy Behavior: The Role of Message Framing,” _Psychol. Bull._ , vol. 121, no. 1, pp. 3–19, 1997.
  * [11] D. Kahneman and A. Tversky, “Prospect Theory: An Analysis of Decision under Risk,” _Econometrica_ , vol. 47, no. 2, p. 263, 1979.
  * [12] R. B. Cialdini, “The Science of Persuasion,” _Scientific American_ , vol. 284, no. 2, pp. 76–81, 2001. [Online]. Available: http://www.jstor.org/stable/26059056
  * [13] Twitter Help Center, “Government and state-affiliated media account labels,” https://help.twitter.com/en/rules-and-policies/state-affiliated, accessed: 2021-5-19.
  * [14] J. W. Pennebaker, R. L. Boyd, K. Jordan, and K. Blackburn, “The Development and Psychometric Properties of LIWC,” Tech. Rep., 2015.
  * [15] B. L. Smith, _Propaganda_ , Jan. 2021.
  * [16] Z. Dong, A. Kapadia, J. Blythe, and L. J. Camp, “Beyond the Lock Icon: Real-time Detection of Phishing Websites Using Public Key Certificates,” in _2015 APWG Symposium on Electronic Crime Research_ , May 2015, pp. 1–12.
  * [17] A. Oest, Y. Safaei, A. Doupé, G.-J. Ahn, B. Wardman, and K. Tyers, “PhishFarm: A Scalable Framework for Measuring the Effectiveness of Evasion Techniques against Browser Phishing Blacklists,” in _2019 IEEE Symposium on Security and Privacy_ , 2019.
  * [18] T. Peng, I. Harris, and Y. Sawa, “Detecting Phishing Attacks Using Natural Language Processing and Machine Learning,” in _2018 IEEE 12th International Conference on Semantic Computing_. ieeexplore.ieee.org, Jan. 2018, pp. 300–301.
  * [19] E. Bursztein and D. Oliveira, “Deconstructing the Phishing Campaigns that Target Gmail Users,” BlackHat 2019, Aug. 2019.
  * [20] R. Basnet, S. Mukkamala, and A. H. Sung, _Detection of Phishing Attacks: A Machine Learning Approach_. Berlin, Heidelberg: Springer Berlin Heidelberg, 2008, pp. 373–383. [Online]. Available: https://doi.org/10.1007/978-3-540-77465-5{\\_}19http://link.springer.com/10.1007/978-3-540-77465-5{\\_}19
  * [21] C. E. Shyni, S. Sarju, and S. Swamynathan, “A Multi-Classifier Based Prediction Model for Phishing Emails Detection Using Topic Modeling, Named Entity Recognition and Image Processing,” _Circuits and Systems_ , vol. 07, no. 09, pp. 2507–2520, 2016.
  * [22] M. Chandrasekaran, K. Narayanan, and S. Upadhyaya, “Phishing email detection based on structural properties,” in _NYS Cyber Security Conference_ , vol. 3. Albany, New York, 2006.
  * [23] A. P. Felt, A. Ainslie, R. W. Reeder, S. Consolvo, S. Thyagaraja, A. Bettes, H. Harris, and J. Grimes, “Improving SSL Warnings: Comprehension and Adherence,” in _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_ , ser. CHI ’15. New York, NY, USA: ACM, 2015, pp. 2893–2902.
  * [24] J. Sunshine, S. Egelman, H. Almuhimedi, N. Atri, and L. F. Cranor, “Crying Wolf: An Empirical Study of SSL Warning Effectiveness,” in _USENIX Security Symposium_ , 2009, pp. 399–416.
  * [25] S. Sheng, B. Magnien, P. Kumaraguru, A. Acquisti, L. F. Cranor, J. Hong, and E. Nunge, “Anti-Phishing Phil: The Design and Evaluation of a Game That Teaches People Not to Fall for Phish,” in _Proceedings of the 3rd Symposium on Usable Privacy and Security_. New York, NY, USA: ACM, 2007, pp. 88–99.
  * [26] A. Vance, B. Kirwan, D. Bjornn, J. Jenkins, and B. B. Anderson, “What Do We Really Know About How Habituation to Warnings Occurs Over Time?: A Longitudinal fMRI Study of Habituation and Polymorphic Warnings,” in _Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems_. New York, NY, USA: ACM, 2017, pp. 2215–2227.
  * [27] D. Akhawe and A. P. Felt, “Alice in Warningland: A Large-scale Field Study of Browser Security Warning Effectiveness,” in _22nd USENIX Security Symposium_ , 2013, pp. 257–272.
  * [28] G. Pennycook and D. G. Rand, “The Psychology of Fake News,” _Trends Cogn. Sci._ , vol. 25, no. 5, pp. 388–402, May 2021.
  * [29] C. Lim, “Checking how fact-checkers check,” _Research & Politics_, vol. 5, no. 3, p. 2053168018786848, Jul. 2018.
  * [30] B. Bago, D. G. Rand, and G. Pennycook, “Fake news, fast and slow: Deliberation reduces belief in false (but not true) news headlines,” _J. Exp. Psychol. Gen._ , vol. 149, no. 8, pp. 1608–1613, Aug. 2020.
  * [31] F. Stajano and P. Wilson, “Understanding Scam Victims: Seven Principles for Systems Security,” _Commun. ACM_ , vol. 54, no. 3, pp. 70–75, Mar. 2011\.
  * [32] D. S. Oliveira, T. Lin, H. Rocha, D. Ellis, S. Dommaraju, H. Yang, D. Weir, S. Marin, and N. C. Ebner, “Empirical Analysis of Weapons of Influence, Life Domains, and Demographic-Targeting in Modern Spam \- An Age-Comparative Perspective,” _Crime Science_ , vol. 8, no. 1, p. 3, Apr. 2019.
  * [33] A. Ferreira and S. Teles, “Persuasion: How phishing emails can influence users and bypass security measures,” pp. 19–31, 2019.
  * [34] D. Oliveira, H. Rocha, H. Yang, D. Ellis, S. Dommaraju, M. Muradoglu, D. Weir, A. Soliman, T. Lin, and N. Ebner, “Dissecting Spear Phishing Emails for Older vs Young Adults: On the Interplay of Weapons of Influence and Life Domains in Predicting Susceptibility to Phishing,” in _Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems_ , ser. CHI ’17. New York, NY, USA: ACM, 2017, pp. 6412–6424.
  * [35] P. Lawson, O. Zielinska, C. Pearson, and C. B. Mayhorn, “Interaction of Personality and Persuasion Tactics in Email Phishing Attacks,” pp. 1331–1333, 2017.
  * [36] N. Akbar, “Analysing persuasion principles in phishing emails,” Master’s thesis, University of Twente, 2014.
  * [37] R. T. Wright, M. L. Jensen, J. B. Thatcher, M. Dinger, and K. Marett, “Influence techniques in phishing attacks: An examination of vulnerability and resistance,” _Information Systems Research_ , vol. 25, no. 2, pp. 385–400, 2014.
  * [38] T. Lin, D. E. Capecci, D. M. Ellis, H. A. Rocha, S. Dommaraju, D. S. Oliveira, and N. C. Ebner, “Susceptibility to Spear-Phishing Emails: Effects of Internet User Demographics and Email Content,” _ACM Trans. Comput.-Hum. Interact._ , vol. 26, no. 5, pp. 32:1—-32:28, jul 2019. [Online]. Available: http://doi.acm.org/10.1145/3336141
  * [39] W. W. Xu, Y. Sang, and C. Kim, “What Drives Hyper-Partisan News Sharing: Exploring the Role of Source, Style, and Content,” _Digital Journalism_ , vol. 8, no. 4, pp. 486–505, May 2020.
  * [40] J. Graham, J. Haidt, and B. A. Nosek, “Liberals and conservatives rely on different sets of moral foundations,” _J. Pers. Soc. Psychol._ , vol. 96, no. 5, pp. 1029–1046, May 2009.
  * [41] “Social Media Advertisements,” https://intelligence.house.gov/social-media-content/social-media-advertisements.htm, accessed: 2020-8-9.
  * [42] F. S. A. J. B. H. Amirkhani, “Fnid: Fake news inference dataset,” 2020. [Online]. Available: https://dx.doi.org/10.21227/fbzd-sw81
  * [43] M. Smiles, _Phishing scam reports archive_ , 2019.
  * [44] U. of Pittsburgh, _Alerts & Notifications, Information Technology_, 2019.
  * [45] U. of Minnesota, _Phishing Scams Targeting the UMN_ , 2019.
  * [46] U. I. Services, _Phish Bowl/Phishing Scams_ , 2019.
  * [47] P. S. . O. of Information Security, _“Phishing_ , 2019.
  * [48] L. University, _Recent Phishing Examples, Library & Technology Services_, 2019\.
  * [49] U. of Arizona, _Phishing Alerts, UA Security_ , 2019.
  * [50] U. of Michigan, _Phishes & Scams_, 2019.
  * [51] R. B. Cialdini, “The psychology of persuasion,” _New York_ , 1993.
  * [52] D. Kahneman and A. Tversky, “Prospect theory,” _Econometrica_ , vol. 12, 1980\.
  * [53] R. P. Fuller and R. E. Rice, “Lights, camera, conflict: Newspaper framing of the 2008 screen actors guild negotiations,” _Journalism & Mass Communication Quarterly_, vol. 91, no. 2, pp. 326–343, 2014.
  * [54] T. Hove, H.-J. Paek, T. Isaacson, and R. T. Cole, “Newspaper portrayals of child abuse: Frequency of coverage and frames of the issue,” _Mass Communication and Society_ , vol. 16, no. 1, pp. 89–108, 2013.
  * [55] A. C. Morey and W. P. Eveland Jr, “Measures of political talk frequency: Assessing reliability and meaning,” _Communication Methods and Measures_ , vol. 10, no. 1, pp. 51–68, 2016.
  * [56] R. E. Rice, A. Gustafson, and Z. Hoffman, “Frequent but accurate: A closer look at uncertainty and opinion divergence in climate change print news,” _Environmental Communication_ , vol. 12, no. 3, pp. 301–321, 2018.
  * [57] W. D. Perreault Jr and L. E. Leigh, “Reliability of nominal data based on qualitative judgments,” _Journal of marketing research_ , vol. 26, no. 2, pp. 135–148, 1989.
  * [58] M. Lombard, J. Snyder-Duch, and C. C. Bracken, “Content analysis in mass communication: Assessment and reporting of intercoder reliability,” _Human communication research_ , vol. 28, no. 4, pp. 587–604, 2002.
  * [59] A. Kühberger, “The Influence of Framing on Risky Decisions: A Meta-analysis,” _Organ. Behav. Hum. Decis. Process._ , vol. 75, no. 1, pp. 23–55, 1998.
  * [60] C. J. Hutto and E. Gilbert, “Vader: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text,” in _Eighth international AAAI Conference on Weblogs and Social Media_ , 2014.
  * [61] M. Steyvers and T. Griffiths, “Probabilistic Topic Models,” _Handbook of latent semantic analysis_ , vol. 427, no. 7, pp. 424–440, 2007.
  * [62] D. M. Blei and J. D. Lafferty, “Topic models,” in _Text Mining: Classification, Clustering, and Applications_ , A. N. Srivastava and M. Sahami, Eds., 2009, pp. 71–89.
  * [63] D. M. Blei and A. Y. N. M. I. Jordan, “Latent Dirichlet Allocation,” _Journal of Machine Learning Research_ , vol. 3, no. Jan, pp. 993–1022, 2003\.
  * [64] T. L. Griffiths and M. Steyvers, “Finding Scientific Topics,” _Proceedings of the National Academy of Sciences of the United States of America_ , vol. 101 Suppl, pp. 5228–35, 2004. [Online]. Available: http://www.ncbi.nlm.nih.gov/pubmed/14872004
  * [65] Y. R. Tausczik and J. W. Pennebaker, “The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods,” _Journal of Language and Social Psychology_ , vol. 29, no. 1, pp. 24–54, mar 2010.
  * [66] K. A. Peace and S. M. Sinclair, “Cold-blooded Lie Catchers? An Investigation of Psychopathy, Emotional Processing, and Deception Detection: Psychopathy and Deception Detection,” _Legal and Criminological Psychology_ , vol. 17, no. 1, pp. 177–191, Feb. 2012.
  * [67] D. Ariely, U. Gneezy, G. Loewenstein, and N. Mazar, “Large Stakes and Big Mistakes,” _Rev. Econ. Stud._ , vol. 76, no. 2, pp. 451–469, 2009.
  * [68] NLTK.org, “Natural Language Toolkit,” 2020. [Online]. Available: {https://www.nltk.org/}
  * [69] H. Shi, M. Gerlach, I. Diersen, D. Downey, and L. Amaral, “A New Evaluation Framework for Topic Modeling Algorithms Based on Synthetic Corpora,” in _Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics (AISTATS)_ , 2019.
  * [70] D. Ramage, D. Hall, R. Nallapati, and C. D. Manning, “Labeled LDA: A Supervised Topic Model for Credit Attribution in Multi-labeled Corpora,” _Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing_ , no. August, pp. 248–256, 2009.
  * [71] A. van der Heijden and L. Allodi, “Cognitive Triaging of Phishing Attacks,” in _28th USENIX Security Symposium_. Santa Clara, CA: USENIX Association, aug 2019, pp. 1309–1326.
  * [72] M. Naili, A. H. Chaibi, and H. H. B. Ghezala, “Comparative study of word embedding methods in topic segmentation,” _Procedia Computer Science_ , vol. 112, pp. 340–349, 2017.
  * [73] R. Fletcher, A. Cornia, L. Graves, and R. K. Nielsen, “Measuring the reach of “fake news” and online disinformation in Europe,” University of Oxford: Reuters Institute for the Study of Journalism, Tech. Rep., Feb. 2018.
  * [74] N. Newman, R. Fletcher, A. Kalogeropoulos, and R. K. Nielsen, “Reuters Institute Digital News Report 2019,” Reuters Institute, Tech. Rep., 2019.

Hanyu Shi was a Postdoc Researcher under Dr. Daniela Oliveira at the
University of Florida during this work. He received his PhD degree with a
special focus on topic modeling algorithms in the department of Chemical and
Biological Engineering at Northwestern University, US. His current research
interests lie in the area of NLP, statistic inference, information retrieval,
etc.
---
Mirela Silva is currently pursuing her PhD at UF under the Dr. Oliveira. Her
research interests include interdisciplinary computer privacy, focused heavily
on the intersection of cyber deception and abuse through the lens of
interindividual differences.
---
Daniel Capecci is currently pursuing his Ph.D. under Dr. Damon Woodard at the
UF. His current research interests include computer vision and machine
learning for hardware assurance, large network analysis and deepfake
detection.
---
Luiz Giovanini is a Postdoc Researcher under Dr. Olivera at UF. He is also an
assistant professor at the Pontifical Catholic University of Parana (PUCPR),
Brazil. He received his PhD degree in Computer Science with focus on applied
machine learning from PUCPR in Brazil. His current research interests include
machine learning for applications in disinformation and cyber security.
---
Lauren Czech is pursuing a Ph.D. under Dr. Oliveira. Her current research
interests include leveraging ML to increase animal welfare.
---
Juliana Fernandes is an Assistant Professor of Advertising at UF. She received
her BA in Journalism from Universidade do Vale do Rio dos Sinos, Brazil, MA
and PhD degrees in Mass Communication from UF. Her current research interests
include uses and effects of negative information in persuasive communication.
---
Daniela Seabra Oliveira is an Associate Professor in the Department of
Electrical and Computer Engineering at UF. She received her BS and MS degree
in Computer Science from the Federal University of Minas Gerais in Brazil. She
received her PhD in Computer Science from UC-Davis, where she specialized in
security and operating systems. Her current research interests are human
factors in cyber security, phishing, and disinformation.
---

##  Appendix A Codebook

### Persuasion

Persuasion constitutes a series of influence principles based on Robert
Cialdini’s work, split into the following categories:

  1. 1.

Authority or Expertise/Source Credibility

  2. 2.

Reciprocation

  3. 3.

Commitment (sub-categories: Indignation, Call to Action)

  4. 4.

Liking

  5. 5.

Scarcity/Urgency/Opportunity

  6. 6.

Social Proof (sub-category: Admonition)

Authority or Expertise/Source Credibility. Humans tend to comply with requests
made by figures of authority and/or with expertise/credibility. The text can
include:

  * •

Literal authority (e.g., law enforcement personnel, lawyers, judges,
politicians)

  * •

Reputable/credible entity that could exert some power over people (e.g., a
bank)

  * •

Indirect authority (especially a fictitious company/person) that builds a
setting of authority

Examples:

  * •

“Tupac Shakur was indeed not just one of the greatest rappers of all time but
a worldly icon whose status in hip-hop culture can never be replaced. His
revolutionary knowledge mixed with street experience made him powerful
unstoppable force that spoke to the hearts of millions of people.”

  * •

“According to data from Mapping Police Violence”

  * •

“Autopsy says”

  * •

“Fox & Friends hosts declare”

Reciprocation. Humans tend to repay, in kind, what another person has provided
them. Text might first give/offer something, expecting that the person/user
will reciprocate. Even if the person does not reciprocate, s/he will still
keep the ”gift.” Therefore, if the user thinks they received a gift, they may
reciprocate the kindness (and may only find out later that the ”gift” was
fake).

Example:

  * •

“Aww! Because you need such a cutie on your timeline!”

Commitment. Once humans have taken a stand, they will feel pressured to behave
in line with their commitment. Text leverages a role assumed by the target and
their commitment to that role. Petitions and donations/charity (gun control,
animal abuse, children’s issue, political issues), or political affiliations
engagements.

Examples:

  * •

“But let’s remember Tupac and his ability to question the social order.
Changes, one of his popular songs, asks everyone to change their lifestyles
for better society. He always asked people to share with each other and to
learn to love each other.”

  * •

“Patriotism comes from your heart… follow its dictates and don’t live a false
life. Join!”

  * •

“We will stand for our right to keep and bear arms!”

  * •

“Black Matters”

Indignation. Still within the definition of commitment, text employing
indignation will also focus on anger or annoyance provoked by what is
perceived as unfair treatment, unjust, unworthy, mean.

Examples:

  * •

“Why should we be a target for police violence and harassment?”

  * •

“Why the pool party in Georgia is a silent story? Why the police was not aware
of a large party? Why this story has no national outrage? Is it ok when a
black teenager dies?”

  * •

“Obama never tried to protect blacks from police pressure”

Call to Action. Still within the definition of commitment, ads/text employing
a call to action will represent an exhortation or stimulus to do something in
order to achieve an aim or deal with a problem. A piece of content intended to
induce a viewer, reader, or listener to perform a specific act, typically
taking the form of an instruction or directive (e.g., buy now or click here).

Examples:

  * •

“Stop racism! We all belong to ONE HUMAN RACE.”

  * •

“We really can change the world if we stay united”

  * •

“We can be heard only when we stand together”

  * •

“White House must reduce the unemployment rates of black population”

  * •

“If this is a war against police - we’re joining this war on the cop’s side!”

  * •

“If we want to stop it, we should fight as our ancestors did it for
centuries.”

Liking. Humans tend to comply with requests from people they like or with whom
they share similarities. Forms of liking:

  * •

Physical attractiveness: Good looks suggest other favorable traits, i.e.
honesty, humor, trustworthiness

  * •

Similarity: We like people similar to us in terms of interests, opinions,
personality, background, etc.

  * •

Compliments: We love to receive praises, and tend to like those who give it

  * •

Contact and Cooperation: We feel a sense of commonality when working with
others to fulfill a common goal

  * •

Conditioning and Association: We like looking at models, and thus become more
favorable towards the cars behind them

May also come in the form of establishing a familiarity or rapport with the
object of liking

Example:

  * •

“What a beautiful and intelligent child she is. How magnificent is her mind…”

Scarcity/Urgency/Opportunity. Opportunities seem more valuable when their
availability is limited Text can leverage this principle by tricking/asking an
Internet user into clicking on a link to avoid missing out on a “once-in-a-
lifetime” opportunity; creates a sense of urgency.

Examples:

  * •

“Is it time to call out the national guard?” (Urgency)

  * •

“Free Figure’s Black Power Rally at VCU” (Opportunity)

  * •

“CLICK TO GET LIVE UPDATES ON OUR PAGE” (Opportunity)

Social Proof. People tend to mimic what the majority of people do or seem to
be doing. People let their guard and suspicion down when everyone else appears
to share the same behaviors and risks. In this way, they will not be held
solely responsible for their actions (i.e., herd mentality). The actions of
the group drive the decision making process.

Examples:

  * •

“More riots are coming this summer”

  * •

“America is deceased. Islamic terror has penetrated our homeland and now
spreads at a threw. Remember Victims Of Islamic Terror”

Admonition. Within the definition of social proof, admonition pertains to
texts that may include the following:

  * •

Caution, advise, or counsel against something.

  * •

Reprove or scold, especially in a mild and good-willed manner: The teacher
admonished him about excessive noise.

  * •

Urge to a duty or admonish them about their obligations.

Examples:

  * •

“More riots are coming this summer”

  * •

“America is deceased. Islamic terror has penetrated our homeland and now
spreads at a threw Remember Victims Of Islamic Terror”

### Slant

Slant encompasses subjectivity and objectivity.

Subjectivity. Subjective sentences generally refer to personal opinion,
emotion or judgment. The use of popular adverbs (e.g, very, actually), upper
case, exclamation and interrogation marks, hash tags, indicates subjectivity.

Examples:

  * •

“I doubt that it’s true”

  * •

“A beautiful message was seen on the streets of the capitol,”

  * •

“A timely message for today.”

  * •

“No matter what Defense Secretary or POTUS are saying they don’t fool me with
promises of gay military equality as key to the nation’s agenda.”

  * •

“This is something that America has a serious issue with - RACISM!” “Is it
time to call out the national guard?”

  * •

”This makes me ANGRY!”

Objectivity. Objective sentences refers to factual information, based on
evidence, or when evidence is presented. May or may not include statistics.

Examples:

  * •

“It has been discovered that”

  * •

“According to data from Mapping Police Violence”

  * •

“The McKinney Police Department, Chief Of Police Greg Conley said”

### Framing

Gain/Loss Framing refers to the presentation of a message (e.g., health
message, financial options, advertisement etc.) as implying a possible gain
(e.g., refer to possible benefits of performing a behavior) vs. implying a
possible loss (e.g., refer to the costs of not performing a behavior).

Gain. People are likely to act in ways that benefit them in some way. A reward
will increase the probability of a behavior A promise that a product (or
something else) can provide some form of self-improvement or benefit to the
user. This product can come in the form of an ad, job offer, joining a group,
etc.

Examples:

  * •

“Think about the benefits of recycling.”

  * •

“Think about what you can gain if you join.”

Loss. People are likely to act in ways that reduce loss/harm to them. Avoiding
loss will increase the probability of a behavior A promise that a product (or
something else) can help avoid some behavior/outcome.

Examples:

  * •

“Think about the costs of recycling.”

  * •

“Think about what you can lose if you don’t join.”

### Attribution of Blame/Guilt

When the text references an “another” (who/what) for the wrong/bad things
happening. “Who” can be a person, organization, etc., and “what” can be a
cause, object, etc.

Example:

  * •

“…Hillary is a Satan, and her crimes and lies had proved just how evil she
is.”

### Emphasis

Emphasis refers to the use of all caps text, several exclamation points,
several question marks, or anything used to call attention.

Example:

  * •

“Our women are the most powerful!!”

  * •

“LATIN WOMEN CAN DO THINGS TO MEN WITH THERE EYES”

##  Appendix B LIWC Emotional Features Summary

TABLE III: Breakdown of LIWC emotional features for each influence cue and text type; highest values for each LIWC category (columns) per influence cue and text type are highlighted bold text. | Influence Cue or
---
Text Type
Anxiety | Anger | Sadness | Reward | Risk | Time | Money
Authority | 0.23 | 0.69 | 0.36 | 1.69 | 0.78 | 6.68 | 2.21
Commitment | 0.20 | 0.67 | 0.28 | 1.61 | 0.78 | 5.97 | 1.87
Call to Action | 0.18 | 0.60 | 0.26 | 1.59 | 0.94 | 5.75 | 2.19
Indignation | 0.34 | 1.62 | 0.46 | 1.49 | 0.91 | 6.06 | 0.88
Liking | 0.13 | 0.34 | 0.24 | 1.84 | 0.57 | 5.87 | 1.67
Reciprocation | 0.19 | 0.32 | 0.36 | 1.53 | 0.59 | 5.00 | 1.14
Scarcity | 0.18 | 0.51 | 0.30 | 1.67 | 1.01 | 5.84 | 2.73
Social Proof | 0.26 | 0.69 | 0.37 | 1.61 | 0.79 | 5.81 | 1.64
Admonition | 0.34 | 1.14 | 0.39 | 1.52 | 1.09 | 6.43 | 1.52
Emphasis | 0.20 | 0.51 | 0.26 | 1.43 | 0.87 | 5.51 | 2.27
Blame/guilt | 0.30 | 1.22 | 0.47 | 1.47 | 1.00 | 6.85 | 1.60
Gain framing | 0.12 | 0.23 | 0.24 | 1.89 | 1.09 | 5.50 | 3.42
Loss framing | 0.20 | 0.41 | 0.37 | 1.70 | 1.16 | 5.78 | 3.73
Objectivity | 0.23 | 0.70 | 0.36 | 1.60 | 0.74 | 6.71 | 2.31
Subjectivity | 0.22 | 0.61 | 0.32 | 1.60 | 0.78 | 6.05 | 1.96
Fake News | 0.25 | 1.08 | 0.35 | 1.35 | 0.80 | 7.51 | 1.56
(News) Center | 0.19 | 0.34 | 0.39 | 1.54 | 0.60 | 6.20 | 3.01
(News) Left | 0.33 | 1.03 | 0.46 | 2.26 | 0.74 | 8.85 | 1.14
(News) Right | 0.25 | 1.01 | 0.30 | 1.69 | 0.79 | 6.66 | 0.88
Phishing Email | 0.06 | 0.05 | 0.15 | 1.63 | 1.21 | 4.96 | 4.27
IRA Ads | 0.12 | 0.55 | 0.18 | 0.65 | 0.54 | 2.34 | 0.56
Average | 0.20 | 0.59 | 0.32 | 1.56 | 0.75 | 6.02 | 2.11