# You Are What You Talk About:
Inducing Evaluative Topics for Personality Analysis

Josip Jukić Iva Vukojević Jan Šnajder
University of Zagreb, Faculty of Electrical Engineering and Computing
Text Analysis and Knowledge Engineering Lab
Unska 3, 10000 Zagreb, Croatia
{josip.jukic, iva.vukojevic<EMAIL_ADDRESS>
###### Abstract

Expressing attitude or stance toward entities and concepts is an integral part
of human behavior and personality. Recently, evaluative language data has
become more accessible with social media’s rapid growth, enabling large-scale
opinion analysis. However, surprisingly little research examines the
relationship between personality and evaluative language. To bridge this gap,
we introduce the notion of evaluative topics, obtained by applying topic
models to pre-filtered evaluative text from social media. We then link
evaluative topics to individual text authors to build their evaluative
profiles. We apply evaluative profiling to Reddit comments labeled with
personality scores and conduct an exploratory study on the relationship
between evaluative topics and Big Five personality facets, aiming for a more
interpretable, facet-level analysis. Finally, we validate our approach by
observing correlations consistent with prior research in personality
psychology.

##  1 Introduction

Sharing opinions has always been rooted in people’s daily habits, but
nowadays, it has scaled up with a plethora of user-generated texts on social
media (Lee and Ma, 2012). Opinions, as dispositions toward specific entities,
can be the key to understanding human behavior. Oftentimes, there is a need to
predict sentiment or stance toward a particular target of interest, making
opinion analysis useful in marketing research, social and political sciences,
and recommender systems, to name a few. Moreover, analyzing how we express our
opinions, i.e., the linguistic aspect and the psychological disposition of the
opinion holder, is crucial for various downstream tasks, such as personality
analysis and mental health prediction.

Figure 1: An illustrative example of mapping user statements to _evaluative
topics_ , i.e., topics induced from evaluative texts. Based on the prevalence
of such topics, we construct an _evaluative profile_ of each author. We then
measure the correlation between each author profile and personality facets,
such as artistic interests, assertiveness, intellect, and imagination.

In public communication, among other situations, people use linguistic
resources known as evaluative language (Hunston, 2010) to express their
attitudes. Thompson and Hunston (2000) define evaluation as a cover term for
unified phenomena of certainty and goodness/desirability, and evaluative
language includes every expression of the speaker or writer’s attitude or
stance towards, or feelings about the entities or propositions. As evaluation
is associated with psychological disposition Malrieu (1999), evaluative
language can help uncover people’s distinguishing qualities, i.e., recurrent
behavioral, cognitive, or affective tendencies. Such behavioral differences
resulting from interaction with the environment fall within the purview of
personality psychology, where they are typically examined as personality
traits – sets of characteristic patterns of behavior, cognition, and feelings
stable over time and across situations (Funder, 2012). Moreover, personality
traits have been shown to correlate with occupational preferences, political
or religious tendencies, intelligence, personal interests, and opinions
(Larson et al., 2002; Ackerman and Heggestad, 1997; Hyman, 1957).

To the best of our knowledge, evaluative language has not yet been studied in
the context of personality analysis on social media. This can perhaps be
attributed to methodological challenges. One of the difficulties in analyzing
evaluative language is separating the signal from the noise. Although
evaluation in text is pervasive, it remains a challenge to automatically
distinguish evaluative from non-evaluative text. Evaluative expressions come
in all sort of flavors, with varying amounts of explicit lexical stance
markers. Another difficulty is choosing the right level of abstraction both
for personality and evaluative language. If the evaluation targets are too
specific, it is difficult to relate them to personality. Nevertheless, we can
choose the appropriate operationalization of the personality in terms of its
breadth, thus aligning it with the abstraction level of the evaluative
targets.

In this paper, we study how evaluative language on social media can be used
for personality analysis. More specifically, our study aims to explore how the
topics people talk about are related to their personality, as illustrated in
Figure 1. To this end, we introduce the notion of an _evaluative topic_ ,
which is a topic constructed from evaluative text. To address the signal-noise
problem in detecting evaluative text, we develop an iterative filtering
technique based on detecting evaluative expressions and paraphrase mining. We
then leverage topic models to induce evaluative topics from prefiltered
evaluative text. Aiming to maximize coherence and diversity, we test several
topic model families, including probabilistic, decompositional, and neural
topic models. Finally, we link evaluative topics to individual text authors to
construct authors’ _evaluative profiles_.

In the experimental part, we investigate the relationship between evaluation
and personality. Our focus is on the Big Five model of personality (Goldberg,
1981), which includes broad personality traits and their more specific (and
arguably more interpretable) aspects called facets. We present a correlation
study of evaluative topics and Big Five facets on comments from Reddit, one of
the most popular discussion websites, which covers an astoundingly diverse
range of topics while preserving user anonymity. We find correlations between
evaluation and personality that are consistent with prior research in
personality psychology. Finally, we provide empirical evidence that evaluative
statements have a stronger association with personality than non-evaluative
expressions.

In summary, our contribution is threefold: we (1) develop an approach for
evaluative author profiling based on topic models with evaluative filtering,
(2) apply evaluative profiling on Reddit comments, and (3) study how
evaluative author profiles of Reddit users correlate with Big Five personality
facets.

The rest of the paper is structured as follows. Section 2 lays out the
background and related work. We describe our technique for filtering
evaluative text along with topic modeling in Section 3. We conclude the
section by presenting the idea of evaluative author profiles. We describe our
experiments and results in Section 4. Finally, Section 5 overlays the main
conclusions and takeaways of the paper.

##  2 Background and Related Work

#### Evaluative language.

In linguistics, studies of evaluative language are embedded in functional
approaches to language that aim to understand and explain linguistic
structures in terms of the semantic and communicative functions of language
and which assume that its primary function is to be a vehicle for social
interaction (Allan, 2007). Most research in evaluative language covers only a
particular niche of evaluation, such as subjectivity Wiebe et al. (2004) and
stance Kiesling et al. (2018). For instance, Biber and Finegan (1989) describe
lexical and grammatical markers of stance, i.e., categorizations along the
continuum of epistemic and attitudinal meanings, developing groups of markers
for people’s attitude, affect, and assessment. More recently, Pavalanathan et
al. (2017) compiled a lexicon of stance markers on Reddit data. On the other
hand, because sentiment analysis is ubiquitous in NLP, evaluative language has
been viewed through the prism of sentiment Benamara et al. (2017); Pang and
Lee (2008). In our work, we attempt to combine several aspects of evaluative
language, namely sentiment, stance, and opinion markers.

#### Target awareness.

Given the nature of expressing opinions, evaluation is directed toward a
particular target. Among others, sentiment-based approaches to evaluation have
been prevalent, which led to the establishment of aspect-based sentiment
analysis (ABSA), an NLP task that combines aspect extraction with sentiment
classification (Pontiki et al., 2014). Recent research in ABSA has evolved
from traditional perspectives, e.g., using conditional random fields for
aspect extraction as a sequence labeling task Toh and Wang (2014), to neural-
based approaches He et al. (2017); Luo et al. (2019); Hoang et al. (2019);
Tulkens and van Cranenburgh (2020). However, ABSA systems are better suited
for smaller domains. Besides ABSA, there have been target-based approaches in
stance classification Du et al. (2017) and topic-dependent argument
classification Reimers et al. (2019), among others. As we are trying to cover
diverse domains, we opt for a topic-level approach to targets.

#### NLP and personality.

NLP has been used extensively for personality analysis, starting with essay
analysis Pennebaker and King (2000) and followed by social network research
Park et al. (2014); Schwartz et al. (2013). The more recent studies were
mainly conducted with Facebook data. For example, Kulkarni et al. (2018) used
trait-level factor analysis for the Big 5 personality system on Facebook.
However, as far as we know, there is no research that leverages NLP to analyze
personality in the context of evaluative language. Moreover, there is little
research in NLP that covers facet-level personality analysis.

##  3 Methodology

We propose a three-step process from raw text to evaluative profiles. We start
with evaluative filtering to extract evaluative text and then construct topics
from the filtered dataset. Finally, we use the developed evaluative topics to
create the corresponding evaluative profiles.

###  3.1 Dataset

We used Pandora, a dataset with personality scores for Reddit users, extracted
from self-reported personality questionnaire results (Gjurković et al., 2021).
In total, there are 1,608 users with self-reported Big Five scores. These
users have written a total of 1.3M comments, consisting of 14.3M sentences.
Additionally, 127 users self-reported their questionnaire scores for the NEO
PI-R facets.111https://ipip.ori.org/newNEO_FacetsTable.htm NEO PI-R provides
information on six facets of each Big Five personality trait. A facet is a
specific and unique aspect of a broader personality trait. For example,
anxiety and depression are facets of neuroticism, while friendliness and
gregariousness are extraversion facets. After we applied preprocessing, which
included segmenting the comments into sentences, we ended up with 6.5M
sentences. We filtered out non-English comments since we used models pre-
trained on English texts. We attach more details on the preprocessing step in
Appendix A.1.

###  3.2 Evaluative filtering

Although social media abounds with evaluative language, it is challenging to
automatically distinguish evaluative from non-evaluative text. To filter out
non-evaluative expressions, we collected comments with evaluative markers –
lexical or grammatical language forms that express the act of evaluation. We
combined off-the-shelf sentiment analysis tools with opinion and stance
lexicons to cover different aspects of evaluative language. On top of that, we
searched for sentences that contain evaluative patterns, i.e., phrases that
express opinions.

To extract sentiment-laden sentences, we used VADER (Hutto and Gilbert, 2014),
where we summed the positive and negative VADER scores to determine the
overall sentiment score. We used the opinion lexicon devised by Hu and Liu
(2004) and the lexicon of stance markers Pavalanathan et al. (2017). Inspired
by Hunston and Thompson (2000), we compiled a list of evaluative patterns in
the form of regular expressions that we matched across the dataset. These
include phrases such as I like/hate/believe/support, personally, in my
(honest) opinion, etc. Along with variations of the aforementioned phrases, we
extended the regular expressions to support negations and modifiers, as well
as the vernacular language used in the online community (e.g., IMO – in my
opinion, FMPOV – from my point of view). To collect matched sentences, we set
a constraint that a given match must exceed the thresholds for sentiment,
opinion, and stance scores. We took the intersection of matched sentences in
the 50th percentile for each category except sentiment, for which we set an
upper bound. This is because we found that sentences with extreme sentiment
scores often do not have an explicit target, and the target cannot be induced
without additional context.222We decided not to use sentences above the 90th
percentile in sentiment, which typically indicates sentences expressing
emotions with highly implicit targets (e.g., a sentence with many emojis).

After applying evaluative filtering on Pandora, we obtained 29k sentences,
focusing on precision as we used strict pattern matching with high evaluative
scores. To improve recall, we developed quasi-snowballing (QSB), a simple
paraphrase mining technique.333The code for quasi-snowballing is available at
the link: https://github.com/josipjukic/quasi-snowballing QSB is an iterative
procedure that starts with a seed set of evaluative expressions and extends it
with similar statements. We employed sentence transformers444We chose all-
mpnet-base-v2 from https://www.sbert.net/ (Reimers and Gurevych, 2019) to
compute contextualized representations and use these representations to detect
paraphrases. We initialized QSB with the filtered evaluative sentences as the
seed set. Afterward, we used cosine similarity as a criterion to extract
similar sentences according to the similarity threshold $t_{sim}$, which is
adjusted at each step by the similarity growth factor $\gamma$. Inspired by
simulated annealing, the factor $\gamma$ increases the threshold exponentially
to achieve easier convergence as the whole set of sentences grows with each
iteration. At the end of each iteration, we augmented the old seed set with
the newly obtained matches. The process stops when there are no more
candidates. Using QSB, we obtained 310k sentences with evaluative markers
(details in Appendix A.2).

QSB mines expressions that can differ in target, polarity, or intensity. Since
we use a high similarity threshold, in most cases only one of the above three
components will be different in a paraphrase match. Multiple iterations of
paraphrase mining can evolve the original sentence, resulting in more lenient
matches overall, as shown in Table 1.

Original hit |  Paraphrase match | Similarity
---|---|---
Age is inexcusable to lie about IMO. |  I’m not a fan of lying about age. | .86
I support the death penalty, but I would never label myself as pro-life. |  I’m pro-life in the sense that I would rather not have people abort later term when it could be reasonably considered a person… | .77
I’m torn on this one, because I support trans folks 100% and I believe that a trans woman is a woman, end of story, and same for a man. |  I’m supportive of transgender people transitioning and being legally treated as someone of the opposite sex. | .78
Table 1: Examples of QSB paraphrase matching. The first column represents the
comments that are matched to regular expressions with matched text shown in
bold. The second column lists matched paraphrases.

###  3.3 Evaluative topics

In the second step, we applied topic modeling to the pre-filtered evaluative
comments to obtain evaluative topics. We adopted the definition of a topic as
a distribution over a fixed vocabulary of terms (Blei and Lafferty, 2009), and
further defined an evaluative topic as a topic built from target-specific
opinions derived from evaluative language. To produce evaluative topics, we
experimented with traditional probabilistic models as well as neural-based
architectures. Furthermore, since we are dealing with Reddit comments, which
tend to be short, we considered several topic modeling approaches for short
texts.

#### LDA.

From the family of traditional topic models, we opted for the latent Dirichlet
allocation (LDA), a well-established probabilistic topic model (Blei et al.,
2003), which also serves as a strong baseline. Considering that LDA deals
poorly with short texts due to the data sparsity problem (Hong and Davison,
2010), following Zuo et al. (2016), we grouped comments into larger documents.
For each author, we created pseudo-documents grouped by subreddit, i.e., a
forum dedicated to a particular topic on Reddit.

#### BTM.

We tested the biterm topic model (BTM), which is primarily designed for short
texts (Cheng et al., 2014). We fed the model with comment-level text, where we
filtered out non-evaluative sentences from the comment.

#### ABAE.

We tried out a neural-based architecture in Attention-based Aspect Extraction
(ABAE), an autoencoder proposed by He et al. (2017). We include this model as
it is designed specifically for aspect clustering, which we expect to align
well with our objective of building evaluative topics. ABAE exploits word
embeddings to improve coherence and attempts to group together words that
appear in similar contexts. Furthermore, ABAE uses the attention mechanism to
reduce the relative importance of irrelevant words during training. We adopted
ABAE but made a few changes to the training procedure. First, we trained
custom Word2Vec embeddings (Mikolov et al., 2013) on Pandora. We also modified
the reconstruction loss function from the ordinary dot product to match cosine
similarity with values in the $[-1,1]$ interval, which improved the learning
stability. We fed the model with segmented evaluative sentences to learn the
topics.

#### CTM.

Finally, we experimented with the combined topic model (CTM) from Bianchi et
al. (2021), which has been shown to increase coherence compared to other topic
models. CTM is a blend of a variational autoencoder and embeddings from a
sentence transformer.555We used paraphrase-distilroberta-base-v2 to produce
contextual embeddings.

####

Topic models are usually evaluated by means of topic coherence. However, this
method suffers from a validation gap, i.e., automated coherence is not
validated by human experimentation (Hoyle et al., 2021). To mitigate this
problem, we count token co-occurrences on the whole dataset and not only on
the training data, which has been shown to have a stronger association with
human judgment (Ding et al., 2018). Specifically, we used the normalized
pointwise mutual information (NPMI) to evaluate the coherence of the induced
topics. For evaluating diversity, we adopted the metric proposed by Bianchi et
al. (2021), defined as the reciprocal of the standard RBO Terragni et al.
(2021).666RBO uses a weighted ranking to estimate the disjointedness between
the top 10 words for each topic. We used NPMI and IRBO as two criteria of a
multi-objective optimization procedure. We determined the Pareto front of the
trained models, i.e., the set of non-dominated models, from which we chose the
model with the smallest number of topics (Table 2).

| NPMI | IRBO | # topics
---|---|---|---
LDA | $-.1413$ | $.9905$ | $20$
BTM | $-.2159$ | $.8241$ | $30$
ABAE | $-.0521$ | $.9833$ | $20$
CTM | $\mathbf{.0628}$ | $.9981$ | $20$
Table 2: Topic modeling evaluation results. NPMI is a coherence score in
$[-1,1]$ range, with $-1$ indicating that the topic representative terms
occurred together, $0$ indicating that the term occurrences were independent
of each other, and $1$ indicating that the terms co-occurred perfectly with
each other. IRBO measures topic diversity with $1$ for completely different
and $0$ for identical topics. We repeated each experiment $10$ times with
different seeds. Shown in bold is the score that is significantly better than
the scores for the rest of the models (independent Wilcoxon test for each
model pair with $p<.01$, adjusted for family-wise error rate with the Holm-
Bonferroni method).

###  3.4 Evaluative author profiles

In the third step, we leverage evaluative topics to produce evaluative author
profiles. Specifically, we describe each user in terms of topic prevalence,
where user text is distributed as a sentiment-weighted mixture of topics
across different targets. Each user is thus assigned the average topic
prevalence. We compute the average topic distributions for the entire
collection of user sentences, where a given sentence contributes with a value
from the $[0,1]$ interval for each topic.

To formalize our procedure, we begin by defining a topic distribution for a
specific document $\mathbf{d}=\begin{bmatrix}c_{1}c_{2}\ldots c_{K}\\\
\end{bmatrix}^{\mathsf{T}}$, where $\mathbf{d}$ represents a document (e.g.,
sentence, comment), and $c_{k}$ is the corresponding mixture component for the
$k$-th topic. We aggregate the values of the user’s evaluative sentences to
compute the $n$-th user’s components for each topic and concatenate the
aggregations into the vector $\mathbf{u}^{(n)}$:

$\mathbf{u}^{(n)}=\frac{1}{N_{n}}\sum_{i=1}^{N_{n}}\mathbf{d}^{(n,i)}\text{,}$

where $N_{n}$ is the number of the $n$-th user’s documents and
$\mathbf{d}^{(n,i)}$ is the $i$-th document of the $n$-th user. Moreover, we
incorporate the sentiment intensity information to obtain a sentiment-enhanced
representation $\mathbf{v}^{(n)}$:

$\mathbf{v}^{(n)}=\frac{1}{N_{n}}\sum_{i=1}^{N_{n}}s^{(n,i)}\mathbf{d}^{(n,i)}\text{,}$

where $s^{(n,i)}$ is the sentiment intensity for the $i$-th document of the
$n$-th user calculated as the sum of the positive and negative VADER scores.

The use of sentiment intensity in lieu of polarity deserves further
explanation. When considering sentiment polarity, we can discern two types:
the user’s sentiment polarity for a given statement and the implicit polarity
(Russo et al., 2015) of the topic itself. This gives rise to a number of
different ways in which sentiment information can be incorporated into
evaluative representations. However, the inclusion of sentiment polarity makes
it difficult to distinguish whether the topic sentiment is driven primarily by
implicit or explicit polarity. We sought parsimony, primarily to facilitate
explainability, and thus chose to use only sentiment intensity. Future work
may look into combining both types of sentiment.

##  4 Experiments

Our investigation of the association between evaluative language and
personality proceeded in two steps. We first built evaluative author profiles
from Reddit comments from Pandora, as described in the previous section. To
build evaluative topics, we used the CTM model on the filtered evaluative
sentences, as CTM yielded the best result in terms of coherence and diversity
scores. We induced $20$ evaluative topics, as per the results of the
optimization process. Table 3 shows a sample of the topics alongside manually
assigned labels.

In the next step, we carried out a correlation study on the relationship of
Big Five personality facets and evaluative topics. We adopted the Revised NEO
Personality Inventory (NEO PI-R), designed to measure the Big Five personality
traits: openness to experience, conscientiousness, extraversion,
agreeableness, and neuroticism Costa Jr and McCrae (1995). In the first part
of the analysis, we examined individual correlations between particular topics
and facets. In the second part, we used canonical correlation analysis (CCA)
to explore common associations between the entire set of topics on the one
side and the set of all Big Five facets on the other. We chose to focus on
personality facets rather than personality traits, hypothesizing that the
abstraction level of facets aligns better with the granularity of induced
topics. In that sense, the choice of facets supports the Brunswik symmetry
principle, which stipulates that analyzed constructs (in our case, personality
and topics) must have similar levels of generality Wittmann (2012).

###  4.1 Pairwise correlations

We calculated partial pairwise correlations between evaluative author profiles
and Big Five facets with control for gender as a possible confounder. For
gender, we used the values provided with the Pandora dataset, which surpassed
the $F_{1}$ score of $.90$ (Gjurković et al., 2021).777The gender prediction
model of Gjurković et al. (2021) is a binary classifier trained only on a
subset of users that self-reported one of the two binary gender categories.
Consequently, its predictions will be incorrect for users with non-binary
gender. However, we presume that this, along with the fact that even for
binary cases, predictions will not always be correct, has only a limited
effect on correlation estimates partialed out for gender. Figure 2 shows
significant correlations corrected for false discovery rate with the
Benjamini-Hochberg method. We observe numerous small correlations and even
moderate correlations ($>.4)$ for some topic-facet pairs (Bosco et al., 2015).
This is surprising, given that the average correlation in questionnaire-based
studies of individual differences is $.19$ Gignac and Szodorai (2016) and that
text data are noisier than questionnaire data.

To assess the validity of our results, we looked into personality psychology
literature for reference. According to psychological research, facets of
openness to experience are expected to correlate positively with curiosity,
food/drinks, fiction, music, gaming, political, personality, debate, and
occult topics, while they correlate negatively with the religion topic Stewart
et al. (2022); Ozer and Benet-Martinez (2006); Soto (2019); Intiful et al.
(2019); Skimina et al. (2021); Church et al. (2008); Marshall et al. (2015);
Chauvin and Mullet (2021). As Figure 2 shows, the majority of our results are
consistent with mentioned prior research. Three listed associations that we do
not significantly support are positive correlations with debate (hypothesized
direction, but not significant), gaming (hypothesized direction, but not
significant), and occult (opposite sign). However, we note that in most of the
referenced psychological literature, effects were analyzed at the trait level
rather than the facet level.

Label | Representants
---|---
open | people, lot, free, good, permit, time, going, open, big, trying
food/drinks | taste, meat, smell, beer, texture, flavour, savory, drink, palate, sweet
religion | testimony, believe, truth, god, witness, primordial, bible, church, jesus, earth
demeaning | shit, tell, talking, strangers, rude, stupid, bitch, upset, weird, shitty
investing/finance | recommend, pack, vouch, opt, buying, invest, fund, ticket, stock, dealt
fiction | watch, character, read, movies, story, books, thought, cool, new, anime
music | songs, album, pop, favourite, music, voice, listen, lyrics, track, sound
gaming | play, game, level, team, damage, skill, hit, gear, combat, dungeons
social issues | society, human, religion, culture, rights, moral, white, argument, victimhood, black
hatred | despise, hate, passion, stand, fucking, goddamn, nerd, mad, smug, ads
day-to-day | day, work, hours, week, spend, food, money, sleep, home, eat
relationships | relationship, woman, dating, child, partner, man, sex, ex, gay, alimony
argument | opinion, salt, grain, sanctimonious, worthless, controversial, assessment, disclaimer, 180, nuanced
political | government, party, vote, trump, support, country, state, speaker, system, tax
sexual/looks | hot, hair, sexy, gross, facial, body, attractive, wear, porn, dimorphism
personality | type, personality, mbti, emotions, test, cognitive, learning, ideas, brain, jung
debate | understand, discussion, post, saying, person, wrong, think, point, internet, debate
maltreatment | despised, hated, school, dropped, sucked, harass, refuses, skipped, hood, beaten
occult | fate, chime, cat, spirits, guides, luck, desperately, talisman, invite, mirror
Table 3: Evaluative topics produced by the CTM. The topic labels shown in
first column were manually assigned as the most frequent label among five
annotators. The second columns shows the top $10$ words for each topic. Figure
2: Pairwise partial correlations between Big 5 facets (x-axis) and evaluative
topics (y-axis) with control for gender. We show only significant correlations
($p<.01$), determined using Fisher’s z-transformation of correlations and
corrected for false discovery rate with the Benjamini-Hochberg method.

###  4.2 Canonical correlation analysis

As facets are inherently intertwined, we can gain deeper insight from
analyzing the facets jointly with respect to the set of evaluative topics. To
this end, we employed canonical correlation analysis (CCA). The goal of CCA is
to find a linear combination of evaluative profiles on the one side and facets
on the other, with the goal of maximizing the correlation between the newly
created canonical variables. CCA assumes that both variable sets have
multivariate normal distributions, which we confirmed with the Henze-Zirkler
test (Henze and Zirkler, 1990).888Normality was not rejected in both cases,
where $p=.62$ and $p=.51$ for evaluative and personality variables,
respectively. We computed $20$ canonical dimensions and sorted them in
descending order of correlation magnitude.999In general, the number of
canonical dimensions is equal to the number of variables in the set with fewer
variables. We found statistically significant correlations for the first three
pairs of canonical variates with Wilks’ lambda test.101010We observed
significant correlation ($p<.01$) for the following ranges: $120$, $220$, and
$320$.

An additional question we wanted to answer is whether personality analysis
benefits from evaluative topics (constructed from pre-filtered evaluative
text) as opposed to ordinary topics (constructed from all text). To this end,
we applied CCA to three different data subsets: (1) unfiltered text, (2) text
obtained with evaluative filtering, and (3) non-evaluative text, obtained as
the difference between the first and the second set. In addition, we ran CCA
on (4) evaluative text with sentiment intensity in order to investigate
whether evaluative profiles benefit from sentiment information. Figure 3 shows
the canonical correlations of the first three canonical pairs on the four data
subsets. Higher correlations computed by CCA indicate stronger associations
between the two sets of variables. Thus, under the assumption that evaluative
profiles and personality are associated, obtaining higher correlations
supports construct validity of evaluative profiles as construed by our model.
Construct validity, in this case, concerns the extent to which an evaluative
topic accurately assesses what it is designed for. With this in mind, two main
observations emerge from the results: (1) evaluative pre-filtering seems to be
more apt than using unfiltered data for establishing an association with Big 5
personality and (2) sentiment information amplifies the correlations.

To further investigate the relationship between facets and evaluative topics
as well as their individual importance, we computed canonical loadings, i.e.,
canonical structure correlations, by projecting both sets of original
variables onto the first and second canonical dimensions (Figure 4). Canonical
loadings reflect the shared variance of the observed variable and the
canonical variate. We followed the common practice in psychometrics (ter
Braak, 1990) and computed the intra-set variable-to-variate correlation for
the instrument variable (evaluative topic), and the inter-set correlation for
the goal variable (facet). We identified three distinct clusters of facets and
topics. The first one is at the bottom right corner of Figure 4 (music,
fiction; artistic-interests, imagination) and it indicates the openness aspect
of openness to experiences. The bottom-left cluster (maltreatment, demeaning;
vulnerability, immoderation, and excitement-seeking) roughly corresponds to
unpleasantly emotionally charged cluster. Finally, the top left cluster
(political, social issues; activity level, assertiveness) can be interpreted
as social engagement. We further validate the CCA results by showing the
dispersion of facets in the first and second canonical dimensions (Figures 5
and 6 in the Appendix). We find that facets from the same domain are grouped
but that there are also associations between facets from different domains,
which is expected based on prior research Schwaba et al. (2020).

Figure 3: Canonical correlations of the first three canonical pairs ($p<.01$)
estimated on four data subsets: original text (Unfiltered), text obtained with
evaluative filtering (Eval), Eval with sentiment information included when
calculating the evaluative profiles (Eval+Senti), and non-evaluative text
(Non-eval) obtained as the set difference between Non-filtered and Eval sets.
Figure 4: Canonical loadings of the first two dimensions for facets and
evaluative topics. The proximity of individual points indicates the strength
of association between two data points. Specifically, when two points,
regardless of their type (topic or facet), have similar angles with respect to
the origin and similar magnitudes, this hints that the two points contribute
similarly to the canonical variate and have stronger associations.

##  5 Conclusion

The relationship between evaluative language and personality has been
understudied in NLP research. We aim to fill this gap by proposing evaluative
author profiling – linking text of authors to topics obtained from text
filtered for evaluative language. We applied evaluative profiling on a dataset
of Reddit comments with self-reported Big 5 personality facet scores. Using
canonical correlation analysis, we showed that the facets within the same
trait have stronger associations in the canonical space. We found that
evaluative topics have moderate correlations with Big 5 facets. Moreover, we
corroborate the hypothesis that evaluative expressions hold greater
informational value for personality analysis than unfiltered texts.
Additionally, we showed that non-evaluative text has a much weaker association
with Big 5 facets compared to evaluative text. Finally, we observed
correlations consistent with previous research in personality psychology. We
believe that our study can contribute to a better understanding of evaluative
language on social media and how it relates to personality traits.

## Ethical Considerations

Our research has been approved by an academic IRB.

#### Potential harm.

Our study is an exploratory study, hence one cannot generalize based on the
correlations we obtained. Otherwise, that could lead to unsupported
generalization, which may discriminate against certain groups of people.

#### Collecting data from users.

According to the American Psychological Association’s Ethical
Principles,111111https://www.apa.org/ethics/code researchers may waive
informed consent when analyzing archival data (i.e., data collected before the
study began) if disclosure of responses would not expose participants to risk
of criminal or civil liability or harm. In our case, we use Reddit data, where
Reddit users agree via Reddit User Agreement that they will not disclose any
sensitive information to others, and they are informed that their comments are
publicly available. As users may opt-out and delete their data, we removed
deleted user accounts and comments. We also present our results at the group
level rather than the individual level to further protect users’ identities.

#### Misuse potential.

In principle, evaluative profiling could be used for making decisions at the
individual level based on social media textual data, e.g., in micro-targeting.
We strongly advocate against the use of our methods for these or other
ethically questionable applications.

#### Biases.

We note that we used self-reported data from Reddit. As such, this data may
not be perfectly accurate and may include various biases, notably the
acquisition bias. The dataset we use may not be representative of the Reddit
population, and it is certainly not representative of any wider population of
users.

## Limitations

Although our evaluative filtering technique resulted with evaluative topics
that have stronger correlations with Big 5 facets than non-evaluative text,
filtering cannot be validated in isolation without additional annotation.
Additionally, incorporating additional information such as sentiment polarity
can hurt interpretability of the results. A possible solution to circumvent
this problem is to use structural topic models Roberts et al. (2014), which
can model additional information such as user demographics as covariates.
Since we use topic modeling and try to optimize topic diversity and coherence,
which has been critized as of late Hoyle et al. (2021), it can be hard to
choose appropriate representants for the topic and there are no guarantees
that a certain topic will be coherent enough to translate to a meaningful
concept. From the psychology point of view, we conduct only an exploratory
study where we observe correlation, so we cannot make any confirmation of the
results from personality psychology, but only support it with associations.
Moreover, since we conduct a facet-level study and take only the scores from
the same questionnaire to mitigate the noise from different tests, the number
of reports of facet scores is relatively small ($n=127$) compared to the total
number of users with reported traits scores ($n=1,608$).

Our study was limited to English texts. A potential transfer to other
languages would require additional resources, namely adequate lexicons and
sentiment classifier to enable evalautive filtering, making the transfer non-
trivial. Finally, we would need to retrain the topic models on texts in the
target languages.

## Acknowledgements

We thank the reviewers for their comments. We also thank Irina Masnikosa for
her invaluable linguistic advice. This work has been fully supported by the
Croatian Science Foundation under the project IP-2020-02-8671 PSYTXT
(“Computational Models for Text-Based Personality Prediction and Analysis”).

## References

  * Ackerman and Heggestad (1997) Phillip L Ackerman and Eric D Heggestad. 1997.  Intelligence, personality, and interests: evidence for overlapping traits.  _Psychological bulletin_ , 121(2):219.
  * Allan (2007) Keith Allan. 2007.  _The western classical tradition in linguistics_.  Equinox London.
  * Benamara et al. (2017) Farah Benamara, Maite Taboada, and Yannick Mathieu. 2017.  Evaluative language beyond bags of words: Linguistic insights and computational applications.  _Computational Linguistics_ , 43(1):201–264.
  * Bianchi et al. (2021) Federico Bianchi, Silvia Terragni, and Dirk Hovy. 2021.  Pre-training is a hot topic: Contextualized document embeddings improve topic coherence.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 759–766, Online. Association for Computational Linguistics.
  * Biber and Finegan (1989) Douglas Biber and Edward Finegan. 1989.  Styles of stance in english: Lexical and grammatical marking of evidentiality and affect.  _Text - Interdisciplinary Journal for the Study of Discourse_ , 9(1):93–124.
  * Blei and Lafferty (2009) David M Blei and John D Lafferty. 2009.  Topic models.  In _Text mining_ , pages 101–124. Chapman and Hall/CRC.
  * Blei et al. (2003) David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003.  Latent dirichlet allocation.  _J. Mach. Learn. Res._ , 3:993–1022.
  * Bosco et al. (2015) Frank Bosco, Kulraj Singh, James Field, and Charles Pierce. 2015.  Correlational effect size benchmarks.  _Journal of Applied Psychology_ , 100:431–449.
  * Chauvin and Mullet (2021) Bruno Chauvin and Etienne Mullet. 2021.  Individual differences in paranormal beliefs: The differential role of personality aspects.  _Current psychology_ , 40(3):1218–1227.
  * Cheng et al. (2014) Xueqi Cheng, Xiaohui Yan, Yanyan Lan, and Jiafeng Guo. 2014.  Btm: Topic modeling over short texts.  _IEEE Transactions on Knowledge and Data Engineering_ , 26(12):2928–2941.
  * Church et al. (2008) A Timothy Church, Marcia S Katigbak, Jose Alberto S Reyes, Maria Guadalupe C Salanga, Lilia A Miramontes, and Nerissa B Adams. 2008.  Prediction and cross-situational consistency of daily behavior across cultures: Testing trait and cultural psychology perspectives.  _Journal of Research in Personality_ , 42(5):1199–1215.
  * Costa Jr and McCrae (1995) Paul T Costa Jr and Robert R McCrae. 1995.  Domains and facets: Hierarchical personality assessment using the revised neo personality inventory.  _Journal of personality assessment_ , 64(1):21–50.
  * Ding et al. (2018) Ran Ding, Ramesh Nallapati, and Bing Xiang. 2018.  Coherence-aware neural topic modeling.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 830–836, Brussels, Belgium. Association for Computational Linguistics.
  * Du et al. (2017) Jiachen Du, Ruifeng Xu, Yulan He, and Lin Gui. 2017.  Stance classification with target-specific neural attention.  In _Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17_ , pages 3988–3994.
  * Funder (2012) David C Funder. 2012.  Accurate personality judgment.  _Current Directions in Psychological Science_ , 21(3):177–182.
  * Gignac and Szodorai (2016) Gilles E Gignac and Eva T Szodorai. 2016.  Effect size guidelines for individual differences researchers.  _Personality and individual differences_ , 102:74–78.
  * Gjurković et al. (2021) Matej Gjurković, Mladen Karan, Iva Vukojević, Mihaela Bošnjak, and Jan Šnajder. 2021.  PANDORA talks: Personality and demographics on Reddit.  In _Proceedings of the Ninth International Workshop on Natural Language Processing for Social Media_ , pages 138–152, Online. Association for Computational Linguistics.
  * Goldberg (1981) Lewis R Goldberg. 1981.  Language and individual differences: The search for universals in personality lexicons.  _Review of personality and social psychology_ , 2(1):141–165.
  * He et al. (2017) Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. 2017.  An unsupervised neural attention model for aspect extraction.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 388–397, Vancouver, Canada. Association for Computational Linguistics.
  * Henze and Zirkler (1990) Norbert Henze and Bernd Zirkler. 1990.  A class of invariant consistent tests for multivariate normality.  _Communications in Statistics-theory and Methods_ , 19:3595–3617.
  * Hoang et al. (2019) Mickel Hoang, Oskar Alija Bihorac, and Jacobo Rouces. 2019.  Aspect-based sentiment analysis using BERT.  In _Proceedings of the 22nd Nordic Conference on Computational Linguistics_ , pages 187–196, Turku, Finland. Linköping University Electronic Press.
  * Hong and Davison (2010) Liangjie Hong and Brian D Davison. 2010.  Empirical study of topic modeling in twitter.  In _Proceedings of the first workshop on social media analytics_ , pages 80–88.
  * Hoyle et al. (2021) Alexander Hoyle, Pranav Goel, Andrew Hian-Cheong, Denis Peskov, Jordan Boyd-Graber, and Philip Resnik. 2021.  Is automated topic model evaluation broken? the incoherence of coherence.  In _Advances in Neural Information Processing Systems_ , volume 34, pages 2018–2033. Curran Associates, Inc.
  * Hu and Liu (2004) Minqing Hu and Bing Liu. 2004.  Mining and summarizing customer reviews.  In _Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 168–177.
  * Hunston (2010) Susan Hunston. 2010.  _Corpus approaches to evaluation: Phraseology and evaluative language_ , volume 13.  Routledge.
  * Hunston and Thompson (2000) Susan Hunston and Geoffrey Thompson. 2000.  _Evaluation in text: Authorial stance and the construction of discourse: Authorial stance and the construction of discourse_.  Oxford University Press, UK.
  * Hutto and Gilbert (2014) Clayton Hutto and Eric Gilbert. 2014.  Vader: A parsimonious rule-based model for sentiment analysis of social media text.  In _Proceedings of the International AAAI Conference on Web and Social Media_ , volume 8.
  * Hyman (1957) Herbert H Hyman. 1957.  An exploration into opinions and personality.  _World Politics_ , 10(1):144–153.
  * Intiful et al. (2019) Freda Dzifa Intiful, Emefa Gifty Oddam, Irene Kretchy, and Joana Quampah. 2019.  Exploring the relationship between the big five personality characteristics and dietary habits among students in a ghanaian university.  _BMC psychology_ , 7(1):1–7.
  * Kiesling et al. (2018) Scott F. Kiesling, Umashanthi Pavalanathan, Jim Fitzpatrick, Xiaochuang Han, and Jacob Eisenstein. 2018.  Interactional stancetaking in online forums.  _Computational Linguistics_ , 44(4):683–718.
  * Kulkarni et al. (2018) Vivek Kulkarni, Margaret L. Kern, David Stillwell, Michal Kosinski, Sandra Matz, Lyle Ungar, Steven Skiena, and H. Andrew Schwartz. 2018.  Latent human traits in the language of social media: An open-vocabulary approach.  _PLOS ONE_ , 13(11):1–18.
  * Larson et al. (2002) Lisa M. Larson, Patrick J. Rottinghaus, and Fred H. Borgen. 2002.  Meta-analyses of big six interests and big five personality factors.  _Journal of Vocational Behavior_ , 61(2):217–239.
  * Lee and Ma (2012) Chei Sian Lee and Long Ma. 2012.  News sharing in social media: The effect of gratifications and prior experience.  _Computers in human behavior_ , 28(2):331–339.
  * Luo et al. (2019) Ling Luo, Xiang Ao, Yan Song, Jinyao Li, Xiaopeng Yang, Qing He, and Dong Yu. 2019\.  Unsupervised neural aspect extraction with sememes.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_ , pages 5123–5129. International Joint Conferences on Artificial Intelligence Organization.
  * Malrieu (1999) Jean Pierre Malrieu. 1999.  _Evaluative semantics: Language, cognition, and ideology_.  Psychology Press.
  * Marshall et al. (2015) Tara C Marshall, Katharina Lefringhausen, and Nelli Ferenczi. 2015.  The big five, self-esteem, and narcissism as predictors of the topics people write about in facebook status updates.  _Personality and Individual Differences_ , 85:35–40.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_.
  * Ozer and Benet-Martinez (2006) Daniel J Ozer and Veronica Benet-Martinez. 2006.  Personality and the prediction of consequential outcomes.  _Annual review of psychology_ , 57:401.
  * Pang and Lee (2008) Bo Pang and Lillian Lee. 2008.  Opinion mining and sentiment analysis.  _Found. Trends Inf. Retr._ , 2(1–2):1–135.
  * Park et al. (2014) Gregory Park, H. Schwartz, Johannes Eichstaedt, Margaret Kern, Michal Kosinski, David Stillwell, Lyle Ungar, and Martin Seligman. 2014.  Automatic personality assessment through social media language.  _Journal of personality and social psychology_ , 108.
  * Pavalanathan et al. (2017) Umashanthi Pavalanathan, Jim Fitzpatrick, Scott Kiesling, and Jacob Eisenstein. 2017\.  A multidimensional lexicon for interpersonal stancetaking.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 884–895, Vancouver, Canada. Association for Computational Linguistics.
  * Pennebaker and King (2000) James Pennebaker and Laura King. 2000.  Linguistic styles: Language use as an individual difference.  _Journal of personality and social psychology_ , 77:1296–312.
  * Pontiki et al. (2014) Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014.  SemEval-2014 task 4: Aspect based sentiment analysis.  In _Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)_ , pages 27–35, Dublin, Ireland. Association for Computational Linguistics.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Reimers et al. (2019) Nils Reimers, Benjamin Schiller, Tilman Beck, Johannes Daxenberger, Christian Stab, and Iryna Gurevych. 2019.  Classification and clustering of arguments with contextualized word embeddings.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 567–578, Florence, Italy. Association for Computational Linguistics.
  * Roberts et al. (2014) Margaret E Roberts, Brandon M Stewart, Edoardo M Airoldi, K Benoit, D Blei, P Brandt, and A Spirling. 2014.  Structural topic models.  _Retrieved May_ , 30:2014.
  * Russo et al. (2015) Irene Russo, Tommaso Caselli, and Carlo Strapparava. 2015.  SemEval-2015 task 9: CLIPEval implicit polarity of events.  In _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)_ , pages 443–450, Denver, Colorado. Association for Computational Linguistics.
  * Schwaba et al. (2020) Ted Schwaba, Mijke Rhemtulla, Christopher J Hopwood, and Wiebke Bleidorn. 2020.  A facet atlas: Visualizing networks that describe the blends, cores, and peripheries of personality structure.  _PloS one_ , 15(7):e0236893.
  * Schwartz et al. (2013) H. Andrew Schwartz, Johannes C. Eichstaedt, Margaret L. Kern, Lukasz Dziurzynski, Stephanie M. Ramones, Megha Agrawal, Achal Shah, Michal Kosinski, David Stillwell, Martin E. P. Seligman, and Lyle H. Ungar. 2013.  Personality, gender, and age in the language of social media: The open-vocabulary approach.  _PLOS ONE_ , 8(9):1–16.
  * Skimina et al. (2021) Ewa Skimina, Jan Cieciuch, and Włodzimierz Strus. 2021.  Traits and values as predictors of the frequency of everyday behavior: Comparison between models and levels.  _Current Psychology_ , 40(1):133–153.
  * Soto (2019) Christopher J Soto. 2019.  How replicable are links between personality traits and consequential life outcomes? the life outcomes of personality replication project.  _Psychological Science_ , 30(5):711–727.
  * Stewart et al. (2022) Ross David Stewart, René Mõttus, Anne Seeboth, Christopher John Soto, and Wendy Johnson. 2022.  The finer details? the predictability of life outcomes from big five domains, facets, and nuances.  _Journal of personality_ , 90(2):167–182.
  * ter Braak (1990) Cajo ter Braak. 1990.  Interpreting canonical correlation analysis through biplots of stucture correlations and weights.  _Psychometrika_ , 55:519–531.
  * Terragni et al. (2021) Silvia Terragni, Elisabetta Fersini, and Enza Messina. 2021.  Word embedding-based topic similarity measures.  In _Natural Language Processing and Information Systems: 26th International Conference on Applications of Natural Language to Information Systems, NLDB 2021, Saarbrücken, Germany, June 23–25, 2021, Proceedings_ , page 33–45, Berlin, Heidelberg. Springer-Verlag.
  * Thompson and Hunston (2000) Geoffrey Thompson and Susan Hunston. 2000.  Evaluation: an introduction.  In _Evaluation in text: Authorial stance and the construction of discourse_. Oxford University Press.
  * Toh and Wang (2014) Zhiqiang Toh and Wenting Wang. 2014.  DLIREC: Aspect term extraction and term polarity classification system.  In _Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)_ , pages 235–240, Dublin, Ireland. Association for Computational Linguistics.
  * Tulkens and van Cranenburgh (2020) Stéphan Tulkens and Andreas van Cranenburgh. 2020.  Embarrassingly simple unsupervised aspect extraction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3182–3187, Online. Association for Computational Linguistics.
  * Wiebe et al. (2004) Janyce Wiebe, Theresa Wilson, Rebecca Bruce, Matthew Bell, and Melanie Martin. 2004\.  Learning subjective language.  _Computational Linguistics_ , 30(3):277–308.
  * Wittmann (2012) Werner W Wittmann. 2012.  Principles of symmetry in evaluation research with implications for offender treatment.  _Antisocial behavior and crime. Contributions of developmental and evaluation research to prevention and intervention_ , (2011):357–368.
  * Zuo et al. (2016) Yuan Zuo, Junjie Wu, Hui Zhang, Hao Lin, Fei Wang, Ke Xu, and Hui Xiong. 2016.  Topic modeling of short texts: A pseudo-document view.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’16, page 2105–2114, New York, NY, USA. Association for Computing Machinery.

##  Appendix A Technical details

###  A.1 Dataset

We selected the comments from users with Big Five self-reports on Pandora. We
kept only the comments in English and removed unlikely candidates for
evaluative expressions, including comments that were too short – those with
fewer than five words – and noisy comments – those consisting of 50% or more
non-alphanumeric characters. We segmented the comments into sentences using
spaCy’s English pipeline
en_core_web_lg121212https://spacy.io/models/en#en_core_web_lg to proceed at a
finer level. Once again, we discarded texts that were too short, but this time
the sentences with fewer than three words.

###  A.2 Evaluative filtering

When we employed QSB to mine paraphrases of the initial set of evaluative
expressions, we aimed for a tenfold increase in size of the seed set (29k
sentences), so we tuned the values of $t_{sim}$ and $\gamma$. We used grid
search with the parameter range $t_{sim}\in[0.50,0.90]$ with step $0.05$ and
$\gamma\in[0.90,1.10]$ with a step of $0.05$ in both cases, which led us to
$t_{sim}=0.7$ and $\gamma=1.05$. QSB yielded 310k sentences with evaluative
markers.

###  A.3 Topic modeling

We used scikit-learn implementation of LDA.131313https://scikit-
learn.org/stable/modules/generated/sklearn.decomposition.LatentDirichletAllocation.html
We adapt the code from the original papers for BTM Cheng et al. (2014), ABAE
He et al. (2017), and CTM Bianchi et al. (2021). The neural-based
architectures in ABAE and CTM have $100,020$ and $19,794,280$, respectively.
We trained ABAE for $5$ epochs and CTM for $20$ epochs, with average running
times of $71.4$ and $590.8$ minutes over $10$ different runs.

We used the same preprocessing for all topic models that we experimented with.
Specifically, we removed punctuation and lowercased the text. We also
eliminated stop words, URLs, emails, digits, and currency symbols, after which
we lemmatized the tokens. We repeated each experiment $10$ times with
different seeds. We conducted a grid search with the number of topics as a
hyperparameter in the $[5,100]$ range with step $5$. For the rest of the
hyperparameters, we used default values.

###  A.4 Computing infrastructure

We conducted our experiments on $2\times$ AMD Ryzen Threadripper 3970X 32-Core
Processors and $2\times$ NVIDIA GeForce RTX 3090 GPU’s with $24$GB of RAM. We
used PyTorch version $1.9.0$ and CUDA $11.4$.

##  Appendix B Additional analysis

In Figure 2, there are cases where certain evaluative topics show significant
correlation with several facets from the same trait. For instance, the topic
social issues correlates positively and significantly with anger,
immoderation, and anxiety facets from the neuroticism trait. On the other
hand, there are a number of cases where the correlation between facets from
the same trait and a topic is of a different sign. This corroborates our
hypothesis that topics align better with facets than with traits.

Figure 5: Facet projection onto the first canonical variate. The values on the
y-axis represent the canonical loadings of facets in the first canonical
dimension. Figure 6: Facet projection onto the second canonical variate. The
second dimension provides a perfect separation of conscientiousness and
neuroticism, with a mixture of facets from the remaining traits.


# CheXternal: Generalization of Deep Learning Models for Chest X-ray
Interpretation to Photos of Chest X-rays and External Clinical Settings

Pranav Rajpurkar<EMAIL_ADDRESS>Stanford UniversityUSA ,  Anirudh
Joshi<EMAIL_ADDRESS>Stanford UniversityUSA ,  Anuj Pareek
<EMAIL_ADDRESS>Stanford UniversityUSA ,  Andrew Y. Ng
<EMAIL_ADDRESS>Stanford UniversityUSA and  Matthew P. Lungren
<EMAIL_ADDRESS>Stanford UniversityUSA

(2021)

###### Abstract.

Recent advances in training deep learning models have demonstrated the
potential to provide accurate chest X-ray interpretation and increase access
to radiology expertise. However, poor generalization due to data distribution
shifts in clinical settings is a key barrier to implementation. In this study,
we measured the diagnostic performance for 8 different chest X-ray models when
applied to (1) smartphone photos of chest X-rays and (2) external datasets
without any finetuning. All models were developed by different groups and
submitted to the CheXpert challenge, and re-applied to test datasets without
further tuning. We found that (1) on photos of chest X-rays, all 8 models
experienced a statistically significant drop in task performance, but only 3
performed significantly worse than radiologists on average, and (2) on the
external set, none of the models performed statistically significantly worse
than radiologists, and five models performed statistically significantly
better than radiologists. Our results demonstrate that some chest X-ray
models, under clinically relevant distribution shifts, were comparable to
radiologists while other models were not. Future work should investigate
aspects of model training procedures and dataset collection that influence
generalization in the presence of data distribution shifts.

Generalizability, Distribution Shifts, Chest X-ray Interpretation, Radiology,
Clinical Deployment

††journalyear: 2021††copyright: rightsretained††conference: ACM Conference on
Health, Inference, and Learning; April 8–10, 2021; Virtual Event,
USA††booktitle: ACM Conference on Health, Inference, and Learning (ACM CHIL
’21), April 8–10, 2021, Virtual Event, USA††price: ††doi:
10.1145/3450439.3451876††isbn: 978-1-4503-8359-2/21/04††ccs: Applied computing
Health informatics††ccs: Computing methodologies Image representations Figure
1. We measured the diagnostic performance for 8 different chest X-ray models
when applied to (1) smartphone photos of chest X-rays and (2) external
datasets without any finetuning. All models were developed by different groups
and submitted to the CheXpert challenge, and re-applied to test datasets
without further tuning.

##  1\. Introduction

Chest X-rays are the most common imaging examination in the world, critical
for diagnosis and management of many diseases. With over 2 billion chest
X-rays performed globally annually, many clinics in both developing and
developed countries lack sufficient trained radiologists to perform timely
X-ray interpretation. Automating cognitive tasks in medical imaging
interpretation with deep learning models could improve access, efficiency, and
augment existing workflows (Rajpurkar et al., 2018; Nam et al., 2018; Singh et
al., 2018; Qin et al., 2018). However, poor generalization due to data
distribution shifts in clinical settings is a key barrier to implementation.

First, a major obstacle to clinical adoption of such technologies is in model
deployment, an effort often frustrated by vast heterogeneity of clinical
workflows across the world (Kelly et al., 2019). Chest X-ray models are
developed and validated using digital X-rays with many deployment solutions
relying on heavily integrated yet often disparate infrastructures (Qin et al.,
2019; Lakhani and Sundaram, 2017; Kallianos et al., 2019; Kashyap et al.,
2019; Shih et al., 2019; Andronikou et al., 2011; Schwartz et al., 2014). One
appealing solution to scaled deployment across disparate clinical frameworks
is to leverage the ubiquity of smartphones. Interpretation of medical imaging
via cell phone photography is an existing “store-and-forward telemedicine”
approach in which one or more photos of medical imaging are captured and sent
as email attachments or instant messages by practitioners to obtain second
opinions from specialists in routine clinical care (Goost et al., 2012;
Vassallo et al., 1998). Smartphone photographs have been shown to be of
sufficient diagnostic quality to allow for medical interpretation, thus
leveraging deep learning models in automated interpretation of photos of
medical imaging examinations may serve as an infrastructure agnostic approach
to deployment, particularly in resource limited settings. However, significant
technical barriers exist in automated interpretation of photos of chest
X-rays. Photographs of X-rays introduce visual artifacts which are not
commonly found in digital X-rays, such as altered viewing angles, variable
lighting conditions, glare, moiré, rotations, translations, and blur (Phillips
et al., 2020). These artifacts have been shown to reduce algorithm performance
when input images are perceived through a camera (Kurakin et al., 2016). The
extent to which such artifacts reduces the performance of chest X-ray models
has not been well investigated.

A second major obstacle to clinical adoption of chest X-ray models is that
clinical deployment requires models trained on data from one institution to
generalize to data from another institution (Kelly et al., 2019; Chen et al.,
2019). Early work has shown that chest X-ray models may not generalize well
when externally validated on data from a different institution and are
possibly vulnerable to distribution shift stemming from change in patient
population or rely on non-medically relevant cues between institutions (Zech
et al., 2018). However, the difference in diagnostic performance of more
recent chest X-ray models to external datasets has not been investigated.

We measured the diagnostic performance for 8 different chest X-ray models when
applied to (1) photos of chest X-rays, and (2) chest X-rays obtained at a
different institution. Specifically, we applied these models to a dataset of
smartphone photos of 668 X-rays from 500 patients, and a set of 420 frontal
chest X-rays from the ChestXray-14 dataset collected at the National
Institutes of Health Clinical Center (Wang et al., 2017). All models were
developed by different groups and submitted to the CheXpert challenge, a large
public competition for digital chest X-ray analysis (Irvin et al., 2019).
Models were evaluated on their diagnostic performance in binary
classification, as measured by Matthew’s Correlation Coefficient (MCC) (Chicco
and Jurman, 2020), on the following pathologies selected in Irvin et al.
(2019): atelectasis, cardiomegaly, consolidation, edema, and pleural effusion
(Irvin et al., 2019).

We found that:

  1. (1)

In comparison of model performance on digital chest X-rays to photos, all 8
models experienced a statistically significant drop in task performance on
photos with an average drop of 0.036 MCC. In comparison of performance of
models on photos compared to radiologist performance, three out of eight
models performed significantly worse than radiologists on average, and the
other five had no significant difference.

  2. (2)

On the external set (NIH), none of the models performed statistically
significantly worse than radiologists. On average over the pathologies, five
models performed significantly better than radiologists. On specific
pathologies (consolidation, cardiomegaly, edema, and atelectasis), there were
some models that achieved significantly better performance than radiologists.

Our systematic examination of the generalization capabilities of existing
models can be extended to other tasks in medical AI, and provide a framework
for tracking technical readiness towards clinical translation.

Figure 2. MCC differences of 8 chest X-ray models on different pathologies
between photos of the X-rays and the original X-rays with 95% confidence
intervals. Figure 3. MCC differences of the same models on photos of chest
X-rays compared to radiologist performance with 95% confidence intervals.
Figure 4. Comparison of the average AUC of 8 individual models on photos of
chest X-rays compared to on standard images

Metric | Comparison | Average | Pleural Effusion | Edema | Atelectasis | Consolidation | Cardiomegaly
---|---|---|---|---|---|---|---
AUC | Photos | 0.856 (0.840,0.869) | 0.950 (0.932,0.968) | 0.917 (0.884,0.943) | 0.882 (0.856,0.912) | 0.914 (0.865,0.946) | 0.921 (0.900,0.940)
| Standard | 0.871 (0.855,0.883) | 0.960 (0.944,0.975) | 0.926 (0.892,0.950) | 0.885 (0.858,0.910) | 0.918 (0.879,0.948) | 0.934 (0.914,0.951)
| Standard - Photos | 0.016 (0.012,0.019) | 0.011 (0.004,0.019) | 0.009 (0.001,0.018) | 0.003 (-0.006,0.013) | 0.005 (-0.009,0.016) | 0.013 (0.006,0.023)
MCC | Photos | 0.534 (0.507,0.559) | 0.571 (0.526,0.631) | 0.556 (0.481,0.639) | 0.574 (0.505,0.634) | 0.316 (0.246,0.386) | 0.580 (0.522,0.630)
| Standard | 0.570 (0.543,0.599) | 0.621 (0.575,0.670) | 0.550 (0.474,0.637) | 0.587 (0.529,0.640) | 0.336 (0.264,0.418) | 0.643 (0.584,0.695)
| Standard - Photos | 0.036 (0.024,0.048) | 0.049 (0.020,0.070) | -0.006 (-0.039,0.033) | 0.012 (-0.016,0.041) | 0.020 (-0.011,0.047) | 0.063 (0.036,0.084)

Table 1. AUC and MCC performance of models and radiologists on the standard
X-rays and the photos of chest X-rays, with 95% confidence intervals.

Comparison | Average | Pleural Effusion | Edema | Atelectasis | Consolidation | Cardiomegaly
---|---|---|---|---|---|---
Photos | 0.534 (0.507,0.559) | 0.571 (0.526,0.631) | 0.556 (0.481,0.639) | 0.574 (0.505,0.634) | 0.316 (0.246,0.386) | 0.580 (0.522,0.630)
Radiologists | 0.568 (0.542,0.597) | 0.671 (0.618,0.727) | 0.507 (0.431,0.570) | 0.548 (0.496,0.606) | 0.359 (0.262,0.444) | 0.566 (0.511,0.620)
Radiologists - Photos | 0.035 (0.009,0.065) | 0.099 (0.056,0.145) | -0.049 (-0.136,0.029) | -0.027 (-0.086,0.050) | 0.042 (-0.056,0.124) | -0.014 (-0.069,0.029)

Table 2. MCC performance of models on the photos of chest X-rays, radiologist
performance, and their difference, with 95% confidence intervals.

##  2\. Methods

###  2.1. Photos of Chest X-rays

We collected a test set of photos of chest x-rays, described in Phillips et
al. (2020). In this set, chest X-rays from each CheXpert test study were
displayed on a non-diagnostic computer monitor. Chest X-rays were displayed in
full screen on a computer monitor with $1920\times 1080$ screen resolution and
a black background. A physician was instructed to capture the photos, keeping
the mobile camera stable and center the lung fields in the camera view. A
time-restriction of 5 seconds per image was imposed to simulate a busy
healthcare environment. Subsequent inspection of photos showed that they were
taken with slightly varying angles; some photos included artefacts such as
Moiré patterns and subtle screen glare. Photos were labeled using the ground
truth for the corresponding digital X-ray image. The reference standard on
this set was determined using a majority vote of 5 board-certified
radiologists. Three separate board-certified radiologists were used for the
comparison against the models and all radiologists used the original chest
X-ray images for making their diagnoses, rather than the photos.

###  2.2. Running Models on New Test Sets

CheXpert used a hidden test set for official evaluation of models. Teams
submitted their executable code, which was then run on a test set that was not
publicly readable to preserve the integrity of the test results. We made use
of the CodaLab platform to re-run these chest X-ray models by substituting the
hidden CheXpert test set with the datasets used in this study.

###  2.3. Evaluation Metrics

Our primary evaluation metric was Matthew’s Correlation Coefficient (MCC), a
statistical rate which produces a high score only if the prediction obtained
good results in all of the four confusion matrix categories (true positives,
false negatives, true negatives, and false positives); MCC is proportionally
both to the size of positive elements and the size of negative elements in the
dataset (Chicco and Jurman, 2020).

We reported the average MCC of 8 models for five pathologies, namely
atelectasis, cardiomegaly, consolidation, edema, and pleural effusion.
Additionally, in experiments comparing the models on standard chest X-rays to
photos of chest X-rays, we reported the AUC and MCC of the models. In
experiments comparing models to board-certified radiologists, we reported the
difference in MCC for each of the five pathologies.

##  3\. Results

###  3.1. Model Performance on Photos of Chest X-rays vs Original X-rays.

####  3.1.1. Performance Drop In Application To Photos

In comparison of model performance on digital chest X-rays to photos, all
eight models experienced a statistically significant drop in task performance
on photos with an average drop of 0.036 MCC (95% CI 0.024, 0.048) (See Figure
2, Table 1). All models had a statistically significant drop on at least one
of the pathologies between native digital image to photos. One model had a
statistically significant drop in performance on three pathologies: pleural
effusion, edema, and consolidation. Two models had a significant drop on two
pathologies: one on pleural effusion and edema, and the other on pleural
effusion and cardiomegaly. The cardiomegaly and pleural effusion tasks led to
decreased performance in five and four models respectively.

####  3.1.2. Performance on Photos In Comparison to Radiologist Performance on
Standard Images

In comparison of performance of models on photos compared to radiologist
performance, three out of eight models performed significantly worse than
radiologists on average, and the other five had no significant difference (see
Figure 3). On specific pathologies, there were some models that had a
significantly higher performance than radiologists: two models on
cardiomegaly, and one model on edema. Conversely, there were some models that
had a significantly lower performance than radiologists: two models on
cardiomegaly, and one model on consolidation. The pathology with the greatest
number of models that had a significantly lower performance than radiologists
was pleural effusion (seven models).

####  3.1.3. Performance drop in context of radiologist performance

Our results demonstrated that while most models experienced a significant drop
in performance when applied to photos of chest X-rays compared to the native
digital image, their performance was nonetheless largely equivalent to
radiologist performance. We found that although there were thirteen times that
models had a statistically significant drop in performance on photos on the
different pathologies, the models had significantly lower performance than
radiologists only 6 of those 13 times. Comparison to radiologist performance
provides context in regard to clinical applicability: several models remained
comparable to radiologist performance standard despite decreased performance
on photos. Further investigation could be directed towards understanding how
different model training procedures may affect model generalization to photos
of chest X-rays, and understanding etiologies behind trends for changes in
performance for specific pathologies or specific artifacts.

Figure 5. MCC differences in performance of models on the CheXpert test set,
with 95% confidence intervals (higher than 0 is in favor of the models being
better). Figure 6. MCC differences in performance of the same models compared
to another set of radiologists across the same pathologies on an external
institution’s (NIH) data.

####  3.1.4. Implication

While using photos of chest X-rays to input into chest X-ray algorithms could
enable any physician with a smartphone to get instant AI algorithm assistance,
the performance of chest X-ray algorithms on photos of chest X-rays has not
been thoroughly investigated. Several studies have highlighted the importance
of generalizability of computer vision models with noise in (Hendrycks and
Dietterich, 2019). Dodge and Karam (2017) demonstrated that deep neural
networks perform poorly compared to humans on image classification on
distorted images. Geirhos et al. (2019), Schmidt et al. (2018) have found that
convolutional neural networks trained on specific image corruptions did not
generalize, and the error patterns of network and human predictions were not
similar on noisy and elastically deformed images.

Institution | Comparison | Average | Pleural Effusion | Edema | Atelectasis | Consolidation | Cardiomegaly
---|---|---|---|---|---|---|---
CheXpert | Radiologists | 0.568 (0.542,0.597) | 0.671 (0.618,0.727) | 0.507 (0.431,0.570) | 0.548 (0.496,0.606) | 0.359 (0.262,0.444) | 0.566 (0.511,0.620)
| Models | 0.570 (0.543,0.599) | 0.621 (0.575,0.670) | 0.550 (0.474,0.637) | 0.587 (0.529,0.640) | 0.336 (0.264,0.418) | 0.643 (0.584,0.695)
| Models - Radiologists | 0.002 (-0.028,0.030) | -0.05 (-0.092,-0.007) | 0.043 (-0.033,0.114) | 0.039 (-0.029,0.106) | -0.022 (-0.104,0.076) | 0.077 (0.040,0.135)
NIH | Radiologists | 0.537 (0.515,0.555) | 0.642 (0.590,0.690) | 0.618 (0.549,0.669) | 0.469 (0.423,0.515) | 0.455 (0.385,0.509) | 0.492 (0.443,0.530)
| Models | 0.578 (0.551,0.601) | 0.673 (0.605,0.734) | 0.662 (0.582,0.742) | 0.529 (0.454,0.595) | 0.551 (0.499,0.623) | 0.517 (0.466,0.567)
| Models - Radiologists | 0.041 (0.010,0.072) | 0.032 (-0.019,0.078) | 0.044 (-0.028,0.124) | 0.060 (-0.003,0.126) | 0.096 (0.027,0.155) | 0.025 (-0.028,0.078)

Table 3. MCC performance of models and radiologists on the CheXpert and NIH
sets of chest X-rays, and their difference, with 95% confidence intervals.
Figure 7. Overall change in performance of models (blue) and radiologists
(orange) across CheXpert and the external institution dataset (NIH).

###  3.2. Comparison of Models and Radiologists on External Institution

We measured the change in diagnostic performance of the same eight chest X-ray
models on chest X-rays obtained at a different institution. We applied these
models, trained on the CheXpert dataset from the Stanford Hospital, to a set
of 420 frontal chest X-rays labeled as part of Rajpurkar et al. (2018). These
X-rays are sourced from the ChestXray-14 dataset collected at the National
Institutes of Health Clinical Center (Wang et al., 2017), and sampled to
contain at least 50 cases of each pathology according to the original labels
provided in the dataset. The reference standard on this set (NIH) was
determined using a majority vote of three cardiothoracic subspecialty
radiologists; six board-certified radiologists were used for comparison
against the models.

####  3.2.1. Performance on external institution in comparison to radiologists

On the external set (NIH), none of the models performed statistically
significantly worse than radiologists (see Figure 6). On average over the
pathologies, five models performed significantly better than radiologists. On
specific pathologies, there were some models that achieved significantly
better performance than radiologists: six models on consolidation, three
models on cardiomegaly, four on edema, and two on atelectasis, one on pleural
effusion.

####  3.2.2. Implication

Our finding that these models perform comparably to or at a level exceeding
radiologists differs from a previous study which reported that a chest X-ray
model failed to generalize to new populations or institutions separate from
the training data, relying on institution specific and/or confounding cues to
infer the label of interest (Zech et al., 2018). Our findings may be
attributed to the improvement in the generalizability of chest X-ray models
owing to larger and higher-quality datasets that have been publicly released
(Irvin et al., 2019; Johnson et al., 2019) Future work should investigate
specific aspects of model training and dataset quality and size that lend to
these differences, and whether self-supervised training procedures (Sowrirajan
et al., 2020) increase generalizability across institutions.

####  3.2.3. Performance change in context of radiologist performance

Comparing performances on the CheXpert and NIH test sets, we found that on the
NIH data set, in 16 instances models had a significantly better performance
than radiologists; on the internal CheXpert test set, we observed that in 6
instances, models had a significantly higher performance than radiologists
(see Figure 5). This difference may be attributed to a variety of factors
including the difference in prevalence of pathologies or the difficulty in
identifying them in the external test set compared to the internal set. We are
able to contextualize the generalization ability of models to external
institutions by comparing their differences to a radiologist performance
benchmark, rather than provide a comparison of their absolute performances,
which would not control for these possible differences. For instance, when
considering cardiomegaly (see Figure 7), we observe a drop in model
performance, which in isolation would indicate poor generalizability. However,
in light of a similar drop in radiologist performance, we may be able to
attribute the difference to differences in difficulties between the two
datasets.

##  4\. Discussion

The purpose of this work was to systematically address the key translation
challenges for chest X-ray models in clinical application to common real-world
scenarios. We found that several chest X-ray models had a drop in performance
when applied to smartphone photos of chest X-rays, but even with this drop,
some models still performed comparably to radiologists. We also found that
when models were tested on an external institution’s data, they performed
comparably to radiologists. In both forms of clinical distribution shifts we
found that high-performance chest X-ray interpretation models trained on
CheXpert produced clinically useful diagnostic performance.

Our work makes significant contributions over another investigation of chest
X-ray models (Rajpurkar et al., 2020). While their study considered the
differences in AUC of models when applied to photos of X-rays, they did not
(1) compare the resulting performances against radiologists, (2) investigate
the drop in performances on specific tasks, or (3) analyze drops in
performances of individual models across tasks. Finally, while they compared
the performance of models to radiologists on an external dataset, they did not
investigate the change in performance of models between the internal dataset
and the external dataset.

Strengths of our study include our systematic investigation of generalization
performance of several chest X-ray models developed by different teams.
Limitations of our work include that our study is still retrospective in
nature, and prospective studies would further advance understanding of
generalization under distribution shifts. Our systematic examination of the
generalization capabilities of existing models can be extended to other tasks
in medical AI (Duan et al., 2019; Huang et al., 2020; Kiani et al., 2020;
Varma et al., 2019; Topol, 2019), and provide a framework for tracking
technical readiness towards clinical translation.

## References

  * (1)
  * Andronikou et al. (2011) Savvas Andronikou, Kieran McHugh, Nuraan Abdurahman, Bryan Khoury, Victor Mngomezulu, William E Brant, Ian Cowan, Mignon McCulloch, and Nathan Ford. 2011.  Paediatric radiology seen from Africa. Part I: providing diagnostic imaging to a young population.  _Pediatric radiology_ 41, 7 (2011), 811–825.
  * Chen et al. (2019) David Chen, Sijia Liu, Paul Kingsbury, Sunghwan Sohn, Curtis B. Storlie, Elizabeth B. Habermann, James M. Naessens, David W. Larson, and Hongfang Liu. 2019\.  Deep learning and alternative learning strategies for retrospective real-world clinical data.  _npj Digital Medicine_ 2, 1 (Dec. 2019), 43\.  https://doi.org/10.1038/s41746-019-0122-0
  * Chicco and Jurman (2020) Davide Chicco and Giuseppe Jurman. 2020.  The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation.  _BMC genomics_ 21, 1 (2020), 6.
  * Dodge and Karam (2017) Samuel Dodge and Lina Karam. 2017.  A Study and Comparison of Human and Deep Learning Recognition Performance under Visual Distortions. In _2017 26th International Conference on Computer Communication and Networks (ICCCN)_. 1–7.  https://doi.org/10.1109/ICCCN.2017.8038465
  * Duan et al. (2019) Tony Duan, Pranav Rajpurkar, Dillon Laird, Andrew Y. Ng, and Sanjay Basu. 2019.  Clinical Value of Predicting Individual Treatment Effects for Intensive Blood Pressure Therapy: A Machine Learning Experiment to Estimate Treatment Effects from Randomized Trial Data.  _Circulation: Cardiovascular Quality and Outcomes_ 12, 3 (March 2019).  https://doi.org/10.1161/CIRCOUTCOMES.118.005010
  * Geirhos et al. (2019) Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. 2019.  ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness.  _arXiv:1811.12231 [cs, q-bio, stat]_ (Jan. 2019).
  * Goost et al. (2012) Hans Goost, Johannes Witten, Andreas Heck, Dariusch R Hadizadeh, Oliver Weber, Ingo Gräff, Christof Burger, Mareen Montag, Felix Koerfer, and Koroush Kabir. 2012\.  Image and diagnosis quality of X-ray image transmission via cell phone camera: a project study evaluating quality and reliability.  _PLoS One_ 7, 10 (2012), e43402.
  * Hendrycks and Dietterich (2019) Dan Hendrycks and Thomas Dietterich. 2019.  Benchmarking Neural Network Robustness to Common Corruptions and Perturbations.  _arXiv:1903.12261 [cs, stat]_ (March 2019).
  * Huang et al. (2020) Shih-Cheng Huang, Tanay Kothari, Imon Banerjee, Chris Chute, Robyn L Ball, Norah Borus, Andrew Huang, Bhavik N Patel, Pranav Rajpurkar, Jeremy Irvin, et al. 2020\.  PENet—A scalable deep-learning model for automated diagnosis of pulmonary embolism using volumetric CT imaging.  _NPJ digital medicine_ 3, 1 (2020), 1–9.
  * Irvin et al. (2019) Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, Jayne Seekins, David A. Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, Curtis P. Langlotz, Bhavik N. Patel, Matthew P. Lungren, and Andrew Y. Ng. 2019\.  CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 33 (July 2019), 590–597.  https://doi.org/10.1609/aaai.v33i01.3301590
  * Johnson et al. (2019) Alistair E. W. Johnson, Tom J. Pollard, Seth J. Berkowitz, Nathaniel R. Greenbaum, Matthew P. Lungren, Chih-ying Deng, Roger G. Mark, and Steven Horng. 2019.  MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports.  _Scientific Data_ 6, 1 (Dec. 2019), 317\.  https://doi.org/10.1038/s41597-019-0322-0
  * Kallianos et al. (2019) K. Kallianos, J. Mongan, S. Antani, T. Henry, A. Taylor, J. Abuya, and M. Kohli. 2019\.  How far have we come? Artificial intelligence for chest radiograph interpretation.  _Clinical Radiology_ 74, 5 (May 2019), 338–345.  https://doi.org/10.1016/j.crad.2018.12.015
  * Kashyap et al. (2019) Satyananda Kashyap, Mehdi Moradi, Alexandros Karargyris, Joy T. Wu, Michael Morris, Babak Saboury, Eliot Siegel, and Tanveer Syeda-Mahmood. 2019.  Artificial intelligence for point of care radiograph quality assessment. In _Medical Imaging 2019: Computer-Aided Diagnosis_ , Vol. 10950. International Society for Optics and Photonics, 109503K.  https://doi.org/10.1117/12.2513092
  * Kelly et al. (2019) Christopher J. Kelly, Alan Karthikesalingam, Mustafa Suleyman, Greg Corrado, and Dominic King. 2019\.  Key challenges for delivering clinical impact with artificial intelligence.  _BMC Medicine_ 17, 1 (Dec. 2019), 195\.  https://doi.org/10.1186/s12916-019-1426-2
  * Kiani et al. (2020) Amirhossein Kiani, Bora Uyumazturk, Pranav Rajpurkar, Alex Wang, Rebecca Gao, Erik Jones, Yifan Yu, Curtis P. Langlotz, Robyn L. Ball, Thomas J. Montine, Brock A. Martin, Gerald J. Berry, Michael G. Ozawa, Florette K. Hazard, Ryanne A. Brown, Simon B. Chen, Mona Wood, Libby S. Allard, Lourdes Ylagan, Andrew Y. Ng, and Jeanne Shen. 2020.  Impact of a deep learning assistant on the histopathologic classification of liver cancer.  _npj Digital Medicine_ 3, 1 (Dec. 2020), 23\.  https://doi.org/10.1038/s41746-020-0232-8
  * Kurakin et al. (2016) Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. 2016\.  Adversarial examples in the physical world.  _CoRR_ abs/1607.02533 (2016).  _eprint: 1607.02533.
  * Lakhani and Sundaram (2017) Paras Lakhani and Baskaran Sundaram. 2017.  Deep Learning at Chest Radiography: Automated Classification of Pulmonary Tuberculosis by Using Convolutional Neural Networks.  _Radiology_ 284, 2 (April 2017), 574–582.  https://doi.org/10.1148/radiol.2017162326
  * Nam et al. (2018) Ju Gang Nam, Sunggyun Park, Eui Jin Hwang, Jong Hyuk Lee, Kwang-Nam Jin, Kun Young Lim, Thienkai Huy Vu, Jae Ho Sohn, Sangheum Hwang, Jin Mo Goo, and others. 2018.  Development and validation of deep learning–based automatic detection algorithm for malignant pulmonary nodules on chest radiographs.  _Radiology_ 290, 1 (2018), 218–228.  Publisher: Radiological Society of North America.
  * Phillips et al. (2020) Nick A. Phillips, Pranav Rajpurkar, Mark Sabini, Rayan Krishnan, Sharon Zhou, Anuj Pareek, Nguyet Minh Phu, Chris Wang, Andrew Y. Ng, and Matthew P. Lungren. 2020\.  CheXphoto: 10,000+ Smartphone Photos and Synthetic Photographic Transformations of Chest X-rays for Benchmarking Deep Learning Robustness.  arXiv:2007.06199 [eess.IV]
  * Qin et al. (2018) Chunli Qin, Demin Yao, Yonghong Shi, and Zhijian Song. 2018\.  Computer-aided detection in chest radiography based on artificial intelligence: a survey.  _BioMedical Engineering OnLine_ 17, 1 (Aug. 2018), 113.  https://doi.org/10.1186/s12938-018-0544-y
  * Qin et al. (2019) Zhi Zhen Qin, Melissa S. Sander, Bishwa Rai, Collins N. Titahong, Santat Sudrungrot, Sylvain N. Laah, Lal Mani Adhikari, E. Jane Carter, Lekha Puri, Andrew J. Codlin, and Jacob Creswell. 2019.  Using artificial intelligence to read chest radiographs for tuberculosis detection: A multi-site evaluation of the diagnostic accuracy of three deep learning systems.  _Scientific Reports_ 9, 1 (Oct. 2019), 1–10.  https://doi.org/10.1038/s41598-019-51503-3
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Jeremy Irvin, Robyn L. Ball, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis P. Langlotz, Bhavik N. Patel, Kristen W. Yeom, Katie Shpanskaya, Francis G. Blankenberg, Jayne Seekins, Timothy J. Amrhein, David A. Mong, Safwan S. Halabi, Evan J. Zucker, Andrew Y. Ng, and Matthew P. Lungren. 2018.  Deep learning for chest radiograph diagnosis: A retrospective comparison of the CheXNeXt algorithm to practicing radiologists.  _PLOS Medicine_ 15, 11 (Nov. 2018), e1002686.  https://doi.org/10.1371/journal.pmed.1002686
  * Rajpurkar et al. (2020) Pranav Rajpurkar, Anirudh Joshi, Anuj Pareek, Phil Chen, Amirhossein Kiani, Jeremy Irvin, Andrew Y. Ng, and Matthew P. Lungren. 2020\.  CheXpedition: Investigating Generalization Challenges for Translation of Chest X-Ray Algorithms to the Clinical Setting.  arXiv:2002.11379 [eess.IV]
  * Schmidt et al. (2018) Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. 2018\.  Adversarially Robust Generalization Requires More Data.  In _Advances in Neural Information Processing Systems 31_ , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.). Curran Associates, Inc., 5014–5026.
  * Schwartz et al. (2014) Adam B Schwartz, Gina S Siddiqui, John L Barbieri, Amana F Akhtar, Woojin K Kim, Ryan A Littman-Quinn, Emily S Conant, Narainder D Gupta, Bryan A Pukenas, Parvati H Ramchandani, and et al. 2014\.  The accuracy of mobile teleradiology in the evaluation of chest X-rays.  _Journal of Telemedicine and Telecare_ (Oct. 2014).  Publisher: Journal of Telemedicine and Telecare.
  * Shih et al. (2019) George Shih, Carol C. Wu, Safwan S. Halabi, Marc D. Kohli, Luciano M. Prevedello, Tessa S. Cook, Arjun Sharma, Judith K. Amorosa, Veronica Arteaga, Maya Galperin-Aizenberg, Ritu R. Gill, Myrna C.B. Godoy, Stephen Hobbs, Jean Jeudy, Archana Laroia, Palmi N. Shah, Dharshan Vummidi, Kavitha Yaddanapudi, and Anouk Stein. 2019\.  Augmenting the National Institutes of Health Chest Radiograph Dataset with Expert Annotations of Possible Pneumonia.  _Radiology: Artificial Intelligence_ 1, 1 (Jan. 2019), e180041.  https://doi.org/10.1148/ryai.2019180041
  * Singh et al. (2018) Ramandeep Singh, Mannudeep K. Kalra, Chayanin Nitiwarangkul, John A. Patti, Fatemeh Homayounieh, Atul Padole, Pooja Rao, Preetham Putha, Victorine V. Muse, Amita Sharma, and Subba R. Digumarthy. 2018\.  Deep learning in chest radiography: Detection of findings and presence of change.  _PLoS ONE_ 13, 10 (Oct. 2018).  https://doi.org/10.1371/journal.pone.0204155
  * Sowrirajan et al. (2020) Hari Sowrirajan, Jingbo Yang, Andrew Y. Ng, and Pranav Rajpurkar. 2020.  MoCo Pretraining Improves Representation and Transferability of Chest X-ray Models.  arXiv:2010.05352 [cs.CV]
  * Topol (2019) Eric J. Topol. 2019\.  High-performance medicine: the convergence of human and artificial intelligence.  _Nature Medicine_ 25, 1 (Jan. 2019), 44–56.  https://doi.org/10.1038/s41591-018-0300-7
  * Varma et al. (2019) Maya Varma, Mandy Lu, Rachel Gardner, Jared Dunnmon, Nishith Khandwala, Pranav Rajpurkar, Jin Long, Christopher Beaulieu, Katie Shpanskaya, Li Fei-Fei, Matthew P. Lungren, and Bhavik N. Patel. 2019.  Automated abnormality detection in lower extremity radiographs using deep learning.  _Nature Machine Intelligence_ 1, 12 (Dec. 2019), 578–583.  https://doi.org/10.1038/s42256-019-0126-0
  * Vassallo et al. (1998) DJ Vassallo, PJ Buxton, JH Kilbey, and M Trasler. 1998\.  The first telemedicine link for the British Forces.  _Journal of the Royal Army Medical Corps_ 144, 3 (1998), 125–130.
  * Wang et al. (2017) Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. 2017.  Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2097–2106.
  * Zech et al. (2018) John R. Zech, Marcus A. Badgeley, Manway Liu, Anthony B. Costa, Joseph J. Titano, and Eric Karl Oermann. 2018.  Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study.  _PLOS Medicine_ 15, 11 (Nov. 2018), e1002683.  https://doi.org/10.1371/journal.pmed.1002683