[table]capposition=top

# [SoK] The Challenges of Machine Learning for Trust and Safety:
A Case Study on Misinformation Detection

Madelyne Xiao
Princeton University  Jonathan Mayer
Princeton University

###### Abstract

We examine the disconnect between scholarship and practice in applying machine
learning to trust and safety problems, using misinformation detection as a
case study. We systematize literature on automated detection of misinformation
across a corpus of 270 well-cited papers in the field. We then examine subsets
of papers for data and code availability, design missteps, reproducibility,
and generalizability.

We find significant shortcomings in the literature that call into question
claimed performance and practicality. Detection tasks are often meaningfully
distinct from the challenges that online services actually face. Datasets and
model evaluation are often non-representative of real-world contexts, and
evaluation frequently is not independent of model training. Data and code
availability is poor. Models do not generalize well to out-of-domain data.

Based on these results, we offer recommendations for evaluating machine
learning applications to trust and safety problems. Our aim is for future work
to avoid the pitfalls that we identify.

##  1 INTRODUCTION

Online services face a daunting task: There is an unceasing deluge of user-
generated content, on the order of hundreds of thousands of posts per minute
on certain well-known social media platforms [259]. Some of that content is
false, hateful, harassing, extremist, or otherwise problematic. How can
platforms reliably and proactively identify these “trust and safety” issues?

Machine learning has proven an attractive approach in the academic literature,
leading to large bodies of scholarship on misinformation detection, toxic
speech classification, and other core trust and safety challenges [146, 341,
342]. The conceptual appeal of machine learning is that it could address the
massive scale of user-generated content on large platforms and the capacity
constraints of small platforms. Recent work claims impressive performance
statistics, and that near-term deployment is practical. As a case in point,
OpenAI recently announced that it was deploying a content moderation system
based on GPT-4 to detect problematic content online; according to the company,
performance on detection tasks exceeds that of human moderators with “modest
training.” Even so, OpenAI’s head of safety systems admitted that human
advisors would still be necessary to “adjudicate borderline cases” [339]. This
accords with our finding that, in practice, trust and safety functions at
online services remain largely manual—driven by user reports and carried out
by human moderators. Additionally, thresholds (e.g., detection precision or
recall) for determining where and when automated processes require human
intervention are underspecified [336, 337].

In this work, we investigate the significant disconnect between scholarship
and practice in applications of machine learning to trust and safety problems.
Our project is inspired by recent research that has identified shortcomings in
machine learning applications for many problem domains, including information
security [315, 320]. We use misinformation detection as a case study for trust
and safety problems, because the topic has recently generated a rich
literature with diverse methods and claimed successes. Misinformation
detection also has substantive complexities that are common for trust and
safety problems: linguistic and cultural nuance, sensitivity to context, and
rapidly evolving circumstances.

Evaluating responses to misinformation is societally urgent: The veracity of
published news is vitally important to democratic processes, public health,
and press integrity. This was particularly evident in 2016, during the U.S.
presidential election, and again in 2020, in the first year of the COVID-19
pandemic. During those years, false and misleading information was rampant on
well-known social media platforms [287, 316, 317, 319]. In response to public
outcry over coordinated misinformation campaigns during the 2016 presidential
election, Twitter and Facebook announced new measures in 2017 to counteract
the spread of misinformation, including automated misinformation detection
methods that would surface potentially false rumors for further vetting by
third-party fact-checking organizations [324, 323, 325].

These machine learning approaches do not appear to have been successful.
Leaked Facebook documents from the time of the COVID-19 pandemic revealed that
the platform’s “ability to detect VH [vaccine hesitancy] comments is bad in
English—and basically non-existent elsewhere” [336]. Employees acknowledged
that automated interventions—including machine learning classifiers—were
grossly underequipped to counter vaccine-related misinformation.

Meanwhile, in the academic literature, publications routinely report
optimistic performance figures for machine learning models to detect
misinformation. In the literature review that we conduct for this work, among
publications that report performance metrics, about 70% of papers report over
80% accuracy on at least one detection task (see SI); some of these works
report near-perfect performance. What explains this discrepancy between
scholarship and practice?

We seek to answer four discrete research questions, which collectively shed
light on the shortcomings of automated misinformation detection.

  * RQ1.

How well-suited are existing misinformation detection methods to the needs of
online services, specifically social media platforms that host user-generated
content?

  * RQ2.

Are there identifiable errors in dataset curation and method development in
ML-driven misinformation detection studies?

  * RQ3.

How reproducible are published ML-driven misinformation detection methods?

  * RQ4.

How generalizable are published ML-driven misinformation detection methods to
out-of-domain data (i.e., text corpora that discuss topics that don’t appear
in the training set)?

We address these research questions in three ways. First, we conduct a broad
literature review and synthesis. We taxonomize existing ML-driven
misinformation detection systems, with a focus on feature types, detection
targets, and errors in method development and dataset design. Second, we
attempt to obtain code and data for a subset of prior work. Third, we test
several widely-cited approaches for replication and generalizability. We
arrive at the following results by applying these methods.

  * •

Detection tasks in scholarship are often steps removed from the misinformation
content moderation challenges that platforms face. Detection targets may be of
limited consequence or may be readily identifiable through manual means.

  * •

Datasets used in publications are often unrealistic. Model evaluation often
lacks independence from training and rarely involves close emulation of a
real-world deployment.

  * •

Data and code availability problems pervade the literature, inhibiting
replication. Where these are available, we are generally able to replicate
prior work.

  * •

Prior work has poor generalizability when classifying content beyond what was
included in the training dataset.

Based on these observations, we recommend a rubric for future work that
proposes ML models to identify trust and safety concerns.

Paper organization. We organize this work as follows:

  * •

Section 2 describes our methodologies for 1) selecting and curating papers in
our corpus of misinformation detection papers, 2) conducting outreach to paper
authors, and 3) performing replication analyses. We also discuss the
information taxonomies that we produce from our inductive coding of our paper
corpus.

  * •

In Section 3, we situate this project within academic literature that
critiques machine learning-driven solutions to subfield-specific problems. In
particular, we motivate our choice of automated misinformation detection as a
case study for understanding the efficacy of ML-driven solutions to trust and
safety problems.

  * •

We present the major takeaways of our literature review and replication
analyses in Section 4. In Section 4.2, we focus on observations from our
reading and coding of papers. In particular, we discuss issues pertaining to
method-target fit, dataset curation, and method design (RQ1, RQ2) within each
information scope. (We define and discuss our process for developing
information scopes in Section 2.) In Section 4.3, we illustrate the issues
identified in our literature review with a series of replication studies, with
a focus on generalizability and reproducibility of results (RQ3, RQ4).

  * •

In Section 5, we conclude with 1) a discussion of findings from our literature
review and replication studies and 2) generalizable recommendations for
evaluating ML-driven interventions in trust and safety settings, synthesized
from our own reading and replication studies.

Through this work, we hope to underscore 1) the prevalence and severity of
reproducibility and ML-driven method design issues in the existing
misinformation detection literature, and 2) the need for careful and
preemptive evaluation of ML-driven methods at the point of problem
formulation. We contribute information taxonomies, our annotated corpus of 270
research papers, recent datasets, and frameworks for evaluating ML-driven fact
classification tasks.

##  2 METHODS

In this section, we describe the following: 1) terminology and taxonomies, 2)
our methodology for selecting papers for our literature review, and 3) our
approach to replication analysis and author outreach.

Definitions. In the absence of a single, unified definition of misinformation,
we develop a working definition of misinformation from existing social science
and media studies literature as any piece of information that is verifiably
misleading, false, or suggestive of falsehood (e.g., hedging, leading
questions) [243, 21, 20, 22, 18]. Within this definition, we also include
misinformation that is transparently false, but not intended to mislead:
satire, parody, spoofs. We limit our analysis to English-language
misinformation.111We acknowledge that the nature of misinformation narratives
and misinformation spread varies with language and geography. We note that
disinformation is commonly used to refer to incorrect information written with
the intent to deceive, while misinformation refers to incorrect information in
general; for the sake of completeness and concision, we refer to intentionally
and unintentionally false information as misinformation for this work. We
occasionally use false news, false information, and misinformative news
interchangeably in this work, but avoid fake news, a politically fraught term.
Additionally, a good deal of existing work in computational linguistics
examines the automatic detection of lying or deliberately deceptive language
in text [307, 309, 308]. While the authors of some misinformative texts might
write with the intention to deceive, this is not a baseline assumption we make
when analyzing texts—in general, we are interested in all verifiable claims,
regardless of authorial intent.

Taxonomies. Through an iterative process of reading and inductive coding of
our papers, we develop a taxonomy of five “information scopes.” As we read
each paper in our corpus, we noted the operative unit of misinformation
detection for the classifier or method described—the smallest semantic or
organizational unit of information that the method attempted to classify as
true or false—and sorted each paper into emergent categories. One reader made
three separate coding passes over the corpus, varying the objectives of her
annotation on each pass: In the first pass, she specifically sought to
identify taxonomies for classification and detection; on the second, she noted
method design errors and performance scores; on the final pass, she noted
actual versus stated detection targets. Our final taxonomy comprises five
information scopes: claims, news articles, social media posts, users/authors,
and websites. With the exception of claims, we consider these scopes in the
context of online services—e.g., evaluation of news articles and websites
whose links might be posted to a social media platform. We note three
principle approaches to misinformation detection: direct verification of
claims, detection of deceptive language, and analysis of propagation patterns
or context signals. We also note the types of features included in each
model’s feature set, and sort observed feature sets into five categories:
textual, network-based, infrastructural, user- or author credential-based, and
multimodal features. We also note performance and accuracy scores of published
methods, if the authors report this information. Additional details about our
corpus construction and paper coding process can be found in Section 2; our
annotation guide can be found in our SI.

We adopt the terminology and taxonomies of previous work in our examination of
ML errors [304, 305, 315]. In the context of data mining, Kaufman et al.
define leakage as “the introduction of information about the target of a data
mining problem that should not be legitimately available to mine from” [305].
(An example of this, encountered in both machine learning and data science
contexts, is training of a model on inputs that also belong to a test
dataset.) Here, we cast a broader net and examine other, related design errors
in the context of ML-based methods, including those described by Kapoor et al.
(their work also includes dependencies and duplicates in datasets, temporal
leakage, and poor feature selection within their taxonomy of ML design errors)
[304]. In general, these errors lead to inflated model performance in an
experimental setting and underperformance in real-world deployments. Though
the precise nature of these errors varies by dataset, information scope, and
choice of model, we broadly categorize them as errors related to dataset
curation, feature set selection and model training. Concise visual summaries
of our text corpus and taxonomies are available in Section 4.1 and our SI.

###  2.1 Literature Survey Design

Corpus construction. To seed our corpus, we manually curated a selection of 23
highly-cited survey papers that provide comprehensive overviews of the state
of automated misinformation detection at the time of writing [143, 144, 145,
146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160,
161, 162, 163, 164, 165]. We relied on these papers to ground our own
understanding of existing approaches to automated misinformation detection,
and to identify detection methods that have been well-received by the research
community. We searched for survey papers in Google Scholar with queries
“survey misinformation detection" and “survey fake news detection" and
collected the most-frequently cited papers within the past 10 years.222This
time frame was naturally enforced by a lack of well-cited older publications,
and was not fixed before we began our sampling process. We then inspected each
paper’s reference section for relevant titles.

We supplemented this core corpus of highly-cited papers with publications
surfaced by Google Scholar queries. In particular, we filtered Google Scholar
search results on the following terms: “misinformation detection [$x$]” and
“automated fact checking [$x$],” where $x\in\\{\text{claims, posts, news
articles, sources, websites}\\}$. We collected the 50 most highly-cited papers
in the set resulting from the union of search results returned by both search
queries for each $x$. In order to counter potential bias towards older
publications, we collected those papers with the highest citation rates per
year.

We note that many papers surfaced by these queries were published in
computational linguistics and NLP research venues. To ensure that security-
oriented approaches to detection were also represented in our corpus, we
oversampled publications from four prominent security research venues (USENIX,
IEEE S&P, NDSS, and ACM CCS) that address detection. To account for the
paucity of literature that explicitly discusses misinformation detection in
these venues, we included papers that consider adjacent topics, such as spam
and Sybil detection. Our final corpus comprises 270 papers published between
2009 and 2023, inclusive.

Corpus curation. As a first-pass relevancy check, we used automated keyword
matching to confirm that all collected papers did, in fact, address
misinformation and automated fact-checking methods; in the case of any
ambiguity, we skimmed the abstract of the paper in question. We then annotated
research papers that passed this check for scope of detection method (e.g.,
article-level, claim-level) and feature set under analysis (e.g., stylometric
features, source credibility); our annotation guide appears in our online SI.

###  2.2 Reproducibility and Replicability

Toward understanding the reproducibility of existing detection methods (RQ3),
we conducted replication analyses of a selection of published methods in our
corpus; where necessary, we contacted paper authors to request access to data
and code. We describe our methodology for both of these processes in this
section.

Paper selection and author outreach. We sorted our full text corpus by
information scope. Within each scope, we sorted papers in order of decreasing
citation count. We then proceeded as follows:

  1. 1.

We attempted to source the full methods and datasets for the most cited paper
within each information scope.

  2. 2.

If we were unable to find this information during an independent web search,
we reached out to the paper’s lead author(s) to request access.

  3. 3.

If this request was unsuccessful—if the author did not respond, or confirmed
that the dataset or code was no longer available—we returned to step 1 for the
next most highly-cited paper in our corpus within that scope.

Replication analyses. In order to reproduce results published in a selection
of papers and perform cross-cutting analyses on out-of-domain and out-of-
sample datasets, we conduct a series of replication analyses on a subset of
papers. We chose methods from disparate information scopes, and which consider
a variety of different feature types. We attempted to reproduce results
reported in each paper on the datasets mentioned therein (we contacted authors
when necessary to obtain datasets and code). We developed the following
framework for evaluating the reproducibility and generalizability of existing
methods:

  * •

Reproducibility. We reproduce published results with code and data reported in
the original publication. We evaluate availability of code and data and, where
possible, compare our analysis outcomes with those reported in the original
paper (RQ3).

  * •

Explainability. Toward understanding the contributions of specific feature
types to overall classifier performance, and why certain approaches work (or
don’t work), we perform feature ablation studies when appropriate.

  * •

Replicability and generalizability. Model perfor-mance on novel datasets will
be useful for determining the generalizability of existing detection methods
to different contexts (RQ4). For models that were explicitly tested on
specific misinformation narratives (e.g., 2020 stolen election narratives), on
specific timeframes, or on specific types of misinformation (parody, satire,
etc), we develop current datasets to test method performance on diverse
information domains. As much as possible, we attempt to imitate real world
contexts.

Ethical Considerations Datasets used for this study were already publicly
available or were obtained with permission from study authors. Datasets were
anonymized by their original authors. We submitted our outreach study as a
protocol and received approval from our institutional review board.

##  3 RELATED WORK

Relevance to existing trust and safety operations. We situate our work within
existing literature—published in both academic and non-academic venues—that
describes trust and safety operations in commercial settings (i.e., social
media platforms). Though Twitter and Facebook have previously advertised
human-in-the-loop processes for automatically flagging potentially
misinformative posts for manual vetting by third-party fact-checking
organizations, criteria for deploying tiered responses and seeking human
intervention are murky [323]. (A larger volume of user reports might be
necessary for takedowns versus warnings, and classifier recall thresholds for
requiring human intervention might be higher for COVID-19 narratives versus
celebrity gossip, for instance.) Furthermore, evidence from the leaked
Facebook Files suggests that automated content moderation methods rely heavily
on proxy signals for identifying hate speech and misinformation on the
platform; as a result, automated approaches were generally employed for first-
line detection of putative rumoring narratives, with the burden of actual fact
verification still falling to human content moderators [336].

Critical examinations of ML-driven methods. This work is the latest entry in a
body of literature that critically examines ML-driven solutions to subfield-
specific problems, including security and privacy issues [315, 320, 304, 342].
Prior studies in the same vein have evaluated ML approaches to network
security and established frameworks for explaining the mechanics of black box
ML- and AI-driven approaches [320]. In particular, Jacobs et al. describe
three instances of model underspecification in published ML solutions for
detecting patterns in network traffic: evidence of shortcut learning, presence
of spurious correlations, and vulnerability to out-of-distribution samples
[320]. These three issues approximately map on to methods design issues we
discuss in this work; namely, overfitting on training data, data dependencies,
and generalizability to out-of-domain data. This study also considers recent
work that examines the risks of automated content moderation powered by large
language models [340].

Feasibility of detection. Most of the work we review presupposes that
automated detection of misinformation is possible at all. We note that this
is, in itself, a strong assumption to make: Published work in psychology,
linguistics, and philosophy of language has previously called into question
the feasibility of using semantic and syntactic features to determine the
veracity of text statements and the usefulness of binary true/false
classifications of information, particularly in the absence of unified
definitions of misinformation, disinformation, and information [338, 190]. On
the other hand, some researchers maintain that misinformative texts are
characterized by indelible “fingerprints” that distinguish them from non-
misinformative texts: for instance, emotional language and reduced lexical
diversity [168, 311]. We use this debate to inform our discussion of extra-
textual feature sets for misinformation detection, and to call into question
established NLP approaches to misinformation detection.

##  4 ANALYSIS

###  4.1 Overview

As described in Section 1, we organize our paper analysis by research
question. Within our discussion of each research question, we discuss
information scope-specific issues pertaining to method and dataset design. Our
taxonomy of information scopes is as follows: claims, news articles, social
media posts, users/authors, and whole websites. For each paper under analysis
in each information scope, we additionally note the type of feature set(s)
that the detection method examines. We describe our taxonomy of features in
Methods (Section 2).

Across all five information scopes, we identify potential areas of need for
automated interventions; evaluate the goodness of fit of existing methods to
those needs; and interrogate the model design and data curation approaches of
those methods. In our analysis of method-target fit, we find that 1) the
actual utility of automated interventions for context-rich information scopes
(news articles, whole websites) is unclear; 2) the vast majority of methods
skirt the difficulties of performing actual fact verification by detecting
proxy signals for false rumors, including evidence of virality, anomalous
patterns of user engagement, and emotional language; and 3) authors often
train text-based methods on narrowly-defined information domains, and fail to
control for data dependencies that might inflate model performance. In our
analysis of data curation and method design, we find that 1) testing on
current or real-time data is rare; 2) a priori assumptions about user
behaviors are rampant; and 3) methods that purport to employ content-agnostic
multimodal features lack explainability and adequate feature normalization.

In Figure 1, we present a summary of our study corpus composition, organized
by year and information scope. From 2016 to 2017, we note an appreciable
increase—both as a proportion of our corpus, and as a percentage of overall
search results—in the number of published papers that examine misinformation
at the level of whole news articles and social media posts. Though we have not
conducted an in-depth investigation of publication trends, it seems likely
that this spike was in response to the proliferation of misinformation on
social media during the 2016 presidential elections (note, in particular, the
dramatic uptick in raw numbers of news article- and social media post-scoped
papers). Interest in misinformation detection remains high, despite decreased
representation of papers post-2019 in our corpus (more than 2,000
misinformation detection preprints and papers have been published since the
start of 2023, according to Google Scholar).

Figure 1: Corpus breakdown by year of publication and information scope. Note:
A few papers belong to multiple information scopes.

###  4.2 Systematization of literature

We present takeaways from our reading and coding of misinformation detection
literature. In this systematization section, we pay particular attention to
issues pertaining to RQ1 and RQ2: method-target fit and data curation.

####  4.2.1 Method-target fit

We discuss the usefulness of available and proposed detection methods to
individual information scopes: claims, news articles, social media
users/posts, and whole websites. We find that detection tasks at the claim and
post scopes stand to benefit the most from automated interventions, absent the
context that news readers might use to evaluate the credibility of whole news
articles or websites. Unfortunately, we also find that claim- and post-scoped
methods in existing detection literature have limited generalizability and
efficacy in early detection.

Claims. We define a single “claim” as the smallest semantic unit of fact or
misinformation. There is a real need for fast, reliable misinformation
detection at this scope, particularly in real time settings: Research
deployments of automated methods have been applied in the context of checking
statements made by political candidates during debates in real-time, and for
flagging possibly spurious statements on social media (e.g., tweets) [60, 27,
278, 5]. At this scope, automated misinformation detection methods proceed
along one of the following three lines: 1) stylometric, semantic, and
syntactic analysis of text contents, 2) knowledge-based comparison of
statements to ground-truth claims (fact-checking), or 3) crowd- or context-
informed inference of text credibility. Of all information scopes, we find
that claim-scoped detection methods are most consistent in their attempts to
perform actual fact-checking of text statements. Features are mostly textual.
Claim-scoped papers constitute about 15% of our corpus.

Off-the-shelf NLP methods, including part-of-speech (POS) tagging and
sentiment analysis, have limited use at this scope, for the simple reason that
single claims are often insufficient inputs (we provide a brief summary of
claim-level NLP methods in this section and elaborate further on whole-text
applications in our discussion of article-scoped approaches).

At the level of single claims, semantic feature analysis is limited to the
identification of structured statements, usually as a precursor to knowledge
graph (KG) construction. At the most basic level, these claims might take the
form of subject-predicate-object (SPO) statements (e.g., "I like pie") [17,
15]. In general, nouns (subjects and objects) become nodes on the KG, and
predicates become edges. Some KG-based methods estimate the probability of the
existence of an edge between two distinct nodes and identify shortest paths
between edges in order to “fact-check” SPO statements [17, 155]. Other claim-
scoped approaches use word embedding methods to quantify the semantic
“distance” between ground-truth claims and claims to be checked [261, 17, 277,
101]. While these distance metrics are, at best, a kind of proxy for the
veracity of text statements—semantic proximity as an indicator for the
likelihood of a relationship between two entities—the methods that employ them
still bear the greatest resemblance to manual fact-verification, compared to
methods that appear in other information scopes. The efficacy of these
approaches is highly dependent upon 1) choice of ground-truth reference and 2)
the coverage and granularity of data included in KGs. We discuss article-
scoped approaches to detection next.

News articles. We consider all news-oriented writing of length 100 words or
greater to fall within this scope. Full-length news articles have sufficient
text contents to make certain off-the-shelf NLP approaches (e.g., stance
detection, authorship detection) feasible, absent any external ground-truth
reference for fact verification. This is the second-largest scope we consider,
and comprises about 24% of the full paper corpus. Most methods at this scope
rely on purely textual features to perform classification.

We observe that methods at the article scope generally adopt one of three
approaches: 1) pure NLP analysis of article contents to identify language
features particular to misinformative or deceptive language; 2) analysis of
article contents and headlines to identify potentially clickbait-y titles or
discussion related to suspected misinformation narratives; and 3) analysis of
user responses in conjunction with analysis of text contents. (There is a
rough correspondence with 1) and 3) for claim-scoped approaches, described in
the previous section.) Features are mostly textual or user/author-based. We
notice that, of these three approaches, authors whose methods align with 1)
and 2) claim that article- or content-only approaches are better-suited for
real-time or early detection of misinformation (detection of propagation
patterns generally requires a degree of virality) [35, 33, 83, 62, 47].
Meanwhile, authors whose methods consider multimodal feature sets in addition
to pure text contents argue that classifiers trained only on text contents
risk overfitting to their training data, and do not generalize well to out-of-
domain datasets [104, 220, 61].

The actual utility of automated methods for misinformation detection for news
articles is unclear. On moderated online fora, for instance, manual review of
new content is frequently very effective at early detection and removal of
misinformative articles: A 2016 study of Wikipedia hoaxes found that 90% of
hoax articles were flagged at the source within one hour of “patrolling” by
trusted editors. Though time-to-detection did increase drastically thereafter,
this was applicable to only a small percentage of hoax articles—the authors
claim that one in a hundred hoax articles survives past one year [7]. This
finding mirrors our own observation, from our review of the Facebook and
Twitter Files, that a good deal of misinformation detection is still performed
manually by human content moderators [337, 336].

Social media posts and users. We discuss social media post- and user-scoped
methods together because we encountered similar data curation and methods
design issues within both scopes. Social media post-scoped papers formed
approximately 24% of our corpus (66 papers). User-scoped papers formed
approximately 15% of our corpus (40 papers). Feature sets for methods at both
scopes are generally multimodal, and include textual, user/author-based, and
network-based features. As mentioned previously, there is a real need for fast
detection of misinformation and false rumors at this scope, given the volume
of content posted to popular social media platforms. Even so, we observe that
a majority of methods at this scope rely on patterns of sharing and
propagation that implicitly associate virality with the existence of rumoring
narratives [3, 214, 35]. We note that these assumptions disallow early
detection of misinformation. Similar issues have arisen in the security
literature, in regards to spam and Sybil account detection. For the detection
of both Sybils and accounts that spread misinformation, anomalous patterns of
account registration and user interaction serve as proxies for the presence of
bad actors [183, 179]; early detection requires authors to formulate a priori
assumptions about the nature of these patterns. We discuss these assumptions
in greater detail in Sections 4.2.2 and 4.2.3.

Websites. As with article-scoped methods, we noted intervention fit issues at
the whole website scope (comprising only 2% of the full corpus, or 6 papers;
feature sets are infrastructural, textual, and network-based). Past research
has demonstrated that a few well-established news websites likely constitute
the super-majority of most readers’ news diets [9]. In oversampling from
fringe and novel news sites, many article- and website-scoped misinformation
detection methods likely overestimate the extent to which such content is
represented in the typical reader’s news diet, even though many paper authors
motivate their methods by citing increasing reader distrust of mainstream
media outlets. In fact, these fringe sites are likely in the long tail of the
distribution of sites most commonly viewed by news readers.

Still, methods for detecting suspicious website infrastructure are useful,
particularly on social media platforms, where links to news articles are often
shared without additional context. All authors within the whole website scope
state that a primary motivation for their work is rapid inference of article-
level factuality from source-level credibility labels [119, 240, 1, 37, 137,
286, 271]. Additionally, more than 70% of paper authors within the news
article scope state that the intended goal of their automated detection method
is the detection of misinformative news articles shared on social media; 9% of
these authors state that early detection of misinformation is an express goal
of their work. Tellingly, about 82% of authors cited the 2016 presidential
election as motivation for development of their automated detection method,
e.g. [19, 49, 220].

####  4.2.2 Proxy targets for detection

We discuss targets for detection across all five information scopes. In
particular, we note that the vast majority of methods do not perform actual
fact verification, and instead detect proxy signals for false rumors (e.g.,
evidence of virality, anomalous patterns of user engagement), and that authors
often train and test text-based methods on single rumoring narratives or
information domains.

Within the claim scope, certain context-aware methods consider extra-textual
features to inform their classification. A number of methods rely on the
“wisdom of crowds” to crowd-source credibility scores or (counter-)evidence
for claims [114, 121]. We note a number of papers that essentially automate
the manual fact-checking procedure that a human web user might follow. These
methods proceed by 1) querying a database or search engine for a certain
claim, 2) seeking the sourcing for this claim among search results, and 3)
verifying the credibility of these sources, or counting the raw number of
relevant hits [202, 15, 48]. Equating hits or mentions with positive proof of
veracity is a dangerous (and possibly false) equivalency, particularly when
existing misinformation literature also assumes that misinformative claims
tend to be outstandingly viral [3].

At the article scope, we note that 44% of papers claiming to detect
misinformation actually detect second- or nth-order effects of the existence
of misinformation. That is, in lieu of actual fact verification, many methods
detect proxy signals that might be indicative of the existence of
misinformation, then rely on a priori assumptions about these signals—e.g.,
user behavior, stylometric properties—to associate them with misinformation.
For instance, a number of methods detect genre signals (satire, parody,
hyperpartisanship), and assume that genre, writing style, or political
inclination are synonymous with (mis)informative reportage [226, 11, 14, 19].
A significant number of methods include text sentiment in their feature sets,
or explicitly report on the advantages of including sentiment analysis in
misinformation detection methods—in general, these authors associate evidence
of strong emotion (and strongly negative emotion, in particular) with
misinformation [38, 19, 14, 67, 261, 220, 222, 62, 82].

We argue that overreliance on text sentiment—and, more generally, unimodal
feature sets and linguistic properties of text—as a proxy indicator for
misinformation makes detection methods more vulnerable to adversarial attacks.
Only one paper within the article scope tested in an adversarial setting: Its
authors found that, while stylometry-based misinformation detection had an
accuracy rate greater than 80% on routine tasks, this score dropped to about
50% in adversarial cases [200]. Malicious actors might edit a factually
correct news story to be misinformative, or include subtly misinformative
statements in the body of an otherwise true news article, without
substantively changing authorial style. We investigate the efficacy of this
circumvention method in our replication analysis, in Section 4.3.2.

At the social media post and user scopes, many methods that examine user
behavior assume that certain propagation patterns are unique to engagement
with misinformative texts [299, 104]. Multiple methods select tweet topics and
hashtags that were retweeted a certain number of times by users, or that
achieved “virality” according to some threshold of popularity predetermined by
the paper authors. It is perhaps unsurprising, then, when the same papers
conclude that high levels of user buy-in or peaks in user interactions with
these tweet topics are signals for misinformation [239, 230, 23]. As we
discuss in greater detail in Section 4.2.3, a priori assumptions about user
engagement with misinformation or the popularity of misinformative narratives
might 1) preferentially surface already-viral misinformation, making the
resulting detection methods that are trained on these biased datasets unusable
for early detection, or 2) yield overoptimistic predictions of detection
efficacy in an experimental setting. It is unclear if these assumptions are
necessary and useful constraints for delineating detection targets, or if they
detrimentally limit detection capacity and increase susceptibility to
circumvention. Techniques for detecting spam and Sybil accounts in security
literature seem to impose similar constraints on the assumed activity of
dishonest nodes: They assume, for instance, that Sybils will form well-
connected neighborhoods [183], or (seemingly contradictorily) that compromised
Sybils will refrain from connecting with additional Sybils, to avoid detection
[179].

In general, these methods—at the user and post scopes and whole website
scopes—employ multimodal feature sets for detection—for the former two, these
feature sets include user credentials, user graphs, and sharing/post
propagation statistics; for the latter, these feature sets include
infrastructure features such as site age and URL structure. While
multimodality might go some way toward preventing potentially malicious actors
from taking advantage of automated detection (as we describe in preceding
paragraphs, in regards to unimodal feature sets), well-considered feature
normalization is crucial for ensuring the explainability and efficacy of
models employing these multimodal features, but is often absent or undisclosed
[37, 300].

####  4.2.3 Data curation and method design

We discuss issues with dataset curation and classifier design. In particular,
we note that most authors do not test on current or real-time data; methods
that detect propagation patterns also tend to make a priori assumptions about
user behaviors; and that methods employing multimodal features often lack
model explainability and adequate feature normalization.

Lack of uniform standards for describing misinformation. The few papers in our
corpus that propose to perform actual fact verification generally rely on
existing compendia of real and fake news sites in order to establish ground
truth. Unfortunately, choice of ground truth site or labeled dataset can
significantly influence the outcome of analysis (Bozarth et al. found that
perceived prevalence of misinformation in a corpus of 2016 election news
varied from 2% to 40%, depending on choice of ground-truth reference website
[16]). This is due in large part to lack of uniform standards across these
sites: These external references do not follow any standard taxonomy for
describing misinformation—some sites’ classifications are far more granular
than others—or only label news articles from specific information domains
(e.g. politics) [282, 283].

KG-based methods generally construct their graphs from static snapshots of
existing online knowledge databases. Though it is easier to quality-control
these references—they are generally single-source, or, in the case of DBpedia,
are fairly reputable—there is empirical evidence that hard-coding predicates
or subjects into structured representations of knowledge databases yields
classifiers with limited versatility. Additionally, it is unclear how scalable
and deployable a knowledge graph-based method might be in a real-time setting:
It would be impractical to continually update such a graph in a rapidly
evolving news landscape, where communication is often semantically
inconsistent and non-textual media are common (images and videos are common
elements of social media posts, for instance). Furthermore, there might not be
a ground-truth reference available online to serve as a point of comparison
for breaking news or primary-source statements. In fact, of the 40 papers we
analyzed that discussed methods with network-based components, very few (about
7%) tested on real-time data [45, 61, 154]. We discuss the limitations of KG-
based approaches in our replication analysis (see Section 4.3.1).

Lack of current datasets. While well-annotated, current datasets are in short
supply across all information scopes, this deficit is particularly glaring at
the article scope: Most news outlets that consistently produce factually
correct news articles do not produce comparably large corpora of verifiably
misinformative news articles, and vice versa. In the absence of well-labeled
corpora of truthful and misinformative articles, many authors use existing
data. A few datasets appear to be go-tos for methods at this scope, despite
the time-specificity of news and rumors; the LIAR dataset (first presented in
[30]) is among these [167, 222, 62]. The dataset was released online in 2017,
and includes manually labeled statements from the PolitiFact fact-checking
website published over the course of a decade (from 2007 to 2017). The many
papers that reference the LIAR dataset were published between 2017 and 2023,
and rarely (in fact, only two papers within this scope) perform additional
testing on current or real-time data. Other datasets, including those
published by BuzzFeed News, Gossip Cop, Media Bias/Fact Check, and the Fake
News Challenge, are also temporally distant from the papers that cite them
[261, 104, 167, 62]. As a result, reported performance on these datasets is
not indicative of method performance in a current or real-world context.

Overfitting to information domains. Much as claim-scoped methods risk
overfitting to particular syntactic structures, misinformation detection
methods scoped to full articles are often overfitted to specific information
domains or rumoring narratives. We observe a number of methods that select one
or more topics of interest (e.g., the 2016 presidential election, the Boston
Marathon bombing), train a classifier on these topics, then test this
classifier on a different set of texts that discuss the same topic [260, 203,
134]. These methods do well when tested on in-domain texts; many report
accuracy well above 80% (see SI). As we show in our replication analysis
(Section 4.3.2), these methods perform significantly worse on out-of-domain
texts.

Limited social media data accessibility. We note that a majority of published
papers did not include links to their methods or datasets (and, in many cases,
these methods and data were not made available to us, even after we reached
out to the lead author of the paper to request access). About 60% of user- and
post-scoped methods (40 papers) relied on data obtained through the Twitter
API, and classify individual tweets or user accounts as misinformative or
factually correct (e.g., [278, 246, 96, 169, 51]). Unfortunately, in April
2023, Twitter shut down its free academic API—the platform now only permits
access to a limited number of tweets each month with a paid subscription plan.
We identified multiple Twitter datasets posted pre-API shutdown that were
missing full tweet contents or user data. These omissions are due to the
Twitter API’s Terms of Service, which only permit limited disclosure of
“…Tweet IDs, Direct Message IDs, and/or User IDs…” [292].

In total, we attempted to locate datasets and code for 50 different papers.
Thirty-six (72%) of these analyzed tweet corpora, and 42 (84%) of these were
user- or post-scoped. We were able to independently source complete methods or
data for fourteen (28%) of these. Of the 27 authors we eventually contacted
about data availability, nine responded; six of those authors were able to
provide method code or partial datasets.

Strong assumptions about user behavior. Misinformation researchers commonly
reference the SIR (susceptible-infected-recovered) epidemiological model and
its variations to model information spread—analogously, newsreaders who have
not been exposed to a misinformation narrative are “susceptible”; “infected”
newsreaders are those who buy into a misinformation narrative; and “recovered”
news readers were once (but are no longer) credulous of a misinformation
narrative [125, 312]. Many papers within this scope make normative theoretical
assumptions about user behaviors: some works assume, for instance, a
homogeneous population of newsreaders, with identical probabilities of
“infection” and reinfection; others assume that, once infected, a user cannot
be re-infected [193]. One method assumes that, in the event that a true and a
false story reach a user simultaneously, the user will believe the true story
first [251]. We noted several methods that did not distinguish between social
media posts that actively promoted false rumors and those that merely
discussed or even debunked them [125, 55]; other methods calculated composite
statistics with questionable feature normalization; e.g., one method
calculated a credibility score for a whole news event as an average of the
credibility scores for all posts that discussed this event (notably, the
dataset was designed to have a balance, 50/50, of true and false rumor tweets)
[247]. Ideally, such a score might be weighted by post visibility or
originating account followership.

We note that most of these assumptions yield conservative estimates of
misinformation spread. In general, when making theoretical estimates of real-
world behaviors, we would recommend the opposite approach (e.g., assumption of
heightened susceptibility to infection and possibility of reinfection). While
we can point out errors in theoretical assumptions about user behaviors, it is
difficult to evaluate possible real-world impacts of these assumptions on
detection efficacy, particularly in the absence of accessible current data. We
previously discussed the extent of accessibility and availability issues
within our corpus (see Limited social media data accessibility).

Dataset bias and inconsistency. For both training and testing, methods scoped
to whole website detection generally relied on published lists of websites
with accompanying credibility scores [37, 300, 1, 286]. As we discussed
previously, however, these references did not have uniform taxonomies for
classifying site credibility, and it was often the case that these pre-labeled
lists were biased towards older, more visible real news and fake news outlets,
or were restricted to specific information domains [37]. We note papers that
perform website infrastructure analysis on contemporaneous snapshots (circa
2019) of websites in their corpora, even though the text-based features for
the same analysis were drawn from datasets published in 2016 and 2017 [286].
This constitutes a serious source of temporal leakage. Hounsel et al. perform
testing in real-time—one of the few studies we reviewed, and the only study at
this scope, that managed to do so [1]. We note, however, that there is an
imbalance in the proportion of inactive misinformation and real news sites in
the training set for this study—34% of the sites in the misinformation
training set were active, while all of the websites in the real news training
set were active. As result, the classifier might have been overfit to features
specific to those inactive websites.

###  4.3 Replication Studies

We conduct replicability and reproducibility studies on three different papers
within our corpus (our selection criteria are described in Section 2.2), at
the claim, article, and whole website scopes. We conducted an author outreach
survey in lieu of a replication study for user- and post-scoped methods, in
light of data accessibility issues (see Section 4.2.3). For each work, we
evaluate replicability and generalizability (RQ3, RQ4). Where possible, we 1)
replicate reported results; 2) inspect datasets for potential dependencies; 3)
perform ablation analyses to understand individual feature performance; and 4)
test on current data.

####  4.3.1 Ciampaglia et al. (2015)

We replicate the published paper from 2015 by Ciampaglia et al., in which the
authors design a knowledge graph-based method called knowledge linker [17] for
evaluating the veracity of claims. In particular, the authors construct
confusion matrix and Wikipedia Knowledge Graph (WKG) representations of
databases of subject-predicate-object (SPO) statements extracted from
Wikipedia information boxes (the authors claim that the “least controversial”
information can be found there), then used semantic proximity (derived from
path length between subject and object nodes) to assign truth values to
individual claims. The authors reported > 0.90 ROC for classifier performance
on test sets of statements about country capitals, film directors, and
presidential spouses (predicates are is-capital-of, is-married-to, is-
directed-by); notably, this type of information is likely to be found in a
Wikipedia information box, or in the body text of a Wikipedia article about a
prominent person or place. While we were able to reproduce a result from the
original paper, we were unable to generate an updated confusion matrix on
current data, as data pre-processing code was not made available to us.

Reproducibility of published results. Original data were obtained from the
paper authors; we were able to independently locate their method code on
Github. We replicated the movies and directors confusion matrix presented in
the original paper (the link prediction method was broken). We present a heat
map representation of this result in Figure 2 (a larger version of this
graphic is in our SI). We note that knowledge linker’s confusion matrix method
is predicate-agnostic—high scores only denote high probability of a
connection, in general, between a subject and an object. The nature of this
connection is unspecified. As such, this approach to fact checking surfaces
many more false positives than false negatives. Within the movies and
directors test data represented in Figure 2, we note four prominent false
positive results (annotated in the figure): Francis Ford Coppola wrote the
screenplay for “Patton” (A); George Cukor was replaced as a director of “Gone
with the Wind” (B); and Robert Redford starred in “The Sting” and “Out of
Africa” (C, D). While each person is strongly connected to his respective
film, he did not direct this film. This approach might be useful for surfacing
checkable statements or putative relationships; more granular automated or
manual checking of these statements is needed to determine actual factuality,
however.

Figure 2: Replication analysis of Ciampaglia et al.: Heatmap of movie-director
relationships, from a DBpedia dataset. Data is not binarized. Labeled points
A, B, C, and D are false positive relationships.

Generalizability of published results. A KG-based approach has limited
generalizability, by design—the breadth of checkable statements is determined
by what information is contained within the KG. We can only query
relationships between films and their directors within the confusion matrix
presented in Figure 2; in order to query COVID-19 misinformation, for
instance, we would have to construct a new KG from an entirely new set of SPO
statements. Knowledge linker’s KG approach has additional shortcomings: The
link prediction task is currently agnostic to negation, so that is-A and is-
not-A relationships might register as equally likely; and the method is
agnostic to non-binary predicates (e.g., is-mostly-A, is-A-20%-of-the-time).
While Ciampaglia et al. explicitly state that knowledge linker is a proof-of-
concept for more sophisticated KG methods, we argue that solely KG-based
approaches to misinformation detection are difficult to scale and make
generalizable. Designers of KG-based methods necessarily make tradeoffs
between the granularity of the relationships hard-coded into KG edges and the
speed and generalizability of their approach. Methods incorporating additional
textual or contextual features might be faster and more accurate in a rapidly
shifting news ecosystem. We discuss these feature types and methods in the
sections that follow.

####  4.3.2 Nasir et al. (2021)

We reproduced results from a study published in 2021 by Nasir et al [134]. The
authors propose a neural net-based approach to the classification of news
articles. The method employs a hybrid deep learning model that combines
convolutional and recurrent neural networks for the classification of real and
fake news. The authors report results for tests on two datasets: the ISOT
dataset (45,000 news stories, equally distributed across true and false
categories, as labeled by PolitiFact) and the FA-KES dataset (804 news stories
about the Syrian war, 426 true and 376 false) [296, 297]. We were able to
replicate original paper results and run the method on updated datasets of
news articles.

Reproducibility of published results. Both ISOT and FA-KES are readily
available online, and we were able to run the original method of Nasir et al.
without issue [296, 297]. The method code was not publicly available online
(the authors sent us their code base after we emailed them a request for
access). Weights and parameters for their trained model are published in the
original paper [134]. We were able to reproduce results originally reported in
their published paper (see ISOT-original and FA-KES-original in Table I).

Data dependencies and overfitting on training data. Nasir et al. report
exceptionally high classification precision on the ISOT dataset—approximately
1.00, or nearly absolute accuracy. (By comparison, the classifier has about
0.521 accuracy on the FA-KES news data after training on ISOT.) While the
original paper by Nasir et al. explicitly describes the provenance of ISOT
data—all true articles were sourced from Reuters—the authors do not mention
that the body text of all true-labeled articles also includes the outlet name
and a geotag (e.g., “BEIJING (Reuters)”). The inclusion of this geotag-Reuters
name pair was consistent across all true articles. This text was absent from
all false-labeled articles. While it seems likely that the CNN simply learned
a shortcut—the presence of a source name in all factual articles, and the
absence of this source name in all misinformative articles—a scrubbed version
of the dataset without geotag-Reuters name pairs posted comparable accuracy
scores (see ISOT-scrubbed in Table I). As such, we hypothesized that the CNN
was actually learning other, more subtle linguistic signatures of Reuters
house style.

House style as a confounder. To investigate house style as a possible
confounder for misinformation detection, we excerpted 100 articles from both
Reuters and The New York Times. We randomly selected these articles from both
outlets’ RSS feeds in May 2023. We sourced articles for both corpora from the
following verticals: U.S. and world politics, economics, science, and
entertainment. Excerpt lengths ranged from 100 to 300 words. These corpora,
each comprising 100 true news stories, are labeled “Reuters-original” and
“NYTimes-original” in Table I.

Dataset (size) | Features | Acc. | FPR | FNR
---|---|---|---|---
ISOT (45,000) | original | 0.995 | 0.00 | 0.00
scrubbed | 0.987 | 0.01 | 0.00
FA-KES (804) | original | 0.521 | 0.151 | 0.843
Reuters (100) | original | 0.727 | 0.71 | –
modified | 0.507 | 0.02 | 1
NYTimes (100) | original | 0.673 | 0.705 | –
modified | 0.492 | 0.735 | 0.286
ChatGPT (250) | original | 0.939 | 0.02 | –
TABLE I: Replication analysis of Nasir et al. (2021). We tested the method of
Nasir et al. on both datasets discussed in the original paper, and on novel
datasets from Reuters and The New York Times [134].

We then selectively edited 50% of the news articles within each corpus. We
changed proper nouns, negated verbs, and altered reported statistics so that
the factual content of these articles was no longer accurate but house style
and tone were preserved. (An example of our editing approach appears in Figure
3.) We call these altered text corpora “Reuters-modified” and “NYTimes-
modified” in Table I. We ran the classifier on each text corpus three times
and took the average of its performance across all three trials; we report
these results in Table I. The classifier had 0.727 accuracy on Reuters-
original and 0.673 accuracy on NYTimes-original; this difference is not
significant (p > 0.05). Additionally, the classifier had about 0.50 (random)
accuracy on both modified datasets. The classifier’s false positive and false
negative rates on modified and original Reuters and NYTimes corpora tell a
more interesting story, however: Overwhelmingly, false NYTimes articles were
classified as true (FPR = 0.02 and 0.735 for Reuters-modified and NYTimes-
modified, respectively), while all true Reuters articles were classified as
false (FNR = 1 and 0.286 for Reuters-modified and NYTimes-modified,
respectively). These results indicate that the classifier was significantly
more conservative in its assignment of true labels for the Reuters dataset
than it was for the NYTimes dataset; additionally, within each modified
corpus, the classifier did not effectively differentiate between true and
untrue news articles. We did not investigate the particular stylistic features
of each corpus that might have produced (anti-)conservative label
distributions; for this work, however, it is sufficient to note that 1)
classifier performance in an adversarial setting was no better than random,
and 2) house style does appear to have a significant impact on classifier
sensitivity to misinformative texts.

Investigating LLM-generated texts. We generated a corpus of 250 “fake” news
articles by prompting ChatGPT to write “a fictional, $\\{100,200,300\\}$ word
account of any event in the style of Reuters” [333]. Classifier performance on
this task exceeded 90% accuracy—comparable to its performance on the original
ISOT dataset. We did not attempt to construct a ChatGPT-generated corpus of
true news articles, in light of ongoing issues with the veracity of
information provided by the LLM and our own limited bandwidth for manually
fact-checking such a dataset [334]. Additionally, in the course of conducting
this experiment, we noted that a number of “fake” Reuters news stories
appeared to be carefully edited accounts of actual events; we present an
instance of this in our SI. Though we do not address LLM-powered detection of
misinformation in this study, we note that a recent paper by DeVerna et al.
reported that ChatGPT might actively decrease belief in true news headlines
that it mislabels as false; notably, only 15% of true headlines were correctly
labeled as such by ChatGPT on a test set of 20 headlines [340].

China’s factory activity contracted faster than expected in May on weakening
demand, heaping pressure on policymakers to shore up a patchy economic
recovery. The official manufacturing purchasing managers’ index (PMI) was 48.8
from 49.2 in April, according to data from the National Bureau of Statistics
(NBS), its lowest in five months and below the 50-point mark that separates
expansion from contraction… China’s factory activity expanded faster than
expected in May on increasing demand, heaping pressure on policymakers to
continue a strong economic recovery. The official manufacturing purchasing
managers’ index (PMI) was 49.2 from 48.8 in April, according to data from the
National Bureau of Statistics (NBS), its highest in five months and above the
30-point mark that separates expansion from contraction…

Figure 3: Replication analysis of Nasir et al.: We edit the factual content of
published news articles without significantly altering house style. Excerpted
text from a Reuters news article is in the top panel; our edited version is in
the bottom panel [330]. Text that differs between both versions is in color.

These article-scoped findings suggest that writing style is a serious
confounder in the classification of news article credibility; moreover, our
replication analyses suggest that malicious actors could produce
misinformative texts that are undetectable to style-based detection methods by
injecting misinformative statements into articles that sound authoritative.
Multimodal methods attempt to address text-based data dependencies by
introducing feature types that are presumably agnostic to topic- and style-
related confounders. In the next section, we present a replication analysis of
a multimodal detection method that jointly considers text contents, social
media presence, and website infrastructure.

Dataset (size) | All features |  articles |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---|---
|  traffic |  |  |  |  |  |  |  |  |
|  twitter |  |  |  |  |  |  |  |  |
|  wikipedia |  |  |  |  |  |  |  |  |
|  url |  |  |  |  |  |  |  |  |
|  $-$ $+$ |  |  |  |  |  |  |  |  |  |
Full corpus (1066) | 0.654 | 0.631 | 0.644 | 0.654 | 0.508 | 0.648 | 0.550 | 0.627 | 0.606 | 0.638 | 0.533
Med. corpus (400) | 0.623 | 0.608 | 0.630 | 0.620 | 0.488 | 0.635 | 0.500 | 0.590 | 0.588 | 0.623 | 0.495
Small corpus (250) | 0.636 | 0.632 | 0.596 | 0.632 | 0.524 | 0.608 | 0.512 | 0.588 | 0.536 | 0.624 | 0.516
Left bias (398) | 0.691 | 0.683 | 0.671 | 0.688 | 0.668 | 0.686 | 0.628 | 0.678 | 0.683 | 0.678 | 0.636
Center (263) | 0.913 | 0.810 | 0.890 | 0.913 | 0.700 | 0.924 | 0.741 | 0.920 | 0.776 | 0.890 | 0.635
Right bias (405) | 0.279 | 0.267 | 0.252 | 0.279 | 0.173 | 0.286 | 0.230 | 0.274 | 0.205 | 0.272 | 0.121
Full corpus (1066) | 0.569 | 0.523 | 0.595 | 0.569 | 0.399 | 0.580 | 0.440 | 0.552 | 0.538 | 0.577 | 0.373
Med. corpus (400) | 0.563 | 0.517 | 0.580 | 0.560 | 0.420 | 0.578 | 0.478 | 0.585 | 0.545 | 0.568 | 0.360
Small corpus (250) | 0.456 | 0.424 | 0.560 | 0.452 | 0.364 | 0.500 | 0.408 | 0.400 | 0.496 | 0.444 | 0.436
Low cred. (256) | 0.590 | 0.516 | 0.633 | 0.590 | 0.641 | 0.629 | 0.445 | 0.609 | 0.633 | 0.590 | 0.473
Mixed cred. (268) | 0.407 | 0.340 | 0.474 | 0.407 | 0.0522 | 0.414 | 0.258 | 0.362 | 0.276 | 0.414 | 0.198
High cred. (542) | 0.349 | 0.336 | 0.408 | 0.349 | 0.255 | 0.369 | 0.271 | 0.341 | 0.316 | 0.351 | 0.218

TABLE II: Replication analysis of Baly et al.: Dropout(–) and feature
importance(+) analyses of subsets of Baly et al.’s EMNLP18 dataset, stratified
by political leaning and credibility. Most (secondmost) performant feature, as
determined by its contribution to overall classifier accuracy on the full
feature set, is highlighted in darker (lighter) hues. Fact and bias
classification task performances are reported in the top and bottom halves of
the table, respectively.

####  4.3.3 Baly et al. (2018)

We reproduce results from a study published in 2018 by Baly et al [37]. In
summary: the authors propose a multimodal approach to the classification of
news websites. Baly et al. analyzed website contents, associated social media
accounts, and Wikipedia pages in order to perform two classification tasks:
fact and political bias classification. The authors developed a dataset of
1066 websites manually labeled for their political leaning (extreme-left,
left, left-center, center, right-center, right, and extreme-right) and degree
of credibility (low, mixed, high). These labels were extracted from the Media
Bias/Fact Check (MBFC) database [295]. We were able to reproduce original
study results and perform ablation analyses on existing datasets. We were
unable to run the method on an updated dataset, as feature extraction code was
not available.

Reproducibility of published results. Classifier code is available on Github.
All features extracted for the original analysis were captured in a series of
json files. On the one hand, this meant that we were able to readily reproduce
results reported in the paper; on the other, this meant that certain elements
of the dataset (follower counts on social media, Wikipedia page contents) were
out of date. As we did not find documentation in the method repository for re-
extraction of these features, we were restricted to conducting our tests on
data that were already available. We decided to bin bias labels into left,
center, and right categories, as the seven-way taxonomy initially applied to
the dataset by MBFC yielded small label classes (e.g., only 21 websites
labeled as ‘extreme-left’) and poor classifier performance on the bias
classification task (less than 40% accuracy on the full dataset, employing all
features). We labeled all “extreme-left,” “left,” and “left-center” sites as
“left”; retained the labels of all “center”-labeled sites; and labeled all
“extreme-right,” “right,” and “right-center” sites as “right.” Classifier
performance on the resulting three-way bias classification task accords with
the results reported by Baly et al. on the same task.

Limited model explainability. From our reading of the paper by Baly et al. and
our analysis of their code base, we find only limited discussion of feature
normalization and the contributions of individual feature sets to overall
model performance. The authors trained their model on a set of news articles
from an undisclosed source; in order to extract feature vectors for the
article feature set for each site in the EMNLP18 dataset, the authors took the
unweighted average of 141 GloVe features over the set of retrieved articles
(article selection process is not disclosed). There is no discussion of the
semantic features included within this set of 141, or of how these features
were chosen.

Multimodal features: help or hindrance? We performed an ablation study of the
method on the EMNLP18 dataset and analyzed the method’s bias and fact-checking
classification functions separately [37]. Specifically, we stratified the
original EMNLP18 dataset by political leaning and credibility, as labeled by
MBFC, and analyzed the performance of 1) the full feature set, 2) individual
features and 3) ablated feature sets (removing one feature type per test). Our
results are summarized in Section 4.3.2. We find that, on 11 out of 12 test
datasets, classifier performance using only text-based features (articles and
wikipedia, derived from articles randomly sampled from the website in
question, and the site’s corresponding Wikipedia page, respectively) was
comparable to performance on the full feature set. On five out six datasets,
bias classification accuracy on text-only features actually outperformed bias
classification on the whole feature set (see the bottom half of Section
4.3.2), which suggests that the introduction of non-textual feature sets was
detrimental to classifier performance.

Target-problem mismatch. Upon manual review of the original Media Bias/Fact
Check dataset, we found that about 13% (137/1066) of websites in the EMNLP18
dataset did not qualify as news websites—and, in fact, MBFC labeled about 80%
(109/137) of these as ‘Organizations or Foundations’ [295]. Many of these
websites are operated by think tanks, non-governmental organizations,
political action committees, or special interest groups; they include the
World Health Organization, Citizens United, and the National Rifle
Association. We removed these non-news sites from the EMNLP18 dataset and re-
ran the classifier on the sanitized dataset, using all features and five-fold
cross-validation. The classifier achieved an accuracy of 0.602 on its fact
classification task, about 5% worse than its performance on the full dataset
(0.654). It achieved only 0.320 accuracy on its bias classification
task—significantly worse than its performance on the full dataset (0.569).

Other characteristics of the websites included in the EMNLP18 dataset,
including country of origin and information domain, called into question the
appropriateness of the classification task proposed by Baly et al. to the data
on hand: About 16% of websites (170/1066) are non-U.S. in origin, and about 3%
of websites (27/1066) report on science news. Bias classification would
ideally be sensitive to the unique political climates of different nation-
states (a “left” label in Egypt is arguably different from a “left” label in
Norway, for instance) and the basic relevance of political bias, in
particular, to the information published on the website (in lieu of a
political leaning label, MBFC labels some science news sites as “Pro-science”
or “Pseudo-science”). Fact classification would ideally be sensitive to the
objectives—stated or not—of published material on these websites, many of
which are opinion-editorial sites.

In lieu of pure fact- or bias-classification, it seems likely that the method
of Baly et al. learned at least a few shortcuts in the EMNLP18 dataset that
serve as confounders to actual fact or bias detection: Absent these signals,
the classifier performed appreciably worse on certain tasks. The composition
of EMNLP18 suggests that, while the authors intended to build a multimodal
classifier, they might have instead constructed a text classifier that learned
textual dependencies pertaining to house style, tone, and topic, such as those
discussed in our analysis of Nasir et al.

##  5 DISCUSSION

We focus our discussion of results on those issues identified by our
literature review and investigated in greater depth in our replication
analyses: fit, data curation, explainability, reproducibility, and
generalizability. Additionally, we provide recommendations for evaluating ML-
driven trust and safety interventions and link these recommendations to major
takeaways of the present study.

###  5.1 Results

As we discussed in the Introduction to this work, the sheer volume of content
posted to social media platforms and on news sites suggests that there might
be a real need for fast, automated misinformation detection methods. As we
found in the course of our literature review and replication analyses,
however, 1) the actual benefits of automated interventions are limited, in
certain cases, when weighed against the potential costs of implementation and
method error, and 2) there were serious discrepancies between existing
detection methods’ proposed and actual performance. The nature of these
differences varied by detection target and information scope. We summarize our
findings below.

RQ1: Fit. Very few methods that claimed to detect misinformation performed
actual fact verification: Instead, they targeted nth order proxy signals that
were frequently steps away from promised detection targets. These differences
were particularly noticeable in methods that relied heavily on text- and
network-based features to perform classification. In those cases,
semantic/syntactic signatures and propagation patterns served as proxies for
the existence of misinformation. We demonstrated, through our own replication
studies, that it is easy to circumvent approaches that rely on style-based
cues to perform proxy detection. Additionally, in the absence of current or
complete social media post data, we argued that most network-based methods are
poorly-equipped to perform early detection of false rumors—this is by design,
as the propagation patterns these methods purport to detect are frequently
only noticeable after a significant degree of rumor spread or virality.

RQ2: Data curation and model explainability. Lack of access to current, well-
annotated datasets remains a serious problem for current and future
misinformation research. Across existing datasets, taxonomies for classifying
misinformation were inconsistent. Testing on contemporaneous data was uncommon
among those papers we annotated, and testing in real-time settings was even
rarer. Proof-of-concept experiments oftentimes did not control for data
dependencies. Authors describing black-box methods—particularly those
employing neural nets or other forms of unsupervised learning—did not disclose
feature sets retrieved by their methods. Model explainability for these
approaches—if available at all—was very limited. For methods employing
multimodal feature sets, temporal consistency across disparate feature sets
was poorly controlled, and feature normalization was often questionable.

RQ3: Reproducibility. We noted widespread code and data availability issues:
Fewer than 30% of our attempts to locate code and data, or obtain this
information from authors, were successful. The code and datasets we were able
to retrieve were frequently unusable or out of date. In fact, we were able to
reproduce and replicate results on published and current data for only one of
our replication studies. In that single case, we found that the article-scoped
method performed no better than random (0.50 accuracy) on a current, mixed-
domain dataset (Section 4.3.2). Additionally, for the significant number of
methods that purported to detect false rumors on Twitter, the shutdown of the
Twitter Academic API poses an existential risk to reproducibility of results
and usefulness of dehydrated datasets.

RQ4: Generalizability. Methods that were trained on single-source or single-
domain datasets appeared to perform well on data from the same source, or
within the same domain; these methods, unsurprisingly, performed poorly on
multi-source or out-of-domain topics. These discrepancies are closely tied to
undisclosed or uncontrolled-for data dependencies, which we discuss in RQ2.
Though we did not attempt to test detection methods on out-of-scope data—e.g.,
we did not test claim-scoped methods on datasets of whole news articles—it
seems unlikely that these methods would fare well in out-of-scope contexts,
particularly in light of their already-poor performance on in-scope datasets.

###  5.2 Recommendations for practitioners

We offer the following recommendations for approaching and implementing
potential ML-driven interventions in trust and safety operations. We relate
each recommendation to a finding from our literature review or replication
studies.

Evaluate intervention fit and detection targets at the point of problem
formulation. More than 40% of article-scoped methods detect nth order targets
for misinformation. Some of these proxy targets, including house style and
text sentiment, are easily circumvented (Section 4.3.2). What signals—or proxy
signals—will the intervention detect? What are the potential risks of a false
negative or false positive result (and are they worth the potential benefits
of e.g., greater speed)? What is the actual detection target, and how closely
does it approximate the classification task to be performed? In an adversarial
setting, would a malicious actor be able to circumvent the automation?

Test on current data, and in real time. Fewer than 10% of methods within our
whole corpus tested on contemporaneous data, and fewer than 5% tested in a
real-time setting. To understand how an ML intervention might work in
practice, we recommend testing in a real-time context—or, at the very least,
on a current dataset. As much as possible, the testing environment should
emulate the deployment setting.

Consider possible dependencies and confounders in training and test data. In
our replication analyses of Baly et al. and Nasir et al., we observed
instances of possible shortcut learning that inflated model performance on
experimental datasets.  Carefully evaluate training and test datasets for
dependencies that might become false signals or sources of noise, that might
limit the generalizability of the classifier, or that might induce unexpected
shortcuts during the model training process. Remove (or otherwise account for)
these dependencies as possible.

Make model explainability a priority. In our replication analysis of Baly et
al., we noted serious issues with model explainability and feature
normalization: Contributions of individual feature types to overall model
performance were unclear, and the addition of non-textual feature sets did not
appear to improve classification performance by a significant margin. We
recommend that authors prioritize model explainability at the point of design
and construction. In the case of methods employing multimodal feature sets, we
recommend that authors 1) make explicit the normalization process employed by
their method to preprocess data and produce feature vectors, 2) clearly
describe the contribution of individual feature sets to overall model
performance.

Make code and data freely available. We noted widespread issues with data
availability and code functionality in our author outreach and replication
studies (Section 4.3). For methods that claim to conduct actual fact-
verification, third-party reference databases should be current and
comprehensive. Maintain and update code and data described in published work.
Clearly describe data provenance and code functionality in code releases, and
include instructions for use.

## Acknowledgments

The authors would like to thank Anne Kohlbrenner for assisting with paper
coding; Sarah Scheffler for reviewing early drafts of this manuscript; and
Mona Wang for retrieving Twitter data used for a replication study.

## Availability

Code and data used for our analyses, as well as our full paper corpus,
discussion of open problems, and additional figures, are included in our
online SI (available at https://github.com/citp/sok_misinformation).

## References

  * [1] A. Hounsel, J. Holland, B. Kaiser, K. Borgolte, N. Feamster, and J. Mayer, ‘Identifying Disinformation Websites Using Infrastructure Features’, FOCI, 2020.
  * [2] S. Helmstetter and H. Paulheim, ‘Weakly Supervised Learning for Fake News Detection on Twitter’, IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), Aug. 2018.
  * [3] S. Vosoughi, D. Roy, and S. Aral, ‘The spread of true and false news online’, Science, vol. 359, no. 6380, pp. 1146–1151, Mar. 2018.
  * [4] J. Maddock, K. Starbird, H. Al-Hassani, D. E. Sandoval, M. Orand, and R. M. Mason, ‘Characterizing Online Rumoring Behavior Using Multi-Dimensional Signatures’, CSCW, Mar. 2015.
  * [5] A. Friggeri, L. A. Adamic, D. Eckles, and J. Cheng, ‘Rumor Cascades’, Proceedings of the 8th International Conference on Weblogs and Social Media, ICWSM 2014, vol. 8, pp. 101–110, 05 2014.
  * [6] J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal, ‘FEVER: a Large-scale Dataset for Fact Extraction and VERification’, in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 2018, pp. 809–819.
  * [7] S. Kumar, R. West, and J. Leskovec, ‘Disinformation on the Web: Impact, Characteristics, and Detection of Wikipedia Hoaxes’, in Proceedings of the 25th International Conference on World Wide Web, Montréal, Québec, Canada, 2016, pp. 591–602.
  * [8] V. Qazvinian, E. Rosengren, D. R. Radev, and Q. Mei, ‘Rumor has it: Identifying Misinformation in Microblogs’, in Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, 2011, pp. 1589–1599.
  * [9] P. R. Center, ‘Social Media Fact Sheet’, Pew Research Center, Sep. 2022.
  * [10] P. Association, ‘Blue ticks for all: Twitter allows users to apply to be verified’, The Guardian, Jul. 2016.
  * [11] H. Rashkin, E. Choi, J. Y. Jang, S. Volkova, and Y. Choi, ‘Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking’, in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017, pp. 2931–2937.
  * [12] J. Cheng, L. Adamic, P. A. Dow, J. M. Kleinberg, and J. Leskovec, ‘Can Cascades Be Predicted?’, in Proceedings of the 23rd International Conference on World Wide Web, 2014, pp. 925–936.
  * [13] T. Schuster, R. Schuster, D. J. Shah, and R. Barzilay, ‘Limitations of stylometry for detecting machine-generated fake news’, Computational Linguistics, vol. 46, pp. 499–510, Jun. 2020.
  * [14] M. Potthast, J. Kiesel, K. Reinartz, J. Bevendorff, and B. Stein, ‘A Stylometric Inquiry into Hyperpartisan and Fake News’, in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018, pp. 231–240.
  * [15] A. Magdy and N. Wanas, ‘Web-Based Statistical Fact Checking of Textual Documents’, in Proceedings of the 2nd International Workshop on Search and Mining User-Generated Contents, Toronto, ON, Canada, 2010, pp. 103–110.
  * [16] L. Bozarth, A. Saraf, and C. Budak, ‘Higher Ground? How Groundtruth Labeling Impacts Our Understanding of Fake News about the 2016 U.S. Presidential Nominees’, Proceedings of the International AAAI Conference on Web and Social Media, vol. 46, no. 2, pp. 48–59, May 2020.
  * [17] G.L. Ciampaglia, P. Shiralkar, LM Rocha, J. Bollen, F. Menczer, and A. Flammini, ‘Computational Fact Checking from Knowledge Networks’, PLoS ONE, vol. 10, no. 6, Jun. 2015.
  * [18] E. C. Tandoc, Z. W. Lim, and R. Ling, ‘Defining “Fake News”: A typology of scholarly definitions’, Digital Journalism, vol. 6, no. 2, pp. 137–153, Feb. 2018.
  * [19] B. D. Horne and S. Adali, ‘This Just In: Fake News Packs a Lot in Title, Uses Simpler, Repetitive Content in Text Body, More Similar to Satire than Real News’, in AAAI CWSM’17, 2017.
  * [20] L. Haiden and J. Althuis, ‘The Definitional Challenges of Fake News’, in SBP-BRiMS 18, 2018.
  * [21] A. Gelfert, ‘Fake News: A Definition’, Informal Logic, vol. 38, no. 1, pp. 84–117, Mar. 2018.
  * [22] D. Klein and J. Wueller, ‘Fake News: A Legal Perspective’, Social Science Research Network, Rochester, NY, Mar. 2017.
  * [23] F. Yang, Y. Liu, X. Yu, and M. Yang, ‘Automatic detection of rumor on Sina Weibo’, in Proceedings of the ACM SIGKDD Workshop on Mining Data Semantics - MDS ’12, 2012, pp. 1–7.
  * [24] S. Kumar, R. West, and J. Leskovec, ‘Disinformation on the Web: Impact, Characteristics, and Detection of Wikipedia Hoaxes’, in Proceedings of the 25th International Conference on World Wide Web - WWW ’16, 2016, pp. 591–602.
  * [25] J. Ma, W. Gao, and K.-F. Wong, ‘Detect Rumors in Microblog Posts Using Propagation Structure via Kernel Learning’, in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2017, pp. 708–717.
  * [26] R. Mihalcea and C. Strapparava, ‘The lie detector: explorations in the automatic recognition of deceptive language’, in Proceedings of the ACL-IJCNLP 2009 Conference Short Papers on - ACL-IJCNLP ’09, 2009, p. 309.
  * [27] S. Jiang and C. Wilson, ‘Linguistic Signals under Misinformation and Fact-Checking: Evidence from User Comments on Social Media’, Proceedings of the ACM on Human-Computer Interaction, vol. 2, no. CSCW, pp. 1–23, Nov. 2018.
  * [28] J. Norregaard, B. D. Horne, and S. Adali, ‘NELA-GT-2018: A Large Multi-Labelled News Dataset for The Study of Misinformation in News Articles’, arXiv:1904. 01546 [cs], Apr. 2019.
  * [29] J. Kim, B. Tabibian, A. Oh, B. Schölkopf, and M. Gomez-Rodriguez, ‘Leveraging the Crowd to Detect and Reduce the Spread of Fake News and Misinformation’, in Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining - WSDM ’18, 2018, pp. 324–332.
  * [30] W. Y. Wang, ‘“Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection’, in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2017, pp. 422–426.
  * [31] K. Shu, S. Wang, and H. Liu, ‘Understanding User Profiles on Social Media for Fake News Detection’, in 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), 2018, pp. 430–435.
  * [32] N. Ruchansky, S. Seo, and Y. Liu, ‘CSI: A Hybrid Deep Model for Fake News Detection’, in Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, 2017, pp. 797–806.
  * [33] F. Qian, C. Gong, K. Sharma, and Y. Liu, ‘Neural User Response Generator: Fake News Detection with Collective User Intelligence’, in Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, 2018, pp. 3834–3840.
  * [34] Y. Liu and Y.-F. B. Wu, ‘Early Detection of Fake News on Social Media Through Propagation Path Classification with Recurrent and Convolutional Networks’, p. 8.
  * [35] K. Shu, S. Wang, and H. Liu, ‘Beyond News Contents: The Role of Social Context for Fake News Detection’, in Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining - WSDM ’19, 2019, pp. 312–320.
  * [36] Y. Chen, N. J. Conroy, and V. L. Rubin, ‘Misleading Online Content: Recognizing Clickbait As “False News”’, in Proceedings of the 2015 ACM on Workshop on Multimodal Deception Detection, 2015, pp. 15–19.
  * [37] R. Baly, G. Karadzhov, D. Alexandrov, J. Glass, and P. Nakov, ‘Predicting Factuality of Reporting and Bias of News Media Sources’, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3528–3539, 2018.
  * [38] N. Dhamani et al., ‘Using Deep Networks and Transfer Learning to Address Disinformation’, arXiv:1905. 10412 [cs], May 2019.
  * [39] X. Zhou and R. Zafarani, ‘Fake News: A Survey of Research, Detection Methods, and Opportunities’, arXiv:1812. 00315 [cs], Dec. 2018.
  * [40] D. Sáez-Trumper, ‘Fake tweet buster: a webtool to identify users promoting fake news ontwitter’, in HT, 2014.
  * [41] N. Vo and K. Lee, ‘The Rise of Guardians: Fact-checking URL Recommendation to Combat Fake News’, in The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval - SIGIR ’18, 2018, pp. 275–284.
  * [42] L. Borges, B. Martins, and P. Calado, ‘Combining Similarity Features and Deep Representation Learning for Stance Detection in the Context of Checking Fake News’, arXiv:1811. 00706 [cs, stat], Nov. 2018.
  * [43] J. Amador, A. Oehmichen, and M. Molina-Solana, ‘Characterizing Political Fake News in Twitter by its Meta-Data’, arXiv:1712. 05999 [cs, stat], Dec. 2017.
  * [44] C. Buntain and J. Golbeck, ‘Automatically Identifying Fake News in Popular Twitter Threads’, 2017 IEEE International Conference on Smart Cloud (SmartCloud), pp. 208–215, Nov. 2017.
  * [45] Y. Yang, L. Zheng, J. Zhang, Q. Cui, Z. Li, and P. S. Yu, ‘TI-CNN: Convolutional Neural Networks for Fake News Detection’, arXiv:1806. 00749 [cs], Jun. 2018.
  * [46] M. Tosik, A. Mallia, and K. Gangopadhyay, ‘Debunking Fake News One Feature at a Time’, arXiv:1808. 02831 [cs], Aug. 2018.
  * [47] X. Zhou, A. Jain, V. V. Phoha, and R. Zafarani, ‘Fake News Early Detection: A Theory-driven Model’, arXiv:1904. 11679 [cs], Apr. 2019.
  * [48] N. Hassan, F. Arslan, C. Li, and M. Tremayne, ‘Toward Automated Fact-Checking: Detecting Check-worthy Factual Claims by ClaimBuster’, in Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD ’17, 2017, pp. 1803–1812.
  * [49] P. Bourgonje, J. Moreno Schneider, and G. Rehm, ‘From Clickbait to Fake News Detection: An Approach based on Detecting the Stance of Headlines to Articles’, in Proceedings of the 2017 EMNLP Workshop: Natural Language Processing meets Journalism, 2017, pp. 84–89.
  * [50] P. Shiralkar, A. Flammini, F. Menczer, and G. Luca Ciampaglia, ‘Finding Streams in Knowledge Graphs to Support Fact Checking’, 2017, pp. 859–864.
  * [51] S. Yang, K. Shu, S. Wang, R. Gu, F. Wu, and H. Liu, ‘Unsupervised Fake News Detection on Social Media: A Generative Approach’, 2019.
  * [52] M. Della Vedova, E. Tacchini, S. Moret, G. Ballarin, M. Di Pierro, and L. de Alfaro, ‘Automatic Online Fake News Detection Combining Content and Social Signals’, 2018.
  * [53] K. Popat, S. Mukherjee, A. Yates, and G. Weikum, ‘DeClarE: Debunking Fake News and False Claims using Evidence-Aware Deep Learning’, in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 22–32.
  * [54] E. Tacchini, G. Ballarin, M. L. Della Vedova, S. Moret, and L. de Alfaro, ‘Some Like it Hoax: Automated Fake News Detection in Social Networks’, arXiv:1704. 07506 [cs], Apr. 2017.
  * [55] E. Tacchini, G. Ballarin, M. L. D. Vedova, S. Moret, and L. de Alfaro, ‘Automated Fake News Detection in Social Networks’, p. 15, 2017.
  * [56] S. Krishnan and M. Chen, ‘Cloud-Based System for Fake Tweet Identification’, in 2019 IFIP/IEEE Symposium on Integrated Network and Service Management (IM), 2019, pp. 720–721.
  * [57] V. Pérez-Rosas, B. Kleinberg, A. Lefevre, and R. Mihalcea, ‘Automatic Detection of Fake News’, arXiv:1708. 07104 [cs], Aug. 2017.
  * [58] S. Volkova, E. Ayton, D. L. Arendt, Z. Huang, and B. Hutchinson, ‘Explaining Multimodal Deceptive News Prediction Models’, 2019, p. 4.
  * [59] S. Shah and M. Goyal, ‘Anomaly Detection in Social Media Using Recurrent Neural Network’, in Computational Science – ICCS 2019, 2019, pp. 74–83.
  * [60] A. Addawood, A. Badawy, K. Lerman, and E. Ferrara, ‘Linguistic Cues to Deception: Identifying Political Trolls on Social Media’, in Proceedings of the Thirteenth International AAAI Conference on Web and Social Media, 2019, p. 11.
  * [61] S. S. Park and K. C. Lee, ‘A Comparative Study of Text analysis and Network embedding Methods for Effective Fake News Detection’, Journal of Digital Convergence, vol. 17, no. 5, pp. 137–143, May 2019.
  * [62] A. M. P. Braşoveanu and R. Andonie, ‘Semantic Fake News Detection: A Machine Learning Perspective’, in Advances in Computational Intelligence, 2019, pp. 656–667.
  * [63] D. M. Nguyen, T. H. Do, R. Calderbank, and N. Deligiannis, ‘Fake News Detection using Deep Markov Random Fields’, in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 1391–1400.
  * [64] L. Burbach, P. Halbach, M. Ziefle, and A. Calero Valdez, ‘Who Shares Fake News in Online Social Networks?’, in Proceedings of the 27th ACM Conference on User Modeling, Adaptation and Personalization - UMAP ’19, 2019, pp. 234–242.
  * [65] M. Ramezani, M. Rafiei, S. Omranpour, and H. R. Rabiee, ‘News Labeling as Early as Possible: Real or Fake?’, arXiv:1906. 03423 [cs], Jun. 2019.
  * [66] X. Zhou and R. Zafarani, ‘Network-based Fake News Detection: A Pattern-driven Approach’, arXiv:1906. 04210 [cs], Jun. 2019.
  * [67] C. Zhang, A. Gupta, C. Kauten, A. V. Deokar, and X. Qin, ‘Detecting Fake News for Reducing Misinformation Risks Using Analytics Approaches’, European Journal of Operational Research, Jun. 2019.
  * [68] X. Dong, U. Victor, S. Chowdhury, and L. Qian, ‘Deep Two-path Semi-supervised Learning for Fake News Detection’, arXiv:1906. 05659 [cs], Jun. 2019.
  * [69] J. Sánchez-Junquera, P. Rosso, M. Montes-y-Gómez, and S. P. Ponzetto, ‘Unmasking Bias in News’, arXiv:1906. 04836 [cs], Jun. 2019.
  * [70] Y. Wang, H. Han, Y. Ding, X. Wang, and Q. Liao, ‘Learning Contextual Features with Multi-head Self-attention for Fake News Detection’, in Cognitive Computing – ICCC 2019, 2019, pp. 132–142.
  * [71] T. Saikh, A. Anand, A. Ekbal, and P. Bhattacharyya, ‘A Novel Approach Towards Fake News Detection: Deep Learning Augmented with Textual Entailment Features’, in Natural Language Processing and Information Systems, 2019, pp. 345–358.
  * [72] Anoop K, Deepak P, and Lajish VL. ‘Emotion Cognizance Improves Fake News Identification’, arXiv:1906. 10365 [cs], Jun. 2019.
  * [73] M. Lim and S. Park, ‘A Study on the Preemptive Measure for Fake News Eradication Using Data Mining Algorithms: Focused on the M Online Community Postings’, Journal of Information Technology Services, vol. 18, no. 1, pp. 219–234, 2019.
  * [74] A. Ahmad, J. Webb, K. C. Desouza, and J. Boorman, ‘Strategically-Motivated Advanced Persistent Threat: Definition, Process, Tactics and a Disinformation Model of Counterattack’, Computers & Security, Jul. 2019.
  * [75] M. Seref and O. Seref, ‘Rhetoric Mining for Fake News: Identifying Moves of Persuasion and Disinformation’, AMCIS 2019 Proceedings, Jul. 2019.
  * [76] N. Papernot, P. McDaniel, A. Sinha, and M. Wellman, ‘SoK: Security and Privacy in Machine Learning’, IEEE Explore, 2018.
  * [77] K. Shu and H. Liu, ‘Detecting Fake News on Social Media’, Synthesis Lectures on Data Mining and Knowledge Discovery, vol. 11, no. 3, pp. 1–129, Jul. 2019.
  * [78] A. Debnath, R. Rahman, M. Mofijul Islam, and M. Abdur Razzaque, ‘A Hierarchical Learning Model for Claim Validation’, in Proceedings of International Joint Conference on Computational Intelligence, 2020, pp. 431–441.
  * [79] J. Reyes and L. Palafox, ‘Detection of Fake News based on readability’, p. 6.
  * [80] M. A. Stefanone, M. Vollmer, and J. M. Covert, ‘In News We Trust?: Examining Credibility and Sharing Behaviors of Fake News’, in Proceedings of the 10th International Conference on Social Media and Society, 2019, pp. 136–147.
  * [81] A. Heydari, J. Zhang, S. Appel, X. Wu, and P. G. Ranade, ‘YouTube Chatter: Understanding Online Comments Discourse on Misinformative and Political YouTube Videos’, p. 32.
  * [82] J. Kapusta, Ľ. Benko, and M. Munk, ‘Fake News Identification Based on Sentiment and Frequency Analysis’, in Innovation in Information Systems and Technologies to Support Learning Research, 2020, pp. 400–409.
  * [83] S. Hosseinimotlagh and E. E. Papalexakis, ‘Unsupervised Content-Based Identification of Fake News Articles with Tensor Decomposition Ensembles’, p. 8.
  * [84] K. Popat, ‘“Credibility Analysis of Textual Claims with Explainable Evidence”’, p. 134.
  * [85] T. Hamdi, H. Slimi, I. Bounhas, and Y. Slimani, ‘A Hybrid Approach for Fake News Detection in Twitter Based on User Features and Graph Embedding’, in Distributed Computing and Internet Technology, 2020, pp. 266–280.
  * [86] D. Pacheco, P.-M. Hui, C. Torres-Lugo, B. T. Truong, A. Flammini, and F. Menczer, ‘Uncovering Coordinated Networks on Social Media’, arXiv:2001. 05658 [physics], Jan. 2020.
  * [87] R. K. Kaliyar, A. Goswami, P. Narang, and S. Sinha, ‘FNDNet- A Deep Convolutional Neural Network for Fake News Detection’, Cognitive Systems Research, Jan. 2020.
  * [88] N. X. Nyow and H. N. Chua, ‘Detecting Fake News with Tweets’ Properties’, in 2019 IEEE Conference on Application, Information and Network Security (AINS), 2019, pp. 24–29.
  * [89] F. Pierri, C. Piccardi, and S. Ceri, ‘Topology comparison of Twitter diffusion networks effectively reveals misleading information’, Scientific Reports, vol. 10, no. 1, pp. 1–9, Jan. 2020.
  * [90] H. Reddy, N. Raj, M. Gala, and A. Basava, ‘Text-mining-based Fake News Detection Using Ensemble Methods’, International Journal of Automation and Computing, Feb. 2020.
  * [91] J. Cao, P. Qi, Q. Sheng, T. Yang, J. Guo, and J. Li, ‘Exploring the Role of Visual Content in Fake News Detection’, arXiv:2003. 05096 [cs], Mar. 2020.
  * [92] K. Shu et al., ‘Leveraging Multi-Source Weak Social Supervision for Early Detection of Fake News’, arXiv:2004. 01732 [cs, stat], Apr. 2020.
  * [93] H. Kudarvalli and J. Fiaidhi, ‘Detecting Fake News using Machine Learning Algorithms’, Apr. 2020.
  * [94] D. Freelon, M. Bossetta, C. Wells, J. Lukito, Y. Xia, and K. Adams, ‘Black Trolls Matter: Racial and Ideological Asymmetries in Social Media Disinformation’, Social Science Computer Review, p. 0894439320914853, Apr. 2020.
  * [95] A. Uppal, V. Sachdeva, and S. Sharma, ‘Fake news detection using discourse segment structure analysis’, in 2020 10th International Conference on Cloud Computing, Data Science Engineering (Confluence), 2020, pp. 751–756.
  * [96] M. Previti, V. Rodriguez-Fernandez, D. Camacho, V. Carchiolo, and M. Malgeri, ‘Fake News Detection Using Time Series and User Features Classification’, in Applications of Evolutionary Computation, vol. 12104, P. A. Castillo, J. L. Jiménez Laredo, and F. Fernández de Vega, Eds. Cham: Springer International Publishing, 2020, pp. 339–353.
  * [97] Y. Zhou, Y. Zhang, and J. Yao, ‘Satirical News Detection with Semantic Feature Extraction and Game-theoretic Rough Sets’, arXiv:2004. 03788 [cs], Apr. 2020.
  * [98] P. Atanasova, J. G. Simonsen, C. Lioma, and I. Augenstein, ‘Generating Fact Checking Explanations’, arXiv:2004. 05773 [cs], Apr. 2020.
  * [99] L. Tian, X. Zhang, Y. Wang, and H. Liu, ‘Early Detection of Rumours on Twitter via Stance Transfer Learning’, in Advances in Information Retrieval, 2020, pp. 575–588.
  * [100] J. George, S. M. Skariah, and T. Aleena Xavier, ‘Role of Contextual Features in Fake News Detection: A Review’, in 2020 International Conference on Innovative Trends in Information Technology (ICITIIT), 2020, pp. 1–6.
  * [101] Y. S. Kartal, B. Guvenen, and M. Kutlu, ‘Too Many Claims to Fact-Check: Prioritizing Political Claims Based on Check-Worthiness’, arXiv:2004. 08166 [cs], Apr. 2020.
  * [102] Z. Chen and J. Freire, ‘Proactive Discovery of Fake News Domains from Real-Time Social Media Feeds’, in Companion Proceedings of the Web Conference 2020, 2020, pp. 584–592.
  * [103] M. L. Gordon, K. Zhou, K. Patel, T. Hashimoto, and M. S. Bernstein, ‘The Disagreement Deconvolution: Bringing Machine Learning Performance Metrics In Line With Reality’, p. 14, 2021.
  * [104] A. Silva, L. Luo, S. Karunasekera, and C. Leckie, ‘Embracing Domain Differences in Fake News: Cross-domain Fake News Detection using Multimodal Data’, arXiv:2102. 06314 [cs], Feb. 2021.
  * [105] L. Kurasinski and R.-C. Mihailescu, ‘Towards Machine Learning Explainability in Text Classification for Fake News Detection’, in 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA), 2020, pp. 775–781.
  * [106] P. Carvalho et al., ‘Assessing News Credibility: Misinformation Content Indicators’, In Review, Mar. 2021.
  * [107] M. Chalkiadakis, A. Kornilakis, P. Papadopoulos, E. P. Markatos, and N. Kourtellis, ‘The Rise and Fall of Fake News sites: A Traffic Analysis’, arXiv:2103. 09258 [cs], Mar. 2021.
  * [108] B. Krämer, ‘Stop studying “fake news” (we can still fight against disinformation in the media)’, Studies in Communication and Media, vol. 10, no. 1, pp. 6–30, 2021.
  * [109] N. Lee et al., ‘On Unifying Misinformation Detection’, arXiv:2104. 05243 [cs], Apr. 2021.
  * [110] K. Pelrine, J. Danovitch, and R. Rabbany, ‘The Surprising Performance of Simple Baselines for Misinformation Detection’, arXiv:2104. 06952 [cs], Apr. 2021.
  * [111] B. Jabiyev, K. Onarlioglu, S. Pehlivanoglu, and E. Kirda, ‘FADE: Detecting Fake News Articles on the Web’, p. 10, 2021.
  * [112] J. Simko et al., ‘Towards Continuous Automatic Audits of Social Media Adaptive Behavior and its Role in Misinformation Spreading’, in Adjunct Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization, 2021, pp. 411–414.
  * [113] Y. Yang, T. Davis, and M. Hindman, ‘Visual Misinformation on Facebook’, p. 8.
  * [114] K. Roitero et al., ‘Can the Crowd Judge Truthfulness? A Longitudinal Study on Recent Misinformation about COVID-19’, arXiv:2107. 11755 [cs], Jul. 2021.
  * [115] K. Roitero et al., ‘The COVID-19 Infodemic: Can the Crowd Judge Recent Misinformation Objectively?’, Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 1305–1314, Oct. 2020.
  * [116] M. Soprano et al., ‘The Many Dimensions of Truthfulness: Crowdsourcing Misinformation Assessments on a Multidimensional Scale’, arXiv:2108. 01222 [cs], Aug. 2021.
  * [117] A. Kim, P. Moravec, and A. Dennis, ‘Behind the Stars: The Effects of News Source Ratings on Fake News in Social Media’, SSRN Electronic Journal, Jan. 2017.
  * [118] P. Resnick, A. Alfayez, J. Im, and E. Gilbert, ‘Informed Crowds Can Effectively Identify Misinformation’, arXiv:2108. 07898 [cs], Aug. 2021.
  * [119] P. Juneja and T. Mitra, ‘Auditing E-Commerce Platforms for Algorithmically Curated Vaccine Misinformation’, in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, New York, NY, USA: Association for Computing Machinery, 2021, pp. 1–27.
  * [120] M. Janicka, M. Pszona, and A. Wawer, ‘Cross-Domain Failures of Fake News Detection’, Computación y Sistemas, vol. 23, no. 3, Oct. 2019.
  * [121] Z. Kou, L. Shang, Y. Zhang, and D. Wang, ‘HC-COVID: A Hierarchical Crowdsource Knowledge Graph Approach to Explainable COVID-19 Misinformation Detection’, Proceedings of the ACM on Human-Computer Interaction, vol. 6, no. GROUP, p. 36:1-36:25, Jan. 2022.
  * [122] R. Wang et al., ‘RumorLens: Interactive Analysis and Validation of Suspected Rumors on Social Media’, in Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, 2022, pp. 1–7.
  * [123] N. Hassan et al., ‘Data in, fact out: automated monitoring of facts by FactWatcher’, Proceedings of the VLDB Endowment, vol. 7, no. 13, pp. 1557–1560, Aug. 2014.
  * [124] J. Thorne and A. Vlachos, ‘Automated Fact Checking: Task Formulations, Methods and Future Directions’, p. 14.
  * [125] F. Jin, E. Dougherty, P. Saraf, Y. Cao, and N. Ramakrishnan, ‘Epidemiological modeling of news and rumors on Twitter’, in Proceedings of the 7th Workshop on Social Network Mining and Analysis, 2013, pp. 1–9.
  * [126] L. Zeng, K. Starbird, and E. Spiro, ‘#Unconfirmed: Classifying Rumor Stance in Crisis-Related Social Media Messages’, Proceedings of the International AAAI Conference on Web and Social Media, vol. 10, no. 1, pp. 747–750, 2016.
  * [127] G. Eady, T. Paskhalis, J. Zilinsky, R. Bonneau, J. Nagler, and J. A. Tucker, ‘Exposure to the Russian Internet Research Agency foreign influence campaign on Twitter in the 2016 US election and its relationship to attitudes and voting behavior’, Nature Communications, vol. 14, no. 1, p. 62, Jan. 2023.
  * [128] A. T. Kabakuş and M. Şimşek, ‘An Analysis of the Characteristics of Verified Twitter Users’, Sakarya University Journal of Computer and Information Sciences, vol. 2, no. 3, pp. 180–186, Dec. 2019.
  * [129] A. Hearn, ‘Verified: Self-presentation, identity management, and selfhood in the age of big data’, Popular Communication, vol. 15, no. 2, pp. 62–77, Apr. 2017.
  * [130] ‘Twitter Verification requirements - how to get the blue check’. Twitter Help Center.
  * [131] ‘Prolific · Quickly find research participants you can trust’. Prolific.co.
  * [132] A. Arif, K. Shanahan, F.-J. Chou, Y. Dosouto, K. Starbird, and E. S. Spiro, ‘How Information Snowballs: Exploring the Role of Exposure in Online Rumor Propagation’, in Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, 2016, pp. 466–477.
  * [133] S. van der Linden, ‘Misinformation: susceptibility, spread, and interventions to immunize the public’, Nature Medicine, vol. 28, no. 3, pp. 460–467, Mar. 2022.
  * [134] J. A. Nasir, O. S. Khan, and I. Varlamis, ‘Fake news detection: A hybrid CNN-RNN based deep learning approach’, International Journal of Information Management Data Insights, vol. 1, no. 1, p. 100007, Apr. 2021.
  * [135] M. Glockner, Y. Hou, and I. Gurevych, ‘Missing Counter-Evidence Renders NLP Fact-Checking Unrealistic for Misinformation’. arXiv, Oct-2022.
  * [136] ‘RumorLens: Interactive Analysis and Validation of Suspected Rumors on Social Media |Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems’.
  * [137] A. Mughaid et al., ‘An intelligent cybersecurity system for detecting fake news in social media websites’, Soft Computing, vol. 26, no. 12, pp. 5577–5591, Jun. 2022.
  * [138] F. Alam et al., ‘A Survey on Multimodal Disinformation Detection’. arXiv, Sep-2022.
  * [139] F. Miró-Llinares and J. C. Aguerri, ‘Misinformation about fake news: A systematic critical review of empirical studies on the phenomenon and its status as a “threat”’, European Journal of Criminology, vol. 20, no. 1, pp. 356–374, Jan. 2023.
  * [140] P. Resnick, A. Alfayez, J. Im, and E. Gilbert, ‘Informed Crowds Can Effectively Identify Misinformation’. arXiv, Feb-2022.
  * [141] ‘Assessing News Credibility: Misinformation Content Indicators’. Mar-2021.
  * [142] K. Roitero et al., ‘Can the Crowd Judge Truthfulness? A Longitudinal Study on Recent Misinformation about COVID-19’, Personal and Ubiquitous Computing, vol. 27, no. 1, pp. 59–89, Feb. 2023.
  * [143] J. Y. Khan, M. T. I. Khondaker, S. Afroz, G. Uddin, and A. Iqbal, ‘A benchmark study of machine learning models for online fake news detection’, Machine Learning with Applications, vol. 4, p. 100032, Jun. 2021.
  * [144] D. Fallis, ‘A Functional Analysis of Disinformation’, iConference 2014 Proceedings, Mar. 2014.
  * [145] X. Zhou and R. Zafarani, ‘A Survey of Fake News: Fundamental Theories, Detection Methods, and Opportunities’, ACM Computing Surveys, vol. 53, no. 5, p. 109:1-109:40, Sep. 2020.
  * [146] F. Alam et al., ‘A Survey on Multimodal Disinformation Detection’, arXiv:2103. 12541 [cs], Mar. 2021.
  * [147] R. Oshikawa, J. Qian, and W. Y. Wang, ‘A Survey on Natural Language Processing for Fake News Detection’, arXiv:1811. 00770 [cs], Nov. 2018.
  * [148] N. K. Conroy, V. L. Rubin, and Y. Chen, ‘Automatic deception detection: Methods for finding fake news’, Proceedings of the Association for Information Science and Technology, vol. 52, no. 1, pp. 1–4, 2015.
  * [149] K. Sharma, F. Qian, H. Jiang, N. Ruchansky, M. Zhang, and Y. Liu, ‘Combating Fake News: A Survey on Identification and Mitigation Techniques’, ACM Transactions on Intelligent Systems and Technology, vol. 10, no. 3, p. 21:1-21:42, Apr. 2019.
  * [150] M. R. Islam, S. Liu, X. Wang, and G. Xu, ‘Deep learning for misinformation detection on online social networks: a survey and new perspectives’, Social Network Analysis and Mining, vol. 10, no. 1, p. 82, Sep. 2020.
  * [151] A. A. A. Ahmed, A. Aljarbouh, P. Donepudi, and M. Choi, ‘Detecting Fake News using Machine Learning: A Systematic Literature Review’, Psychology (Savannah, Ga. ), vol. 58, pp. 1932–1939, Jan. 2021.
  * [152] K. Shu, A. Sliva, S. Wang, J. Tang, and H. Liu, ‘Fake News Detection on Social Media: A Data Mining Perspective’, p. 15.
  * [153] A. Habib, M. Z. Asghar, A. Khan, A. Habib, and A. Khan, ‘False information detection in online content and its role in decision making: a systematic literature review’, Social Network Analysis and Mining, vol. 9, no. 1, p. 50, Sep. 2019.
  * [154] J. Zhang, L. Cui, Y. Fu, and F. B. Gouza, ‘Fake News Detection with Deep Diffusive Network Model’, arXiv:1805. 08751 [cs, stat], May 2018.
  * [155] N. R. de Oliveira, P. S. Pisa, M. A. Lopez, D. S. V. de Medeiros, and D. M. F. Mattos, ‘Identifying Fake News on Social Networks Based on Natural Language Processing: Trends and Challenges’, Information, vol. 12, no. 1, p. 38, Jan. 2021.
  * [156] K. Shu, S. Wang, D. Lee, and H. Liu, ‘Mining Disinformation and Fake News: Concepts, Methods, and Recent Advancements’, in Disinformation, Misinformation, and Fake News in Social Media: Emerging Research Challenges and Opportunities, K. Shu, S. Wang, D. Lee, and H. Liu, Eds. Cham: Springer International Publishing, 2020, pp. 1–19.
  * [157] F. Miró-Llinares and J. C. Aguerri, ‘Misinformation about fake news: A systematic critical review of empirical studies on the phenomenon and its status as a “threat”’, European Journal of Criminology, p. 1477370821994059, Apr. 2021.
  * [158] Y. Chen, N. K. Conroy, and V. L. Rubin, ‘News in an online world: The need for an “automatic crap detector”’, Proceedings of the Association for Information Science and Technology, vol. 52, no. 1, pp. 1–4, 2015.
  * [159] M. Fernandez and H. Alani, ‘Online Misinformation: Challenges and Future Directions’, in Companion of the The Web Conference 2018 on The Web Conference 2018 - WWW ’18, 2018, pp. 595–602.
  * [160] D. A. Martin, J. N. Shapiro, and M. Nedashkovskaya, ‘Recent Trends in Online Foreign Influence Efforts’, p. 34.
  * [161] S. Shelke and V. Attar, ‘Source detection of rumor in social network – A review’, Online Social Networks and Media, vol. 9, pp. 30–42, Jan. 2019.
  * [162] B. Guo, Y. Ding, L. Yao, Y. Liang, and Z. Yu, ‘The Future of Misinformation Detection: New Perspectives and Trends’. arXiv, Sep-2019.
  * [163] J. Thorne and A. Vlachos, ‘Automated Fact Checking: Task Formulations, Methods and Future Directions’, in Proceedings of the 27th International Conference on Computational Linguistics, 2018, pp. 3346–3359.
  * [164] A. Zubiaga, M. Liakata, R. Procter, G. W. S. Hoi, and P. Tolmie, ‘Analysing How People Orient to and Spread Rumours in Social Media by Looking at Conversational Threads’, PLOS ONE, vol. 11, no. 3, p. e0150989, Mar. 2016.
  * [165] V. L. Rubin, Y. Chen, and N. K. Conroy, ‘Deception detection for news: Three types of fakes’, Proceedings of the Association for Information Science and Technology, vol. 52, no. 1, pp. 1–4, 2015.
  * [166] F. Yu, Q. Liu, S. Wu, L. Wang, and T. Tan, ‘A Convolutional Approach for Misinformation Identification’, pp. 3901–3907, 2017.
  * [167] A. Roy, K. Basak, A. Ekbal, and P. Bhattacharyya, ‘A Deep Ensemble Framework for Fake News Detection and Multi-Class Classification of Short Political Statements’, in Proceedings of the 16th International Conference on Natural Language Processing, 2019, pp. 9–17.
  * [168] B. Amado, F. Ramón Fariña, ‘Undeutsch hypothesis and Criteria Based Content Analysis: A meta-analytic review’, in European Journal of Psychology Applied to Legal Context, 2015., pp. 3-12.
  * [169] S. Antoniadis, I. Litou, and V. Kalogeraki, ‘A Model for Identifying Misinformation in Online Social Networks’, in On the Move to Meaningful Internet Systems: OTM 2015 Conferences, 2015, pp. 473–482.
  * [170] B. Al Asaad and M. Erascu, ‘A Tool for Fake News Detection’, in 2018 20th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC), 2018, pp. 379–386.
  * [171] A. C. Nied, L. Stewart, E. Spiro, and K. Starbird, ‘Alternative Narratives of Crisis Events: Communities and Social Botnets Engaged on Social Media’, in Companion of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, 2017, pp. 263–266.
  * [172] X. Zhang and A. A. Ghorbani, ‘An overview of online fake news: Characterization, detection, and discussion’, Information Processing & Management, vol. 57, no. 2, p. 102025, Mar. 2020.
  * [173] M. Christodorescu, S. Jha, S. A. Seshia, D. Song and R. E. Bryant, ‘Semantics-aware malware detection,’ 2005 IEEE Symposium on Security and Privacy (S&P’05), Oakland, CA, USA, 2005, pp. 32-46, doi: 10.1109/SP.2005.20.
  * [174] S. T. King and P. M. Chen, ‘SubVirt: implementing malware with virtual machines,’ 2006 IEEE Symposium on Security and Privacy (S&P’06), Berkeley/Oakland, CA, 2006, pp. 14 pp.-327, doi: 10.1109/SP.2006.38.
  * [175] P. Paudel, J. Blackburn, E. De Cristofaro, S. Zannettou and G. Stringhini, ‘LAMBRETTA: Learning to Rank for Twitter Soft Moderation,’ in 2023 IEEE Security & Privacy, San Francisco, CA, USA, 2023 pp. 311-326.
  * [176] M.H. Saeed, S. Ali, J. Blackburn, E. De Cristofaro, S. Zannettou, and G. Stringhini, ‘TROLLMAGNIFIER: Detecting State-Sponsored Troll Accounts on Reddit,’ in 2022 IEEE Security & Privacy (SP), San Francisco, CA, USA, 2022 pp 2161-2175.
  * [177] M. Zurko, ‘Disinformation and Reflections From Usable Security,’ in IEEE Security & Privacy, vol. 20, no. 03, pp. 4-7, 2022.
  * [178] K. Thomas, C. Grier, J. Ma, V. Paxson and D. Song, ‘Design and Evaluation of a Real-Time URL Spam Filtering Service,’ in 2011 IEEE Security & Privacy, Oakland, CA, USA, 2011, pp. 447-462, doi: 10.1109/SP.2011.25.
  * [179] G. Danezis and P. Mittal, ‘SybilInfer: Detecting Sybil Nodes using Social Networks’ in NDSS Symposium 2009, San Diego, CA, USA, 2009.
  * [180] G. Wang, T. Konolige, C. Wilson, X. Wang, H. Zheng, and B. Y. Zhao, ‘You are how you click: clickstream analysis for Sybil detection,’ in 22nd USENIX conference on Security (SEC’13). USENIX Association, USA, 241–256.
  * [181] G. Wang, T. Wang, H. Zhang, and B. Y. Zhao, ‘Man vs. machine: practical adversarial detection of malicious crowdsourcing workers,’ in 23rd USENIX conference on Security Symposium (SEC’14). USENIX Association, USA, 239–254.
  * [182] T. Kim, N. Park, J. Hong, and S. Kim, ‘Phishing URL Detection: A Network-based Approach Robust to Evasion,’ Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security (CCS ’22). Association for Computing Machinery, New York, NY, USA, 1769–1782. https://doi.org/10.1145/3548606.3560615
  * [183] D. Yuan, Y. Miao, N. Z. Gong, Z. Yang, Q. Li, D. Song, Q. Wang, and X. Liang. 2019. ‘Detecting Fake Accounts in Online Social Networks at the Time of Registrations,’ in ACM SIGSAC Conference on Computer and Communications Security (CCS ’19). Association for Computing Machinery, New York, NY, USA, 1423–1438. https://doi.org/10.1145/3319535.3363198
  * [184] S. Nilizadeh, F. Labrèche, A. Sedighian, A. Zand, J. Fernandez, C. Kruegel, G. Stringhini, and G. Vigna, ‘POISED: Spotting Twitter Spam Off the Beaten Paths,’ in ACM SIGSAC Conference on Computer and Communications Security (CCS ’17). Association for Computing Machinery, New York, NY, USA, 1159–1174. https://doi.org/10.1145/3133956.3134055
  * [185] C. Grier, K. Thomas, V. Paxson, and M. Zhang, ‘@spam: the underground on 140 characters or less,’ in ACM conference on Computer and communications security (CCS ’10). Association for Computing Machinery, New York, NY, USA, 27–37. https://doi.org/10.1145/1866307.1866311
  * [186] C. Whittaker, B. Ryner, M. Nazif, ‘Large-Scale Automatic Classification of Phishing Pages,’ in NDSS Symposium 2010, San Diego, CA, USA, 2010.
  * [187] A. Pitsillidis , K. Levchenko, C. Kreibich, C. Kanich, G. M. Voelker, V. Paxson, N. Weaver, S. Savage, ‘Botnet Judo: Fighting Spam with Itself,’ in NDSS Symposium 2010, San Diego, CA, USA, 2010.
  * [188] B. Stone-Gross, M. Cova, L. Cavallaro, B. Gilbert, M. Szydlowski, R. Kemmerer, C. Kruegel, and G. Vigna, ‘Your botnet is my botnet: analysis of a botnet takeover,’ in 6th ACM conference on Computer and communications security (CCS ’09). Association for Computing Machinery, New York, NY, USA, 635–647. https://doi.org/10.1145/1653662.1653738
  * [189] H. Yu, P. B. Gibbons, M. Kaminsky and F. Xiao, ‘SybilLimit: A Near-Optimal Social Network Defense against Sybil Attacks,’ 2008 IEEE Symposium on Security and Privacy (sp 2008), Oakland, CA, USA, 2008, pp. 3-17, doi: 10.1109/SP.2008.13.
  * [190] A. Vrij, ‘Detecting lies and deceit: pitfalls and opportunities’, Wiley Publishers, Chichester, England, 2008.
  * [191] V. Singh, R. Dasgupta, D. Sonagra, K. Raman, and I. Ghosh, ‘Automated Fake News Detection Using Linguistic Analy- sis and Machine Learning’.
  * [192] S. Kaur, P. Kumar, and P. Kumaraguru, ‘Automating fake news detection system using multi-level voting model’, Soft Computing, vol. 24, no. 12, pp. 9049–9069, Jun. 2020.
  * [193] N. P. Nguyen, G. Yan, M. T. Thai, and S. Eidenbenz, ‘Containment of misinformation spread in online social networks’, in Proceedings of the 4th Annual ACM Web Science Conference, 2012, pp. 213–222.
  * [194] T. Mitra and E. Gilbert, ‘CREDBANK: A Large-Scale Social Media Corpus With Associated Credibility Annotations’, Proceedings of the International AAAI Conference on Web and Social Media, vol. 9, no. 1, pp. 258–267, 2015.
  * [195] K. Shu, L. Cui, S. Wang, D. Lee, and H. Liu, ‘dEFEND: Explainable Fake News Detection’, in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019, pp. 395–405.
  * [196] C. G. Harris, ‘Detecting Deceptive Opinion Spam Using Human Computation’, 2012.
  * [197] M. Aldwairi and A. Alwahedi, ‘Detecting Fake News in Social Media Networks’, Procedia Computer Science, vol. 141, pp. 215–222, Jan. 2018.
  * [198] Abdullah-All-Tanvir, E. M. Mahir, S. Akhter, and M. R. Huq, ‘Detecting Fake News using Machine Learning and Deep Learning Algorithms’, in 2019 7th International Conference on Smart Computing & Communications (ICSCC), 2019, pp. 1–5.
  * [199] S. Aphiwongsophon and P. Chongstitvatana, ‘Detecting Fake News with Machine Learning Method’, in 2018 15th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON), 2018, pp. 528–531.
  * [200] S. Afroz, M. Brennan, and R. Greenstadt, ‘Detecting Hoaxes, Frauds, and Deception in Writing Style Online’, in 2012 IEEE Symposium on Security and Privacy, 2012, pp. 461–475.
  * [201] H. Ahmed, I. Traore, and S. Saad, ‘Detecting opinion spams and fake news using text classification’, SECURITY AND PRIVACY, vol. 1, no. 1, p. e9, 2018.
  * [202] B. A. Galitsky, ‘Detecting Rumor and Disinformation by Web Mining’, 2015.
  * [203] Z. Jin, J. Cao, H. Guo, Y. Zhang, Y. Wang, and J. Luo, ‘Detection and Analysis of 2016 US Presidential Election Related Rumors on Twitter’, in Social, Cultural, and Behavioral Modeling, 2017, pp. 14–24.
  * [204] A. Zubiaga, A. Aker, K. Bontcheva, M. Liakata, and R. Procter, ‘Detection and Resolution of Rumours in Social Media: A Survey’, ACM Computing Surveys, vol. 51, no. 2, p. 32:1-32:36, Feb. 2018.
  * [205] H. Ahmed, I. Traore, and S. Saad, ‘Detection of Online Fake News Using N-Gram Analysis and Machine Learning Techniques’, in Intelligent, Secure, and Dependable Systems in Distributed and Cloud Environments, 2017, pp. 127–138.
  * [206] S. K. Maity, A. Chakraborty, P. Goyal, and A. Mukherjee, ‘Detection of Sockpuppets in Social Media’, in Companion of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, 2017, pp. 243–246.
  * [207] P. Hernon, ‘Disinformation and misinformation through the internet: Findings of an exploratory study’, Government Information Quarterly, vol. 12, no. 2, pp. 133–139, Jan. 1995.
  * [208] L. van de Guchte, S. Raaijmakers, E. Meeuwissen, and J. Spenader, ‘Near Real-Time Detection of Misinformation on Online Social Networks’, in Disinformation in Open Online Media, 2020, pp. 246–260.
  * [209] M. Dong, L. Yao, X. Wang, B. Benatallah, Q. Sheng, and H. Huang, ‘DUAL: A Deep Unified Attention Model with Latent Relation Representations for Fake News Detection: 19th International Conference, Dubai, United Arab Emirates, November 12-15, 2018, Proceedings, Part I’, 2018, pp. 199–209.
  * [210] M. Konte, N. Feamster, and J. Jung, ‘Dynamics of Online Scam Hosting Infrastructure’, in Passive and Active Network Measurement, 2009, pp. 219–228.
  * [211] K. Starbird, ‘Examining the Alternative Media Ecosystem Through the Production of Alternative Narratives of Mass Shooting Events on Twitter’, Proceedings of the International AAAI Conference on Web and Social Media, vol. 11, no. 1, pp. 230–239, May 2017.
  * [212] K. Shu, S. Wang, and H. Liu, ‘Exploiting Tri-Relationship for Fake News Detection’, Dec. 2017.
  * [213] T. Mihaylov, I. Koychev, G. Georgiev, and P. Nakov, ‘Exposing Paid Opinion Manipulation Trolls’. arXiv, Sep-2021.
  * [214] M. Tambuscio, G. Ruffo, A. Flammini, and F. Menczer, ‘Fact-checking Effect on Viral Hoaxes: A Model of Misinformation Spread in Social Networks’, in Proceedings of the 24th International Conference on World Wide Web, 2015, pp. 977–982.
  * [215] C. M. M. Kotteti, X. Dong, N. Li, and L. Qian, ‘Fake news detection enhancement with data imputation’, Computer Information Systems Faculty Publications, Oct. 2018.
  * [216] S. Tschiatschek, A. Singla, M. Gomez Rodriguez, A. Merchant, and A. Krause, ‘Fake News Detection in Social Networks via Crowd Signals’, in Companion of the The Web Conference 2018 on The Web Conference 2018 - WWW ’18, 2018, pp. 517–524.
  * [217] A. Kesarwani, S. S. Chauhan, and A. R. Nair, ‘Fake News Detection on Social Media using K-Nearest Neighbor Classifier’, in 2020 International Conference on Advances in Computing and Communication Engineering (ICACCE), 2020, pp. 1–4.
  * [218] Y. Long, Q. Lu, R. Xiang, M. Li, and C.-R. Huang, ‘Fake News Detection Through Multi-Perspective Speaker Profiles’, in Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), 2017, pp. 252–256.
  * [219] S. Kumar, R. Asthana, S. Upadhyay, N. Upreti, and M. Akbar, ‘Fake news detection using deep learning models: A novel approach’, Transactions on Emerging Telecommunications Technologies, vol. 31, no. 2, p. e3767, 2020.
  * [220] I. Ahmad, M. Yousaf, S. Yousaf, and M. O. Ahmad, ‘Fake News Detection Using Machine Learning Ensemble Methods’, Complexity, vol. 2020, p. e8885861, Oct. 2020.
  * [221] M. Granik and V. Mesyura, ‘Fake news detection using naive Bayes classifier’, in 2017 IEEE First Ukraine Conference on Electrical and Computer Engineering (UKRCON), 2017, pp. 900–903.
  * [222] B. Bhutani, N. Rastogi, P. Sehgal, and A. Purwar, ‘Fake News Detection Using Sentiment Analysis’, in 2019 Twelfth International Conference on Contemporary Computing (IC3), 2019, pp. 1–5.
  * [223] A. Thota, P. Tilak, S. Ahluwalia, and N. Lohia, ‘Fake News Detection: A Deep Learning Approach’, vol. 1, no. 3, 2018.
  * [224] S. S. Jadhav and S. D. Thepade, ‘Fake News Identification and Classification Using DSSM and Improved Recurrent Neural Network Classifier’, Applied Artificial Intelligence, vol. 33, no. 12, pp. 1058–1068, Oct. 2019.
  * [225] M. Farajtabar et al., ‘Fake News Mitigation via Point Process Based Intervention’, in Proceedings of the 34th International Conference on Machine Learning, 2017, pp. 1097–1106.
  * [226] V. Rubin, N. Conroy, Y. Chen, and S. Cornwell, ‘Fake News or Truth? Using Satirical Cues to Detect Potentially Misleading News’, in Proceedings of the Second Workshop on Computational Approaches to Deception Detection, 2016, pp. 7–17.
  * [227] A. Dey, R. Z. Rafi, S. Hasan Parash, S. K. Arko, and A. Chakrabarty, ‘Fake News Pattern Recognition using Linguistic Analysis’, in 2018 Joint 7th International Conference on Informatics, Electronics & Vision (ICIEV) and 2018 2nd International Conference on Imaging, Vision & Pattern Recognition (icIVPR), 2018, pp. 305–309.
  * [228] L. Wu, Y. Rao, H. Yu, Y. Wang, and A. Nazir, ‘False Information Detection on Social Media via a Hybrid Deep Model’, in Social Informatics, 2018, pp. 323–333.
  * [229] S. Kumar and N. Shah, ‘False Information on Web and Social Media: A Survey’, Apr. 2018.
  * [230] K. Wu, S. Yang, and K. Q. Zhu, ‘False rumors detection on Sina Weibo by propagation structures’, in 2015 IEEE 31st International Conference on Data Engineering, 2015, pp. 651–662.
  * [231] Nguyen, V.-H. A. Sugiyama, K. A. Nakov, P. A. Kan, and Min-Yen, ‘FANG |Proceedings of the 29th ACM International Conference on Information & Knowledge Management’. .
  * [232] N. Sitaula, C. K. Mohan, J. Grygiel, X. Zhou, and R. Zafarani, ‘Credibility-Based Fake News Detection’, in Disinformation, Misinformation, and Fake News in Social Media: Emerging Research Challenges and Opportunities, K. Shu, S. Wang, D. Lee, and H. Liu, Eds. Cham: Springer International Publishing, 2020, pp. 163–182.
  * [233] H. Zhang, A. Kuhnle, J. D. Smith, and M. T. Thai, ‘Fight Under Uncertainty: Restraining Misinformation and Pushing out the Truth’, in 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), 2018, pp. 266–273.
  * [234] T. Mihaylov, G. Georgiev, and P. Nakov, ‘Finding Opinion Manipulation Trolls in News Community Forums’, in Proceedings of the Nineteenth Conference on Computational Natural Language Learning, 2015, pp. 310–314.
  * [235] Y. Liu and Y.-F. B. Wu, ‘FNED: A Deep Network for Fake News Early Detection on Social Media’, ACM Transactions on Information Systems, vol. 38, no. 3, p. 25:1-25:33, May 2020.
  * [236] N. Chawla and W. Wang, Eds., Proceedings of the 2017 SIAM International Conference on Data Mining. Philadelphia, PA: Society for Industrial and Applied Mathematics, 2017.
  * [237] L. Wu, J. Li, X. Hu, and H. Liu, ‘Gleaning Wisdom from the Past: Early Detection of Emerging Rumors in Social Media’, 2017, pp. 99–107.
  * [238] B. Ni, Z. Guo, J. Li, and M. Jiang, ‘Improving Generalizability of Fake News Detection Methods using Propensity Score Matching’. arXiv, Jan-2020.
  * [239] C. Castillo, M. Mendoza, and B. Poblete, ‘Information credibility on twitter’, in Proceedings of the 20th international conference on World wide web, 2011, pp. 675–684.
  * [240] F. Ribeiro et al., ‘Media Bias Monitor: Quantifying Biases of Social Media News Outlets at Large-Scale’, Proceedings of the International AAAI Conference on Web and Social Media, vol. 12, no. 1, Jun. 2018.
  * [241] Q. Liu, F. Yu, S. Wu, and L. Wang, ‘Mining Significant Microblogs for Misinformation Identification: An Attention-Based Approach’, ACM Transactions on Intelligent Systems and Technology, vol. 9, no. 5, p. 50:1-50:20, Apr. 2018.
  * [242] H. Zhang, M. A. Alim, X. Li, M. T. Thai, and H. T. Nguyen, ‘Misinformation in Online Social Networks: Detect Them All with a Limited Budget’, ACM Transactions on Information Systems, vol. 34, no. 3, p. 18:1-18:24, Apr. 2016.