1]Vector Institute, Toronto, Ontario, Canada

2]RBC Royal Bank, Toronto, Ontario, Canada

3]Toronto Metropolitan University, Toronto, Ontario, Canada

#  FakeWatch : A Framework for Detecting Fake News to Ensure Credible
Elections

Shaina Raza<EMAIL_ADDRESS>Tahniat Khan
<EMAIL_ADDRESS>Drai Paulen-Patterson<EMAIL_ADDRESS>Veronica
Chatrath<EMAIL_ADDRESS>Mizanur Rahman
<EMAIL_ADDRESS>Oluwanifemi Bamgbose
<EMAIL_ADDRESS>[  [  [

###### Abstract

In today’s technologically driven world, the rapid spread of fake news,
particularly during critical events like elections, poses a growing threat to
the integrity of information. To tackle this challenge head-on, we introduce
FakeWatch , a comprehensive framework carefully designed to detect fake news.
Leveraging a newly curated dataset of North American election-related news
articles, we construct robust classification models. Our framework integrates
a model hub comprising of both traditional machine learning (ML) techniques
and cutting-edge Language Models (LMs) to discern fake news effectively. Our
overarching objective is to provide the research community with adaptable and
precise classification models adept at identifying the ever-evolving landscape
of misinformation. Quantitative evaluations of fake news classifiers on our
dataset reveal that, while state-of-the-art LMs exhibit a slight edge over
traditional ML models, classical models remain competitive due to their
balance of accuracy and computational efficiency. Additionally, qualitative
analyses shed light on patterns within fake news articles. This research lays
the groundwork for future endeavors aimed at combating misinformation,
particularly concerning electoral processes. We provide our labeled data and
model publicly for use and reproducibility.

###### keywords:

Large language models, fact-checking, elections, fake news

##  1 Introduction

Fake news encompasses false or misleading information presented as if it were
true, with the intent to deceive or manipulate. This deceptive practice is
achieved through misinformation, which is spread without deceptive intent, and
disinformation, which is deliberately created to mislead (Zhou and Zafarani,
2020, Raza and Ding, 2022). Fake news spreads through channels including
traditional media, social media, websites, and online platforms, encompassing
fabricated stories, distorted facts, sensationalized headlines, and
selectively edited content. The motivation for creating and spreading
misinformation varies from financial gain, to advancing a specific agenda, to
acting as a tool for propaganda and influencing public opinion, to sowing
confusion.

The negative consequences of fake news are evident in real-world scenarios.
During the 2022 Ukraine-Russia conflict, termed World War Wired for its heavy
social media documentation, fake news has been a major issue (Wright et al.,
2023). Fake news is widespread on online platforms, as highlighted by a
NewsGuard report, showing that new TikTok users quickly encounter misleading
war content (Brown, 2022). With the ongoing conflict, showing truth from
falsehood remains a challenge for citizens, exacerbated by social media
platforms’ struggle to control fake news.

During the 2020 Munich Security Conference, the Director-General of the World
Health Organization (WHO) remarked, We are not just fighting an epidemic; we
are fighting an infodemic, underscoring the alarming speed at which COVID-19
misinformation was spreading, compared to the virus itself (Muhammed T and
Mathew, 2022). Over 6,000 people were hospitalized in the first three months
of 2020 due to COVID-19 misinformation, with false claims of microchip
implants contributing to vaccine hesitancy (Benenson, 2021).

Fake news is not limited to the aforementioned events; it also impacts various
aspects of society. Whether during elections, public health crises, or
conflicts between nations, combating fake news has become crucial, emphasizing
the role of artificial intelligence (AI) classifiers in identifying and
mitigating misinformation. Integrating advanced technology and collaborative
efforts is essential for navigating the information landscape and fostering a
resilient, informed society.

Our research in fake news detection advances existing transformer-based model
studies, particularly those by Raza and Ding (2022) and Alghamdi et al.
(2023), with a focus on the dynamic context of North American elections.
Previous insights from Kaliyar et al. (2021), Aïmeur et al. (2023), and Hamed
et al. (2023) highlight deep learning techniques, but reveal shortcomings in
handling data and concept drift (Raza and Ding, 2019). Data drift, the
performance decline of AI models with new data, and concept drift, the
evolution of data patterns, pose challenges in maintaining classifier
accuracy, especially critical in the rapidly changing arena of election news.
To solve these challenges, our research focuses on the 2024 US elections,
tailoring our data preparation methods to address data and concept drift.

We present three significant contributions that form the core of our work:

  1. 1.

We have introduced a novel 2024 US Elections dataset, curated using targeted
keywords and themes, and annotated through a combination of large language
models (LLMs) and human verification. Existing datasets primarily focus on US
Elections in 2016 and 2022 (Allcott and Gentzkow, 2017, Grinberg et al., 2019,
Gruppi et al., 2023). These datasets may not adequately address potential
shifts in data and concepts during the upcoming 2024 US lections. Hence, our
new dataset is designed to address this gap for making robust fake news
classifiers for the 2024 Presidential US Elections. Data prepared for this
paper is available
here111https://huggingface.co/datasets/newsmediabias/fake_news_elections_labelled_data.

  2. 2.

We present an extensive range of machine learning (ML) and deep learning (DL)
classifiers for fake news, creating a versatile and comprehensive hub. We have
designed this model hub to provide invaluable and adaptable resources in the
fight against misinformation. We release the best performing classification
model on our data publicly for
research222https://huggingface.co/newsmediabias/FakeWatch.

  3. 3.

Our study showcases the effectiveness of our classification methodology
through both qualitative and quantitative lenses. We show the numerical
performance of our models using quantifiable measures. We also test a central
hypothesis: that specific linguistic patterns—emotional tone, pronoun usage,
cognitive complexity, and temporal orientation—serve as distinguishable
markers between fake and real news. We aim to provide a more holistic
evaluation through this approach.

To the best of our knowledge, we are the first to create a unique dataset and
fake news classifier for the upcoming 2024 US elections, using AI to counter
misinformation in this domain. Our dataset is specifically designed to address
concept and data drifts, ensuring accurate analysis in the fast-changing
election environment. While the data curation involves employing keywords
specific to the United States, our research approach is applicable to the
broader context of fake news detection across North America. Our findings in
the paper not only affirm the precision of our classification approach but
also contribute significantly to the evolving field of misinformation studies.
By releasing our model and data, we offer invaluable insights for the
development of more resilient media literacy initiatives.

##  2 Related Works

Fake news detection is a subtask of text classification (Liu et al., 2019),
commonly defined as the task of classifying news as either real or false. The
term fake news refers to false or misleading information presented as
authentic, with the intention to deceive or mislead people (Lu et al., 2022,
Zhou and Zafarani, 2020, Allcott and Gentzkow, 2017, Raza and Ding, 2022).
Fake news takes various forms, including clickbait (misleading headlines),
disinformation (intentionally misleading the public), misinformation (false
information regardless of motive), hoax, parody, satire, rumor, deceptive
news, and others, as discussed in Zhou and Zafarani (2020).

The recent developments in fake news classification highlight the integration
of ML and DL techniques to improve accuracy and efficiency. A recent deep
learning-focused study by Lu et al. (2022) shows how these methods can enhance
classifier performance. A related work by Arora and Sikka (2023) provides a
critical analysis of various algorithms used for fake news classification,
shedding light on their effectiveness. Another work by Bonny et al. (2022)
demonstrates the potential of ML classifiers in identifying fake news within
English news datasets. In a similar work, a model is trained on detecting
political news and fake information (Raza, 2021). These studies represent the
state-of-the-art efforts in combating misinformation with AI. However, a key
challenge in building an AI-based fake news classifier is related to datasets,
feature representation, and data fusion (Hamed et al., 2023).

Datasets: The quality and diversity of datasets used for training and
evaluating fake news detection models is a key challenge. Ensuring datasets
are representative is crucial for models to generalize to real-world scenarios
(Raza and Schwartz, 2023). A comprehensive and balanced dataset is required to
capture the multifaceted nature of fake news (Asr and Taboada, 2019).
Benchmark datasets like FakeNewsNet (Shu et al., 2020), Fakeddit (Nakamura et
al., 2019), and NELA-GT (Gruppi et al., 2023) are used for fake news
detection.

In response to the issue of COVID-19-related misinformation, Alghamdi et al.
(2023) implemented transformer-based models for fake news detection. Elections
and fake news have always been a topic of interest, and many datasets have
been proposed for this purpose (Allcott and Gentzkow, 2017, Grinberg et al.,
2019, Gruppi et al., 2023). These datasets are from the 2016 and 2022 US
elections and may not be able to combat data and concept drifts in the 2024 US
elections, which led us to develop a new dataset along this line.

Feature Representations: There are a number of challenges related to selecting
relevant features, considering the dynamic nature of fake news content, such
as the temporal evolution of linguistic patterns (Raza and Ding, 2019), the
influence of multimedia elements (Qi et al., 2019), and the contextual nuances
that contribute to more accurate feature representation (Hamed et al., 2023).
Raza and Ding (2022) and Hamed et al. (2023) recognize the difficulty in
identifying fake news due to the interplay between content and social factors,
and propose a transformer-based approach, focusing on news content and social
contexts.

Data Fusion: The integration of multiple data sources, or data fusion, is
another critical challenge. Combining information from various modalities,
such as textual content (Hamed et al., 2023), user metadata (Raza and Ding,
2022), network structures (Aïmeur et al., 2023), and using LMs (Kaliyar et
al., 2021) can make fake news detection systems more robust. Our work draws
inspiration from prior research, with a particular emphasis on data and
feature representations. Our novel approach includes curating fresh election
agenda data and uniquely applying LLM and ML techniques.

There are also multimodal fake news detection methods (Yang et al., 2023) that
can detect the intramodality and cross-modal correlation between image regions
and text fragments. Combining information from various modalities, such as
textual content (Shu et al., 2020), user metadata (Raza and Ding, 2022), and
network structures (Hamed et al., 2023), and using language models (Chen and
Shu, 2023) can make fake news detection systems more efficient and robust. Our
work draws inspiration from prior research, with a particular emphasis on data
and feature representations.

##  3 Methodology

###  3.1 Defining Fake News

Fake news detection can be defined as a binary classification problem, where
each news item is assigned a label indicating its correctness. Mathematically,
this can be represented as follows:

Let $N$ be a set of news items, where each news item, $n\in N$, is represented
by a feature vector, $x_{n}$, containing its characteristics (e.g., text,
metadata, etc.). There is a corresponding label, $y_{n}$, for each news item,
_n_ , where $y_{n}\in\\{0,1\\}$.

$y_{n}=1$ means the news item is fake (false information).

$y_{n}=0$ means the news item is real (true information).

A classification model, _f_ , is trained to map the feature vector, $x_{n}$,
to its corresponding label, $y_{n}$ (i.e., vector ${f(x}_{n})\rightarrow
y_{n}$). The goal of this model is to accurately predict the label for each
news item based on its features, thus distinguishing between fake and real
news.

###  3.2 FakeWatch Framework

Figure 1: We present FakeWatch , a framework to detect biases within textual
data. In our proposed four-layer framework, data is first gathered from
diverse sources and is then constructed into a quality-focused corpus. The
data is processed by various machine learning models, and evaluated based on
metrics such as accuracy, precision, recall, and F1 score.

We present FakeWatch , an innovative framework designed to detect biases
within textual data, as illustrated in Figure 1. This framework is structured
into four distinct layers: (i) the data collection layer, (ii) the corpus
construction layer, (iii) the model development layer, and (iv) the evaluation
layer. Each layer is designed to integrate with the others, providing an
effective and comprehensive approach for detecting fake news in text.

####  3.2.1 Data Layer

In this layer we integrate data from two distinct sources: 1) Google RSS for
data curation and 2) NELA-GT-2022 dataset (Gruppi et al., 2023), an existing
benchmark.

Data Curation: We curated data from Google RSS by carefully selecting
keywords, categorized into groups such as race/ethnicity-related terms,
religious terms, geographical references, historical and political events, and
other terms associated with racial discourse. From the Google RSS feeds, we
use web scraping and news feed analysis to gather and categorize a wide array
of news data from the US over a six-month period (Apr. 20, 2023 - Oct. 20,
2023).

Our search query is as follows:

Search Query: (2024 U.S. elections OR presidential candidates 2024) AND
(race relations OR ethnic diversity OR racial discourse) AND
(religious freedom OR religious discrimination) AND
(political events OR historical events OR geographical impact) AND
(voting patterns OR electoral college OR campaign strategies) AND
(news analysis OR media coverage)
DATES: [2023-04-20 to 2023-10-20]
SOURCE: News Feeds

From the newspaper library, we parse a selection of articles from each feed,
focusing on: the first five sentences, the source of publication, publication
date, and the article’s URL. We then systematically categorize each article
based on its corresponding keyword group and store it in a structured format.

After gathering and pre-processing data from the sources, we consolidate all
datasets into a single data frame, resulting in a comprehensive dataset
consisting of 9000 records, prioritizing quality over quantity. To safeguard
users’ privacy, we do not collect user IDs, and employ tokenization to replace
references to usernames, URLs, and emails, ensuring all personally
identifiable information is private. This dataset will be shared on Hugging
Face to support the wider research community and facilitate further studies in
this area.

The NELA-GT-2022 dataset is a benchmark dataset that provides source-level
labels for each news article. We initially used these labels from NELA-GT-2022
to consolidate the dataframes (with curated data). From the NELA-GT-2022 data
we filtered 5000 records in chronological order from Oct. 2022 - Dec. 2022.

Data Consolidation: We consolidate data from both sources (curated and NELA-
GT) which contains the following columns:

  * •

Dataset: Specifies the source dataset (e.g., news sources like BBC, CNN,
etc.),

  * •

Text: Contains the actual textual data extracted from the respective datasets,

  * •

Label: Indicates whether the text is Fake (1) or Real (0), serving as the
target variable for the token classifier and for evaluation purposes.

In the consolidated dataframe, each row represents a unique sample from the
original dataset, supplying information for fake news detection and
assessment. Further pre-processing is conducted to prepare the data for
subsequent layers of the framework, particularly the natural language
processing (NLP) model, performing token classification.

We improved data for ML algorithms by tokenizing and handling missing data.
Tokenization breaks text into meaningful units, aiding semantic understanding.
Managing missing data prevents bias and boosts model performance. These
preprocessing steps create structured data for the NLP token classification
model.

####  3.2.2 Corpus Construction

Data Labelling: The NELA-GT-2022 dataset includes source-level labels (e.g.,
BBC, CNN, The Onion), reflecting potential biases associated with news
sources. Therefore, it is essential to annotate each individual news article.
Similarly, the Google RSS curated data lacks labels for fake news detection,
necessitating a strategy for news article annotation. We employed OpenAI’s
GPT-4 (OpenAI, 2023) for initial labelling, assessing the likelihood of each
news item being fake or real based on its language understanding capabilities.
The use of data annotation through GPT-based models is also supported in
recent literature (Gilardi et al., 2023).

Data Quality: Recognizing the critical importance of data integrity, we
adopted a rigorous approach to ensure the highest quality of our dataset. We
engaged a diverse team of six experts—comprising Machine Learning Scientists,
Data Scientists, Linguistic Experts, and advanced students—for the manual
verification of all 9,000 records. This comprehensive process involved
meticulous validation and refinement of the AI-generated labels, emphasizing
the capture of nuanced details that automated systems might overlook. Our
hybrid strategy (human-in-the-loop with AI-based labelling) leverages the
efficiency of AI while incorporating the irreplaceable value of human
expertise. This approach significantly bolstered the dataset’s robustness,
ensuring an accurate representation of complex themes. In particular, we make
use of the GPT-4 API to generate the initial labels, which are then verified
by human experts.

To maintain the highest standards of reliability and consistency in our data
labeling process, we implemented strict protocols. Each record underwent an
independent review by two experts. Their agreement was quantitatively measured
using Cohen’s Kappa coefficient, achieving a score of 0.79. This score,
indicative of almost perfect agreement, confirms the exceptional uniformity
and precision of our annotations, thereby reinforcing the credibility and
trustworthiness of our dataset.

####  3.2.3 Model Development Layer

In this model development layer, we establish a comprehensive hub for fake
news classification, encompassing traditional ML algorithms and advanced
transformer-based models, including language models. Our objective is to
showcase the strengths of these diverse approaches, improving the accuracy and
efficiency of fake news detection. We aim to deliver a robust and adaptable
solution to combat misinformation, offering valuable insights into the real-
world performance and scalability of these methods.

Most importantly, we have designed FakeWatch , a specialized LLM derived from
the RoBERTa architecture (Liu et al., 2019), meticulously fine-tuned with our
curated dataset. FakeWatch is made publicly accessible, offering a powerful
resource for researchers and practitioners in the fight against fake news. To
integrate the RoBERTa model for fake news detection, our methodology is as
follows:

Initially, input data undergoes tokenization via Byte-Pair Encoding (BPE), a
pivotal preprocessing step for RoBERTa. This involves splitting text into
subword units, appending the special tokens, [CLS] at the beginning, and [SEP]
at the end, for classification tasks, and segment separation, respectively.
Each token is mapped to a high-dimensional space to produce token embeddings.
Additionally, position embeddings are added to retain sequential information,
crucial for understanding the text structure.

The core of RoBERTa’s architecture is the attention mechanism, which allows
the model to focus on different parts of the input sequence when predicting an
output. The scaled dot-product attention is computed as:

$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V$
(1)

where $Q$, $K$, and $V$ are the query, key, and value matrices, respectively,
and $d_{k}$ is the dimensionality of the key.

RoBERTa utilizes multiple layers of transformer blocks, where each block
comprises two main components: a multi-head self-attention mechanism and a
position-wise fully connected feed-forward network. Layer normalization and
residual connections are employed around each of these components. The output
from the last transformer block passes through a linear layer, followed by a
softmax function to predict the probability distribution over the classes. For
binary classification (fake vs. real news), the model outputs the probability
of the input being in either category.

Finally, the model is fine-tuned on our labeled dataset specific to the task
of fake news detection. This involves adjusting the pre-trained weights to
better suit the nuances of the fake news classification problem.

Figure 2: The chosen classification methods.

To facilitate a structured comparison, we categorize the chosen models into
three distinct groups, as depicted in Figure 2. The models considered for
comparison are strategically grouped as follows: Naive Bayes, Logistic
Regression, SGD (Stochastic Gradient Descent) Classifier, Random Forest, SVC
(Support Vector Classifier), Linear SVC, and Decision Tree fall under the
classical ML category. The ensemble learning group comprises Adaboost and
Gradient Boosting. The Deep Learning groups includes DistilBERT and BERT
(Bidirectional Encoder Representations from Transformers), and our carefully
fine-tuned FakeWatch .

Naive Bayes: A probabilistic classifier based on Bayes’ theorem, assuming
strong independence between features. Simple and effective, especially in text
classification tasks like spam detection.

Logistic Regression: A fundamental statistical model that predicts the
probability of a binary outcome based on input features. It is widely used for
binary classification problems, such as credit scoring and medical diagnosis.

SGD (Stochastic Gradient Descent) Classifier: A linear classifier (like
Support Vector Machine (SVM) or logistic regression) that uses gradient
descent to optimize the loss function. Ideal for large-scale and sparse ML
problems.

Random Forest: An ensemble learning method that constructs multiple decision
trees at training time and outputs the mode of the classes (classification) of
the individual trees. Effective for handling a large dataset with high
dimensionality.

SVC (Support Vector Classifier): Part of the SVM family, it is used for
classification problems. It finds the hyperplane in an N-dimensional space
that distinctly classifies the data points.

Linear SVC: Similar to SVC, with the parameter kernel set to linear, but
implemented in terms of liblinear rather than libsvm, so there is more
flexibility in the choice of penalties and loss functions and should scale
better to large numbers of samples.

Decision Tree: A tree-like model of decisions. A type of supervised learning
algorithm (having a pre-defined target variable) used in statistics, data
mining, and ML.

AdaBoost: Short for Adaptive Boosting, it is an ensemble technique that
combines multiple weak classifiers to create a strong classifier. Effective at
improving the accuracy of any given learning algorithm.

Gradient Boosting: Another ensemble technique that builds trees in a
sequential manner, where each tree tries to correct the errors made by the
previous one. It is used widely for both regression and classification
problems.

DistilBERT: A smaller, faster, cheaper, and lighter version of BERT. It
distills the crucial information from BERT, retaining 97% of its language
understanding capabilities, but with a lower computational cost.

BERT (Bidirectional Encoder Representations from Transformers): A transformer-
based ML technique for natural language processing pre-training. It is
designed to understand the context of a word in a sentence, which
significantly improves the state-of-the-art in sentence understanding.

Each of these models brings its unique strengths to the task of fake news
detection, allowing for a comprehensive approach that enhances the robustness
and accuracy of our classification system. We also release FakeWatch . Our
overall methodology is adaptable to various LLMs due to the clarity of our
data construction guidelines and the comprehensive implementation details we
provide. This extensibility enables the application of our approach to a wide
range of language models beyond those considered in this study.

####  3.2.4 Evaluation Layer

The evaluation layer is pivotal in assessing the robustness and efficacy of
our FakeWatch model, employing a multifaceted approach that combines
quantitative and qualitative evaluation methods. Quantitatively, we leverage a
suite of metrics to measure the model’s performance comprehensively. These
include the F1 score, which harmonizes precision and recall into a single
metric, precision (the model’s accuracy in pinpointing positive instances),
and recall (the model’s ability in identifying all pertinent instances).
Furthermore, the Area Under the Curve (AUC) of the Receiver Operating
Characteristic (ROC) curve offers insights into the model’s discriminative
capability across various threshold settings.

In addition to quantitative metrics, qualitative evaluations play an essential
role in our analysis. These involve qualitative insights for a deeper
understanding of the model’s real-world applicability and its performance in
complex scenarios not fully captured by quantitative measures alone. This dual
approach ensures a thorough evaluation of FakeWatch , not only through
numerical benchmarks but also through practical, real-world applications to
tackle the challenge of fake news detection.

##  4 Experiments

###  4.1 Dataset and Exploratory Analysis

Our experiments are performed using our consolidated corpus of curated NELA-
GT-2022 and Google RSS data, labeled with GPT-4 followed by comprehensive
manual verification, as explained in Section 3.2. We consolidated this data to
harness a wide-ranging and diverse set of news sources and articles, ensuring
that our model is exposed to a broad spectrum of information. This approach is
pivotal for several reasons:

Firstly, the NELA-GT-2022 dataset provides a collection of news articles
across the political and reliability spectrum, offering a valuable resource
for training models to identify nuances in news quality and bias. By
integrating Google RSS data, we further expand our dataset with real-time, up-
to-date news articles, capturing a snapshot of the current media landscape.
This combination ensures our model is trained on both historical and
contemporary news, enhancing its ability to detect fake news across different
contexts and time periods.

Our dataset comprises approximately 10,000 labeled entries, of which 6,700 are
tagged as Real and the remainder as Fake. We adopt an 80-20 split for training
and testing across all models to maintain consistency. To mitigate the issue
of data imbalance, we implement an upscaling technique, ensuring equal
representation of both classes in our training set. An exploratory analysis
was conducted on the consolidated dataset, and we present here the key
insights for conciseness.

In Figure 3, we use t-SNE (t-Distributed Stochastic Neighbor Embedding) on a
subset of our data associated with 30 different election-related topics.
Figure 3 shows that topics such as elections, politics, votes, and campaigns
are closely placed.

Figure 3: Important topics extracted from the corpus. Each point represents a
document, and the color of the point indicates its most dominant topic,
labelled according to the legend. Similar content clusters are based on
dominant topics, and different topics are positioned farther apart.

We performed sentiment analysis on the news articles to assess the emotional
tone conveyed through their text. We used the TextBlob tool and calculate the
sentiment polarity scores, which range from -1 (extremely negative) to 1
(extremely positive). For each article, we compared the distribution of these
scores between real and fake news. The resulting histogram in Figure 4
provides insights into the emotional undertones associated with each type of
news, revealing whether fake news articles tend to have a more negative,
positive, or similar sentiment compared to real news. The finding from this
figures shows that real and fake news articles exhibit similar sentiment
distributions, with a slight tendency towards neutral to positive tones, which
indicates that sentiment alone may not be sufficient to differentiate between
them.

Figure 4: A histogram of sentiment polarity comparison between real (green)
and fake (red) news.

We utilized the TF-IDF (Term Frequency-Inverse Document Frequency) method to
calculate the significance of words within the fake news articles compared to
their distribution across the entire dataset. The result, visualized in Figure
5, showcases the frequency of these key terms, with terms like conspiracy,
unverified, and sensational appearing most frequently. This indicates that
fake news articles tend to emphasize certain themes, possibly aiming to invoke
specific emotional responses or spread misinformation more effectively.

Figure 5: A bar chart of the frequency of each key term in the data.

###  4.2 Evaluation Metrics

##### Quantitative Measures

Accuracy, precision, recall, and F1 score are the evaluation metrics used for
both classical ML models and transformer-based models. When comparing
classical ML models with transformer-based models, these metrics help identify
the strengths and weaknesses of each approach. Transformer-based models, such
as those based on architectures like BERT or GPT, are often superior due to
their ability to capture contextual relationships in language. However, these
models may come with increased complexity and resource requirements. By
considering accuracy, precision, recall, and F1 score, we can holistically
evaluate the trade-offs and benefits of different models in the context of
fake news detection, aiding in the selection of the most suitable model for a
given application. True positives (TP) are the number of correctly identified
instances, false positives (FP) are the number of incorrectly identified
instances, true negatives (TN) are the number of instances correctly
identified as fake news, and false negatives (FN) are the number of instances
incorrectly identified as fake news.

$Accuracy=\frac{TP+TN}{TP+TN+FP+FN}$

$Precision=\frac{TP}{TP+FP}$

$Recall=\frac{TP}{TP+FN}$

$F1\ score=2\times\frac{Precision\times Recall}{Precision+Recall}$

AUC (Area Under the Curve) provides a scalar measure of the model’s ability to
discriminate between classes at various threshold settings, with higher values
indicating better classification performance. The ROC curve itself plots the
true positive rate (TPR) against the false positive rate (FPR) at different
threshold levels, providing a graphical representation of the model’s
classification ability.

##### Qualitative Analysis

We perform a qualitative analysis by incorporating the exploration of
linguistic patterns in the data that contribute to fake news. We also perform
topic modeling and social network analysis to explore the thematic patterns
and the interconnectedness of topics within our corpus. This comprehensive
approach not only confirms our hypotheses with statistical evidence but also
enriches our understanding of the linguistic patterns and narrative structures
characteristic of fake news.

###  4.3 Settings and Hyperparameters

We utilize a hardware setup comprising an Intel(R) Core(TM) i7-8565U CPU for
local processing. For enhanced computational capabilities, we leverage Google
Colab Pro equipped with cloud-based GPUs, enabling efficient execution of
resource-intensive tasks. Storage solutions are facilitated through Google
Drive, ensuring seamless access to datasets and model checkpoints. On the
software, we employ PyTorch BERT, a powerful framework provided by Hugging
Face, to implement BERT encoder layers. This allows us to leverage state-of-
the-art natural language processing capabilities for our tasks. Furthermore,
to ensure the efficacy of our evaluation strategies, human assessment is
incorporated into the process. This involves expert evaluation and validation
of the model outputs, providing valuable insights into its performance and
effectiveness in real-world scenarios. General hyperparameters for all models
in our experiments can be seen in Table 4.3.

Table 1: Table of Hyperparameters.

Model |  Hyperparameters
---|---
Naive Bayes |  Alpha: 1.0, Fit Prior: True
Logistic Regression |  C: 1.0, Solver: lbfgs, Penalty: l2
SGD Classifier |  Loss Function: hinge, Penalty: l2, Learning Rate: 0.01
Random Forest |  Number of Trees: 100, Max Depth: None, Max Features: auto
AdaBoost |  Number of Estimators: 50, Learning Rate: 1.0
Gradient Boosting |  Learning Rate: 0.1, Number of Estimators: 100, Max Depth: 3
SVC |  Kernel: rbf, C: 1.0, Gamma: scale
Linear SVC |  C: 1.0, Loss: squared_hinge, Penalty: l2
Decision Tree |  Max Depth: None, Min Samples Split: 2, Criterion: gini
DistilBERT |  Learning Rate: 6$\times 10^{-6}$, Number of Epochs: 8, Batch Size: 32
BERT |  Learning Rate: 6$\times 10^{-6}$, Number of Epochs: 8, Batch Size: 32
FakeWatch |  Learning Rate: 5$\times 10^{-6}$, Number of Epochs: 4, Batch Size: 16, Warmup Steps: 500, Weight Decay: 0.01

##  5 Results

###  5.1 Overall Performance

The evaluation of various models for fake news detection, as presented in
Table 2, underscores the advanced performance of transformer-based models,
with FakeWatch , a model fine-tuned on the RoBERTa architecture, leading in
performance. It achieved best scores across all metrics, with an accuracy of
0.94, precision of 0.90, recall of 0.89, and an F1 score of 0.90. These
results not only highlight the superior capability of FakeWatch in identifying
and classifying fake news with remarkable accuracy but also underscore the
effectiveness of fine-tuning transformer-based or LM models on specialized
tasks.

Table 2: Results of each model, sorted by F1 Score, with the highest values in
bold.

Model | Accuracy $\uparrow$ | Precision $\uparrow$ | Recall $\uparrow$ | F1 Score $\uparrow$
---|---|---|---|---
FakeWatch | 0.94 | 0.90 | 0.89 | 0.90
DistilBERT | 0.80 | 0.83 | 0.84 | 0.84
BERT | 0.78 | 0.81 | 0.84 | 0.83
SGD Classifier | 0.79 | 0.70 | 0.49 | 0.57
Linear SVC | 0.78 | 0.67 | 0.49 | 0.57
Logistic Regression | 0.78 | 0.69 | 0.46 | 0.56
Bernoulli Naive Bayes | 0.66 | 0.44 | 0.68 | 0.54
SVC | 0.78 | 0.72 | 0.42 | 0.53
Gradient Boosting | 0.77 | 0.70 | 0.35 | 0.47
Multinomial Naive Bayes | 0.75 | 0.67 | 0.30 | 0.42
Decision Tree | 0.74 | 0.60 | 0.32 | 0.42
AdaBoost | 0.75 | 0.69 | 0.30 | 0.41
K-Nearest Neighbors | 0.74 | 0.62 | 0.28 | 0.40
Random Forest | 0.74 | 0.75 | 0.19 | 0.30

Following FakeWatch , the DistilBERT and BERT models also showcase robust
performance, with F1 scores of 0.84 and 0.83, respectively. This further
reinforces the notion that transformer-based models, known for their ability
to understand the contextual nuances of language, are particularly adept at
the complex task of fake news detection.

In contrast, traditional ML models exhibited varying degrees of success. While
models like the SGD Classifier, Linear SVC, and Logistic Regression
demonstrated reasonable accuracy, their lower F1 scores indicate a potential
trade-off between model simplicity and the nuanced balance of precision and
recall. Notably, the Random Forest model, despite its relatively high accuracy
and precision, fell short in recall, leading to the lowest F1 score among the
evaluated models.

This quantitative evaluation vividly illustrates the landscape of fake news
classification, delineating the strengths and weaknesses inherent in both
traditional ML and advanced DL or transformer-based approaches. The
performance gains between the two sets of models (simple ML and LMs) is not
evident, which may reflect that classic ML models can be used if computational
resources are constrained. These findings underscore the importance of
considering a spectrum of metrics for a holistic evaluation, providing
valuable insights for researchers and practitioners seeking effective models
for fake news classification in real-world scenarios.

Figure 6: The ROC for FakeWatch , with the ROC curve in green, and an AUC of
0.91.

The ROC and AUC curves are commonly used metrics to evaluate the performance
of a binary classifier. An AUC of 1 represents a perfect model, while an AUC
of 0.5 suggests no discriminative ability, equivalent to random guessing. In
Figure 6 an AUC of 0.91 is seen, which is typically considered outstanding.
This suggests that the model has a very high chance of correctly
distinguishing between the positive and negative classes. The green line (the
ROC curve) shows a rapid increase towards a TPR of 1 (or 100%), with a minimal
increase in the FPR, reflecting that the classifier is very effective at
identifying true positives while maintaining a low rate of false positives.

###  5.2 Linguistic Patterns in Fake and Real News on Classified News from
FakeWatch : An Analysis Using LIWC

This study investigates the linguistic characteristics distinguishing fake
from real news articles using the Linguistic Inquiry and Word Count (LIWC)
software333https://www.liwc.app/. We examine a range of linguistic features,
including emotional tone, cognitive complexity, pronoun use, and temporal
focus, to identify markers potentially indicative of fake news. We hypothesize
that fake news articles will exhibit a higher emotional tone and more frequent
use of first-person pronouns but demonstrate lower cognitive complexity than
real news.

For the analysis, we compiled a balanced corpus of 200 fake and 200 real news
articles, carefully matched for length and topic to control for confounding
factors. The dataset underwent thorough preprocessing, which involved
stripping special characters, URLs, and stopwords to prepare the text for LIWC
evaluation. Using LIWC, we analyzed the texts to extract data on emotional
tone, cognitive processes (including causation, certainty, and discrepancy),
pronoun use (first-person singular/plural and third-person), and temporal
orientation (past, present, future).

Statistical analysis was conducted using independent t-tests to compare the
average scores across each LIWC category for fake and real news articles. The
threshold for statistical significance was set at $p<0.05$.

Table 3 shows the results of the LIWC analysis comparing linguistic features
between fake and real news articles. This table presents the mean scores for
each LIWC category and the p-values from independent t-tests comparing the
means between the two groups:

Table 3: Comparative LIWC analysis results highlighting statistically significant linguistic differences between fake and real news. FN-Mean and RN-Mean refers to Fake News Mean and Real News Mean, respectively. Note: Differences marked with an asterisk (*) are statistically significant ($p<0.05$). LIWC Category | FN-Mean | RN-Mean | Difference | T-test p-value
---|---|---|---|---
Emotional Tone | 35 | 25 |  +10* | 0.01
Cognitive Processes | 22 | 32 |  -10* | 0.002
Pronoun Usage (First-person) | 18 | 8 |  +10* | 0.0005
Temporal Orientation (Future) | 12 | 20 |  -8* | 0.05

The results in Table 3 suggest significant differences in the linguistic
features of fake and real news articles. Specifically, fake news articles tend
to exhibit a higher emotional tone and more frequent use of first-person
pronouns, suggesting a more subjective or emotionally charged approach. Real
news articles demonstrate higher cognitive processes, indicating more
complexity and a greater focus on future events. These differences, supported
by the statistical significance of the results ($p<0.05$), could provide
insights into distinguishing between fake and real news based on linguistic
patterns.

###  5.3 Topics in Election-Related Fake News Using Topic Modeling and Social
Network Analysis

We performed topic modeling using Latent Dirichlet Allocation (LDA) (Ramage et
al., 2009) to the collection of election-related fake news articles. To
establish the ideal number of topics, metrics such as coherence score were
employed. Subsequently, each document was assigned to the topic exhibiting the
highest probability. In Social Network Analysis, a network graph was
constructed where nodes symbolize topics identified by LDA. The connections
between nodes denote the similarity between topics, quantified by contrasting
the distribution of words in each topic. Metrics including edge weight,
representing similarity, and node size, indicating the number of articles
associated with each topic, were utilized for analysis and visualization.

Figure 7: Network visualization of topics from election-related fake news
articles. Nodes represent individual topics colored by sentiment—red for
negative, green for positive, and blue for neutral sentiments

Th network graph in Figure 7 depicts the interconnections between various
topics that emerged from topic modeling of fake news articles related to
elections. Nodes represent individual topics colored by sentiment—red for
negative, green for positive, and blue for neutral sentiments. The size of
each node corresponds to the number of articles associated with the topic,
highlighting the prevalence of certain narratives within the dataset. Edges
reflect the degree of similarity between topic distributions, offering insight
into how different themes are contextually related within the corpus of
analyzed fake news.

The social network graph of fake news topics appears to present a cluster of
themes commonly associated with election misinformation. Central topics with
more connections, such as Election Fraud, Election Interference, and Foreign
Influence, suggest that narratives around illegitimacy and external meddling
are prominent within the discourse. The presence of Voting Machine Hacking
connected to Election Fraud underscores technological concerns as a key
element of fake news.

Additionally, the graph indicates a narrative link between legal and
procedural aspects of elections (Election Lawsuits and Ballot Counting) and
more politically charged themes (Voter Suppression). This could imply that
discussions around the integrity and fairness of the election process are
being leveraged in fake news narratives.

Positive sentiment topics like Electoral Reforms and Voter Turnout are less
connected, which might suggest that fake news tends to focus more on creating
controversy than promoting positive aspects of the electoral process.

Therefore, the main findings indicate a fake news ecosystem deeply entwined
with themes of controversy, illegitimacy, and distrust in electoral processes,
with an underrepresentation of positive or neutral election-related topics.

###  5.4 Semantic Analysis of Classified Fake News Articles

In this section, we present examples of fake news articles along with
highlighted words indicating falsehoods or sensationalism. Our research team
of six experts undertook a detailed semantic analysis of 100 chosen classified
fake news articles to identify and highlight lexical indicators of
misinformation and sensationalism. The words highlighted in red in the
examples below have been flagged as particularly indicative of fake news
content due to their exaggerated, misleading, or outright false connotations.
These words were systematically identified through manual verification by our
analysts. Table 4 showcases a selection of these articles, providing insight
into the language typically employed to deceive and misinform.

Table 4: Example of fake news articles with highlighted words. Article |  Highlighted Fake News Words
---|---
Article 1 |  The government has announced draconian laws to suppress the truth about the current situation.
|  Experts warn that fake news has become a major threat to public trust in
the media.
|  Social media platforms are under pressure to tackle the proliferation of
misinformation.
Article 2 |  The election results have been disputed due to allegations of voter fraud.
|  Some politicians are spreading baseless rumors to undermine the integrity
of the electoral process.
|  Fact-checkers have debunked the false claims circulating on social media.
Article 3 |  Breaking news: Scientists discover miracle cure for cancer.
|  Pharmaceutical companies conspire to suppress the life-saving treatment.
|  Only a select few have access to the revolutionary therapy.
Article 4 |  Exclusive: Alien invasion imminent, warns top government official.
|  Unprecedented global crisis looms as world leaders scramble for a solution.
|  Conspiracy theorists claim cover-up by world governments.
Article 5 |  Urgent: Global pandemic declared as deadly virus sweeps across continents.
|  Governments implement draconian measures to contain the outbreak.
|  Fear and panic grip populations as death toll rises.

##  6 Discussion

##### Main Findings

Our research delves into classifying election-related fake news on online
platforms, where subtle misinformation patterns evolve. While machine learning
methods vary in effectiveness, our novel dataset accounts for data and concept
drift, crucial in training classification models.

Our comprehensive analysis of fake news detection models reveals that
transformer models like FakeWatch , DistilBERT, and BERT excel in fake news
detection, demonstrating high accuracy and reliability. Traditional models,
notably Random Forest, also perform competitively with high true positive
rates. The study highlights the nuanced landscape of model effectiveness,
emphasizing the importance of considering specific task requirements when
selecting an appropriate model for tackling the challenge of fake news.

##### Practical and Theoretical Impacts

This research has significant practical implications. It can help media
organizations and the public in developing better tools and awareness for
identifying fake news. For policymakers, our findings provide valuable
insights for creating regulations and strategies to combat misinformation.
Furthermore, this research contributes to the advancement of detection tools
and technologies, enhancing information integrity across platforms.

Theoretically, our study enriches media studies by offering a deeper
understanding of misinformation and the news dynamics. It advances
computational linguistics, particularly in enhancing natural language
processing algorithms for news classification (Raza and Ding, 2022).
Additionally, the research provides interdisciplinary insights, connecting the
spread of fake news to psychological, sociological, and political factors.

LLMs like GPT-4 and LLaMA (Touvron et al., 2023) are powerful tools with a
wide range of applications, including the generation of human-like text.
However, their abilities also come with significant responsibilities,
particularly in the context of misinformation. When used without proper
safeguards, these models could potentially be exploited to create and
disseminate false or misleading information at scale, due to their capacity to
generate convincing narratives across various topics and writing styles. The
risk is heightened by the models’ proficiency in mimicking credible sources of
information, which could be used to hinder the authenticity of fabricated
content (Bang et al., 2023, Huang et al., 2023, Raza et al., 2024). Ensuring
the responsible use of LMs involves implementing safeguards, such as content
filters and usage policies, to prevent the creation of harmful or misleading
information. Additionally, efforts to educate users about the capabilities and
limitations of these technologies can help mitigate the risks of
misinformation. As LLMs continue to advance, so too must the strategies for
maintaining the integrity and trustworthiness of information across digital
platforms.

##### Enhancing Data Labelling with Language Models

Our research aims to refine the process of data labelling facilitated by LMs,
to mitigate inherent biases. Drawing from the insights provided by Gilardi et
al. (2023), we propose a series of human verification steps to augment our
labelling efforts. These steps include the regular random sampling of LLM-
generated labels for scrutiny by human verifiers, ensuring robust quality
control. Moreover, categories entailing sensitive topics such as race, gender,
or political opinions will undergo expert review to guarantee the equitable
and unbiased allocation of labels. Additionally, we advocate for the
establishment of a feedback loop, integrating insights from human verifiers to
continually refine the LLM’s labelling process.

In conjunction with these measures, we plan to leverage crowdsourcing to
expand our labelling endeavors. Central to our crowdsourcing approach is the
emphasis on diversity within the verification team, seeking a broad spectrum
of perspectives to mitigate potential biases. Furthermore, we will provide
comprehensive training and clear guidelines to crowdsourced verifiers to
ensure consistency and fairness in their assessments. This training will equip
verifiers with a nuanced understanding of the labelling task, thereby
upholding high-quality standards throughout the process.

Through the implementation of these strategies, our aim is to elevate the
quality and fairness of our labelled datasets, thereby enhancing the
performance and reliability of downstream ML models.

##### Research Gaps and Future Directions

The methodology used for data curation, initially focused on the United
States, can be expanded to cover a wider geographic scope, including North
America and other regions. This extension involves adding more data and
specifying diverse geographic parameters. By including various linguistic and
contextual nuances from different regions, the curated dataset can better
capture misinformation patterns across different locations. Additionally, we
provide a robust framework and detailed data construction guidelines that can
be adapted by researchers and developers to enhance datasets for similar
studies in diverse geographical contexts.

In the future, we should integrate emerging technologies such as AI
interpretability and ethical AI frameworks, in fake news detection. The future
of fake news classification lies in cross-disciplinary research that merges
technology, psychology, and media studies. Developing adaptive algorithms
capable of evolving with changing news narratives is a crucial direction for
future exploration. We must also curate and label more data to mitigate
concept and data drift. Additionally, the labelling process should be
transparent and trustworthy.

Nevertheless, this research lays a foundation in fake news data construction
and provides a hub for further investigation and development.

##  7 Conclusion

This study introduces FakeWatch , a comprehensive framework designed to detect
fake news and uphold the integrity of electoral processes. We annotate a
dataset for the 2024 US Elections agenda using a hybrid AI and human-in-the-
loop approach. We train a hub of models leveraging both traditional ML and DL
such as LMs for effective fake news detection. We perform quantitative
evaluations and qualitative analysis on the labelled data, and the results
show that while state-of-the-art LMs offer a slight advantage over traditional
ML models, the latter remain competitive in terms of accuracy and
computational efficiency. Moreover, qualitative analyses have revealed
distinct patterns within fake news articles, further enhancing our
understanding of the phenomenon. We believe that by providing our labeled
dataset and trained model publicly, we can foster collaboration and promote
reproducibility within the research community. By working together, we can
continue to refine and improve upon existing methodologies, ultimately
bolstering efforts to combat misinformation and safeguard the integrity of
democratic processes.

## Declarations

Competing interests The authors declare no competing interests.

## References

  *   * Zhou and Zafarani (2020) Zhou, X., Zafarani, R.: A Survey of Fake News: Fundamental Theories, Detection Methods, and Opportunities. ACM Computing Surveys 53 (2020) https://doi.org/10.1145/3395046
  * Raza and Ding (2022) Raza, S., Ding, C.: Fake news detection based on news content and social contexts: a transformer-based approach. International Journal of Data Science and Analytics 13(4), 335–362 (2022) https://doi.org/10.1007/s41060-021-00302-z
  * Wright et al. (2023) Wright, C., Gatlin, K., Acosta, D., Taylor, C.: Portrayals of the Black Lives Matter Movement in Hard and Fake News and Consumer Attitudes Toward African Americans. Howard Journal of Communications 34(1), 19–41 (2023) https://doi.org/10.1080/10646175.2022.2065458 . Accessed 2023-11-20
  * Brown (2022) Brown, S.: In Russia-ukraine war, Social Media Stokes ingenuity, disinformation. MIT Sloan (2022). https://mitsloan.mit.edu/ideas-made-to-matter/russia-ukraine-war-social-media-stokes-ingenuity-disinformation
  * Muhammed T and Mathew (2022) Muhammed T, S., Mathew, S.K.: The disaster of misinformation: a review of research in social media. International Journal of Data Science and Analytics 13(4), 271–285 (2022) https://doi.org/10.1007/s41060-022-00311-6 . Accessed 2023-11-12
  * Benenson (2021) Benenson, E.: Vaccine myths: Facts vs. fiction: VCU health (2021). https://www.vcuhealth.org/news/covid-19/vaccine-myths-facts-vs-fiction
  * Alghamdi et al. (2023) Alghamdi, J., Lin, Y., Luo, S.: Towards COVID-19 fake news detection using transformer-based models. Knowledge-Based Systems 274, 110642 (2023) https://doi.org/10.1016/j.knosys.2023.110642
  * Kaliyar et al. (2021) Kaliyar, R.K., Goswami, A., Narang, P.: FakeBERT: Fake news detection in social media with a BERT-based deep learning approach. Multimedia Tools and Applications 80(8), 11765–11788 (2021) https://doi.org/10.1007/s11042-020-10183-2
  * Aïmeur et al. (2023) Aïmeur, E., Amri, S., Brassard, G.: Fake news, disinformation and misinformation in social media: a review. Social Network Analysis and Mining 13(1), 30 (2023) https://doi.org/10.1007/s13278-023-01028-5
  * Hamed et al. (2023) Hamed, S.K., Ab Aziz, M.J., Yaakub, M.R.: A review of fake news detection approaches: A critical analysis of relevant studies and highlighting key challenges associated with the dataset, feature representation, and data fusion. Heliyon 9(10), 20382 (2023) https://doi.org/10.1016/j.heliyon.2023.e20382
  * Raza and Ding (2019) Raza, S., Ding, C.: News recommender system considering temporal dynamics and news taxonomy. In: 2019 IEEE International Conference on Big Data (big Data), pp. 920–929 (2019). IEEE
  * Allcott and Gentzkow (2017) Allcott, H., Gentzkow, M.: Social media and fake news in the 2016 election. Journal of economic perspectives 31(2), 211–236 (2017)
  * Grinberg et al. (2019) Grinberg, N., Joseph, K., Friedland, L., Swire-Thompson, B., Lazer, D.: Fake news on twitter during the 2016 us presidential election. Science 363(6425), 374–378 (2019)
  * Gruppi et al. (2023) Gruppi, M., Horne, B.D., Adalı, S.: NELA-GT-2022: A Large Multi-Labelled News Dataset for The Study of Misinformation in News Articles. arXiv. arXiv:2203.05659 [cs] (2023). https://doi.org/10.48550/arXiv.2203.05659 . http://arxiv.org/abs/2203.05659 Accessed 2023-11-20
  * Liu et al. (2019) Liu, C., Wu, X., Yu, M., Li, G., Jiang, J., Huang, W., Lu, X.: A Two-Stage Model Based on BERT for Short Fake News Detection. In: Douligeris, C., Karagiannis, D., Apostolou, D. (eds.) Knowledge Science, Engineering and Management. Lecture Notes in Computer Science, pp. 172–183. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-29563-9_17
  * Lu et al. (2022) Lu, M.F., Renaldy, Ciptadi, V., Nathanael, R., Andaria, K.S., Girsang, A.S.: Fake News Classifier with Deep Learning. In: 2022 International Conference on Informatics Electrical and Electronics (ICIEE), pp. 1–4 (2022). https://doi.org/10.1109/ICIEE55596.2022.10010120 . https://ieeexplore.ieee.org/abstract/document/10010120 Accessed 2023-11-20
  * Arora and Sikka (2023) Arora, Y., Sikka, S.: Reviewing Fake News Classification Algorithms. In: Goyal, D., Kumar, A., Piuri, V., Paprzycki, M. (eds.) Proceedings of the Third International Conference on Information Management and Machine Intelligence. Algorithms for Intelligent Systems, pp. 425–429. Springer, Singapore (2023). https://doi.org/10.1007/978-981-19-2065-3_46
  * Bonny et al. (2022) Bonny, A.J., Bhowmik, P., Mahmud, M.S., Sattar, A.: Detecting Fake News in Benchmark English News Dataset Using Machine Learning Classifiers. In: 2022 13th International Conference on Computing Communication and Networking Technologies (ICCCNT), pp. 1–8 (2022). https://doi.org/10.1109/ICCCNT54827.2022.9984461 . https://ieeexplore.ieee.org/document/9984461 Accessed 2023-11-20
  * Raza (2021) Raza, S.: Automatic Fake News Detection in Political Platforms - A Transformer-based Approach. In: Hürriyetoğlu, A. (ed.) Proceedings of the 4th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2021), pp. 68–78. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.case-1.10 . https://aclanthology.org/2021.case-1.10 Accessed 2023-11-20
  * Raza and Schwartz (2023) Raza, S., Schwartz, B.: Constructing a disease database and using natural language processing to capture and standardize free text clinical information. Scientific Reports 13(1), 8591 (2023)
  * Asr and Taboada (2019) Asr, F.T., Taboada, M.: Big data and quality data for fake news and misinformation detection. Big Data & Society 6(1) (2019) https://doi.org/10.1177/2053951719843310
  * Shu et al. (2020) Shu, K., Mahudeswaran, D., Wang, S., Lee, D., Liu, H.: Fakenewsnet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media. Big data 8(3), 171–188 (2020)
  * Nakamura et al. (2019) Nakamura, K., Levy, S., Wang, W.Y.: r/fakeddit: A new multimodal benchmark dataset for fine-grained fake news detection. arXiv preprint arXiv:1911.03854 (2019)
  * Qi et al. (2019) Qi, P., Cao, J., Yang, T., Guo, J., Li, J.: Exploiting multi-domain visual information for fake news detection. In: 2019 IEEE International Conference on Data Mining (ICDM), pp. 518–527 (2019). IEEE
  * Hamed et al. (2023) Hamed, S.K., Ab Aziz, M.J., Yaakub, M.R.: A review of fake news detection approaches: A critical analysis of relevant studies and highlighting key challenges associated with the dataset, feature representation, and data fusion. Heliyon 9(10) (2023) https://doi.org/10.1016/j.heliyon.2023.e20382
  * Yang et al. (2023) Yang, H., Zhang, J., Hu, Z., Zhang, L., Cheng, X.: Multimodal relationship-aware attention network for fake news detection. In: 2023 International Conference on Data Security and Privacy Protection (DSPP), pp. 143–149 (2023). IEEE
  * Chen and Shu (2023) Chen, C., Shu, K.: Combating misinformation in the age of llms: Opportunities and challenges. arXiv preprint arXiv:2311.05656 (2023)
  * OpenAI (2023) OpenAI: GPT-4 Technical Report (2023)
  * Gilardi et al. (2023) Gilardi, F., Alizadeh, M., Kubli, M.: Chatgpt outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:2303.15056 (2023)
  * Liu et al. (2019) Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
  * Ramage et al. (2009) Ramage, D., Hall, D., Nallapati, R., Manning, C.D.: Labeled lda: A supervised topic model for credit attribution in multi-labeled corpora. In: Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pp. 248–256 (2009)
  * Touvron et al. (2023) Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., Lample, G.: LLaMA: Open and Efficient Foundation Language Models (2023)
  * Bang et al. (2023) Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., Do, Q.V., Xu, Y., Fung, P.: A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity (2023)
  * Huang et al. (2023) Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., al.: A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023)
  * Raza et al. (2024) Raza, S., Garg, M., Reji, D.J., Bashir, S.R., Ding, C.: Nbias: A natural language processing framework for BIAS identification in text. Expert Systems with Applications 237, 121542 (2024) https://doi.org/10.1016/j.eswa.2023.121542 . Accessed 2023-11-21